ExT5: Towards Extreme Multi-Task Scaling for Transfer Learning
Authors: Vamsi Aribandi
Yi Tay
Tal Schuster
Jinfeng Rao
Huaixiu Steven Zheng
Sanket Vaibhav Mehta
Honglei Zhuang
Vinh Q. Tran
Dara Bahri
Jianmo Ni
Jai Gupta
Kai Hui
Sebastian Ruder
Donald Metzler
Venue: ICLR 2022 Poster (2022)
Keywords: Natural Language Processing
Transfer Learning
Multi-task Learning

Abstract: Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces ExMix (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using ExMix, we study the effect of multi-task pre-training at the largest scale to date, and analyze co-training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised ExMix. Via extensive experiments, we show that ExT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMix. ExT5 also significantly improves sample efficiency while pre-training.

Published as a conference paper at ICLR 2022
EXT5: TOWARDS EXTREME MULTI-TASK SCALING FOR TRANSFER LEARNING
Vamsi Arihandi* Yi Tay', Tal Schuster, Jinfeen Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder*, Donald Metzler Google Research, *DeepMind faribandi, yitay)Erpogle.com
ABSTRACT
Despite the recent success of multi-task learning and transfer learning for natural language processing (NLP), few works have systematically studied the effect of scaling up the number of tasks during pre-training. Towards this goal, this paper introduces EXMIX (Extreme Mixture): a massive collection of 107 supervised NLP tasks across diverse domains and task-families. Using EXMIX, we study the effect of multi-task pre-traiming at the largest scale date, and analyze co- training transfer amongst common families of tasks. Through this analysis, we show that manually curating an ideal set of tasks for multi-task pre-tra pre-training is not straightforward, and that multi-task scaling can vastly improve models on its own. Finally, we propose ExT5: a model pre-trained using a multi-task objective of self-supervised span denoising and supervised EXMIX. Via extensive exper- iments, we show that EXT5 outperforms strong T5 baselines on SuperGLUE, GEM, Rainbow, Closed-Book QA tasks, and several tasks outside of ExMIX. ExT5 also significantly improves sample efficiency while pre- training..
INTRODUCTION
Transfer learning (Schmidhuber, 1987; Pratt et al., 1991; Caruana et al., 1995) has been the cor- nerstone of recent progress in natural language processing (Ruder et al., 2019; Devlin et al., 2019; Raffel et al., 2020). While self-supervised pre-training has been shown to be highly effective at exploiting large amounts of unlabeled data without relying on human annotation, there is still much to explore regarding transfer learning in a multi-task co-training setup.
Prior seminal works like T5 (Raffel al., 2020) and MT-DNN (Liu et al., 2019a) have demon- strated degree of promise in the paradigm of multi-task co-training (Caruana, 1997). However, the challenge of catastrophic forgetting remains. Tasks often have be carefully selected in order to demonstrate positive affinity with regards to downstream transfer. In many cases, it is not unreason- able to expect negative transfer (Rosenstein al., 2005; Vu et al., 2020). This makes the process of empirically curating set of tasks to include in transfer learning setup both computationally prohibitive and specific to downstream tasks.
While standard pre-training typically employs variant of the self-supervised language modeling objective (Raffel et al., 2020), certain types of skills such as commonsense knowledge are only acquired at a slow rate even using massive amounts of unlabeled data (Zhang et al., 2021). As ever larger models are trained, the development of much more sample-efficient pre-training settings becomes thus more important, and could be addressed via multi-task learning.
For the first time, we explore and propose Extreme Multi-task Scaling - new paradigm for multi- task pre- pre-training, Compared to the largest prior work (Aghajanyan et al., 2021). study doubles the number of tasks and focuses on multi-task pre-training rather than fine-tuning, which enables a direct comparison standard pre-tnain Our proposal is based on the insight that despite negative transfer being common during fine-tuning, a massive and diverse collection of pre-training tasks is generally preferable an expensive search for the best combination of pre-training tasks.
"Google AI Resident. 'Equal contribution. Scbastian now at Google Research. Sanket returned to CMU.

---

Published as a conference paper at ICLR 2022
To this end, we introduce ExMIx: massive collection of 107 supervised NLP tasks be included in a multi-task pre- pre-training setup. We process all tasks in an incoder-decodde friendly format to readily support the sharing of all parameters across all tasks. We postulate that ensembling effect across as many tasks, distributions and domains as possible results in a consistently net-positive outcome. This echoes early multi-task learning results (Caruana, 1997; Baxter, 2000) indicating that a bias that is learned on sufficiently many tasks is likely to generalize to unseen tasks drawn from the same environment. Moreover, our experiments verify that our EXMIX mixture outperforms best-effort mixture of manually curated tasks.
Finally, we propose ExT5: T5 model (Raffel al., 2020) pre-trained on mixture of supervised ExMIx and self-supervised C4 span denoising. ExT5 outperforms state-of-the-ar T5 models on well-established benchmarks such as SuperGI.UR (Wang et al., 2019a), GEM (Gehrmann et al., 2021), and Rainbow (Lourie et al., 2021); as well as Closed-Bood QA (Roberts et al., 2020) tasks. Notably, our experimental findings also suggest that including ExMIx may reduce the number of pre-training steps required to achieve strong performance, bringing about substantial sample effi- ciency benefits.
To summarize, this paper contributes the following:
We propose EXMIX ($2): collection of 107 supervised NLP tasks for Extreme Multi-task Scaling, formatted for encoder-decoder training. ExMIx has approximately twice as many tasks the largest prior study to date (Aghajanyan al., 2021), totaling 18M labeled examples across diverse task families.
Given this large collection of tasks, we conduct rigorous empirical studies evaluating trans- fer between common task families ($2.1). Our experiments show that curating pre-training mixture based on fine-tuning transfer is not straightforward ($2.2). Hence, efficiently search- ing for the best subset of tasks include in multi-task pre-training setup is challenging and prohibitive.
Using ExMIX, we pre-train model alongside the C4 ppan-denoising objective introduced by Raffel et al. (2020), resulting in a new pre-trained model which we call EXT5 ($3). EXT5 outperforms state-of-the-art T5 on well-established benchmarks such as SuperGI.UR, GEM, Rainbow, Closed Book Question Answering, and several other tasks that are outside of ExMix ($3.2), while also being more sample-efficient ($2.6).
THE EXMIX TASK COLLECTION
To explore the Extreme Task Scaling paradigm, we introduce ExMIx (Extreme Mixture), a collection of 107 diverse English NLP tasks totaling 18M ex- amples. Following Raffel et al. (2020), we format all tasks as text-to-text examples to readily allow for multi-task training. This unified format also enables simple implementations without the need for task- specific heads/losses, loss scaling, or explicit gra- dient accumulation for heterogenous batches as in prior works (Liu al., 2019a; Aghajanyan et al., 2021). When selecting examples from ExMIx, ex- amples from each dataset are sampled proportionate the individual dataset's size, with each dataset's sampling rate capped at 3 10t maximum effective examples ensure balance be- tween large and small datasets. We refer readers to Appendix A for a comprehensive breakdown of ExMIx. Additionally, we discuss future multilingual variants of ExMIX and ExT5 in $5.
164
Figure 1: ExMIX task sizes in log scale. The dashed line is the 3 x 10" sampling rate cap.
2.1 TRANSFER RELATIONS BETWEEN EXMIX TASKS
As discussed in $1, our goal is to pre-train model on EXMIX improve downstream performance. One natural question to ask is which tasks have negative impact on downstream performance? Specifically, there subset of ExMIX that leads better representations when used for multi- task pre-training? Obviously, testing all possible 2XEMI combinations is impractical since the
2

---

Published as a conference paper at ICLR 2022
pre-training process is expensive to run. Instead, experiment with the less expensive co-training procedure (i.e., multi-task fine-tuning), using representative subsets of similar tasks. Later, in $2.2, explore whether these results can be used to inform the task selection for multi-task pre-training.
To study transfer amongst task-families in ExMIX, we construct subsets (Table 1) of tasks each that are par- titioned along their task family. Using these subsets, evaluate transfer among task families in multi-task learning (co-training) setup. While other types of tasks are available ExMIX, we did not include them because they were not diverse enough to be representative of task family, and would scale the number of models needing to be trained at a quadratic rate.
Family
Datasets
CNNDuilyMall
Summanization
XSum
Lingua
Schoma-gaee dialogue
Dialogne
Wizand-at-Wiiipee
Dialoglne-TO
ANLI
NLI
MNLI
GNL
IMDB
Classifiattin
GeEmitions
Civil Comments
ATIS FuntqL
Experimental Setting We fine-tune model each pair of task families (i.e., 6 datasets at a time). To ensure a fair balance of tasks, sample tasks proportional within their family, but uniformly between task families. For ex- ample, while evaluating how classification tasks and NLI tasks transfer amongst each other, the sampling ratio of MNLI:ANLI will be proportional (approximately 2.4:1), but the overall ratio of NLI examples to classification ex- amples will be 1:1. For reference, we also train a model on each individual task family using proportional exam- ple mixing (Sanh et al., 2019).
Semantic Parsing
GED to FarQL.
COGS
PhyscaliQQ
Commonsemse
SucialiQA
WineCrande
Natual Questions
Book QAA
Trivia QA
Hotpot QA
SQuAD
Realing Comprehenvion
TweetQA
Table I: Representative datasets used for task-family transfer learning exper- iments ($2.1).
In total, this results in F+ (5) models trained, where Fis the number of task families. Our experiments use F F== 8 as shown in Table 1, resulting in 34 models trained in total. Each model is fine-tuned on top of the released T5.1. checkpoint for 200k steps using batch size of 128 and constant learning rate of 10 3
Observations We summarize our results in Table We observe that although there exist particu- lar task-family pairs that show positive transfer (e.g., co-training with NLI helps most other tasks), negative transfer is more common when training across task families compared to intra-family train- ing. 21 out of the 56 inter-family relationships perform worse than aatra-family models with the same data budget, which grows to 38 out of 56 for fixed compute-budget. While the abundance of negative transfer among diverse task families is an expected result, interesting trends manifest the individual relationships. For example, summarization tasks generally seem to hurt performance on most other task families; and CBQA tasks are highly sensitive to multi-task fine-tuning.
We also report correlations for intra- intra-family datasets in Figure 2 using the same models as in Table 2. In most cases, we see positive correlations between datasets the same family. In a few cases, however, we observe opposite trend. For example, fine-tuned models that performed better on the GEM schema-guuder dialog dataset achieved lower scores on KILT Wizard-of Wikipedia.
This initial exploratory analysis highlights both the potential of EXMIX as a tool to systematically study task relationships, as well as the potential challenges in leveraging multi-task learning naively on top of pre-trained representations.
2.2 CAN FINE-NUNING TASK RELATIONSHIPS HELP CURATE A PRE-TRAINING MIXTURE?
Our observations in $2.1 showed that multi-task co-training on top of existing pre-trained check- points not straightforward, and often results in negative transfer. However, the uncovered task relationships might help efficiently search for an ideal subset of ExMIX for multi-task pre-training. To this end, we select a set of the most promising task families to be included in a multi-task pre- training setup, ranking task families by the average relative improvement they provide to other target families (the last column in Table 2). Specifically, we include NLI, commonsense, classification, and closed-book QA tasks from EXMIX to form mixture of 48 tasks to include in multi-task
3

---

Published as a conference paper at ICLR 2022
SUM
DLG
NLJ
CLS
SEM
CMNS
CBQA
RC
Avva
27.89
SUM
29.56
37.81
60.45
77.10
78.25
61.92
7.84
65.37
-6.9%
38.56
64.01
N11
28.61
40,00
67.23
77.29
77.72
07.60
15.24
06.40
+4.3%
SEM
29.50
38.86
62.46
76.83
72.79
72.09
57.84
12:44
63.52
-2.5%
68.23
65.35
14
CBQA
29.75
39.29
64.96
77.66
75.21
06.84
19.98
06.37
+1.2%
63.70
-2.4%
AVGgdiing
63.77
Table 2: Co-training transfer among task families. The entry at (row i, column j) indicates average performance family using model co-trained families and For intra-family, models (diagonal cells) we report results upto 100k steps (consistent data- bata-budget) and 200k steps (consistent compute-budget). Averages calculated excluding the intra-family models (i.e. the diagonal cells). The last column denotes the average gain that a source family provides to other task families.
1990000
35535A>
25022025 SOHD
CNN-99
TOP
CEO
AAWAAA 5/0A
P2LAADS4 Sceld
Gat
Pafy
190002
AUTE MANDOAA
SOLAT TOSA
NO
PERAINNA WOW
JAWAAAA T10A
(a) SUM
(b) DLG
(c)
(d) CLS
(e) SEM
(1) CMNS
(g) CBQA
(h) RC
Figure 2: Within- family correlations for each dataset in task family, using models from Table 2. Performance on datasets from some task families are highly correlated (c.g., NLD) whereas other task families have more erratic results across their datasets (e.g., Dialogue)
pre-training setup. We then fine-tune the resulting model SuperGLUE, comparing it to T5.1.1 and ExT5 in Table 3.
While this model narrowly outperformed 15.1.1, it did yield better results than including all of ExMIX the multi-task pre-training mixture, as we report in $3.2. Moreover, did not outperform random selection of 55 tasks on average, as we report our task scaling experiments ($2.5).
We conclude that the negative transfer during multi-task fine-tuning does not necessarily inhibit pre-training. While we cannot directly conclude that an ideal subset of ExMIX does not exist to be mixed with self-supervised pre-training for spe- cific downstream tasks, our experiments show that randomly including more diverse pre-training tasks generally improves downstream performance.
Mixture
Tasks
SuperGLUF
Vanilla
0
76.1
Best-effort Random-55 ($2.5)
48
76.4
55
77.0
EXMiX (53.2)
107
79.9
must also be noted that the end-goal is to find mix- ture that leads to general praitrened model that can be used for large variety of downstream tasks, and that a setup to find a pre- pre-training mixture tailored for SuperGI
Table 3: A best-effort mixture from fine- tuning transfer results does not beat increas- ing the number of tasks.
GUE UE would be different.
4

---

Published as a conference paper at ICLR 2022
2.3 MULTI-TASK PRE-TRAINING VS PRE-FINETUNING
Instead of pre-training on EXMIX, another way to leverage multi-task learning is as an intermediate step between pre-training and fine-tuning. This is referred pre-finetunin by Aghajanyan al. (2021). We conduct controlled experiments to compare pre-training with pre-finetuning. We begin with standard T5 base checkpoint and pre-finetune it with ExMIX. After this phase, we fine-tune on SuperGLUE.
Table compares pre-finetuning and our proposed multi-task pre-training. We also report the total com- pute (in total number of tokens processed) by the model in both schemes. The results show that multi- task pre-training significantly superior to pre- finetuning. A potential hypothesis is that multi-task pre-training narrows the gap between pre-training and finetuning data distributions, the pre-training stage more closely resembles fine-tuning. Con- versely, segregating pre-training and pre-finetuning into two different stages may induce catastrophic forgetting of the pre-training task. Hence, in ExT5, we opt for multi-task pre-training over pre-finetuning.
Method
Compute
SuperGL.U
Vanilla
1.2M
76.1
Pre-finetuning (200k)
1.4M
78.
Multi-task Pre-training
1.2M
Table 4: Comparison of Pre- tre-finetunin and Multi-task Pre-training on ExMIX.
2.4 How MUCH LABELED DATA SHOULD BE MIXED?
In this section, we explore how the ratio of C4 to EXMIX examples during massive multi-task pre- training affects performance. As mentioned later
80.0
77.5
75.0
$3, this controlled by a hyperparameter R, where a pre-training batch will have approximately R times as many C4 examples compared to ExMIx. From our results in Figure we find that despite ExMIX improving downstream performance when mixed with self-supervised C4 pre-training many rates, a model trained with R = O suffers greatly in comparison. This result is significant, as shows that while EXMIX improves the pre-training pro- cess, sele-supe training over large unstruc- tured corpus is still crucial.
22.5
Figure 3: How the ratio of C4 span denois- ing examples to ExMIX affects SuperGLUE results on EXT5 RASE. The dashed line is per- formance without using EXMIX (R - o))
2.5 DOES ADDING MORE TASKS HELP? TASK SCALING EXPERIMENTS
In this section, we explore how model performance changes the number of tasks included in mas- sive multi-task pre- pre-training setup is scaled up. We choose random sets of 30, 55, and 80 tasks (each superset the last), pre-train a BASE-sized model for 524k steps, and fine-tune them on SuperGLUE. We train our models with batch sizes of 128 and 512 and R 2 (the ratio of C4 EXMIX examples) as this configuration worked best for our BASE-sized models ($2.4). We repeat this over three random seeds (for random subset selection), and report av- erage scores in Figure 4.
128
$12
30
55
80
307
NUM TASKS
Figure Scaling the number of tasks dur- ing multi-task pre-training generally helps. The shaded area portrays standard deviation across three random subset selections.
Overall, with large batches, we can see that increas- ing the number of tasks being mixed generally helps downstream performance. This reinforces our intuition that task scaling indeed helps. With small batches, there is less of an upward trend, signifying that large batches are essential for a large number of tasks. This is intuitive, given that multi-tas task learning may cause gradients to be noisy (Yu et al., 2020). Another explanation as to why this happens is that large-batch training can offer benefits for single-task models (Smith al., 2018) trend formalized by McCandlish al. (2018).
5

---

Published as a conference paper at ICLR 2022
2.6 IMPROVING SAMPLE EFFICIENCY WITH EXMIX
We hypothesize that extreme multi-task scaling also improves the sample efficiency of pre-training. To test this, we exclude SuperGLUE from ExMIX, pre-
82
train a large model for 200k steps, and fine-tune on SuperGL.UE. at several intervals during early pre- training stages. We find that EXMIX pre-training significantly more sample-efficient than vanilla self- supervised pre-training. Note that at only 20k pre- training steps, our ExT5 model already achieves 75.8 SuperGLUE score, which outperforms a fully pre- trained BERT large model by about +4% (Wang et al., 2019a).
MODEL
76
15.11
20k
100K
200k
STEP
Figure SuperGLUE score of EXT5, ExT5LAROe arrr vs To a function of number of pre- training steps.
3 THE EXT5 MODEL
To overcome the challenges of multi-task co-training at scale, i.c. negative transfer and catastrophic forgetting explored in $2.1, the rest of this paper revisits the multi-task pre-training paradigm troduced by Raffel et al. (2020) via extreme multi-task scaling. This section introduces ExT5: pre-trained sequence-to-sequee to-sequence Transformer -ecodrrdecoodr model (Vaswani et al., 2017) based on the popular T5 framework.
3.1 TRAINING ExT5
Pre-training We pre-train on mixture of C4 and EXMIX ($2). and combine them with hyper- parameter that the ratio at which C4 examples are sampled with respect to ExMIX examples. The C4 objective we use is the same as that used by Raffel et al. (2020), and every task optimizes cross-entropy loss. We pre-train ExT5 on the same number of
the standard steps as T5, and ExT5 sees an identical number of tokens to the released T5 models. Concretely, we pre-train our models for 1M total steps with a batch size of 2048 and sequence length 512, resulting in total of approximately IT tokens seen by the model during pre-training (both unsupervised and supervised inclusive). We use the T..1. architecture (Shazeer, 2020) for all of our experiments which uses GEGLU-activated layers instead of ReLU in classic Transformer models (Vaswani al., 2017). For optimization, we use Adafactor with an inverse square root learning rate schedule that kicks in after a constant phase of 0.01 for 10k steps. EXT5 also uses the same tokenizer as T5.
Fine-tuning We follow the same fine-tuning procedure for T5 and ExT5 for fair comparison, al- though we found that ExT5 generally benefitted from a smaller learning rate while fine- tin-ttunig (10 (10-4 worked well for ExT5 vs 10 33 for T5 variants). Fine-grained details can be found in Appendix B.
3.2 EXPERIMENTAL SETUP
Our experiments consider both within-mixture and out-of-mixture tasks (i.e., whether a task is in- cluded ExMix). Within-mixture tasks measure the amount the task benefits from multi-task pre-training and extreme task scaling. Similar to the co-trained models in Raffel et al. (2020), we continue to fine-tune on target task from pre-trained ExT5 checkpoint. For out-of-mixture tasks, we consider possibly new unseen tasks or collections that were not included in the ExMIX mixture to test the effect of generalizing to unseen tasks. For the sake of brevity, the fine-grained details of these experimental setups can be found in the Appendix.
3.3 EXPERIMENTAL RESULTS
WITHIN-MIXTURE RESULTS
We report results SuperGLUE (Table 5), GEM (Table 6), Rainbow (Table 7). MsMarco (Ta- ble 8) and CBQA datasets (Table 9). On the whole, we observe that ExT5 consistently outperforms strong T5 baselines across range of model sizes. On SuperGLUUF, achieve +5%. 2.3% and

---

Published as a conference paper at ICLR 2022
+0.7% gain BASE, LARGE and XL respectively. On GEM, ExT5 outperforms TS 6 out of 9 collections while remaining on-par on the other collections. Notably, the gain on datasets such WebNLG approximately +11% ROUGE for the large model and generally range from +1% to +6% on different collections. On Rainbow, ExT5 outperforms our own run of TS by +0.7% on average and +4.6% improvement over the best multi-task (sequential) setup in Lourie et al. (2021). Finally, on question answering and ranking, ExT5 substantially outperforms T5 at two different sizes.
Model
BoolQ
CB
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
1
82.3
91.7/92.9
60.0
76.9/39.6
80.9/80.2
84.5
69.3
81.7
76.1
Ex ExT5RASE
82.6
98.7/98.2
73.0
79.5/45.4
80.8/80.0
87.0
71.3
83.7
79.9
L5.L.
88.3
94.3/96.4
87.0
85,4/55.1
89.2/88.5
90.6
73.5
88.5
85.3
ARGR
88.4
98.7/98.2
89.0
85.5/58.0
88.6/87.9
93.1
73.4
96.2
87.3
x
89.6
92.0/96.4
96.0
88.2/64.1
92.4/91.7
91.7
74.3
95.2
88.7
ExT5x1.
89.4
100/100
97.0
87.5/62.7
91.4/90.9
94.2
74.6
93.3
89.4
15.1.
90.4
100.0/100.0
99.0
88.6/63.9
91.0/90.1 93.3/92.7
92.1
78.5
95.2
90.2
ExT5xxI.
91.1
94.9/96.4
98.0
89.4/66.0
95.7
77.3
96.2
90.6
Table 5: Comparisons of T5 and ExT5 on SuperGIUUE validation sets.
Model
Metric
WebNLG
DART
SGD
E2E
CG
ToTTo
WiA-A
WiA-T
WLE
METEOR
0.323
0.364
0.325
0.383
0.201
0.366
0.302
0.368
0.189
T5. 1.1 nasE
ROUGE-2
39.46
45.62
36.25
47.40
17.32
49.8
38.58
51.54
19.19
BLEU
29.06
34.75
33.44
43.17
8.34
39.59
29.53
42.71
14.72
METEOR
0.349
0.367
0.330
0.382
0.206
0.368
0.306
0.367
0.192
ExTS
ROUGE-2
45.07
46.87
37.46
47.32
18.13
50.17
39.10
51.35
19.41
BLEUU
32.36
35.15
34.34
42.71
9.39
40.01
30.04
43.39
14.96
METEOR
0.344
0.363
0.324
0.382
0.202
0.368
0.301
0.362
0.196
T..
ROUGE-2
43.31
45.22
36.17
46.60
17.01
49,90
38.37
50.52
20.47
BLEU
31,67
34.31
33.15
42.57
8.38
39.79
29.30
42.12
15.55
METEOR
0.365
0.376
0.330
0.381
0,214
0.369
0.300
0.358
0.204
ExT5EE
ROUGE-2
48.17
48.14
37.77
46.70
19.04
50.33
37.98
50.38
21.16
BLEU
35.03
36.62
34.74
42.25
9.68
40.14
29.23
41.39
16.64
Table 6: Comparisons of T5 and ExT5 on GEM (English).
Model
oNLI
CosmosQA
HellaSwag
PIQA
SocialIQA
Winogrande
AVG
TSLARGE (multitask)"
78.40
81.10
81.30
80.70
74.80
72.10
78.07
TS,ARGE (scquential)"
79.50
83,20
83.00
82.20
75.50
78.70
80.35
T..L.
82.51
85.59
88.57
85.53
78.51
79.79
83.42
EXTSLARGE
82.25
85.86
88.99
85.04
79.73
82.53
84.07
% Gain
-0.3%
+0.3%
+0.5%
-0.6%
+1.6%
+3.4%
+0.8%
Table 7: Results the Rainbow Commonsense Reasoning benchmark validation sets. Results with are from Lourie et (2021).
Model
MRR 10
Model
NQ
WQ
TQA
TA (Nogueira et al., 2020)
0.393
I.
27.3
29.5
28.5
EXT5
0.402
EXT5
28.6
30,5
30.7
% Gain
+2.3%
% Gain
+4.8%
+3.4%
+7.7%
x5 (Nogueira et al., 2020)
0.398
15.
29.5
32.4
36.0
ExT5x1.
0,403
xxT5xt.
30.6
35.2
37.0
% Gain
+ +1.3%
% Gain
+3.7%
+8.6%
+2.8%
Table 8: Results on MSMarco.
Table 9: Results on CBQA dev sets.

---

Published as a conference paper at ICLR 2022
OUT-OF- RESULTS
We are also interested in evaluating EXT5 on tasks outside of ExMIX, and hypothesize that the extreme multi-task pre-training of EXT5 will lead to better performance on new unseen settings. Concretely, we fine-tune and evaluate on Machine Translation: translating sentences from English to other languages (Bojar et al., 2014; 2015; 2016); Reasoning: answering scientific questions on ARC (Clark et al., 2018); and Named Entity Recognition: extracting all entities from sentences on the C0NLL-20033 NER dataset (Tjong Kim Sang & De Meulder, 2003).
Machine Translation EnFr
QA ARC
NER
Model
EnDe
EnRo
Dev
Test
Raffel al. (2020)
26.98
39.82
27.65
T
28.30
41.01
28.11
26.45
92.55
85.75
ExTSaASE
28.32
41.89
28.38
36.35
92.68
86.53
% Gain
+10%
+2.1%
+1.0%
+37.4%
+0.13%
+0.91 %
T5.1 LARGE
28.68
41.34
29.01
55.80
92.80
86.56
Ex EXTSL
28.98
42.71
29.49
63.99
93.63
87.34
% Gain
+1.0%
+3.3%
+1.7%
+14.7%
+0.90%
+0.90%
Table 10: Experimental results on tasks that are not in ExMIx. For ARC, we report test scores on the challenge set with retrieval. For NER, we report accuracy on a sentence level (see Appendix B.2).
Table 10 summarizes the results on the out-of-mixture tasks. Across tasks, we see that EXT5 outperforms upon T5 baselines. The largest improvement is on the ARC scientific reasoning task, perhaps due to the large amount of QA tasks in ExMIX. Though, the trend is consistent also with the NER and MT tasks that do not have any similar dataset ExMIX. This suggests that the representations learned by EXT5 are more general adaptable to a new objective, even when the language.
output is in
This improved generalization of ExT5 is very encouraging from a practical stand point, since pre- training again with ExMIx U {t) for any target task t would be very expensive. Instead, see that the extreme multi-task pre-training of ExT5 already provides improved results. Therefore, it might only be worth repeating pre-training when the collection of training datasets grows by significant amount (see $2.5).
4 RELATED WORK
Improving NLP models with Multi-task Learning Collobert Weston (2008) leverage multi- task learning for relatively simple tasks like Part-of-Speech tagging. Phang et al. (2019) use an intermediate fine-tuning stage using four tasks with large datasets for Natural Language Under- standing. Similarly, Liu et al. (2019a) proposed MT-DNN, which uses setup at a scale of around 30 tasks and up to 440M parameters. Most recently, Aghajanyan et al. (2021) use around 50 tasks and models of sizes upto 440M parameters. Gururangan et al. (2020) take an alternative approach, which is to continue pre-training a model but use domain-specific data as an intermediate step. Mc- Cann al. (2018) proposed unified framework similar to that of T5. Recently, Wei et al. (2021) also illustrated how a multi-task learning stage can greatly improve the zero-shot prompting perfor- mance of large language models the scale of -37B parameters. Efforts have also been made to tailor pre-training objectives to specific tasks, e.g., question answering (Ram et al., 2021; Jia et al., 2021), dialogue (Li al., 2020), and span selection tasks (Joshi al., 2020).
Relationships amongst different tasks Bingel & Søgaard (2017) conducted a study similar to ours in $2.1 but for more traditional NLP tasks like chunking, CCG tagging, POS tagging, etc. More recently, Vu et al. (2020) conducted an in-depth study of relationships between various classifica- tion/regression, question-answerte framework to predict such relationships. Khashabi et al. (2020) also conducted similar experiments but specific to question-answent
and sequence-labeling tasks, and proposed a task-embedding
datasets/formats, resulting in a strong QA model known as Uni- fiedQA that is also based on the T5 framework. Outside of NLP, Zhang & Yeung (2010) introduced a convex optimization objective for learning task relationships, and Li et al. (2018) explore and exploit task relationships on variety of diverse datasets.
8

---

Published as a conference paper at ICLR 2022
Choosing which tasks to transfer from Our experiments in $2.2 attempted empirically select a set of tasks to transfer from. Along these lines, Ruder & Plank (2017) use Bayesian Optimization method with similarity measures to automatically select relevant data from different domains. Sim- ilarly, Guo et al. (2019) use multi-armed bandits to select tasks and a Gaussian Process to control the mixing rates for the selected tasks. Another strand of recent work selects appropriate transfer languages based on manually defined features (Lin et al., 2019; Sun et al., 2021). Aside from the NLP domain, Fifty et al. (2021) proposed method select which tasks to transfer based on task gradients. All of the aforementioned works select data tailored to downstream task of inter- est. a general pre-trained model was attempted to be trained in similar fashion, computational bottlenecks similar to those motivating $2.1 and $2.2 would arise.
Pre-trained Transformers Transformer models (Vaswani et al., 2017) such as T5 (Raffel et al., 2020), BERT (Devlin et al., 2019) and GPT-3 (Brown al., 2020) rely large unlabeled corpus for self-supervised learning. Given the wild success of the suitable pre-training tasks has also become an active area of research (Lewis al., 2019; Lan al., 2019; Chang et al., 2020; Zhang et al., 2019; Lourie et al., 2021). While there has been evidence that supplementary pre-training tasks can help improve performance, this work is the first massive-scale multi-task pre-trained model.
paradigm, the search for
Scaling Laws Scaling laws for Transformers have attracted much attention recently, especially pertaining model size (Kaplan et al., 2020; Zhai et al., 2021; Tay et al.. 2021a). In Kaplan et al. (2020), the authors further investigate scaling with respect to dataset size (on the same pre-training corpus). To this end, this work can be interpreted an attempt of scaling up with respect to the number of high quality, diverse labeled tasks that can be used for pre-training.
5 EPILOGUE
Limitations Despite our best efforts to evaluate on as many representative tasks as possible while also maintaining a balance among task partitions for given set of transfer learning experiments, any study that explicitly abstracts datasets into "task families" is highly dependent on nuances pertaining to the nature, domain, and expressiveness of the task family's representative datasets. For this paper, the subsets were constructed so as to include diverse set of datasets to evaluate on, and we tried to partition task samiifaissess to be as mutually exclusive as possible. However, it must be acknowledged that no dataset is perfectly isolated, and any set of them only a proxy for a larger "task family". On separate note, lexical metrics like BLEU/ROUGE are useful but do not paint the full picture of how well model truly performs on text-generation tasks.
Future Work We believe that a multilingual version of EXT5 would be a natural extension of this work. Such model will require extra care with regard to balancing not only task families, but also task languages. A multilingual version of EXMIX could provide a more robust foundation for the analysis of task families in existing works that analyze how multilingual NLP models transfer amongst different languages (Kudugunta et al., 2019; Hu et al., 2020; Wang et al., 2021). For exam- ple, it would be interesting to understand whether our results in $2.1 hold across different languages (and language families), and to explore cross-lingual cross-task generalization. We also hypothesize that modeling innovations that introduce inductive biases designed to exploit multi-task learning se- tups (Ha et al., 2016; Tay et al., 2021b) can push the boundary of the strong performance displayed by EXT5. Other solutions like gradient manipulation (Yu et al., 2020; Wang et al., 2021) might also further improve extreme multi-task scaling, albeit at the cost of more complex implementations.
Conclusion This paper explores how supervised multi-task learning at massive scale can be used to improve existing self-supervised pre-training strategies for NLP models, and does so by introducing EXMIX ($2) and EXT5 ($3). Our experiments showed that while negative transfer is common when fine-tuning on diverse tasks ($2.1), scaling up the number of tasks to include in multi-task task pre-training setup enables strong downstream performance ($3.2) with better sample- efficiency ($2.6). We hope that this paper motivates future research on how existing labeled datasets can be used further improve NL.P models within the pre-train/fine-tun paradigm.

---

Published as a conference paper at ICLR 2022
ACKNOWLEDGEMENTS
The authors would like to thank Mostafa Dehghani and Adhi Kuncoro for valuable comments and insights. We would also like to thank the authors of Mesh Tensorflow (Shazeer al., 2018) and T5 (Raffel et al., 2020), their high-quality code and paper enabled this work.
AUTHOR CONTRIBUTIONS
Vamsi co-led the project and was primarily responsible for the design, implementation, and engi- neering effort behind it. Vamsi proposed and ran key experiments including (but not limited to): task-family transfer analysis, early proof-of-concepts for extreme task-scaling and ExT5, task- scaling analysis, etc. Vamsi also wrote most of the paper, and was (jointly) responsible for the overall framing of the paper.
Yi served the co-lead of the paper and proposed the initial idea. Yi was responsible for most of the downstream ExT5 experiments, including SuperGLUE, Rainbow, CBQA, and Machine translation, along with large-scale pre-training of ExT5. Yi also wrote large portions of the paper.
Tal was (jointly) responsible for the overall framing the paper, and wrote large portions of it. Tal contributed the Vitamin C task to the EXMIX mixture, along with running out-of-mixture experi- ments for Named Entity Recognition.
Jinfeng contributed the GEM benchmark mixture and was heavily involved in running exper- iments.
Steven contributed the DialoGLUE tasks to ExMIx, helped run substantial number of experiments and contributed substantially to discussions around the paper's framing.
Sanket responsible for and ran experiments GEM, and contributed the Yahoo Answers and Argument mining tasks.
Honglei contributed the MsMarco task ExMIX and helped with benchmarking ExT5 Ms- Marco. Honglei also contributed scripts and pipelines for obtaining reranking results on MsMarco.
Vinh contributed NewsQuiz, AgreeSum, TweetQa and TweetEval to ExMIX and contributed sub- stantially to paper writing and the framing of the paper.
Dara contributed GPT DeepFake tasks and low-resource tasks (which not used in the end).
Jianmo contributed the DocNLI task and helped with running experiments for out-of-mixture tasks.
Jai contributed the Race and MultRCC Eraser tasks to the ExMIX and helped edit the paper.
Kai contributed several retrieval tasks to EXMIX, which were not included eventually.
Sebastian helped substantially with the paper narrative, writing of the paper and brainstorming.
Donald (along with Yi and Vamsi) was heavily involved in framing the early vision the project. Donald was also deeply involved in brainstorming sessions, and provided critical feedback that helped to steer the project in the right direction.
All authors contributed brainstorming and discussion sessions.
ETHICS STATEMENT
Large language models have been shown to capture certain biases about the data they have been pre-trained on (Bender al., 2020). While comprehensive analysis of such biases is outside of the scope of this work, it is a compelling direction to investigate to what extent the inclusion of supervised data during pre-training can help mitigate such biases. An alternative consideration is the addition of diverse values-targeted data (Solaiman Dennison, 2021) during pre-training in order instill beneficial biases in model.
Another challenge when training large models is their energy consumption and environmental im- pact (Strubell et al., 2019). To ablate different task combinations, we performed experiments using
10

---

Published as a conference paper at ICLR 2022
the more computationally efficient fine-tuning setup. We have shown that ExMIX leads to more sample-efficient pre-trainin vre-train compared to standard self-supervision, which we hope will save com- pute in future experiments.
REPRODUCABILITY STATEMENT
All of the modeling and training code used for ExT5 and its variants is already open-sourced part of the Mesh Tensorflow' (Shazeer et al., 2018) and T5² (Raffel et al., 2020) Libraries. Addi- tionally, EXMIX is composed of datasets that already publicly available.
REFERENCES
Amazon reviews dataset.
canssttosspppp:///s....eeooooss readme.. Saaime.hter -tmml.
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-tast representations with pre-finetuning. 2021.
Payal Bajaj, Daniel Campos, Nick Craswell, Deng, Jianfeng Gao, Xiaodong Liu, Rangan Ma- jumder, Andrew McNamara, Bhaskar Mitra, Tri Nguyen, Mir Rosenberg, Xia Song, Alina Stoica, Saurabh Tiwary, and Tong Wang. Ms marco: human generated machine reading comprehension dataset, 2018
Francesco Barbieri, Jose Camacho-Collade Francesco Ronzano, Luis Espinosa-Anke, Miguel Ballesteros, Valerio Basile, Viviana Patti, and Horacio Saggion. Semeval 2018 task 2: Multi- lingual emoji prediction. In Proceedings of The 12th International Workshop on Semantic Eval- uation, pp. 24-33, 2018.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. Tweet- Eval: Unified benehmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1644-1650, Online, 2020. Association for Computational Linguistics. doi: 10.18653/v1/2022 1/2020. 1200 mendings-emmll.145. URL org/ 2020.022222 findinin- 1p. 1 148.
https: http:://acaaathhhccco
Valerio Basile, Cristina Bosco, Elisabetta Fersini, Debora Nozza, Viviana Patti, Francisco Manuel Rangel Pardo, Paolo Rosso, and Manuela Sanguinetti. SemEval-2019 task 5: Multilingual de- tection of hate speech against immigrants and women in Twitter. In Proceedings of the 13th International Workshop on Semantic Evaluation, pp. 54-63, Minncapolis, Minnesota, USA, 2019. Association for Computational Linguistics. doi: 10. 10.18663/v1/S11-2000 URL https: / //www. aclweb. org/ http///nnhoooyyy/ooooooo S10-2077.
Jonathan Baxter. A model of inductive bias learning. Journal of artificial intelligence research, 12: 149-198, 2000.
Emily Bender, Timnit Gebru, Angelina McMillan-Major, and Shmargaret Shmitchell, On the dan- gers of stochastic parrots: can language models be big? In Proceedings of the ACM/IEEE Joint Conference on Digital Libraries, volume 1, pp. 27 172278 Association for Computing Ma- chinery, 2020. ISBN 9781450375856. doi: 10.1145/3442188.344555
Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. Semantic parsing on Freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1533-1544, Seattle, Washington, USA, 2013. Association for Compu- tational Linguistics. URL https: http:///aaaaatttt////a/ org Carrg/DD331111 1160.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Hannah Rashkin, Doug Downey, Wen tau Yih, and Yejin Choi. Abductive commonsense reasoning. In International Conference on Learning Representations, 2020. URL https: //oppn/eeriee...tt///oommaaaaaap
httpe:://gttabb.ccc///essrrfo/// http:///w..hhtc.c...o.....ee.ae.

---

Published as a conference paper at ICLR 2022
Joachim Bingel and Anders Søgaard. Identifying beneficial task relations for multi-task learning in deep neural networks. In Proceedings of the 15th Conference of the European Chapter of the As- sociation for Computational Linguistics: Volume 2, Short Papers, pp. 164-169, Valencia, Spain, April 2017. Association for Computational Linguistics. URL https: : /awllaaatolloooooooooooooooo ogy.
org/ / E 2026.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. Piqa: Reasoning about physical commonsense in natural language. In Thirty- AAAI Conference on Artificial Intelligence, 2020.
Ondfej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Lev- cling, Christof Monz, Pavel Pecina, Matt Post, Herve and Aleš Tamchyna. Findings the 2014 workshop on statistical machine translation. In Pro- ceedings of the Ninth Workshop Statistical Machine Translation, pp. 12-58, Baltimore, Mary- land, USA, June 2014. Association for Computational Linguistics. doi: 1/3111////W14-33302 URL httpt:tpp//..m. /aa/caaaaatttt www.aaaannnnllol.aaaoooaoo orgg /e/g//144330222
Radu Soricut, Lucia Specia,
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Barry Haddow, Matthias Huck, Chris Hokamp, Philipp Koehn, Varvara Logacheva, Christof Monz. Matteo Negri, Matt Post, Car- olina Scarton, Lucia Specia, and Marco Turchi. Findings of the 2015 workshop on statistical machine translation. In Proceedings of the Tenth Workshop on Statistical Machine Translation, pp. 1 1-46, Lisbon, Portugal, September 2015. Association for Computational Linguistics. doi: 10. 101//////11-10000 URL https::tt///.. pestts://aclaante httaa/ccaanttooooooooooo org/W15 -3001.
Ondřej Bojar, Rajen Chatterjee, Christian Federmann, Yvette Graham, Barry Haddow, Matthias Huck, Antonio Jimeno Yepes, Philipp Koehn, Varvara Logacheva, Christof Monz, Matteo Negri, Aurélie Névéol, Mariana Neves, Martin Popel, Matt Post, Raphael Rubino, Carolina Scarton, Lu- cia Specia, Marco Turchi, Karin Verspoor, and Marcos Zampieri. Findings of the 2016 conference on machine translation. In Proceedings of the First Conference Machine Translation: Volume 2. Shared Task Papers. pp. 131-198, Berlin, Germany, August 2016. Association for Compu- tational Linguistics. doi: 10.18653//1/116-23001 URL https: http:::://aaaatttoolooo -16-2301.
org/
Daniel Borkan, Lucas Dixon, Jeffrey Sorensen, Nithum Thain, and Lucy Vasserman. Nuanced metrics for measuring unintended bias with real data for text classification. ion Proceedings of The 2019 World Wide Web Conference, WWW '19, pp. 491-500, New York, NY, USA, 2019. Association for Computing Machinery. ISBN 9781450366755. doi: URL https: /tw.i/oiioo/// orgg 11 404455333366000 3317593.
Compun-
10. N0.1145/33085003331755
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agar- wal, Ariel Herbert-Voss, Gretchen Krueger. Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse. Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCan- dlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020. URL https: http:///cccceeeeeeeeeeeee / 1457c0d6bfcb499 2a53:008bb....1111a.BBBB..222--2 Abst html.
neurips cc httpp//ppp////haaas//
R. Caruana, D. L. Silver, J. Baxter, T. M. Mitchell, L. Y. Pratt, and S. Thrun. Learning to learn: knowledge consolidation and transfer in inductive systems, 1995.
Rich Caruana. Multitask learning. Machine learning, 28(1):41-75, 1997.
Iñigo Casanueva, Tadas Temčinas, Daniela Gerz, Matthew Henderson, and Ivan Vulić. Efficient intent detection with dual sentence encoders, 2020.
Thiago Castro Ferreira, Claire Gardent, Nikolai llinykh. Chris der Lee, Simon Mille, Diego Moussallem, and Anastasia Shimorina. The 2020 bilingual, bi-directional webnlg+ shared task overview and evaluation results (webnlg+ 2020). In Proceedings of the 3rd WebNLG Workshop
12

---

Published as a conference paper at ICLR 2022
Natural Language Generation from the Semantic Web (WebNLG+ 2020), pp. 55-76, Dublin, Ireland (Virtual), 2020. Association for Computational Linguistics.
Daniel Cer, Mona Diab, Eneko Agirre, lñigo Gazpzo, and Lucia Specia. SemEval-2017 task
I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-200 -0EEVl)--01), pp. 1-14, Vancouver, Canada, 2017. Association for Computational Linguisties. doi: 10.18653v1//S77-2 ht https: httpss//acaaatthllogggoooo www.:/oww...r//1--000000
URL
Wei-Cheng Chang, Felix X Yu, Yin-Wen Chang, Yiming Yang, and Sanjiv Kumar. Pre-training tasks for embedding- baaedd large-scale retrieval. arXiv preprint arXiv:2002.03932 2020.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer. QuAC: Question answering in context. In Proceedings of the 2018 Con- ference Empirical Methods in Natural Language Processing, pp. 2174-2184, Brussels, Bel- gium, 2018. Association for Computational Linguistics. doi: 10.18653/v1/D1-- 1241. URL https: /aaalannnhhlogoooo wwwphhooogooooggggoge 101g/D18-1211.1.
Christopher Clark, Kenton Lee, Ming-Was Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. BoolQ: Exploring the surprising difficulty of natural yes/no questions. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 2924-2936, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/ N19-1300. URL https: / /aclantholloyyyyo.. wwwp/19-1g11110000
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try the ai2 reasoning challenge. ArXiv, 1bs//803.05457, 2018.
Ronan Collobert and Jason Weston. A unified architecture for natural language processing: Deep neural networks with multitask learning. In Proceedings of the 25th International Conference on Machine Learning, ICML '08, pp. 160-167, New York, NY, USA, 2008. Association for Computing Machinery. ISBN 978 17055825525 doi: 10.1145/139015. 191.199991166111117 URL https: //doi.org/10....//////
/ //doi. org/ 10. 1ww.11111111151..11 1390177.
Ido Dagan, Oren Glickman, and Bernardo Magnini. The pascal recognising textual entailment chal- lenge. In Joaquin Quinonero-Candela, Ido Dagan, Bernardo Magnini, and Florence d' Alché Buc (eds.), Machine Learning Challenges. Evaluating Predictive Uncertainty, Visual Object Classi- fication, and Recognising Tectual Entailment, pp. 177-190, Berlin, Heidelberg, 2006. Springer Berlin Heidelberg. ISBN 978-3-540-33422-6.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. The CommitmentBank: In- vestigating projection in naturally occurring discourse. 2019. To appear in proceedings of Sinn und Bedcutung 23. Data can be found at http://wwww..ooooom/mmooomommeee
Dorottya Demszky, Dana Movshovitz-Atiaa Jeongwoo Ko, Alan Cowen, Gaurav Nemade, and Sujith Ravi. GoEmotions: A dataset of fine-grained emotions. In Proceedings of the 58th An- nual Meeting of the Association for Computational Linguistics, pp. 4040-4054, Online, 2020. Association for Computational Linguistics. doi: 10. we11116/11/222..ccm 2tt.:/022...mmm..3777..... https: / /oaloatthlloyy. org//0200 acl manlmmmain..... 372.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics; Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4171-4186, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N19-165 1423. URL https: http:://cllattttt lowhoolooyy..... loy.. org/N1 1423.
Jay Young, Sarthak Jain, Nazneen Fatema Rajani, Eric Lehman, Caiming Xiong, Richard Socher, and Byron C. Wallace. ERASER: A benchmark to evaluate rationalized NLP models. In Proceed- ings of the 58th Annual Meeting the Association for Computational Linguistics, pp. 4443-4458, Online, 2020. Association for Computational Linguistics. doi: 10. 10.18653/1122022.ac URL https::////......... / a waaclannnhooo wwwaaannhhooyooooooooo ogg. 2020 acl -ma in. 408.
13

---

Published as a conference paper at ICLR 2022
Emily Dinan, Stephen Roller, Kurt Shuster, Angela Fan, Michael Auli, and Jason Weston. Wizard of Wikipediia: srowledge-powereee conversational agents. In Proceedings of the International Conference on Learning Representations (ICLR), 2019.
William B. Dolan and Chris Brockett. Automatically constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), 2005. URL
ptttps://aclantholoogg org/ I05-5002.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gard- ner. DROP: reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pp. 2368 2378, Minneapolis, Minnesota, 2019. Association for Computational Linguistics. doi: 10.18653/v 1ttp/////199-11--1 URL https: http://caaaattttoooo/oooo org/N 9214..
Ondiej Dušek, David M Howeroft, and Verena Rieser. Semantic Noise Matters for Neural Natural Language Generation. In Proceedings of the 12th International Conference on Natural Language Generation (INLG 2019), pp. 421-426, Tokyo, Japan, 2019, URL https: http:://wwww.. /www//wwwoo/oooo////oo// aclweb. org/anthology / /W1 9 8652/.
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. Multi-news: large- scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 1074-1084, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10. 10.18653/11/P19-1102. URL https: /aaaaaaaaanlllyo.oeee org//1 9- 1102.
Claire Cardie Faisal Ladhak, Esin Durmus and Kathleen McKeown. Wikilingua: A new benchmark dataset for multilingual abstractive summarization, In Findings of EMNLP, 2020, 2020.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. Eli5:
Long form question answering, 2019.
Christopher Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. Efficiently identifying task groupings for multi-task learning. arXiv preprint (wXiv:2109.04111 2021.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura Perez-Beltrachhni.. Creating train- ing corpora for nlg micro-planners. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguisties (Volume 1: Long Papers). pp. 179-188. Association for Compu- tational Linguistics, 2017. doi: 10. 10.18655/w1/117-10077 URL http::/t///tt. //www//wwwwwww aclweb. org/ (aclooyy1110 /01/l////1111111
anthology/PI1
Sebastian Gehrmann, Tosin Adewumi, Karmanya Aggarwal, Pawan Sasanka Ammanamanchi, Anuoluwapo Aremu, Antoine Bosselut, Khyathi Raghavi Chandu, Miruna-Adriana Clinciu, Dipanjan Das, Kaustubh Dhole, Wanyu Du, Esin Durmus, Ondřej Dušek, Chris Chinenye Emezue, Varun Gangal, Cristina Garbacea, Tatsunori Hashimoto. Yufang Hou, Yacine Jer- nite, Harsh Jhamtani, Yangfeng Ji, Shailza Jolly, Mihir Kale, Dhruv Kumar, Faisal Ladhak, Aman Madaan, Mounica Maddela, Khyati Mahajan, Saad Mahamood, Bodhisattwa Prasad Ma- jumder, Pedro Henrique Martins, Angelina McMillan-Majon Simon Mille, Emiel van Miltenburg, Moin Nadeem, Shashi Narayan, Vitaly Nikolaev, Andre Niyongabo Rubungo, Salomey Osei, Ankur Parikh, Laura Berez-Beltracchni, Niranjan Ramesh Rao, Vikas Raunak, Juan Diego Ro- driguez, Sashank Santhanam, João Sedoc, Thibault Sellam. Samira Shaikh, Anastasia Shimorina, Marco Antonio Sobrevilla Cabezudo, Hendrik Strobelt, Nishant Subramani, Wei Xu, Diyi Yang, Akhila Yerukola, and Jiawei Zhou. The GEM benchmark: Natural language generation, its eval- uation and metrics. In Proceedings of the 1st Workshop on Natural Language Generation, Eval- uation, and Metrics (GEM 2021), pp. 96-120, Online, 2021. Association for Computational Lin- guistics. doi: 10.18653/w1/2221.gem 1.m... URL https: / //allll anthology or2/2021. gem-1.10.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. SAMSum corpus: A human- annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop New Frontiers in Summarization, pp. 70-79, Hong Kong, China, 2019. Association for Com- putational Linguistics. doi: 10.18653/v11119 5409. URL https: hwl.antooooggyy....o/ooo .xrg D1 119955099
14

---

Published as a conference paper at ICLR 2022
Alec Go, Richa Bhayani, and Lei Huang. Twitter sentiment classification using distant supervision, 2009. URL http: /tttp::/hhlppp /htlp.eepc....e sent inttitieenti. 140. noom..
Han Guo, Ramakanth Pasunuru, and Mohit Bansal. AutoSeM Automatic task selection and mixing in multi-task learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pp. 3520-3531, Minneapolis, Minnesota, June 2019. Association for Com- putational Linguistics. doi: 10. 1t.11853/11//19111 1355. URL https: /aaalaaaannnooooyyy.o . N1 9- 1355.
Jiaqi Guo, Qian Liu, Jian Hian-Guang Lou, Zhenwen Li, Xueqing Liu, Tao Xie, and Ting Liu. Bench- marking meaning representations in neural semantic parsing. In Proceedings of the 2020 Confer- ence on Empirical Methods in Natural Language Processing (EMNLP), pp. 1520-1540, Online, 2020. Association for Computational Linguistics. doi: 10. 1 ht
118. URL
org/ 2020. emnlp momnlp-mai.... mmiin.18..
Zhaochen Guo and Denilson Barbosa. Robust entity linking via random walks. In Proceedings of the 23rd ACM International Conference on Conference on Information and Knowledge Management, CIKM 2014, Shanghai, China, November 3-7, 2014, pp. 499-4099000 508, 2014. doi: 10.1 00.1145/26618 829. 2661887. URL http: http://oo..ccc.ccc.ccooooom.oo.. acm. org / 10. 1145/2661829..... 2661887.
Sonal Gupta, Rushin Shah, Mrinal Mohit, Anuj Kumar, and Mike Lewis. Semantic parsing for task oriented dialog using hierarchical representations, 2018.
Suchin Gururangan, Ana Marasovié, Swabha Swayamdipta, Kyle Lo, Iz Beltagy. Doug Downey, and Noah A. Smith. Don't t stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pp- 1342-8360, Online, July 2020. Association for Computational Linguistics. doi: 10. 1863vvvvv 2020.acl- main. 740. URL https: hatp:////caaathogy.yo.. org/2020 aal-mmin....ammm 74C.
David Ha, Andrew Dai, and Quoe V. Le. Hypernetworss, 2016.
Jonathan Herzig, Peter Shaw, Ming- Wei Chang, Kelvin Guu, Panupong Pasupat, and Yuan Zhang. Unlocking compositional generalization in pre-trained models using intermediate representations, 2021.
Eduard Hovy, Laurie Gerber, Ulf Hermjakob, Chin- Lin, and Deepak Ravichandran. To- ward semantics-baid answer pinpointing. Proceedings of the First International Conference Human Language Technology Research, 2001. URL https:/ /at./atoaannnoooy....... orr/ H01-1069.
Junjie Hu, Sebastian Ruder, Aditya Siddhant, Graham Neubig, Orhan Firat, and Melvin Johnson. XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Gener- alization. In Proceedings of ICML. 2020, 2020.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Confer- ence on Natural Language Processing (EMNLP-IJCNLPP pp. 2391-2401, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10.18553v1//11--122 1243. URL httpss://eet //cl antho anttooooyy.o...oo.ooooooooooooot ogy. DTgg/11-1145
Harsh Jhamtani, Varun Gangal, Eduard Hovy, and Eric Nyberg. Shakespearizing modern language using copy-enriched sequence to sequence models. In Proceedings of the Workshop on Stylistic Variation, pp. 10-19, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10. 10.118653/1//11-40000 /
URL httph:tt////.. http://aaaantoooooo.ooooo wwgpw17rr///111 hatps////llaaccooooo..oooo///// 4902.
Robin Jia, Mike Lewis, and Luke Zettlemoyer. Question answering infused pre-training of general- purpose contextualized representations, 2021.
15

---

Published as a conference paper at ICLR 2022
Chao Jiang, Mounica Maddela, Wuwei Lan, Yang Zhong, and Wei Xu. Neural CRF model for sentence aligmment in text simplification. In Proceedings of the 58th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 7943-7960, Online, July 2020. Association for Com- putational Linguistics. doi: 10. 00.18653/v1/20200 awl.://0...mmmmai.00000 URL https: htwp///...........o/ ac web. org/ org/anthology. / 2200 acl -mal-maalia m 709.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. TriviaQA: A large scale dis- tantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th An- nual Meeting of the Association for Computational Linguistics (Volume Long Papers), pp- 1601-1611, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/v1/P17-11
URL https:/ www.anthoogg.ooogg....ooo org/P17-1144
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. Span- bert: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77, 2020.
Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models. arXiv preprint arXiv:2001.08361 2020.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Looking be- yond the surface: challenge set for reading comprehension over multiple sentences, In Proceed- ings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume (Long Papers), pp. 252-262, New Or- leans, Louisiana, 2018. Association for Computational Linguistics. doi: 10.18863/v1/N188 1023. URL https:://///. / ac wocccannhhllgg org
/rgg/N18 N1 1023.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, On- line, 2020. Association for Computational Linguistics. doi: 10.18653/v 1/202/ findings-emnlp. 171. URL https: //laaaannnnnoyyyoaaaaaaaaaaalllt wtg//020. findings-eeeiiiii emnlp 17.
Najoung Kim and Tal Linzen. COGS: A compositional generalization challenge based semantic interpretation. In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 9087-9105, Online, 2020. Association for Computational Lin- guistics. doi: 00.18653/1122020emm//p 731. URL https: https////caatthoogyc.. /ttpsp////acaaattllll 2020. emnlp-main 731.
org/
Sneha Kudugunta, Ankur Bapna, Isaac Caswell, and Orhan Firat. Investigating multilingual NMT representations at scale. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Pro- cessing (EMNLP-IJCNLL) pp. 1565 1575, Hong Kong, China, 2019. Association for Compu- tational Linguistics. doi: 10.18653/v1/D19-1111 D1 019--167.
URL https://acae http:///aaaaatggg..a.oommm .x/g/
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming- Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. Natural Questions: A Benchmark for Question Answering Research. Transactions of the Association for Computational Linguistics, 7:453-466, 08 2019. ISSN 2307-387X. doi: URI https::t//// doi. .rgg / 0. 11 tacaaaccc http:ttaall____22211 __2766.
10.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pp. 785-794, Copenhagen, Denmark, 2017. Association for Computational Linguistics. doi: 10. 18653//11DD17-1022 URL https: / //aclanthology wttpthhoooooyyooo111-1-11-
org/D17-108t
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Sori- cut. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv: 1942, 2019.
16

---

Published as a conference paper at ICLR 2022
Stefan Larson, Anish Mahendran, Joseph J. Peper, Christopher Clarke, Andrew Lee, Parker Hill, Jonathan K. Kummerfeld, Kevin Leach, Michael A. Laurenzano, Lingjia Tang, and Jason Mars. An evaluation dataset for intent classification and out-of-scope prediction. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th In- ternational Joint Conference Natural Language Processing (EMNLP-IJCC P-IJCNLPP). pp. 1311- 1316, Hong Kong, China, November 2019. Association for Computational Linguistics. doi: 10. 10.11663//1/D19-1133 URL https: /aa/caaaanthhe lolgttooooon org/D19-111 1131.
Adam D. Lelkes, Vinh Q. Tran, and Cong Yu. Quiz-Style Question Generation for News Stories, pp. 2501-2511 Association for Computing Machinery, New York, NY, USA, 2021. ISBN
9781450383127. htRLt:t::///aa oreeiiorgg11... 100.0.115/342235 34492282.2.4444892.. 3449892.
Hector J Levesque, Emest Davis, and Leora Morgenstern. The Winograd schema challenge. In AAAI Spring Symposium: Logical Formalizations of Commonsense Reasoning, volume 46, pp.
47, 2011.
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising sequence-to-sequenee pre- training for natural language generation, translation, and comprehension. arXiv preprint 134611 2019.
Junlong Li, Zhuosheng Zhang, Hai Zhao, Xi Zhou, and Xiang Zhou. Task-specific objectives of pre-trained language models for dialogue adaptation. 2020.
Ya Li, Xinmei Tian, Tongliang Liu, and Dacheng Tao.
On better exploring and exploiting task rela- tionships in multitask learning: Joint model and feature learning. IEEE Transactions on Neural Networks and Learning Systems, 1975) May 2018. ISSN 2162-2388. doi: 10.1109/ 2onls.001..2696633 URL http:
dx. doi. org/ 10. 1109/TNNLS TNNLS. 2017 01117..26666..
Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou. Chandra Bhagavatula, Yejin Choi, and Xiang Ren. CommonGen: A constrained text generation challenge for generative commonsense reasoning. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 1823-1840, Online, November 2020. Association for Computational Linguistics. URL https: / //www. aclweb weeb. org/ wnw.aaogyoooyyoooeooooo 2220 findings emmlp. 165.
Yu- Yu-Hsiang Lin, Chian-Ya Yu Chen, Jean Lee, Zirui Li, Yuyan Zhang, Mengzhou Xia, Shruti Rijhwani, Junxian He, Zhisong Zhang, Xuezhe Ma, Antonios Anastasopoulos, Patrick Littell, and Graham Neubig. Choosing Transfer Languages for Cross-Lingual Learning. In Proceedings of ACI.2019, 2019. URL http: /ttp:iaaavvvvvv//v orgg/rrgaaa... abs/ /190/955.0 12688.
Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. Multi-task deep neural networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Asso- ciation for Computational Linguistics, pp. 4487-4496, Florence, Italy, 2019a. Association for Computational Linguistics. doi: 10.18653/v 1/t.//////11911/ 1441. URL https: hwtp//aaaaattto..............a.. org/P19-144
Xingkun Liu, Arash Eshghi, Pawel Swietojanski, and Verena Rieser. Benchmarking natural lan- guage understanding services for building conversational agents, 2019b.
Nicholas Lourie, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Unicorn on rainbow: A universal commonsense reasoning model on a new multitask benchmark. AAAI, 2021.
Yiwei Lyu, Paul Pu Liang, Hai Pham, Eduard Hovy, Baraaáss Póczos, Ruslan Salakhutdinov, and Louis-Philiipe Morency. StylePTB: A compositional benchmark for fine-grained controllable text style transfer. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2116-2138, Online, 2021. Association for Computational Linguistics. doi: 10. 10.18653/v1/0 1/2521 htt.2/......n.l.-.1.1 main. 171. URL https:
org/ 22g/2022 2021 maacl-main.. 171.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pp. 142-150, Portland, Oregon, USA, 2011. Association for Computational Linguistics. URL https:/ acl cotthelogy.org/Peee
17

---

Published as a conference paper at ICLR 2022
Sam McCandlish, Jared Kaplan, Dario Amodei, and OpenAl Dota Team. An empirical model of large-batch training, 2018.
Bryan McCann, Nitish Shirish Keskar, Caiming Xiong, and Richard Socher. The natural language decathlon: Multitask learning as question answering, 2018.
Shikib Mehri, Mihail Eric, and Dilek Hakkani-Tur. Dialoglue: A natural language understanding benchmark for task-oriented dialogue, 2020.
Saif Mohammad, Svetlana Kiritchenko, Parinaz Sobhani, Xiaodan Zhu, and Colin Cherry. Semeval- 2016 task 6: Detecting stance in tweets. In Proceedings of the 10th International Workshop Semantic Evaluation (SemEval-2016), pp. 31-41, 2016.
Saif Mohammad, Felipe -ravo-Marquez, Mohammad Salameh, and Svetlana Kiritchenko. Semeval- 2018 task 1: Affect in tweets. Proceedings of the 12th international workshop on semantic evaluation, pp. 1-17, 2018.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. Don't give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 1797-1807, Brus- sels, Belgium, 2018. Association for Computational Linguistics. doi: 10.18653/v 1/D118 1/D18-1206. URL https: https////aaaaattlogooooooooo wwt.//ggg//oo//.eo
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Ad- versarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics. pp. 4885 1985, Online, URL
2020. Association for Computational Linguistics. doi: 12255////2022..clmmaaa https: / / a anthology
aml 441.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. Document ranking with pretrained sequence-to-sequence model. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pp. 708-718, Online, 2020. Association for Computational Linguisties. doi: 1/2200 110266331/2200..mnnnmnn-o-o findings-emnlp633. URL https: https: http:////aaaanng.../////. wwwpiloooooorroooooooooooo 2egg/222 2020. findings-emmipp 63.
Richard Yuanzhe Pang, Adam Lelkes, Vinh Tran, and Cong Yu. AgreeSum: Agreement-orientte multi-document summarization. In Findings of the Association for Computational Linguis- fies: ACI-IJCNIP 2021, pp. 3377-3391, Online, 2021. Association for Computational Linguis- tics. doi: 10. 10.18653/v 1/203/ 1/2021..inninnnn-..eeeeeeea audinng--acl.299 URL https: /at/aaaaaanahhaaaaaaa org/2021. findings acl. 299.
Ankur P Parikh, Xuezhi Wang, Sebastian Gehrmann, Manaal Faruqui, Bhuwan Dhingra, Diyi Yang, and Dipanjan Das. Totto: controlled table-to-text generation dataset. arXiv preprint arXiv:2004. 14373, 2020.
Fabio Petroni, Aleksandra Piktus, Angela Fan, Patrick Lewis, Majid Yazdani, Nicola De Cao, James Thorne, Yacine Jernite, Vladimir Karpukhin, Jean Maillard, Vassilis Plachouras, Tim Rocktäschel, and Sebastian Riedel. KILT: a benchmark for knowledge intensive language tasks. In Proceedings of the 2021 Conference of the North American Chapter of the Associ- ation for Computational Linguistics: Human Language Technologies, pp. 2523-2544, Online, 2021. Association for Computational Linguistics. doi: 10. 18.18663/v 1/tp11/ mainm..mmmmmmmmm.200000 URL https: / /tc//aclllc. anwhhlnnhhhoggyyy org/202/ naacl-maaeea in. 200.
Jason Phang, Thibault Févry, and Samuel R. Bowman. Sentence encoders on stilts: Supplementary
training on intermediate labeled-data tasks, 2019.
Mohammad Taher Pilchvar and Jose Camacho-Collade
WiC: the word-in-context dataset for evaluating context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 1267-1273, Minneapolis, Min- nesota, 2019. Association for Computational Linguistics. doi: 10.18653/v1/N199 1128. URL https: a antho h wetthologg oogyoorg/N 9-1188.
18

---

Published as a conference paper at ICLR 2022
Lorien Y Pratt, Jack Mostow, Candace A Kamm, Ace A Kamm, et al. Direct transfer of learned information among neural networks. In Aaai, volume 91, pp. 584-589, 1991.
Dragomir Radev, Rui Zhang, Amrit Rau, Abhinand Sivaprasad, Chiachun Hsieh, Nazneen Fatema Rajani, Xiangru Tang, Aadit Vyas, Neha Verma, Pranav Krishna, Yangxiaokang Liu, Nadia Ir- wanto, Jessica Pan, Faiaz Rahman, Ahmad Zaidi, Murori Mutuma, Yasin Tarabar, Ankit Gupta, Tao Yu, Yi Chern Tan, Xi Victoria Lin, Caiming Xiong, and Richard Socher. Dart: Open-domain structured data record to text generation. arXiv preprint arXiv:2007.0288 2020.
Alec Radford, Jong Wook Kim, and Jeff Wu. Gpt-2 output dataset. https: / /gtpttgttthhhoggggoogo satpaaadpppt-ttttte
2019.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with unified text-to- text transformer. Journal of Machine Learning Research, 21(140):1-67, 2020. URL http: // //jmlr. wwtp://appes... htt///211222220 074 html.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. SQuAD: 100,000+ ques- tions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pp. 2383-2392, Austin, Texas, 2016. Association for Computational Linguistics. doi: 10.18653//1/D166 1264. URL
org/D16 1264..
Ori Ram, Yuval Kirstain, Jonathan Berant, Amir Globerson, and Omer Levy. Few-shot ques- tion answering by pretraining span selection. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Lunguage Processing (Volume 1: Long Pupers), pp. 3066-3079, Online, August 2021. Association for Computational Linguistics. doi: (..18653/v112000 aac.///.....iooggg2333333 URL ortp/o2lll........oo.../.oo..... 1oo22.acc-long 239.
Abhinav Rastogi, Xiaoxue Zang, Srinivas Sunkara, Raghav Gupta, and Pranav Khaitan. Towards scalable multi-domain conversational agents: The sahema--gudedd dialogue dataset, 2020.
Adam Roberts, Colin Raffel, and Noam Shazeer. How much knowledge can you pack into the parameters of a language model? In Proceedings of the 2020 Conference on Empirical
Methods in Natural Language Processing (EMNLP), pp. 5418-5426, Online, 2020. Associa- tion for Computational Linguistics.
doi: 1/2.6/3///222.mmmllplp. main.437. URL https:
// //aclantholooy.ccr orq/ 2atg//2220 2020. emnlp-main. 437.
Melissa Roemmele, Cosmin Adrian Bejan, and Andrew S. Gordon. Choice of plausible alternatives: An evaluation of commonsense causal reasoning. 201 AAAI Spring Symposium Series, 2011.
Michael T. Rosenstein, Zvika Marx, Leslie Pack Kaelbling, and Thomas G. Dietterich. To transfer or not transfer. In In NIPS'05 Workshop, Inductive Transfer: 10 Years Later, 2005.
Sara Rosenthal, Noura Farra, and Preslav Nakov. Semeval- task 4: Sentiment analysis in twitter. In Proceedings of the 11th international workshop on semantic evaluation (SemEval-20177 pp- 502-518, 2017.
Sebastian Ruder and Barbara Plank. Learning select data for transfer learning with Bayesian optimization. In Proceedings of the 2017 Conference on Empirical Methods in Natural Lan- guage Processing. pp. 372-382, Copenhagen, Denmark, September 2017, Association for Com- putational Linguistics. doi: 10.1663/v1/I1--1033 URL https: pettts://aclannte haca:////laaattooooo..o org. org/ D17-1038.
Sebastian Ruder, Matthew E Peters, Swabha Swayamdipta, and Thomas Wolf. Transfer learning in natural language processing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Tutorials, pp. 15-18, 2019.
Alexander M. Rush, Sumit Chopra, and Jason Weston. neural attention model for abstractive sen- tence summarization. In Proceedings of the 2015 Conference Empirical Methods in Natural Language Processing, pp. 379-389, Lisbon, Portugal, 2015, Association for Computational Lin- guistics. doi: 10.15 1044, URL https: a antho anthology.ong ogy. 1044..
19

---

Published as a conference paper at ICLR 2022
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. ArXiv, abs/1907.106444 2020.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver-
sarial winograd schema challenge at scale. Commun. ACM, 64(9):99-106 106, August 2021. ISSN 0001-0782. doi: 10. 145/347438. URL https: o
1145/3477 1145/3474381.
Victor Sanh, Thomas Wolf, and Sebastian Ruder. A Hierarchical Multi-task Approach for Learning Embeddings from Semantic Tasks, In Proceedings of AAAI 2019, 2019.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. Social IQa: Com- monsense reasoning about social interactions. In Proceedings of the 2019 Conference on Em- pirical Methods Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 4463-4473, Hong Kong, China, Novem- ber 2019. Association for Computational Linguistics. doi: 10.18653/v//D19-14555 URL https: ptttp::/acclattheee www.llanttooooyooogggo...
J. H. Schmidhuber. Evolutionary principles in self-referential learning, or learning how to learn: the meta-meta-... hook. 1987.
Tal Schuster, Adam Fisch, and Regina Barzilay. Get your vitamin C! robust fact verification with contrastive evidence. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 624-643, Online, 2021. Association for Computational Linguistics. doi: 10. 1/665//v1/2021... nnaccl-man.555 URL https: http//aaaaaaa http://caaaannctooceeee org/ 2021 nmacl-ma 52.
Abigail See, Peter J. Liu, and Christopher D. Manning. Get to the point: Summarization with pointer-generator networks. Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 1073-1083, Vancouver, Canada, 2017. Association for Computational Linguistics. doi: 10. 10.18655/1//117-10999 URL https:
// ///aclannhologyyccoom org/ P171/9/9.
Noam Shazeer. Glu variants improve transformer, 2020.
Noam Shazeer, Youlong Cheng, Niki Parmar, Dustin Tran, Ashish Vaswani, Penporn Koanan- takool, Peter Hawkins, HyoukJoong Lee, Mingsheng Hong, Cliff Young, Ryan Sepassi, and Blake A. Hechtman. Mesh-tensorfloww Deep learning for supereomputers. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi and Roman Gar- nett (eds.), (edds),.Avannee Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurlPS 2018, December 3-8, 2018, Montréal, Canada, neurips. wtt.cc//paaa... ptcppapper/201/
pp. 10444, 2018. blahh/a33
URL htttss:/pproceed/ee Abstract. htmtl..t..m.....
Samuel L. Smith, Pieter-Jan Kindermans, Chris Ying, and Quoe V. Le. Don't Itn't decay the learning rate, increase the batch size. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL https::///. /pw.reveeeie.eeee/////e htttp:://oeerrviieee
net porummidd=yyyyBeee
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pp. 1631-1644 Seattle. Washington, USA, 2013. Association for Computational Lin- guistics. URL ht pps: /////aaaanttt //aa/aaannttooooo or1/111111/00
Irene Solaiman and Christy Dennison. Process for adapting language models to society (palms) with values-targeted datasets. arXiv preprint arXiv:2106. 103328 2021.
Christian Stab, Tristan Miller. Pranav Rai, Benjamin Schiller, and Iryna Gurevych. Ukp sentential ulb. tu- darmstadtt de/
argument mining corpus, 2018. URL https:
harddle/tudatallb/223
Emma Strubell, Ananya Ganesh, and Andrew McCallum. Energy and policy considerations for deep learning in nlp. arXiv preprint arXiv:1906.0220 2019.
20

---

Published as a conference paper at ICLR 2022
Jimin Sun, Hwijeen Ahn, Chan Young Park, Yulia Tsvetkov, and David R. Mortensen. Ranking Transfer Languages with Dogmatically-Motiiaae Features for Multilingual Sentiment Analysis. In Proceedings of EACL 2021, 2021.
Yi Tay, Mostafa Dehghani, Jinfeng Rao, William Fedus, Samira Abnar, Hyung Won Chung, Sharan Narang, Dani Yogatama, Ashish Vaswani, and Donald Metzler. Scale efficiently: Insights from pre-training and fine-tuning transformers. arXiv preprint arXiv:2109. 10686. 2021a.
Yi Tay, Zhe Zhao, Dara Bahri, Donald Metzler, and Da-Cheng Juan. Hypergrid transformers: To- wards a single model for multiple tasks. In International Conference on Learning Representa- tions, 2021b. URL https: /pwpr//eepppeeeeeeeee/o http://wwwwwtteeteeteeeeoeecc..e wtr////rrmmieeeee Forummmnhhhhii110000000
James Thoree, Andreas Vlachos, Oana Cocarascu, Christos Christodoulopoulls, and Arpit Mittal. The FEVER2.0 shared task. In Proceedings of the Second Workshop on Fact Extraction and VERification (FEVER), 2018.
Erik F. Tjong Kim Sang and Fien De Meulder. Introduction to the conll-2003 shared task: Language- independent named entity recognition. In Proceedings the Seventh Conference Natural Language Learning ar HL.I-NACL 2003 Volume 4, CONLL '03, pp. 142-147, USA, 2003. Association for Computational Linguistics. doi: 10.3115/111996..111999 19111999776.119995 URL https:/
1119115.
Cynthia Van Hee, Els Lefever, and Véronique Hoste. Semeval-2018 task 3: Irony detection in english tweets. In Proceedings of The 12th International Workshop Semantic Evaluation, pp- 39-50, 2018.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp. paaper/201///0
5998 6008, 2017. URL https: http:://cccceeeeeeeeeeeee / neur ips. 225ee2455//deeee1bddd3e........ abs.eeaaatttaaa-aaattttt........ htmml.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferability across NLP tasks. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7882-7926, Online, 2020. Association for Computational Linguistics. doi: 12065///110220..mmmmmmmo// 2020. emnlp-maiii....memm 635.
URL https: / acc ant ho I ogy. org/
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelz- imer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Advances in Neu- ral Information Processing Systems 32: Annual Conference cessing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pp. 3261-3275, 2019a. URL hwtpttttp//ccm... http:////occeeeins.e.... URtttttss://ppoceeedi neurg......rreeo. neuriiu..p..... pcc/paper//019// 20129/99hhhh Abstract html.
Neural Information Pro-
Alex Wang. Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019, OpenReview.net, 2019b. URL attps://openrevice net ftr.mformmmooo rJ4km2R55
Zirui Wang, Yulia Tsvetkov, Orhan Firat, and Yuan Cao. Gradient vaccine: Investigating and improving multi-task optimization in massively multilingual models. In International Confer- ence on Learning Representations, 2021. URL F1 VE.jjK-1H
net ftt.roormmmmcmmmmm
21

---

Published as a conference paper at ICLR 2022
Alex Warstadt, Amanpreet Singh, and Samuel R. Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-644 641, 2019. doi: 10.1162/ tacl a 00290. URL https: www.atoooonntoooo.oooooooooooooa wrg. wetg//9--00 000-000
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. Finetuned language models are zero-shot learners, 2021.
Adina Williams, Nikita Nangia, and Samuel Bowman. A broad-covent broad-coverage challenge corpus for sen- tence understanding through inference. In Proceedings of the 2018 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pp. 1112-1122, New Orleans, Louisiana, 2018. Association for Com- .
putational Linguistics. doi: 10. (0.18653/v1/N168 1 101. URL https:
N18-1101.
Ellery Wulczyn, Nithum Thain, and Lucas Dixon. Ex machina: Personal attacks seen at scale. In Rick Barrett, Rick Cummings, Eugene Agichtein, and Evgeniy Gabrilovich (eds.), Proceedings of the 26th International Conference on World Wide Web, wwW 2017, Perth, Australia, April 3-7, 2017, pp. 1391-13999 ACM, 2017. doi: 10.1 (0.1145/90389112.3555511 URL attpps:/ddo /doi. org/ 10. 10w11114333333 00345/333331222 3052591.
Wenhan Xiong. Jiawei Wu, Hong Wang, Vivek Kulkarni, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. TWEETQA: A social media focused question answering dataset. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5020-5031, Florence, Italy, 2019, Association for Computational Linguistics. doi: 10.18653/v1/ P19-1496. URL https::///... : / wwc.anthooog..org/P..... 1496.
Wei Xu, Courtney Napoles, Ellie Pavlick, Quanze Chen, and Chris Callison-Burch Optimiz- ing statistical machine translation for text simplification. Transactions of the Association for Computational Linguistics, 4:401-415, 2016. URL https://cocoxu.ce/ www.oocwww.iitu..i.... smt wimmpplfiiii www.fiiaiiin......
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2018.
Wenpeng Yin, Dragomir Radev, and Caiming Xiong. DocNLI: A large-scale dataset for document- level natural language inference. In Findings of the Association for Computational Linguis- tics: ACL-IJCNLP 2021, pp. 4913-4922, Online, 2021. Association for Computational Linguis- tics. doi: 10.18653/v1/0021 htt.://.i.....nnnsaaedeeeel. andingg--acl.435 URL https: http:///aaaaatlto//////a org/2021. Findings-acl.493 435.
Mohamed Amir Yosef, Johannes Hoffart, Ilaria Bordino, Mare Spaniol, and Gerhard Weikum. Aida: An online tool for accurate disambiguation of named entities in text and tables. Proc. VLDB Endow., (000:14450-153,, August 2011. ISSN 2150-8097. doi: 10. 10.14778/3307755.44222.. URL https:tt//// /ooi. wwwg///............ 14778/ 3402 755. 3402793.
Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. Gradient surgery for multi-task learning, 2020.
Marcos Zampieri, Shervin Malmasi, Preslav Nakov, Sara Rosenthal, Noura Farra, and Ritesh Kumar. Semeval-2019 task Identifying and categorizing offensive language in social media (offense- val). In Proceedings of the 13th International Workshop Semantic Evaluation, pp. 75-86, 2019.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. HellaSwag: Can a ma- chine really finish your sentence? In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 4791-4800, Florence, Italy, July 2019. Association for Com- putational Linguistics. doi: 10. 10.18653/v1/P19 URL https:/ / /aclaaaanttt/ aaclaatoonootyyy... org/ P1 1472.
Xiaohua Zhai, Alexander Kolesnikov, Neil Houlsby, and Lucas Beyer. Scaling vision transformers. arXiv preprint arXiv: 045600 2021.
22

---

Published as a conference paper at ICLR 2022
Rui Zhang and Joel Tetreault. This email could save your life: Introducing the task of email subject line generation. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pp. 446-456, Florence, Italy, 2019. Association for Computational Linguistics. doi: 10.18653/v1/P19- 1043. URL https: a anthol .gr.llyyo.oooroogo.... 9-10433 1043.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 1810. /2885, 2018.
Xiang Zhang. Junbo Jake Zhao, and Yann LeCun. Character-level convolutional networks for text classification. In Corinna Cortes, Neil D. Lawrence, Daniel D. Lee, Masashi Sugiyama, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 28: Annual Confer- ence on Neural Information Processing Systems 2015, December 7-12, 2015, Montreal, Que- neurips cc/paper/ Abst
bec, Canada, pp. 649 649-657, 2015. URL rat html. htt.://hhhe222ee8822eeffffffooo.
Yian Zhang, Alex Warstadt, Xiaocheng Li, and Samuel Bowman. When do you need bil- lions of words of pretraining data? In Proceedings of the 59th Annual Meeting of the As- sociation for Computational Linguistics and the 11th International Joint Conference on Natu- ral Language Processing (Volume 1: Long Papers), pp. 1112-1125, Online, August 2021. As- sociation for Computational Linguistics. doi: 10.18633/v12021 -ac-2oooo.----09000 URL https: hww.hhooooyy..rg/22111//
//a /aclaanhhology.ooo org. wer222aaccloooo acl.oog.cl-oo..900 90ng.90.
Yu Zhang and Dit-Yan Yan Yeung. A convex formulation for learning task relationships in multi-task learning. In Proceedings of the Twenty-Sixth Conference Uncertainty in Artificial Intelligence, U0I'10, pp. 733-742, Arlington, Virginia, USA, 2010. AUAI Press. ISBN 9780974903965.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. Ernie: Enhanced language representation with informative entities. arXiv preprint /rXiv:9905.07/29, 2019.
23

---

Published as a conference paper at ICLR 2022
A DATASETS
Dataset(s)
Description
No. Train Datasets
[2]
Citation
GLUE
General Language Understanding
949,101
Wang et al. (2019b)
SuperGLUE
General Language Understanding
185,673
Wang et (2019a)
KILT
Snnwledge-Intenstte Language Tasks
3,129.859
Petroni et al. (2021)
Rainbow
Commonsense Reasoning
324,742
Lourie et al. (2021)
GEM (en)
Natural Language Generation
1,067,955
Gehrmann et al. (2021)
DialoGLUEE
Dialogue Understunding
76,122
Mehri et al. (2020)
TweetEval
Twitter Classification Benchmark
120,104
Barbieri al. (2020)
CNN/Dailymail
News Summarization
287,113
See et al. (2017)
XSum
News Summarization
203,577
Narayan et al. (2018)
Multi-News
News Summarization
44,972
Fabbri et (2019)
AESLC
Email Summarization
14,436
Zhang & Tetreault (2019)
Gigaword
Summarization
3,803,957
Rush et al. (2015)
SamSum
Dialogue Summarization
14.372
Gliwa al. (2019)
ANLI
Adverserial NLI
162,865
Nic et al. (2020)
ESNLI
Explainable
549,367
De et (2020)
AgreeSum Entailment
AntilleSSummay NLI
7,750
Pung et al. (2021)
DocNII
Document NI.I
942,314
Yin et al. (2021)
Vitamin C
Fact-checking NLI
370,653
Schuster et (2021)
Web Questions
QA (open)
3778
Berant et al. (2013)
SQuAD
QA (context)
87,599
Rajpurkar et (2016)
QuAC
QA (context)
83,568
Choi al. (2018)
DROP
QA (Discrete Reasoning)
77,409
Dua et al. (2019)
RACE
School QA (MCQ)
113,013
Lai et (2017)
Eraser MultiCC
Explainable QA (MCQ)
24.029
DeYoung et al. (2020)
TweetQA
QA (context)
10692
Xiong et al. (2019)
NewsQuizQA
Question Answer Generation
16,160
Lelkes et al. (2021)
Amazon Reviews
Review Classification
100,000
uma
GoEmotions
Emotion Classification
43,410
Demszky et al. (2020)
IMDb Reviews
Sentiment Classification
25,000
Maas et (2011)
Sentiment140
Sentiment Classification
1,600,000
Go et al. (2009)
Yelp Reviews
Sentiment Classification
560,000
Zhang et (2015)
AGNews
News Classification
120,000
Zhang et al. (2015)
TreqQC
Question Classification
5000
Hovy et al. (2001)
Civil Comments
Toxicity Classification
1,804,874
Borkan et (2019)
Wiki Toxicity
Toxicity Classification
159,571
Wulczyn et. (2017)
Yahoo! Answers
Topic Classitication
140,000
Zhang et al. (2015)
UKP Arg. Mining
Argument Classification
18,341
Stab al. (2018)
Parsing to FunQI.
Semantic Parsing
5,565
Guo et al. (2020)
Passing to interm, repe.
Semantic Parsing
117,318
Herzig et al. (2021)
COGS
Semantic Parsing (Comp. Gen.)
24.155
Kim & Linzen (2020)
GPT Deepfake detection
Generated-tex: classification
500,000
Radford et al. (2019)
StylePTB
Style Transfer
53,546
Lyu et al. (2021)
Shakespearizing English
Style Transfer
2
36,790
Jhamtani al. (2017)
MS-MARCO
Pointwise Ranking
100,000
Bajuj et al. (2018)
Total
ExMIx
18,085,040
Table 11: All of the training datasets used to construct ExMix.
Table 11 summarizes the 107 datasets included in ExMIx. Some of the lines in the table represent existing benchmarks that group several tasks together. From each collection, we use the datasets that include English training data:
GLUE: CoLAA (Warstadt et al., 2019), SST-2 (Socher et al., 2013), MRPC (Dolan & Brockett, 2005), QQP, STS-B (Cer et al., 2017), MNLI (Williams et al., 2018), QNLI (Converted from Rajpurkar et al. (2016), RTE (Dagan et al., 2006), WNLI (Sakaguchi et al., 2020).
SuperGLUE: BoolQ (Clark et al., 2019), CB (De Marneffe et al., 2019), COPA (Roemmele et al., 2011). MultiRC (Khashabi et al., 2018), ReCoRD (Zhang et al., 2018), RTE (Dagan et al., 2006), WIC (Pilehvar & Camacho-Collados 2019), WSC (Levesque et al., 2011).
KILT: FEVER (Thorne et al., 2018). AIDA (Yosef et al., 2011). WNED (Guo & Barbosa, 2014), T-REx (Guo & Barbosa, 2014), NQ (Kwiatkowski et al., 2019), HoPo (Yang et al.,
2018), TQA (Joshi et al., 2017), E1.1 (Fan et al., 2019), WoW (Dinan et al., 2019).
Rainbow: oNLI (Bhagavatula et al., 2020), CosmosQA (Huang et al., 2019), HellaSWAG (Zellers et al., 2019), PIQA (Bisk et al., 2020), SocialIQA (Sap et al., 2019), WinoGrande (Sakaguchi et al., 2021).
-EM (en): Wiki-Lingua (Faisal Ladhak & McKeown, 2020), WenNLG (Gardent et al., 2017; Castro Ferreira et al., 2020), CommonGEN (Lin et al., 2020), E2E (Dušek et al., 2019), DART
24

---

Published as a conference paper at ICLR 2022
(Radev al., 2020), ToTTo (Parikh et al., 2020), Wiki-Auto (Jiang al., 2020), TurkCor-
pus(Xu et al., 2016)
DialoGLUE: Banking77 (Casanueva et al., 2020), HWU64 (Liu et al., 2019b), CLINC150
(Larson et al., 2019), SGD (Rastogi et al., 2020), TOP (Gupta et al., 2018).
TvaetEval: Emotion Recognition (Mohammad al., 2018), Emoji Prediction (Barbieri al., 2018), Irony Detection (Van Hee et al., 2018), Hate Speech Detection (Basile et al., 2019), Offensive Language Identification (Zampieri et al., 2019), Sentiment Analysis (Rosenthal al., 2017), Stance Detection (Mohammad et al., 2016).
B EXPERIMENTAL DETAILS
This section describes the experimental details
B.1 IMPLEMENTATION DETAILS
Our models were trained using Mesh Tensorflow (Shazeer et al., 2018) using the T5 library (Raffel
et al., 2020).
B.2 DATASET EXPERIMENTAL SETUP
This section reports the dataset and experimental setup each individual target task/dataset.
SuperGLUE We finetune on entire SuperGLUE as a mixture with proportionate sampling in
similar fashion to (Raffel et al., 2020). We finetune for total of 200k steps with batch size of 128. When selecting checkpoints on SuperGLUE, we follow the same convention as Raffel et al. (2020) in selecting the best checkpoint for each task for fair comparison models that fine-tuned on the individual tasks instead co-training on all of them.
We report test set results all datasets except CommonGen and ToTTo, which report validation scores. We sweep over learning rates of 10-3, 5 x 10-4 and 10-4 All results computed using GEM-metries'. For each dataset. select the best model checkpoint using average of BLEU, ROUGE-1, ROUGE-2 and ROUGE-L scores on the validation set. We use the greedy decoding strategy to be consistent with the original GEM paper (Gehrmann et al., 2021).
GEM
CBQA We report validation set results, and sweep over learning rates of 10-3 and 10-4
Rainbow We multi-task co-train on all datasets, and sweep over learning rates of 1 and 10-4
WMT Machine Translation We finetune our models on three collections of WMT, namely EnDe, EnFr and EnRo. We use constant learning rate of 10 and dropout of 0.1. We train with batch size of 4096 for maximum of 400k steps and report peak validation BLEU score. We use a beam size of and length penalty of 0.6.
ARC We report scores on the Challenge set, and train with batch size of 32 and sweep over learning rates of 10-3 and 10-4,
convert NER to seq2seq by writing the target as the ordered sequence of tags and entities (for example "When Alice visited New York" "[PER] Alice [LOC] New York"). Accuracy is measured on a sentence level, considering prediction to be correct only it exactly matches the reference sequence.
CoNLL-03 NER
C DETAILED EXPERIMENTAL RESULTS
Many of our experiments $2 used the average SuperGLUE score model for evaluation. We report the full results all datasets below.
Phtpps:// // thub. com beechhha rk /wt.G/MM--iresssssee
25

---

Published as a conference paper at ICLR 2022
Mixture
BoolQ
CB
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
Vanilla
82.3
91.7/92.9
60.0
76.9/39.6
80.9/80.2
84.5
69.3
81.7
76.1
Best-effort
81.7
89.4/92.9
75.0
76.6/37.4
76.4/75.5
82.7
67.1
80.8
76.4
Random-55
81.3
97.3/97.0
67.7
77.0/39.7
76.5/75.6
82.7
69.5
83.3
77.0
EXT5
82.6
98.7/98.2
73.0
79.5/45.4
80.8/80.0
87.0
71.3
83.7
79.9
Table 12: Full SuperGLUE results from $2.2
Method
BoolQ
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
Vanilla
82.3
91.7/92.9
60.0
76. 9
80.9/80.2
84.5
69.3
81.7
76.1
Pre-fine f--ffnettninn
82.2
85. 1/89/
74.0
79.8/45.1
79.2/78.3
87.7
69.6
82.7
78.1
Multi-task pre-training
82.6
98.7/98.2
73.0
79.5/45.4
80.8/80.0
87.0
71.3
83.7
79.9
Table 13: Full SuperGLUE results from $2.3
R
BoolQ
CB
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
76.8
58.6/83.9
63.0
66.6/22.6
53.0/52.1
75.5
63.2
73.1
65.0
1
82.1
93.7/94.6
75.0
78.0/41.7
76.7/75.7
85.6
68.8
76.9
77.3
2
82.6
98.7/98.2
73.0
79.5/45.4
80.8/80.0
87.0
71.3
83.7
79.9
81.3
96.0/94.6
73.0
75.2/38.8
77.4/76.6
84.8
68.8
83.7
77.6
5
81.9
89.4/92.9
74.0
75.5/35.6
76.2/75.3
85.6
69.1
76.9
76.2
10
81.2
93.2/96.4
77.0
75.6/37.6
76.5/75.6
82.7
70.4
80.8
77.4
20
80.7
93.7/94.6
71.0
74.5/36.5
75.9/74.4
79.8
66.5
84.6
75.9
00
82.3
91.7/92.9
60.0
76.9/39.6
80.9/80.2
84.5
69.3
81.7
76.1
Table 14: Full SuperGLUE results from $2.4
# Tasks
BoolQ
CB
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
Batch Size 128
0
79.3
92.4/92.9
72.0
74.2/32.8
74.9/73.9
79.8
70.2
81.7
75.4
30 (random)
78.7
95.7/95.2
66.0
72.6/30.5
72.8/72.0
77.6
68.4
82.4
74.1
55 (random)
79.4
93.2/94.6
74.3
73.6/33.8
74.1/73.2
80.7
68.8
82.1
75.8
80 (random)
80.0
92.5/94.6
70.0
74.3/34.8
73.9/72.9
80.0
68.1
82.4
75.3
107
79.3
95.0/96.4
74.0
74.0/34.8
73.5/72.5
79.4
70.7
77.9
75.6
Batch Size = 512
0
82.3
91.7/92.9
60.0
76.9/39.6
80.9/80.2
84.5
69.3
81.7
76.1
30 (random)
80.6
93.6/95.8
67.7
74.6/35.4
75.7/74.7
81.2
68.6
83.3
75.8
55 (random)
81.3
97.3/97.0
67.7
77.0/39.7
76.5/75.6
82.7
69.5
83.3
77.0
80 (random)
82.1
94.4/95.8
71.7
76.8/39.4
77.0/76.1
84.7
69.2
83.0
77.6
107
82.6
98.7/98.2
73.0
79.5/45.4
80.8/80.0
87.0
71.3
83.7
79.9
Table 15: Full SuperGLUE results from $2.5
# Pre-train steps
BoolQ
CB
Copa
MultiRC
ReC
RTE
WiC
WSC
AVG
T5.1.1
77.9
93.0/92.9
69.0
73.0/32.3
73.3/72.4
77.6
69.6
79.8
73.8
50k
82.3
100.0/100.0
74.0
76.8/38.0
79.9/79.1
82,3
70.4
83.7
78.7
100k
83.7
95.0/96.4
82,0
80.0/45.9
83.8/83.0
87.0
73.7
84.6
81.6
200k
85.7
100.0/100.0
85.0
81.8/49.0
85.2/84.4
87.7
73.5
88.5
83.8
EXT5
20k
80.3
95.0/96.4
70.0
74.4/35.8
72.9/72.1
82.7
68.5
81.7
75.8
50k
83.1
97.4/96.4
78.0
79.2/43.9
79.6/78.9
88.1
73.4
87.5
81.0
100k
85.3
100.0/100.0
81.0
81.6/48.9
83,7/83.0
89.2
73.2
90.4
83.5
200k
86.5
98.7/98.2
86.0
83.2/53.1
85.4/84.7
91.7
73.4
93.3
85.3
Table 16: Full SuperGLUU results from $2.6
26


Reviews:
[{"review_id": "t5koISr6YEF", "reviewer": "Reviewer_6JD9", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros\n* The scale of the experiments is very impressive and the results are very strong - ExT5 is generally a much stronger model than it's T5 counterpart.\n* From a resource perspective, both the pre-trained ExT5 model and the ExMix collection would be valuable to the NLP research community.\n* Because of how well the pre-training of ExT5 and T5 are controlled to be as similar as possible, the argument that there is significant benefit to supervised pre-training is convincing.\n* The analysis is helpful in demonstrating the role of the number of tasks in the final model performance. There is a clear relationship between the number of tasks used during pre-training and the strength of the downstream model, which motivates the scale of ExMix.\n* The authors take a principled approach to selecting a subset of tasks and demonstrate that it is generally not successful over increasing the number of tasks, highlighting the importance of the number of tasks.\n* Section 2.3 provides strong evidence that multi-task pre-training is stronger than multi-task pre-finetuning.\n* Section 2.6 demonstrates that multi-task pre-training not only results in stronger final models than vanilla pre-training, but that throughout the entirety of training multi-task pre-training beats vanilla pre-training.\n\nCons\n* The methodological novelty of the paper is fairly low. ExMix is a collection of pre-existing datasets and using multi-task signals during pre-training is not new either. The scientific value of this work predominantly comes from the scale of their multi-task setting, and their analyses. It is perhaps not surprising that a larger collection of supervised pre-training datasets would improve performance over these baselines.\n* All of section 2 is evaluated on average SuperGLUE performance only. While SuperGLUE is a strong baseline given the diverse set of tasks it encompasses, there is always the possibility that a subset of tasks within SuperGLUE are particularly well-suited for the ExT5 training setup and, as such, Section 2 risks overfitting to the SuperGLUE baseline.\n* There are some details missing regarding Section 2.4, especially with respect to random seeds. In particular, Figure 3 shows some interesting behavior regarding the hyperparameter R - it is surprising to me that performance does not monotonically decrease after a certain point but rather seems to be trending upwards. What happens if R increases even more? Or is the behavior at the tail end of R (R=5 or 10) due to random noise?\n* Section 2.5 compares different numbers of tasks trained on different batch-sizes. However, the models use the same R parameter across numbers of tasks, which seems unfair to low-task settings given that it was tuned on 107 tasks. Additionally, the models use the same learning rate regardless of batch-size, ostensibly a learning rate that was tuned for a batch-size of 512. It has been previously noted (e.g. by McCandlish et al., 2018) that there is a clear relationship between batch-size and learning rate; thus, if the same learning rates are being used across the 512 and 128 batch-size settings, it's likely that the results are heavily influenced by a poor learning rate for the 128 batch-size setting.", "summary_of_the_paper": "This paper introduces \"ExMix,\" a collection of 107 pre-existing NLP datasets across 8 identified task-families. ExMix is proposed as a pre-training dataset, to be mixed with the typical self-supervised pre-training prevalent in NLP. The authors propose to pre-train a T5 model on a mixture of ExMix and C4 (the self-supervised pre-training dataset for T5), which they term ExT5. To do this, they convert every task to a text-to-text problem, so that all tasks, supervised and self-supervised, use standard sequence-to-sequence cross-entropy loss. They pre-train ExT5 for the same number of steps and on the same number of tokens as T5. They then evaluate how ExT5 and T5 perform when fine-tuned to individual tasks. They find that ExT5 largely outperforms T5 on the majority of downstream evaluated tasks, including tasks whose training data was included in ExMix as well as tasks which were not seen during pre-training.\n\nThe authors perform an experiment to identify whether or not a manually curated subset of ExMix can provide better pre-training generalization than the full set of tasks, due to negative transfer between tasks. To do this, they examine negative transfer between pairs of task families during fine-tuning. They select the task-families which exhibited the highest amount of fine-tuning transfer across all task-pairs, and use only 48 tasks from these task-families as a pre-training objective. They compare this model to a model trained on a slightly larger subset (55) of randomly selected tasks, as well as ExT5 (trained on all 107) tasks. The authors find that their method of task-selection is largely unhelpful for selecting pre-training tasks - the model trained on a random subset of 55 tasks outperforms the manually curated 48-task model, and ExT5 significantly outperforms both. This suggests that, for pre-training, selecting tasks that complement one-another with respect to downstream performance is not straight-forward, but increasing the number of tasks seems to alleviate negative transfer, rather than exacerbate it.\n\nFinally, the authors perform some additional analysis of the training decisions made around ExT5. They find that: (a) pre-finetuning, which introduces multi-task learning as an intermediate step between self-supervised pre-training and fine-tuning, is not better than mixing all tasks for pre-training when using ExT5, (b) ExMix without self-supervised pre-training is significantly worse than self-supervised training without ExMix, suggesting that self-supervised pre-training is still crucial for pre-training, (c) downstream performance generally increases as the number of tasks scales up, and (d) pre-training with ExMix is significantly more sample-efficient than self-supervised pre-training along, i.e. it achieves strong downstream results very early, even 20k steps into pre-training.", "summary_of_the_review": "Overall, the paper fairly convincingly argues that mixing supervised and self-supervised signals during pre-training is extremely beneficial to downstream performance and sample-efficiency. However, there is limited methodological novelty and the benefits of scale and incorporating supervision may not be particularly surprising.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "eyYckJbKq1p", "reviewer": "Reviewer_ri6r", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Since the ideas presented are not new and were actually recently explored in both the T5 paper and Aghajanyan et al., (2021), I really view this paper as a scaling paper. \n\n**Pros**\n\nThe paper presents interesting results on scaling T5 multi-task learning to close to 100 tasks, showing improvements on downstream benchmarks over vanilla T5.\n\n**Cons**\n\n- While the results improve performance over downstream benchmarks, I didn’t learn anything new from the paper. The paper completely reuses the T5 strategy for multi-task learning and just scales it up with more tasks. While this scaling could have been an interesting study, the ablations and empirical investigation don’t ask very interesting questions for this to be useful. I didn't find any significant recommendations for scaling tasks that can be reusable in future work or even applicable to models other than T5. Moreover, prior work (Aghajanyan et al., 2021) also studied most of these questions already, in the context of two pre-trained models, though in a slightly different setting. Over this and much other related work on multi-task and multi-lingual learning, the current paper only seems to contribute new results using the T5 model.\n- Many of the results presented in motivating the work are over random subsets of pre-training tasks and random seeds for training. However, none of the results seem to give an idea of the variance of performance, nor any indication of how many different random subsets of tasks were used for the results.\n- The task scaling experiment (2.5) seems to be core to the contribution of the paper. Similar scaling experiments are also present in Aghajanyan et al., (2021). However, apart from the lack of an exposition of the variance in the quantities presented, I found this analysis to be very underwhelming. The authors can analyze many interesting choices here to see how task scaling affects results. To give one instance, one can consider choices like selecting tasks proportional to their size (instead of random sampling) and evaluating how it affects performance. Without any meaningful consideration of the choices available here, I am not sure what future work can take away from this paper.\n- Often the evaluation is in terms of “average” super-glue performance. Since many improvements are actually marginal (less than 1%), it will be good to provide a spread of the results (at least in the appendix) so the readers know the results are not due to improvement on just a small subset of downstream tasks. Many hyper-parameters and design choices for the pre-training are also difficult to find and not clearly specified.\n\n\n**Questions and suggestions**\n\n- Table 3: best effort has 48 tasks but random-mix is trained with 55, why? What is the variance of random-mix? What is the overlap between best-effort tasks and random tasks?\n- 2.4: Is the amount of data and pre-training steps the same across the models being compared? If not, it will be good to make the amount of data and steps similar when comparing only C4 vs only C4 + ExMix. Moreover, it will actually be more useful to compare with respect to C4 performance at multiple data sizes, since pre-training data is more abundant than supervised multi-task data. \n- Fig 3, adding multi-task over C4 only seems to help a little bit, looking at the spread of SuperGlue scores between the two settings will be more helpful. These numbers (at R=2) seem different than what is reported in Fig 5, can you comment on why?\n- 2.5 and Fig 4: task scaling only helps at 512 batch-size, there doesn’t seem to be any consistent trend at batch-size of 128. This seems to indicate that batch-size is a very crucial axis for ablation. Did you try other batch-sizes, a plot of how performance varies with this parameter will be helpful. Moreover, batch-size here is confounded with the number of tasks that are sampled in a batch. What is the task sampling strategy used here and how does it affect this plot?\n- Fig. 4: What is the shaded area? if it is some statistic like standard deviation across random seeds, why is this not present at 0 and 107.\n- Fig. 4: What is the variance with respect to the random selection of tasks? Will a size-proportional selection of tasks lead to different results, for instance, to select 30 tasks pick the 30 with the largest data?\n- In the super-glue results, improvements diminish at larger model sizes, to the extent that T5-XL performs similarly. This seems like an interesting result and the trend indicates multi-task might become less useful at even larger model sizes. It will be good to explore this further, if you have resources for such an experiment, as I consider this an important aspect of “scaling” which will fall within the scope of this work.\n- Can you comment on your choice of T5 1.1 instead of the original T5 models published with the paper. The original models also include downstream supervised tasks in their pre-training, just like this work.\n\n**Post Author Response**\n\nI am not convinced that the experiments fully support the claim that adding a random mix of more tasks overcomes negative transfer between tasks. The spread of SuperGlue results with the number of tasks is highly inconsistent and performance of some tasks often gets worse at more tasks. The large variance in Fig 4 at 55 and 80 actually indicates task selection is really important to get higher performance, though the best-effort approach, as presented in this paper, might not be the best to get that task mixture. This then requires a more thorough evaluation of the various choices for multi-task task selection. I am also concerned with the large difference in trend at the two different batchsizes, which the authors didn't address. Compared to previous work, I have to disagree with the authors that problems like task heads or loss scaling are mitigated by T5. T5 still generates using a shared softmax classifier (over the whole vocabulary), instead confounding the problem with that of generation. Loss scaling is hidden in how you sample the instances/task for each batch. The main point of difference that I see from the rebuttal is of using multiple seeds in Fig 3. The response also included new results using a larger T5 model. The result on the larger T5 model seems to confirm the trend that there are diminishing returns from multi-tasking at a larger scale. \n\nI am increasing my rating by one point (which in the new system makes it 5) as there are tidbits of interesting results in the paper, which if presented neutrally could have been interesting. However, I am concerned that this will be used as a point of reference to say that randomly adding more tasks always helps, for which the paper doesn't provide sufficient evidence.", "summary_of_the_paper": "The paper presents an analysis of multi-task learning with the T5 model at a large scale. The tasks are formatted as text-to-text tasks and trained alongside the span denoising objective on C4, same as the multi-task strategy used in the original T5 paper. The paper evaluates this approach for a few different sizes of the T5 model on various downstream datasets, compared with vanilla T5 model. The paper shows improvements on average accuracy over multiple downstream benchmarks. In addition, there are some ablations on the number of tasks, the ratio of C4 vs multi-task, and comparing multi-task with pre-finetuning. ", "summary_of_the_review": "In general, I think it is important to carefully study multi-task scaling in the post pre-training era. There are many papers in a similar vein recently and to be useful the paper should concentrate more on asking meaningful questions and rigorously analyzing the results to provide recommendations for future work. I hope the authors take the criticism constructively to improve their work further, concentrating on more analysis rather than more downstream benchmark results.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "TA1Ad-jUohq", "reviewer": "Reviewer_u5L1", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strengths\n\n- Well written and convincing\n- The variety and size of their corpus ExMix\n- The number of interesting research questions explored\n- The improved efficiency of their approach as compared with the baseline methods\n- They evaluate on a large number of varied text suites (SuperGLUE, GEM, Rainbow CommonSense, CBQA) and compare against a strong baseline T5 and mostly perform better. \n\nweakness\n- Nothing significant\n- Selecting subsets of tasks and exploring fine-tuning (2.1, 2.2) takes up a lot of the paper and it did not perform as well as pre-training or impact the performance of pre-training much and so maybe less space could have been dedicated to these sections.\n\n", "summary_of_the_paper": "This paper presents a corpus (ExMix) and a model (ExT5) which extend work on multi-task pretraining for large language models. \n\nTheir ExMix covers 8 families of supervised natural language processing tasks (eg. summarization, classification) with three individual tasks in each of them. These tasks are all converted into one text-to-text task (like prior work does) in order to allow uniform training on mutliple tasks. This is an impressively comprehensive collection of multi-task data sets, twice as large as the previously biggest multi-task corpus. \n\nThey also perform thorough experiments on a variety of intersting research questions. \nThey look at how to use cheaper fine-tuning to select a sub-set of tasks to use for pre-training - and they show that it is best to use all the supervised tasks in pretraining, not just the subset which worked best in fine-tuning. \nThey looked at whether adding and additional fine-tuning step before the final GLUE task fine-tuning would be better than pre-training with all this supervised multi-task data, and they show that in fact it is best to actually pre-train with them. \nIn their pre-training they mix masked-unsupervised learning with the task-supervised learning and they explore what is the right mix of the two, showing that most mixture settings are good, but that without the unsupervised signal, the model performs much worse. Also increasing the number of tasks generally helps downstream performance. \n\nThey also look at the efficiency of their approach. Training on ExMix allows the model to reach the performance of the T5 model trained only on plain text with many fewer (approximately half) the number of steps. \n\nFinally they build a large pre-trained model ExT5 on which they perform really extensive evaluations and the demonstrate that the multi-task pretraining clearly helps and improves on a strong baseline across many tasks, including tasks which have not been included in the ExMix training data like translation. \n", "summary_of_the_review": "This paper pushes forward work on large scale multi-task transfer learning, with a huge number and variety of tasks (107 for training) with many test suits and research questions explored. It is well written and convincing and I would like to see it published in ICLR. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "8LctZkSN5tz", "reviewer": "Reviewer_4oQF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "strength:\n- The paper is well-motivated and clearly presented; \n\n- The authors extensively study the effect of co-training, multi-task learning at scale by leveraging multi-task learning naively on top of pre-trained representation, further testing on multi-task pretraining and pre-finetuning, which suggests a practical way to leverage multi-task learning and how should that improve the sample efficiency of the learned model; \n\n- Substantial ablations are provided for understanding how multi-task data is introduced, how much multi-task data is introduced will affect the SuperGLUE downstream performance using the multi-task pretraining objective. \n\nweakness and question: \n- Given that SuperGLUE may represent one or several language understanding tasks, do the findings regards finetuning transfer, pre-finetuning hold still for other task families like generation? Or is that specifically tailored to SuperGLUE as mentioned in the paper. \n\n- Section 2.5 uses the same learning rate for different batch sizes, which might result in unfair comparison given there are previous works suggesting how batch size, learning rate could have compound effects on the downstream tasks.  \n\n- In section 2.4, will R beyond 10 still lead to consistent improvement of the SuperGLUE performance is unclear. \n\n- Beyond the empirical observations, are there explanations (like task affinity beyond the accuracy correlation) that could explain more clearly the results of co-training by multi-task finetuning on top of pre-trained models? \n\n- The significance of the results on different task/task families can be further validated by reporting std/mean results with multiple seeds as in (Section 2.5) \n\n- The zero-shot performance might also be a point to add given the multi-task prompting / finetuning seems to be helpful both for GPT-families and T5-families [1, 2]. \n\n[1] Wei, Jason, et al. \"Finetuned language models are zero-shot learners.\" arXiv preprint arXiv:2109.01652 (2021).\n\n[2] Sanh, Victor, et al. \"Multitask Prompted Training Enables Zero-Shot Task Generalization.\" arXiv preprint arXiv:2110.08207 (2021).", "summary_of_the_paper": "This paper revisits the idea of multi-task learning for natural language processing (NLP) and scales it up to 107 supervised NLP tasks as EXMIX (Extreme Mixture) across diverse domains and task families. It extensively analyzes the co-training effect between the different families of NLP tasks and proposes a model pre-trained using a multi-task objective of self-supervised span denoising and supervised EXMIX named EXT5. They show that a massive and diverse collection of pre-training tasks is generally preferable to an expensive search for the best combination of pre-training tasks. It further shows the effectiveness of EXT5 on a range of supervised NLP tasks. ", "summary_of_the_review": "The idea of leveraging multi-task learning for NLP has been introduced before as mentioned in the paper and not referred to as [3, 4]. It is great to revisit and extensively test this idea at scale at the paradigm of pre-training methods. It also offers potentials for future works on how existing labeled datasets can be used to further improve NLP models. \n\n[3] McCann, Bryan, et al. \"The natural language decathlon: Multitask learning as question answering.\" arXiv preprint arXiv:1806.08730 (2018).\n\n[4] Subramanian, Sandeep, et al. \"Learning general purpose distributed sentence representations via large scale multi-task learning.\" arXiv preprint arXiv:1804.00079 (2018).\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper explores large scale supervised multi-task training across 107 NLP tasks combined with self-supervised C4 masked span infilling, using the T5 sequence-to-sequence model.  The results improve over prior strong T5 baselines on several NLP benchmarks such as SuperGLUE, GEM, and Rainbow.

The paper's main strengths are the scale and large number of tasks, the release of the trained models and data, as well as the clarity and presentation.  Reviewers had concerns with the novelty, limitations in the evaluation (to just T5, and to just SuperGLUE in portions of the paper), and the potential impact of hyperparameters on the results.  During the discussion period, the authors noted that it is not obvious a priori that their approach would work, and that their evaluations on other tasks made it unlikely to be overfitting to SuperGLUE.  They also noted that running the additional hyperparameter experiments suggested during the reviews were computationally prohibitive.

Overall, despite the drawbacks and relative lack of novelty, the extensive experiments and released models provide significant value and will be of interest to the research community.
Decision: Accept (Poster)