Venue: NAACL 2022

# Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

Vishakh Padmakumar \({}^{1*}\) Leonard Lausen\({}^{2}\) Miguel Ballesteros\({}^{3}\) Sheng Zha\({}^{2}\) He He\({}^{12}\) George Karypis\({}^{2}\)

###### Abstract

Recent work has found that multi-task training with a large number of diverse tasks can uniformly improve downstream performance on unseen target tasks. In contrast, literature on task transferability has established that the choice of intermediate tasks can heavily affect downstream task performance. In this work, we aim to disentangle the effect of scale and relatedness of tasks in multi-task representation learning. We find that, on average, increasing the scale of multi-task learning, in terms of the number of tasks, indeed results in better learned representations than smaller multi-task setups. However, if the target tasks are known ahead of time, then training on a smaller set of related tasks is competitive to the large-scale multi-task training at a reduced computational cost.

+
Footnote †: _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 2542 - 2550

July 10-15, 2022 ©2022 Association for Computational Linguistics

## 1 Introduction

Following the wide success of unsupervised language model pre-training Devlin et al. (2019); Liu et al. (2019); Lewis et al. (2019), recent work on transfer learning has shown that additional supervised multi-task training further improves performance on various downstream NLP tasks Raffel et al. (2019); Khashabi et al. (2020); Aghazjanyan et al. (2021). There are two distinct ways in which the supervised data has been used: increasing the _scale_ of the multi-task step to incorporate more tasks Khashabi et al. (2020); Aghazjanyan et al. (2021) and developing task similarity metrics to incorporate tasks _related_ to the target task Pruksachatkun et al. (2020); Vu et al. (2020).

Aghajanyan et al. (2021) show that a multi-task training step, or _pre-finetuning_ step, with a sufficiently large, diverse set of tasks is an effective task-agnostic second stage of model pre-training before finetuning on target tasks. In particular, they find that using a large number of tasks (e.g., roughly 15 tasks) is crucial in achieving good downstream performance, while pre-finetuning with fewer tasks causes a small performance drop. Meanwhile, work on task transferability Vu et al. (2020); Prusachatkun et al. (2020) has shown that the choice of individual intermediate tasks significantly affects downstream fine-tuning performance--predicting the transfer is challenging and there is high variance depending on the choice of intermediate task.

This motivates us to ask the question if pre-finetuning on a small group of tasks related to the target tasks can obtain comparable performance to large-scale multi-task training. In this work, we present an empirical study to answer this question by extending the task transferability experiment to groups of tasks.

We follow the two-step experimental pipeline from Aghajanyan et al. (2021), where a pre-trained model is first pre-finetuned on a set of tasks and then separately finetuned on various target tasks on which we report performance. In addition, we group our set of 29 pre-finetuning tasks based on task format into 3 groups--classification tasks, sequence labelling tasks and extractive question answering tasks (Figure 1). We perform model pre-finetuning on every combination of these task groups and report performance on target tasks that belong to each group. This allows us to systematically study how the size of the multi-task step and the choice of pre-finetuning tasks affects downstream task performance.

We observe that, on average, large-scale multi-task pre-finetuning results in improved performance on downstream target tasks. We also see that a model trained on related1 pre-finetuning tasks obtains comparable downstream task performance to the large-scale model, at a reduced computational cost2, but pre-finetuning on an unrelated grouping

can result in a severe decline in performance.

Our findings show the interplay between multi-task scaling and task selection--when the target tasks are unknown then multi-task scaling is an effective, if expensive, intermediate step but if the goal is to improve performance on a specific target task set then multi-task training on a smaller set of related tasks is an effective cheaper alternative. These results also hint at the need to better study modeling techniques that mitigate negative transfer between tasks in large-scale multi-task training.

## 2 Multi-task Setup

Our multi-task experiments follow the two step approach from Aghajanyan et al. (2021), shown in Figure 2. A pre-trained model is first pre-finetuned on a set of tasks to obtain a single shared encoder. This model is then finetuned separately on various target tasks on which we report performance.

Grouping of TasksFigure 1 shows our list of 29 pre-finetuning tasks and 12 unseen target tasks (listed in Appendix A). Informed by prior work Vu et al. (2020); Ye et al. (2021); Sanh et al. (2021), we divide these datasets into three groups based on task format--sequence labelling (SL), extractive question answering (QA) and classification (C). As noted in Sanh et al. (2021), grouping of NLP tasks is an imperfect heuristic. Prior work Achille et al. (2019); Vu et al. (2020) formalizes the notion of task similarity using learned task embeddings so an alternate formulation would be to divide the tasks into groups based on these learned embeddings. In this work we focus on a simple, intuitive grouping based on the task format, or output space, of the task.

Research QuestionWe aim to study how the choice of pre-finetuning tasks and the size of the multi-task step, in terms of number of pre-finetuning tasks, affects target task performance. In order to do so, we compare pre-finetuning runs on all combinations of task groups, reporting performance on target tasks from each group.

## 3 Experiments

### Model Details

We use the pre-trained XLM-Roberta model Conneau et al. (2020) for all of our experiments. During pre-finetuning, we learn a shared encoder for all tasks and a task-specific head for each pre-finetuning task. For downstream finetuning, we randomly initialize a new head for each target task. We use the Huggingface Wolf et al. (2020) XLM-Roberta base pre-trained model. The various task specific heads are linear classifiers on the encoder output per the Huggingface implementation. More model details are provided in Appendix B.1.

### Training Details

During pre-finetuning, to ensure that the model does not overfit to any particular task, we follow the sampling approach from Aghajanyan et al. (2021) of ensuring that each batch consists of examples from multiple tasks while maintaining the empirical distribution of examples in the datasets. For our study, the loss function for all the different pre-finetuning tasks is cross entropy loss. Aghajanyan et al. (2021) recommends scaling the losses from each task-specific head based on the size of

Figure 1: List of tasks grouped into sequence labelling (SL), question answering (QA) and classification (C).

Figure 2: High level workflow of the multi-task setup.

 

the label space to ensure more stable training. Our preliminary experiments showed better results without loss scaling so we follow the same for all pre-finetuning runs.

We run pre-finetuning on the powersset of the set of task groups from Figure 1 and for each pre-finetuning run we report performance on all the target tasks. Hyperparameters are kept uniform across pre-finetuning runs and we train with batch size 128 and an early stopping criteria based on validation loss. Once pre-finetuning is completed, we finetune the model on each target task and report the average validation set performance3 across 5 random seeds along with the associated standard deviation. More details on pre-finetuning and target task finetuning are included in Appendix B.2.

Footnote 3: For all sequence labelling target tasks, we report F1 score. Among classification tasks, we report F1 for CB and accuracy for the other tasks. For extractive question answering, we report F1. These choices was made based on the standard evaluation for the task from the Huggingface metrics module.

NotationWe refer to the pre-finetuning runs with their initials--Only-QA means that only question answering tasks were used in pre-finetuning the model and QA+C means that all question-answering and classification tasks were used and so on. We have 3 task groups and run experiments on all 7 possible combinations--Only-SL, Only-QA, Only-C, QA+C, QA+SL, SL+C and SL+QA+C. The baseline is a pre-trained model directly fine-tuned on the target tasks.

## 4 Results

Table 1 shows the results of each pre-finetuning run on all the target tasks. Table 2 contains the results aggregated by task group.

A large-scale target task-agnostic Multi-task step improves downstream performanceFrom Table 1, we see that the multi-task setup containing all 29 pre-finetuning tasks (SL+QA+C) has the best average performance over all the target tasks as well as the best score in 5 individual tasks. This is consistent with observations reported in Aghajanyan et al. (2021) that increasing the scale of the multi-task step results in better downstream performance on average across all tasks. Our results show the same trend on a different set of tasks with a smaller batch size. We also show that this observation holds in a standard multi-task training regime, without the optimization tricks used Aghajanyan et al. (2021), namely loss scaling and regularized finetuning (Aghajanyan et al., 2020).

Related tasks transfer betterTo identify the role of transferability, we aggregate the results on target tasks based on task groups in Table 2. Each row in Table 2 is the average of all the unseen tasks within that group from Table 1. From this, we see that the Only-SL and Only-QA setups are on-par with SL+QA+C on unseen SL and QA tasks respectively indicating that pre-finetuning on a smaller set of related tasks obtains comparable performance to the large-scale multi-task model. We also see that selecting a mismatched set of pre-finetuning tasks significantly hurts downstream task performance. From Table 2, we see a drop of 9.6% compared to the baseline on SL tasks with Only-C pre-finetuning and a 20.3% drop on QA tasks with Only-SL. These results extend those observed in Pruksachatkun et al. (2020) and Vu et al. (2020) to transferability across task groups. With appropriate task selection, we can obtain comparable performance to the large-scale multi-task model at a reduced computational cost. The reduction in computational cost is mainly due to the change in number of pre-finetuning examples. We provide the comparison over various runs in Appendix B.3. Aghajanyan et al. (2021) reported that multi-task learning is detrimental to target task performance at a smaller scale (\(<15\) tasks) . In our study, we see that pre-finetuning on a single group of related tasks always outperforms the results from the mismatched pairwise setup--Only-QA outperforms SL+C on unseen QA tasks, Only-SL outperforms QA+C on SL tasks and Only-C outperforms SL+QA on C tasks. Hence we conclude that at smaller scales, the particular pre-finetuning tasks selected significantly impacts downstream task performance, linking back to transferability literature.

Tasks interact differently, so selecting an optimal subset is hardWhen we look at pre-finetuning runs on pairs of task groups taken together, we see that the SL+C and QA+C pre-finetuning setups perform worse on QA and SL tasks than even the baseline model but the SL+QA setup is competitive with the best SL+QA+C setup across all unseen tasks. This shows that selecting an optimal combination of tasks can be challenging based on task group heuristics. Aribandi et al. (2021) also observed a similar result that, at larger scales, a random subset of tasks often outperforms 

subsets selected using simple heuristics.

Multi-task training reduces the variability of downstream task performanceIn Table 1, we also report the standard deviation in target task performance across 5 random restarts. We see a trend that large-scale multi-task pre-finetuning reduces the variability across runs on all tasks--the SL+QA+C setup has the lowest average and the pairwise setups average lower variation than the single task group setups. Phang et al. (2018) also reported similar findings that multi-task training reduces variability in performance across random restarts. Additionally we observe that the Only-SL, Only-C and Only-QA setups have lower variability on unseen tasks of the same group than other groups, indicating that the downstream performance is more reliable on tasks within the same group. We discuss some limitations of our setup in Appendix C

## 5 Related Work

Large-Scale Multi-task LearningPost the wide success of unsupervised language model pretraining, Phang et al. (2018) showed that intermediate task training on large datasets results in performance improvements on GLUE benchmark. Liu et al. (2019a) showed an improvement over standard pre-training on multiple NLP benchmarks in the multi-task setting. T5 (Raffel et al., 2019) framed various NLP tasks in a text-to-text format and subsequent work (Khashabi et al., 2020; Paolini et al., 2021) and sequence labelling showed that adapting T5-style models to particular domains results in powerful multi-task models. Aghajanyan et al. (2020) found that increasing the scale of a multi-task pre-finetuning step results in uniform improvement across various unseen tasks. Recent work in prompting large LMs has also shown that multi-task training can improve zero-shot performance (Wei et al., 2021; Sanh et al., 2021). Ye et al. (2021) showed that the few-shot performance on unseen tasks can be improved via a supervised multi-task step and recommended further analysis on task similarity and transferability. Our work aims to address this gap and connect large-scale multi-task learning to work on transferability.

Exploring Relationships Between TasksWang et al. (2019) and Pruksachatkun et al. (2020) performed extensive empirical studies to identify the most beneficial intermediate tasks that improve target task performance both yielding mixed results. Changpinyo et al. (2018) observed that jointly

\begin{table}
\begin{tabular}{c c c c c c c c c|c} \hline  & **Baseline** & **Only-SL** & **Only-C** & **Only-QA** & **SL+C** & **SL+QA** & **QA+C** & **SL+QA** \\ \hline
**UNseen SL** & \(80.134\) & \(\mathbf{80.844}\) & \(72.370\) & \(78.693\) & \(79.453\) & \(80.165\) & \(77.378\) & \(80.750\) \\
**Unseen C** & \(68.109\) & \(67.406\) & \(71.404\) & \(63.21\) & \(70.422\) & \(70.068\) & \(70.067\) & \(\mathbf{73.021}\) \\
**UNseen QA** & \(56.692\) & \(45.174\) & \(61.120\) & \(75.252\) & \(57.568\) & \(75.460\) & \(75.035\) & \(\mathbf{75.678}\) \\ \hline
**Average** & \(68.312\) & \(64.475\) & \(68.298\) & \(72.385\) & \(69.147\) & \(75.564\) & \(74.160\) & \(\mathbf{76.483}\) \\ \hline \end{tabular}
\end{table}
Table 2: Results on all unseen tasks aggregated by task format (rows) for each pre-finetuning setup (columns). Each value in this table is an average of the 4 unseen tasks of that particular task format from Table 1. We see the effect of transferability where the Only-SL and Only-QA setups are competitive with SL+QA+C on unseen SL and QA tasks but suffer significantly on mismatched task groups.

\begin{table}
\begin{tabular}{l l l l l l l l l l} \hline
**Dataset** & **Baselines** & **Only-SL** & **Obj-Q** & **SQL+C** & **SLA-QA** & **QuAC+** & **Sl+QA** & **QuAC+** & **SL+QA** \\ \hline
**UNNITI** & \(63.929\) & \(63.132\) & \(66.95\) & \(43.431\) & \(158.052\) & \(60.973\) & \(62.952\) & \(62.610\) & \(67.950\) & \(66.362\) & \(54.950\) \\
**Wilded** & \(80.409\) & \(48.849\) & \(62.714\) & \(45.085\) & \(78.713\) & \(10.641\) & \(87.403\) & \(63.131\) & \(11.07\) & \(70.023\) & \(2.81\) & \(81.635\) \\
**Nell Disease** & \(86.492\) & \(46.870\) & \(65.554\) & \(65.421\) & \(68.562\) & \(68.074\) & \(67.433\) & \(68.476\) & \(\mathbf{0.32}\) & \(86.642\) & \(78.737\) & \(69.737\) \\
**MIT Monte** & \(\mathbf{90.426}\) & \(\mathbf{98.93}\) & \(79.360\) & \(85.430\) & \(68.952\) & \(98.922\) & \(98.947\) & \(92.892\) & \(98.276\) & \(98.931\) & \(2.21\) \\ \hline
**BackD** & \(78.044\) & \(15.076\) & \(70.72\) & \(78.713\) & \(40.48\) & \(77.140\) & \(77.875\) & \(70.567\) & \(76.247\) & \(80.1497\) & \(\mathbf{79.7}\) & \(51.510\) \\
**CB** & \(86.333\) & \(20.218\) & \(21.217\) & \(87.161\) & \(67.980\) & \(10.658\) & \(43.362\) & \(50.003\) & \(90.803\) & \(87.09\) & \(\mathbf{80.55}\approx 1.46\) \\
**Corns** & \(52.166\) & \(49.530\) & \(55.004\) & \(56.553\) & \(33.930\) & \(58.833\) & \(73.566\) & \(32.506\) & \(53.696\) & \(49.24\) & \(53.313\) & \(5.87\) & \(\mathbf{33.33}\approx 2.48\) \\
**WC** & \(60.136\) & \(74.565\) & \(62.410\) & \(64.812\) & \(52.514\) & \(53.977\) & \(65.229\) & \(14.556\) & \(51.771\) & \(65.511\) & \(55.153\) & \(55.165\) & \(\mathbf{67.674}\) \\ \hline
**Spand Adv** & \(47.064\) & \(24.284\) & \(23.284\) & \(55.067\) & \(83.829\) & \(79.681\) & \(53.053\) & \(61.535\) & \(80.80\) & \(82.776\) & \(61.813\) & \(83.488\) \\
**SubJQA** & \(60.917\) & \(72.584\) & \(58.777\) & \(41.742\) & \(61.789\) & \(60.819\) & \(60.718\) & \(62.367\) & \(62.789\) & \(62.886\) & \(\mathbf{0.56}\approx 6.286\) & \(\mathbf{0.56}\approx 6.36\) \\
**QD** & \(37.779\) & \(56.544\) & \(11.561\) & \(56.041\) & \(34.746\) & \(64.333\) & \(12.719\) & \(\mathbf{76.160}\) & \(73.910\) & \(44.76\) & \(31.32\) \\
**Xoud-B** & \(63.522\) & \(46.50\) & \(30.389\) & \(46.538\) & \(8.404\) & \(\mathbf{70.449}\) & \(63.654\) & \(79.624\) & \(78.249\) & \(80.163\) & \(65.3\) \\ \hline
**Average** & \(68.312\) & \(64.475\) & \(68.298\) & \(72.385\) & \(69.147\) & \(75.564\) & \(74.160\) & \(\mathbf{76.483}\) \\ \hline \end{tabular}
\end{table}
Table 1: Results on all the target tasks (rows) for all the pre-finetuning schemes (columns). Each cell value is an average on 5 runs with different seeds and 

learning 11 sequence tagging tasks, with task embeddings, improves performance in around half of them and that the learned task embeddings revealed interesting task relationships such as clusters of semantic and syntactic tasks. Kiperwasser and Ballesteros (2018) showed that learning to perform syntactic tasks such as dependency parsing and part-of-speech tagging along with translation in varying schedules improves translation performance--they used task embedding vectors to identify the tasks to the decoder model. Indeed the idea of using identifying tokens as language embeddings is known to improve translation Johnson et al. (2017) and dependency parsing Ammar et al. (2016) predates the widespread adoption of transformer models for these tasks.

More recently, Vu et al. (2020) proposed two methods to learn task embeddings capable of predicting transferability between source and target tasks-one by pooling the representations of the textual task data from BERT and the other by using the layer-wise gradients of a BERT model. Vu et al. (2021) learned task specific prompts that can benefit each other via prompt transfer. These works largely identify the single most suitable task for each target task, we extend the same to groups of tasks. Our work most closely relates to a concurrent study, Aribandi et al. (2021), that examined the transfer across various task families. Our results compliment theirs using a different base model--they use a T5 style formulation of tasks, we use a shared Roberta encoder approach, showing that the transferability phenomenon is independent to the model architecture. We differ from their work in that we compare transfer on individual task groups with pairs of task groups as well and present results on the variance of performance as a result of multi-task learning.

## 6 Conclusion

In this work, we bring together the lines of exploration on transferability of tasks and large-scale multi-task training. Our results show that when the target tasks are unknown then multi-task scaling offers an effective way to obtain good downstream performance but if the goal is to improve performance on a specific target task set then a smaller set of related tasks is an effective, cheaper alternative. We observe that task groups interact differently when combined and that selecting an optimum subset becomes harder as the size increases. We also see that variability across multiple random restarts decreases on related target tasks and also reduces on increasing the size of the multi-task step.

## Acknowledgements

We would like to thank Stefano Soatto for providing feedback on this draft as well as the anonymous reviewers for their helpful comments.

## References

* Achille et al. (2019) Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. 2019. TaskZvec: Task embedding for meta-learning. In _Proceedings of the IEEE/CVF International Conference on Computer Vision_, pages 6430-6439.
* Aghajanyan et al. (2021) Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre-finetuning. _arXiv preprint arXiv:2101.11038_.
* Aghajanyan et al. (2020) Armen Aghajanyan, Akshat Shrivastava, Anchit Gupta, Naman Goyal, Luke Zettlemoyer, and Sonal Gupta. 2020. Better fine-tuning by reducing representational collapse. _CoRR_, abs/2008.03156.
* Alvarado et al. (2015) Julio Cesar Salinas Alvarado, Karin Verspoor, and Timothy Baldwin. 2015. Domain adaption of named entity recognition to support credit risk assessment. In _Proceedings of the Australasian Language Technology Association Workshop 2015_, pages 84-90.
* Ammar et al. (2016) Waleed Ammar, George Mulcaire, Miguel Ballesteros, Chris Dyer, and Noah A Smith. 2016. Many languages, one parser. _Transactions of the Association for Computational Linguistics_, 4:431-444.
* Aribandi et al. (2021) Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Honglei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jiaiputa, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021. Ext5: Towards extreme multi-task scaling for transfer learning.
* Balasuriya et al. (2009) Dominic Balasuriya, Nicky Ringland, Joel Nothman, Tara Murphy, and James R Curran. 2009. Named entity recognition in wikipedia. In _Proceedings of the 2009 Workshop on The People's Web Meets NLP: Collaboratively Constructed Semantic Resources (People's Web)_, pages 10-18.
* Bos et al. (2017) Johan Bos, Valerio Basile, Kilian Evans, Norothej Verhuizen, and Johannes Bjerva. 2017. The Groningen meaning bank. In _Handbook of linguistic annotation_, pages 463-496. Springer.
* Changpinyo et al. (2018) Soravit Changpinyo, Hexiang Hu, and Fei Sha. 2018. Multi-task learning for sequence tagging: An empirical study. _arXiv preprint arXiv:1808.04151_ 

Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary, Guillaume Wenzek, Francisco Guzman, Edouard Grave, Myle Ott, Luke Zettlemoyer, and Veselin Stoyanov. 2020. Unsupervised cross-lingual representation learning at scale.
* Coucke et al. (2018) Alice Coucke, Alaa Saade, Adrien Ball, Theodore Bluche, Alexandre Caulier, David Leroy, Clement Doumouro, Thibault Gisselbrecht, Francesco Caltagirone, Tibbaut Lavril, et al. 2018. Snips voice platform: an embedded spoken language understanding system for private-by-design voice interfaces. _arXiv preprint arXiv:1805.10190_.
* Derczynski et al. (2017) Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the wnut2017 shared task on novel and emerging entity recognition. In _Proceedings of the 3rd Workshop on Noisy User-generated Text_, pages 140-147.
* Devlin et al. (2019) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In _Association for Computational Linguistics (ACL)_, pages 4171-4186.
* Ding et al. (2021) Ning Ding, Guangwei Xu, Yulin Chen, Xiaobin Wang, Xu Han, Pengjun Xie, Hai-Tao Zheng, and Zhiyuan Liu. 2021. Few-nerd: A few-shot named entity recognition dataset. _arXiv preprint arXiv:2105.07464_.
* Dogan et al. (2014) Rezatta Islamaj Dogan, Robert Leaman, and Zhiyong Lu. 2014. Ncbi disease corpus: a resource for disease name recognition and concept normalization. _Journal of biomedical informatics_, 47:1-10.
* DSTL (2017) DSTL. 2017. Relationship and entity extraction evaluation dataset. [https://github.com/dstl/re3d](https://github.com/dstl/re3d).
* Hakkani-Tur et al. (2016) Dilek Hakkani-Tur, Gokhan Tur, Asli Celikjilmaz, Yun-Nung Chen, Jianfeng Gao, Li Deng, and Ye-Yi Wang. 2016. Multi-domain joint semantic frame parsing using bi-directional RNN-LSTM. In _InterSpeech_.
* Hemphill et al. (1990) Charles T. Hemphill, John J. Godfrey, and George R. Doddington. 1990. The ATIS spoken language systems pilot corpus. In _Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27,1990_.
* Johnson et al. (2017) Melvin Johnson, Mike Schuster, Quoc V Le, Maxim Krikun, Yonghui Wu, Zhifeng Chen, Nikhil Thorat, Fernanda Viegas, Martin Wattenberg, Greg Corrado, et al. 2017. Google's multilingual neural machine translation system: Enabling zero-shot translation. _Transactions of the Association for Computational Linguistics_, 5:339-351.
* Khashabi et al. (2020) Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format boundaries with a single qa system.
* Kiperwasser and Ballesteros (2018) Eliyahu Kiperwasser and Miguel Ballesteros. 2018. Scheduled multi-task learning: From syntax to translation. _Transactions of the Association for Computational Linguistics_, 6:225-240.
* Lewis et al. (2019) Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019. Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension. _arXiv preprint arXiv:1910.13461_.
* Lhoest et al. (2021a) Quentin Lhoest, Albert Villanova del Moral, Patrick von Platen, Thomas Wolf, Mario Sasko, Yacine Jerneitte, Abhishek Thakur, Lewis Tunstall, Suraj Pattil, Mariama Drame, Julien Chaumond, Julien Plu, Joe Davison, Simon Brandeis, Victor Sanh, Teven Le Scaco, Kevin Canwen Xu, Nicolas Patry, Steven Liu, Angelina McMillan-Major, Philipp Schmid, Sylvain Gugger, Nathan Raw, Sylvan Lesage, Anton Lozhkov, Matthew Carrigan, Theo Matussieere, Ledro von Werra, Lysandre Debut, Staes Bekman, and Clement Delangue. 2021a. huggingface/datasets: 1.15.1.
* Lhoest et al. (2021b) Quentin Lhoest, Albert Villanova del Moral, Yacine Jernite, Abhishek Thakur, Patrick von Platen, Suraj Pattil, Julien Chaumond, Mariama Drame, Julien Pu, Lewis Tunstall, Joe Davison, Mario Sasko, Gunjan Chhablani, Bhavitya Malik, Simon Brandeis, Teven Le Scaco, Victor Sanh, Canwen Xu, Nicolas Patry, Angelingna McMillan-Major, Philipp Schmid, Sylvain Gugger, Clement Delangue, Theo Mathussiere, Lysandre Debut, Staes Bekman, Pierric Cistace, Thibault Goehringer, Victor Mustar, Francois Lagunas, Alexander Rush, and Thomas Wolf. 2021b. Datasets: A community library for natural language processing. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 175-184, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Lim et al. (2017) Swee Kiat Lim, Aldrian Obaja Muis, Wei Lu, and Chen Hui Ong. 2017. Malwaretextdb: A database for annotated malware articles. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1557-1567.
* Liu et al. (2019a) Xiaodong Liu, Pengcheng He, Weizhu Chen, and Jianfeng Gao. 2019a. Multi-task deep neural networks for natural language understanding. _arXiv preprint arXiv:1901.11504_.
* Liu et al. (2019b) Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. RoBERTa: A robustly optimized BERT pretraining approach. _arXiv preprint arXiv:1907.11692_.
* Nothman et al. (2013) Joel Nothman, Nicky Ringland, Will Radford, Tara Murphy, and James R Curran. 2013. Learning multilingual named entity recognition from wikipedia. _Artificial Intelligence_, 194:151-175 

* Ohta et al. (2012) Tomoko Ohta, Sampo Pyysalo, Jun'ichi Tsujii, and Sophia Ananiadou. 2012. Open-domain anatomical entity mention detection. In _Proceedings of the workshop on detecting structure in scholarly discourse_, pages 27-36.
* Paolini et al. (2021) Giovanni Paolini, Ben Athiwaratkun, Jason Krone, Jie Ma, Alessandro Achille, RISHITA ANUBHAI, Cicero Nogueira dos Santos, Bing Xiang, and Stefano Soatto. 2021. Structured prediction as translation between augmented natural languages. In _International Conference on Learning Representations_.
* Phang et al. (2018) Jason Phang, Thibault Fevry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. _arXiv preprint arXiv:1811.01088_.
* Pruksachatkun et al. (2020) Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R Bowman. 2020. Intermediate-task transfer learning with pretrained models for natural language understanding: When and why does it work? _arXiv preprint arXiv:2005.00628_.
* Raffel et al. (2019) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. _arXiv preprint arXiv:1910.10683_.
* Rodriguez (2020) Juan Diego Rodriguez. 2020. Datasets for entity recognition. [https://github.com/juand-r/entity-recognition-datasets](https://github.com/juand-r/entity-recognition-datasets).
* Sang and De Meulder (2003) Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independent named entity recognition. _arXiv preprint cs/0306050_.
* Sanh et al. (2021) Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintanq Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, Manan Dey, M Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Elizza Szczechla, Taewoon Kim, Gunjan Chhablani, Nihal Nayak, Debajyooti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang, Mattee Manica, Sheng Shen, Zheng Xin Yong, Harshit Pandey, Rachel Baowden, Thomas Wang, Trishaha Neeraji, Jos Rozen, Abneesh Sharma, Andrea Santiilli, Thibault Fevry, Jason Alan Fries, Ryan Teehan, Stella Biderman, Leo Gao, Tali Bers, Thomas Wolf, and Alexander M. Rush. 2021. Multi-task prompted training enables zero-shot task generalization.
* CSAIL Spoken Language Systems Group (2020) MIT CSAIL Spoken Language Systems Group. 2020. Mit movie corpus. [https://groups.csail.mit.edu/sls/downloads/](https://groups.csail.mit.edu/sls/downloads/).
* Vu et al. (2021) Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer.
* Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. _CoRR_, abs/2005.00770.
* Wang et al. (2019a) Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4465-4476, Florence, Italy. Association for Computational Linguistics.
* Wang et al. (2019b) Alex Wang, Yadda Prukschatchun, Nikita Nangia, Amanneret Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In _Advances in Neural Information Processing Systems (NeurIPS)_.
* Wang et al. (2019c) Alex Wang, Ampreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019c. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _International Conference on Learning Representations (ICLR)_.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V. Le. 2021. Finetuned language models are zero-shot learners.
* Weischedel et al. (2013) Ralph Weischedel, Martha Palmer, Mitchell Marcus, Eduard Hovy, Sameer Pradhan, Lance Ramshaw, Ninawen Xue, Ann Taylor, Jeff Kaufman, Michelle Franchini, et al. 2013. Ontonotes release 5.0 ldc2013t19. _Linguistic Data Consortium, Philadelphia, PA_, 23.
* Wolf et al. (2020) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Loufor, Morgan Funtowicz, Joe Davison, Sam Shlieifer, Patrick von Platen, Clara Ma, Vacine Jentice, Julien Pu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lohoest, and Alexander M. Rush. 2020. Transformers: State-of-the-art natural language processing. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations_, pages 38-45, Online. Association for Computational Linguistics.
* Ye et al. (2021) Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross-task generalization in nlp.
* Zeldes (2017) Amir Zeldes. 2017. The gum corpus: Creating multilayer resources in the classroom. _Language Resources and Evaluation_, 51(3):581-612 

Dataset Details

Sequence Labelling (SL) Tasks:We source the following sequence labeling datasets and use them in the CONLL data format. Rodriguez (2020) provides preprocessed data in the original splits. We report F1 score on all unseen target tasks. The total number of train examples across all 15 datasets is 225,433.

Pre-finetuning tasks:

* English, Spanish, Dutch, German [Sang and De Meulder2003]
* English, Chinese [Weischedel et al.2013]
* ANEM [Ohta et al.2012]
* GUM [Zeldes2017]
* GMB [Bos et al.2017]
* SEC Filings [Alvarado et al.2015]
* Re3d [DSTL2017]
* Malware TextDB [Lim et al.2017]
* Few-NERD [Ding et al.2021]
* SNIPS [Coucke et al.2018]
* ATIS [Hemphill et al.1990] with preprocessing from the repository of Hakkani-Tur et al. (2016)

Unseen Target Tasks:

* WNUT17 [Derczynski et al.2017]
* Wikigold [Balasuriya et al.2009; Nothman et al.2013]
* EN [Spoken Language Systems Group2020]
* NCBI Disease Corpus [Dogan et al.2014]

Classification (C) TasksThe pre-finetuning classifications tasks are from the Glue benchmark [Wang et al.2019c] and the 4 unseen tasks are from the Superglue benchmark [Wang et al.2019b]. We use the versions made available via the Huggingface Datasets library [Lhoest et al.2021b,a]. In total, we have 943984 train examples across the pre-finetuning classification tasks. The unseen target tasks Superglue tasks, again made available via HuggingFace. We retain the original splits and report performance on the validation sets. For Commitment Bank (CB) we report F1 score and accuracy for the other 3 unseen tasks as dictated by the metrics module from HuggingFace.

Question Answering (QA) TasksOur 6 pre-finetuning QA tasks Figure 1 are obtained from the MRQA dataset. We use the versions made available via the HuggingFace Datasets library [Lhoest et al.2021b,a]. In total, we have 435,624 train examples across the pre-finetuning question answering tasks. We use the Books subset of the SubjQA dataset. For QED and SquadAdv, we split the data in a 4:1 ratio for train and validation. These datasets are collected as challenge sets explaining the big difference in performance. For XQuad, we use only the English language data and the original train and validation splits. We report F1 again as the metric of comparison for all QA tasks using the HuggingFace metrics module.

The HuggingFace Datasets library is released under the Apache License 2.0. The license information for all the sequence labelling datasets are available at Rodriguez (2020).

Heterogeneous BatchesWhen we run pre-finetuning on any combination of groups, we pool all the examples from the corresponding datasets and create heterogeneous batches from this pool. This is the chief reason for a gain in terms of computational cost on selecting a smaller subset and we provide statistics to measure this in Appendix B.3.

## Appendix B Training Details

### Model Details

The model we use for our experiments is an XLM-Roberta model. We learn a shared encoder during pre-finetuning along with separate task specific heads for each pre-finetuning task. Our tasks are of three different formats that use the output of the encoder differently to make predictions--for classification we predict a single label for the entire sequence using the representation of the <<> token, for sequence labelling we predict a label for each token in the input sequence and for extractive question answering we demarcate a span of the input sequence that corresponds to the answer for the question. The implementation of each of these follows from standard task-specific heads released 

by Huggingface where the corresponding output is fed to a linear classifier. When we initialize the model, we provide it with a list of pre-finetuning tasks to index the various task-specific heads. For each forward pass, the batch consists of the tokenized input as well as the task indices to be used for all the examples. When we use the model to report performance on a target task, the encoder checkpoint is loaded and a new task-specific head is initialized.

### Pre-Finetuning and Target Task Finetuning Details

During pre-finetuning, the model sees examples from multiple tasks in the train and validation sets. We train each pre-finetuning model with an early stopping condition based on validation loss. We compute the validation loss for each task separately at the end of each epoch and the early stopping condition is when the average of the validation losses of all tasks does not improve for 3 epochs. This scheme is kept uniform for all pre-finetuning runs. We also keep the same batch size, 128, and search space of learning rates for pre-finetuning. We use validation data to search for the best learning rate, sweeping from \(1e^{-3}\) to \(1e^{-5}\). The rest of the Adam optimizer parameters are retained as the default values from Huggingface Trainer Wolf et al. (2020).

For target task finetuning, we load the saved pre-finetuned model and train the model to convergence defined as when the average validation loss does not improve for 3 consecutive epochs. We again sweep for the best learning rate from \(1e^{-3}\) to \(1e^{-5}\) and report performance on the best selection across 5 random restarts.

### Computational Cost

All experiments are run on an Amazon p3.16xlarge EC2 instance containing 8 Tesla V100 GPUs. The relative improvement in computational cost on selecting a smaller subset of tasks is mainly due to having fewer examples since we make use of heterogeneous batches. Following was the per-epoch runtimes during pre-finetuning on the various runs:

* 1131 seconds per epoch
* 1643 seconds per epoch
* 2237 seconds per epoch
* 2661 seconds per epoch
* 3413 seconds per epoch
* 3079 seconds per epoch
* 4884 seconds per epoch

## Appendix C Limitations and Future Work

Grouping of TasksWe note that grouping of NLP tasks is often fuzzy and imprecise. Our chosen grouping was based on task format, as is used in recent work in the field Vu et al. (2020); Sanh et al. (2021). We note that task similarity measures Achille et al. (2019) calculate a more principled relationship between tasks. We acknowledge that a more optimum grouping could be found for each target task set but our results (Table 2) show that our chosen heuristic is a reasonable choice to isolate the effect of transferability. Selecting an optimal grouping would be a combinatorial task which would take significantly more compute. The way we choose to group tasks makes it easy to select 'related' tasks for each target task set. This might become more challenging for non-standard tasks.

Controlling the Size of Each Task GroupOur chosen task groups have an unequal number of examples per task and group. We chose to retain all the examples from the various datasets since controlling for the number of examples also doesn't account for the relative difficulty of tasks (in particular we see that some of our chosen QA tasks seem to be more difficult for the model) but this could be a future line of research.

Potential RisksThe main risks of our project are the risks associated with training large language models. We do not collect the datasets ourselves and use publicly released data which might contain biases against certain protected groups that will be reflected on models trained in this manner. To the best of our knowledge, these are standard datasets and we use them for the released tasks but we do not manually check them for offensive content