TaskWeb: Selecting Better Source Tasks for Multi-task NLP
Authors: Joongwon Kim
Akari Asai
Gabriel Ilharco
Hannaneh Hajishirzi
Venue: EMNLP 2023 Main (2023)
Keywords: NLP
machine learning
task transfer
task selection
multi-task training
TL;DR: We quantify task relationships via pairwise task transfer, propose a new method for selecting helpful source tasks and apply it to multi-task training and improve zero-shot performances.

Abstract: Recent work in NLP has shown promising results in training models on large amounts of tasks to achieve better generalization. However, it is not well-understood how tasks are related, and how helpful training tasks can be chosen for a new task. In this work, we investigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TaskWeb, a large-scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design a new method TaskShop based on our analysis of TaskWeb. TaskShop uses TaskWeb to estimate the benefit of using a source task for learning a new target task, and to choose a subset of helpful training tasks for multi-task training. Our method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. We also use TaskShop to build much smaller multi-task training sets that improve zero-shot performances across 11 different target tasks by at least 4.3%.

TASKWEB: Selecting Better Source Tasks for Multi-task NLP
Joongwon Kim', Akari Asai', Gabriel Ilharco', Hannaneh Hajishirzi
"University of Washington (jwonkim, gamaga, hannaneh]@cs washington.edr
2 Allen Institute for Al
Abstract
Assign score from source task target task
Recent work in NLP has shown promising re- sults in training models on large amounts of tasks to achieve better generalization. How- ever, it is not well-understood how tasks are related, and how helpful training tasks be chosen for new task. In this work, we inves- tigate whether knowing task relationships via pairwise task transfer improves choosing one or more source tasks that help to learn a new target task. We provide TASKWEB, large- scale benchmark of pairwise task transfers for 22 NLP tasks using three different model types, sizes, and adaptation methods, spanning about 25,000 experiments. Then, we design new method TASKSHOP based on our analysis of TASKWEB. TASKSHOP uses TASKWEB timate the benefit of using source task for learning new target task, and choose sub- set of helpful training tasks for muulii-aask train- ing. Our method improves overall rankings and top-k precision of source tasks by 10% and 38%, respectively. We also use TASKSHOP to build much smaller multi-task training sets that improve zero-shot performances across II different target tasks by at least 4.3%.
TASKWEB
source
target
cosmosqa
cb
socialiqa
target examples (S
coemonga
TASKSHOP
helsored
cosmosqa cb: 0.752
Suartz
socialiqa
helpful saurce mulli fask leaming
snil
Imdb
stsb
sociallqa
cosmosqa
Figure I: We use pairwise transfer scores in TASK WEB to score (source, target) pairs where the source task is in TASKWER and the target task is unseen (i.e., access to only few examples). Then, we select helpful tasks and perform multi-task learning for the target task.
pairwise task transfer helps task selection, which define as choosing one
source tasks that better initialize model for an unseen target task shown in Figure We begin from a pairwise setup as is often used to quantify task relationships (Zamir et al., 2019; Vu et al., 2020) and is more tractable than larger combinations of tasks.
1 Introduction
Recent studies have revealed that large language models are able to generalize to unseen tasks when jointly trained many different tasks, with their performance scaling to the size and diversity of the training data (Sanh et al., 2022: Wang al., 2022b; Wei et al., 2022a; Chung et al., 2022; Long- pre et al., 2023). As more and more tasks are added to build senerral-urposs. models, it has been noted that knowing inter-task relationships may be help- ful but that it remains unclear how to select helpful tasks for multi-task learning (Ye et al., 2021; Min al., 2022; Asai et al., 2022; Chan et al., 2022). In this work, investigate whether quantifying the relationship between different NLP tasks via
large-scale benchmark for pairwise task transfers across dif- ferent model architectures (encoder-only, decoder- only, encoder-devoder), parameter count (60M 770M) and adaptation methods including finetun- ing, Adaptcr-tuning (Houlsby et al., 2019) and Bit- Fit (Zaken et al., 2022), resulting in 25,000 trans- fers. From our results, we discover a transitive property where having strong, positive transfers A + B and B - C for tasks A. B and C makes it more likely that A C is also positive transfer.
First, we construct TASK
Then, we introduce a new method TASKSHOp that predicts the transferability from source task to a target task associated with only a few examples. TASKSHOP builds upon the transitive behavior to
'Our cnde is available at https://giithei com/ danieljkim@11 (aseelljjim1111/eaaaaa 18/aakkWe.

---

construct different paths with "pivot" tasks between the source and target tasks. It combines TASK WEB scores between the source and pivot and textual similarity scores between the pivot and target to estimate (source -ttaget) transfers.
access only to a small number of examples from (n I 32). We propose a new task selection method TASKSHOP that builds upon the transitive behavior to select the best source task to transfer to an unseen target task, even without pairwise transfer scores for the target (Section 4.1). We evaluate the overall task rankings and the precision of top-k helpful tasks returned by TASKSHOP (Section 5.1).
We evaluate our methods in both single-task and multi-task settings. First, we show that TASKSHOP assigns better transferability scores both in terms of the overall ranking and identifying top helpful tasks. Then, we demonstrate that models trained on small multi-task sets built with TASKSHOP out- perform models trained on larger sets tasks. We perform additional analyses and discover that there is a tradeoff for building multitask sets of varying sizes with THSKSHOP, and that the proportion of helpful tasks in the training set affects performance. To summarize, our contributions are follows:
multi- Moreover, we extend task selection task setup. By selecting tasks k. > times, we obtain set of l source tasks as a multi-task train- ing set (Section 4.2). We train models on these multi salti--task sets and perform evaluations and analy- ses on 11 different target tasks (Sections 5.2, 5.3).
2.2 Related Work
Pairwise Task Transfer. Pairwise task transfer, also known as intermediate task transfer, is used to quantify relationships between different tasks in computer vision (Zamir et al., 2019; Achille et al., 2019) and NLP (Vu et al., 2020; Poth et al., 2021). It is also used in NLP to study factors impacting task transfer (Pruksachatkun et al., 2020; Albalak et al., 2022) and identify helpful source tasks for parameter-efficien methods (Vu et al., 2022; Su et al., 2022; Asai et al., 2022). Building upon pre- vious work, we address more diverse tasks, models, and adaptation methods.
1. We build and analyze BASKWEB, bench- mark of pairwise transfer experiments across various tasks, models and adaptation methods.
2. We define task selection for single-task and multi-task setups and propose TASkSHOP which uses pairwise transfer scores to predict transfer to an unseen target task.
3. We use TASKSHOP and TASKWER choose helpful source tasks and build small multi- task training sets that result in better zero-shot performance for unseen targets.
Task Selection. Task selection is used in many studies to better initialize models for learning new tasks. Some methods assume access to the entire training set and model (Vu et al., 2020; Poth et al., 2021; Vu et al., 2022; Su et al., 2022), while other methods only access a small portion of the training data (Jang et al., 2023; Paranjape et al., 2023). We build upon the second case in this work.
2 Background and Overview
We use pairwise task transfer to quantify task simi- larities, select better source tasks for unseen tasks and improve performance via multi-task finetuning.
2.1 Overview
Figure 2 depicts how we use task relationships to se- lect better source tasks. We first quantify task rela- tions with pairwise task transfer, which a process of sequentially learning one task- the source task and then another task-the the target task. We use this to build TASK WEB, a collection of 22 diverse, high-resource tasks in NLP and their pairwise task transfer scores across seven different training se- tups (Sections 3.1, 3.2). From our analysis, we find that pairwise task transfer indicates transitive behavior between positive transfers (Section 3.3). We then explore task selection, where for a target task & with n examples and set of source tasks S, we select a helpful task s E S for t. Here, we assume that the target task is unseen, that is, with
Multi- fask Fine-tuning. Multi-task fine-tuning is used to train models that generalize across many tasks (Khashabi al., 2020; Mishra et al., 2022: Sanh et al., 2022). While studies report that adding more tasks generally improve performance, (Agha- janyan et al., 2021; Wei et al., 2022a; Wang et al., 2022b), others report that using a subset of tasks provide better performance (Padmakumar et al., 2022; Chan et al., 2022) but that it is not clear how to identify such subset (Aribandi et al., 2022). Previous work retrieves the top-k relevant source examples based on the target examples (Lin et al., 2022; Ivison al., 2022). In this work, we take a simpler approach and select helpful tasks based on target examples to build multi-task training sets.

---

Retrieve TASKWEB scores from source to plvot
Use score from pivat to target
Assign scores helpful
TASKWEB
wnd
0.15
soceliga
cosmosqa
0.752
source
0.07
0.53
target
single task:
agand VV
0,08
ani
0.38
mrpc
0.670
Section 5.1
0.06
socielige
T
F
0.752
stsb
0.589
pica
0.15
0.12
0.07
snll 0.850
quartz
0.08
target examples (5 32)
multi-task: Section
0: 12
imdb
0.481
Indb
piga
Quartz
pivol examples (S 100)
socialiqa
0.741
Source pivot scores
pivot target scores
TaskShop
task selection
Figure 2: Overview single and multi task selection using TASKSHOP and TASKWER Section describes the pairwise task transfer involved in TASKWEB well as its analysis. Section 4 details TASKSHOP and describes task selection in single task and multi-task setups. Section presents our experiments as well as additional analyses.
3 TASKWEB: A Benchmark for Pairwise Task Transfer
Category
Tasks
NLI/Entailment
ANLI, CB, QNLI RTE, SciTail. SNLI
Previous studies in pairwise task transfer tend to fo- cus on specific models, adaptation methods or task domains (Vu et al., 2020; Poth et al., 2021; Albalak et al., 2022). We introduce TASKWEE. which con- sists of pairwise task transfer experiments that span wide variety of tasks, models, and adaptation methods. TASK WEB can be used as a benchmark to evaluate task transferability, and as a repository for selecting helpful source tasks (Section 4).
Paraphrase
MRPC, QQP. STSB
Sentiment
IMDB, Rotten Tomatoes
Commonsense
COPA, CosmosQA, HellaSwag, PIQA,
Quartz, SocialIQA, Winogrande
Semantics
WIC, WSC
QA
BoolQ, SQuAD2.0
Table I: All tasks used in our pairwise transfer experi- ments, grouped by high-level task categories. Citations for all datasets are provided in Table 8 in the appendix.
3.1 Focus and Experimental Setup
decoder; Raffel et al. 2020), GPT-2 (decoder- only; Radford et al. 2019) and RoBERTa (encoder- only; Liu et al. 2019). We the LM-adapted ver- sions* (Lester et al., 2021) of T5-ssmal//baselarge. as well as GPT-2 medium and RoBERTa-base.
Tasks, To build TASK WEB, we choose a set of 22 representative tasks NLP that span diverse categories and require various forms of knowledge, as shown in Table 1. We perform total of about 25,000 transfers between all pairs of t
Adaptation Settings. We investigate pairwise task transfer with three widely-adopted adapta- tion methods full fine-tuning, Adapter-tuning (Houlsby et al., 2019) and BitFit (Zaken et al., 2022) while fixing T5 base as the base model.
Training Procedure. We finetune a pre-trained language model on the full dataset associated with a source task 8, and further finetune the model on a set of 1,000 random examples of the target task t.3 Then, we compare the performance gain from initializing the model on to finetuning the model on the same subset of t without starting from We repeat this process over eight random seeds to reduce variability (Dodge et al., 2020).
Metrics for Task Transferability. We follow Vu et al. (2020) and use the average percentage change to measure task transfer. Also, we measure the pro- portion of models with positive transfer across all random seeds. We combine both metrics to account for both the magnitude and consistency of transfers across all random seeds. The formal definition is provided in Section A. in the appendix.
Models. We study the impacts of three different model architectures on task transfer-TT (encoder-
We use SQuAD2.0 as only source task due difficulties associated with running SQuAD evaluation for all transfers. "This number was chosen for the model to not overfit to t, but also learn enough from to provide a measure of how it would perform on the task, in line with previous studies.
"The original T5 checkpoints have been trained on datasets that overlap with ours. We aim to separate the effects of multi- task supervised pretraining in our pairwise transfer analysis.

---

target tasks
piqa
0.71
0.:63
0.66
64
0.49
0.58
rtomato
0.03
9.00000000
3.33
0.01
150
0.53
0.71
0.65
0.63
0.58
0.62
0.66
mdd
copa
8.64
06
9.68
6.22
0.06
0.
Scitail
hellopw
02
0.62
quart
4:00
635
6.14
0.0
0.15
150
15
od
of
seli
cosoa
rte
0.02
9.52
0:00
9.03
54
0.04
socigo
0:15
3:0
4.00
0.0
6.9
wic
2.00
307
C.77
winogr
0:05
0.
4.00
0:55
0.05
C3225
W30
2.02
02
0.22
0:4
booka
6.775
wic
and
cosmga
kmob
9000
secioa
Squaa22
vordstr threstos
wilhognd
WSc
source tasks
1567 15b-t
1bbad
Figure 3: (Left) visualization of TASKWEB, our collection of pairwise transfer between 22 different NLP tasks, averaged over seven training setups. Positive transfers are blue and negative transfers are red. All transfers point from the source to the target. (Center) transfer scores between subset of source tasks (three more helpful/three less helpful) and a subset of target tasks. The full set of scores is given in Figure 5 in the appendix. (Top-right) similarities between pairwise transfer results in our experiment of 22 tasks obtained for seven different training setups. (Bottom-right) probability of identifying positive source + target transfers as the minimum threshold for (source pivot, pivot -> target) transfers is increased. Results with all setups are in Figure 14 in the appendix. (5s/b/1: t5-small/basellrgge ft: finetuning, ad: adapter-tuning, bf: BitFit, gpt2: GPT-2 medium, rob: RoBERTa-bes
3.2 Observations from TASK
3.3 Analysis of Mathematical Properties
Results. Figure 3 visualizes TASK WEB-theethe left shows all transfers, and the center gives examples of pairwise transfer scores. All scores are averaged over seven training configurations. Refer to Figures 5 to 12 in the appendix for the full results.
Computing pairwise transfer scores can become costly as more tasks are added. Would be possible to predict transferability beforehand using existing scores? We formulate pairwise task transfer mathematical relationship and investigate two and transitivity.
We note that positive transfers (blue) occur be- tween intuitively similar tasks such as CosmosQA to SocialIQA (+0.15), both of which are multiple- choice commonsense questions. In contrast, neg- ative transfers (red) occur for tasks that seem to require unrelated skills, such from QQP to Cos- mosQA (-0.12). Surprisingly, positive transfers ist between tasks that do seem similar, such as a positive transfer from SocialTQA RTE (+0. 10).
We define commutativity in our setup as whether A + being a positive/negative transfer implies that B + A is also positive/negative transfer. If A + B is known, the commutativity would help us predict B A before performing the transfer. Mcanwhile, we define transitivity in our setup as whether knowing the transfer scores of A - B and - C allows us infer about A + C. This property would also provide us more ficxibility to predict pairwise transfer in advance.
Effects of Training Setup. We investigate how the training setup affects pairwise task transfer. To this end, we build matrices of pairwise transfer scores for each training setup as shown in Figure and compute their normalized dot products.
Commutativity often does not hold. Based on the pairwise transfer scores shown in Figure 3 (cen- ter), we compute the proportion of transfer pairs that exhibit commutativity. Of the 210 unique trans- fer pairs in our setup, we find that 97 exhibit com- mutativity. The results visualized in Figure 13 in the appendix. We uniquely observe from our experiments that pairwise transfer does not display strong signs of commutativity. One possible reason is that while knowledge acquired from task A may be helpful for task B, the reverse may not be true.
Refer to the top-right subfigure of Figure 3. We observe more similar pairwise transfers when 1) the same adaptation method is applied to models of the same class but different sizes, 2) different adaptation methods are applied the same model, For example, T5-base finetune exhibits more simi- lar transfer with T5 smallaale finetune or TS-base adapter/BitFit than GPT-2 or RoBERTa finetune.

---

Transitivity holds for positive transfers. We perform a small experiment where we predict trans- fer A - as positive both A - B and B -> C score above a threshold. Here, we call A the source task, C the target task, and B the "pivot" task.
pairwise transfer score T(s + p) from TASK EB, and average the two scores. We repeat this process over all pivot tasks p E (s} and average the resulting scores. Finally, we linearly interpolate our estimate with a direct estimate F(s -> t) using hyperparameter À tuned on held-out task.
Refer to the bottom-right subfigure of Figure 3. We observe that as stricter criteria is imposed for source pivot and pivot target, the likelihood of observing positive transfers steadily increase across all training setups. For example, the proba- bility of observing positive transfers increases from 88% to 97% when the intermediate thresholds in- crease from 0.01 to 0.04. These results indicate a transitive behavior between positive transfers.
T(s + p) F(p t)
TS(s, = ||S PES)())
Σ
2
+(1 A) F(*
TASKSHOP is directional. One interesting fea- ture of TASKSHOP is its directionality- pre- dictions for A - B differs from B - A. Our method deviates from conventional techniques that use task embeddings and select tasks using cosine similarities, which results in symmetric predictions. Hence our method is more aligned with the non- commutative property observed in Section 3.3.
4 Task Selection for Unseen Target Tasks
Pairwise transfer scores not always available for new target task. We introduce TASKSHOP to estimate transfer from a source task in TASK WFB to an unseen target task with only a small number of examples (Figure 2). Then, we perform task selection in two settings: a single-task setup where we identify helpful source task, and multi-task setup where we locate set of helpful source tasks.
TAsKShop is modular. Another feature of TASKSHOP is its modularity since any task selec- tion method that only uses a small number of target examples can be used for F. Likewise, we utilize recent methods that only use small number of target task examples, thereby excluding methods that require the fine-tuned model or the full train- ing set. Specifically, we use Retrieval -arerall--f--eeeee of-Experts (RoE) from Jang et al. (2023) and the LLM simi- larity method from Paranjape et al. (2023) for F.
4.1 TASkShOP: Selecting Helpful Tasks
The objective of task selection in single-task setup is to predict the benefit of initializing a model on a source task for learning a target task. We intro- duce new method TASKSHOP which uses pair- wise transfer scores to estimate the transfer from source tasks in TASK WBB to unseen target task.
4.2 Extension to Multi-Task Selection
While choosing a single, appropriate source task is beneficial for learning a target task (Vu et al., 2020, 2022). it has also been observed that using multiple source tasks provides additional benefits (Asai et al., 2022). Hence we extend task selection from single-task to multi-task setup.
Setup. Given source task S and an unseen target task t, we seek predict the transferability of s to t. We assume access to pairwise transfer scores between 8 and other source tasks ())s). Meanwhile, we have a small number of examples (n < 32) but no pairwise transfer scores for t.
Given a target task and a task selection method, we first select the top-k highest scoring source tasks Sy = {s1,..., 8&) for t. Here, the task selection method can be TASKSHOP or other methods. We then randomly sample n prompted examples from each task, resulting in small training set of kn ex- amples. Table 6 in the appendix shows examples of top-5 tasks selected by TASKSHOP with F=RoE.
Overview. Our method searches over paths from 8 to l via a set of pivot tasks in TASK where each pivot p forms path + p + t, and averages their scores to estimate s + f. It builds upon our previous observation that the strengths of + p and p >> t help estimate the strength of >
5 Experiments and Results
Method. The TASKSHOP method is summarized in Equation 4.1. Given a pivot task p E S (s) for which transfer s - p is already known, we first use an off-the-shelf task selection method F to obtain F(p + t). F' can be any method that only uses a small number of task examples. Then, we find the
5.1 Single-Task Selection
Comparisons. We compare Retrieval-of- Experts (RoE) from Jang et al. (2023) and LLM- similarity in Paranjape et al. (2023). For Retrieval- of -f-exs, we take 100 examples of the source

---

Method
NLI/Entailment
Paraphrase
Commonsense
Sentiment
QA
Semantics
Mean
LLM similarity
54.75
47.01
63.14
65.71
41,96
56.07
56.69
NDCG
Retrieval-of- Experts
66.53
49.19
65.7
78.21
84,46
54.33
64.52
Ours: TASKSHOPLLM
54.12
52.9
67.26
71.38
51.12
56.48
59.69
Ours: TASKSHOPROE
75.14
49.29
79.49
80.53
85.74
54.22
71.54 (1)
LLM similarity
3.31
1.84
6.92
0.56
3.79
0.78
3.67
@egret
Retrieval-of-
4.79
1,38
6.83
0.14
4.26
1.84
4.11
Ours: TASKSHOPLLM
3.31
0.85
4.37
0.22
3.79
0.86
2.73
Ours: TASKSHOPROF
3,51
1.35
3.76
0.04
2.22
1.67
2.66 (U)
Table 2: Results of task selection experiments. We use TASKWEB to evaluate TASKSHOP and two task selection methods that only use target examples : LLM similarity (Paranjape et al., 2023) and RoE (Jang et al., 2023).
TASKSHOP LLM uses F LLM-similarity and TASKSHOP RoE uses F RoE in Equation 4.1. TASKSHOP ROE exhibits the best performance in task selection both in terms of the overall ranking (NDCG) and top-5 precision (Regret@5). Note that a higher score is better for NDCG (above) and lower score is better for Regret@5 (below).
task and 32 examples of the target task and com- pute the similarity between text embeddings of the prompts. For LLM-similarity, we input a prompt text-davinci-003 (Ouyang et al., 2022) as- sign probability scores to whether the two tasks are similar or not. For TASKSHOP, we use RoE and L.M--imillrrity for F Equation 4.1. More details are provided in Section A.I in the appendix. two metries: normalized discounted cumulative gain (NDCG) and Regret@ k, following Poth et al. (2021). We use NDCG to evaluate the overall rank- ing, and Regret@k k to measure the performance drop of the predicted top-k source tasks from the actual top-k source tasks. We evaluate task selec- tion for all tasks in our setup grouped by categories in Table I, and use TASKWEB for the gold labels.
transitive behavior between positive task transfers, these transfer scores would provide additional in- formation about the transferability from the helpful source tasks to the target. Moreover, our method considers the direction of the pairwise transfer un- like the other methods, thereby better accounting for the non-commutativity observed in Section 3.3.
5.2 Multi- Task Selection
Metrics. To evaluate task selection,
We now investigate whether TASKSHOP can also be used to select multiple source tasks that collec- tively improve target task performance.
Comparisons. We use the following baselines. TO-3B has the same architecture as T5-3B but trained on millions of examples spanning 35 dif- ferent tasks (Sanh et al., 2022). T5-3B + most similar is the LM-adapted T5-3B (Lester et al., 2021) trained a handpicked, similar source task from the same category as each target task. T5-3B + all tasks the LM adMaddeeed T5-3B trained with samples from all 22 tasks from TASK WEB except each target task in a leave-one-out setup.
Experimental Setup. While we use target tasks from WASKWEB to use their transfer scores as la- bels, we wish to simulate a scenario in which there are only 32 examples for each target. Therefore we perform our experiments in leave-one-out setup. where for each experiment we assume access to pairwise scores amongst our set of tasks except for the given target task. In this way, we maintain the assumption that only small number of examples of the target task are available during evaluation.
We then train T5-3B models on small training sets sampled from the five highest-scoring source tasks based the following task selection meth- ods: Retrieval-of- from (Jang et al., 2023), LLM-similarity from (Paranjape et al., 2023) and TASKSHOP RoE with F RoE in Equation 4.1. Finally, we consider the case where TASK WEB scores for the target task are available and select the five highest-scoring source tasks for each target. We train T5-3B on samples from these tasks.
Results. Table 2 reports our results. Combin- ing pairwise transfer scores with LLM and RoF improves both NDCG and R5gret@5 compared their base methods, with the best gains from RoE. We hypothesize that the improvement occurs because the pairwise transfer scores capture the transfcrability between each source task and the set tasks textually similar to the target task. Due to
Training Setup. Given target task and a task selection method, we first select the five highest- scoring source tasks ...,.. $5 for We then ran- domly sample 2,000 prompted examples from each

---

Method
ANLI-RI
ANLI-R2 ANLI-R3
CB
COPA Hellasw.
RTF
StoryC.
WiC
Winogr.
WSC
Mean
TO-3B
35.62
33.36
33.10
62,20
75.50
27.30
61.87
85.13
50.88
50.65
66.02
52.88
T5-3B + most similar
44.50
37.42
39.61
79.07
81.42
41.46
72.83
93.73
50.86
52.83
36.54
57.30
T5-3B + all tasks
41.49
35.32
39.61
79.96
82.08
39.73
74.95
91.93
52.93
57.35
44.44
58.16
38.38
35.44
41.24
75.2
83.17
41.86
65.08
94.04
53.22
50.09
44.76
56.59
LLM "iMi-iilaarity
39,91
34.74
38.84
81.65
80,91
40.85
78.2
93.96
51.35
52.26
55.02
58.88
Ours: TASKSHOPROE
42.86
36.15
41.41
84.52
86.08
41.94
76.73
94.04
51.49
53.0
59.4
60.69
Ours: TASKWEB
40.16
36.15
42.15
82.24
85.25
43.73
77.71
92.69
50.75
55.84
62.82
60.86
Table 3: Results of multi-task learning experiments. We perform all evaluations in zero-shot settings, meaning that we do not fit the model parameters to the target task however, we still assume access to small number of labeled examples of the target. We average results over multiple prompts. The first group corresponds to our baselines. the second group corresponds to two existing task selection methods, as well as TASKSHOP without access to TASK EB scores for the target task (but access to TASKWEB scores between other tasks), and the third group uses TASKW scores for the target task to select source tasks. is from Jang al. (2023) and is from Paranjape al. (2023). i has access to TASK WEB scores directly to the target task. All methods below the dotted line use the top-5 scoring source tasks to build multi-task training sets, while the three above utilize different numbers of source tasks.
task and randomly shuffle all examples to create multitask training set. For the T5-3B most similar baseline, we sample 10,000 examples of the similar task in the same category in order to ensure that the size of the training set the same as the size of the multitask training sets in our other experiments. Meanwhile, for the T5-3B + all tasks baseline, we select 21 tasks except the target and use 2,000 ex- amples from each task. We provide more training details in the appendix.
while ensuring that the target task is unseen in our setup. By doing so, we simulate using TASKSHOP and TASK WEB across various categories of target tasks with access only to their examples (n S 32). We perform all evaluations in a zero-shot setting.
Results. Table summarizes the results of our experiments. The middle section details the per- formances of task selection methods that assume no access to pairwise transfer scores to the target. Two out of three methods improve target task per- formance compared to all baselines. Most notably, TASKSHOP outperforms both baselines well as other task selection methods, improving by 14.7% over TO-3B and by 4.3% over our strongest base- line while using small portion of the training set. Finally, we observe that using the top-5 source tasks for each target according to TASK WEB con- sistently improves target performance. Our results support previous observations that using smaller multi-task training sets with more careful task selection strategy can improve target performance (Pruksachatkun et al., 2020; Chan al., 2022).
As it is costly to compute pairwise transfer scores with bigger language models, we use TASK WEB scores from T5-large. This is based observation that models with similar archi- tectures and adaptation methods share more similar transferabilities (Section 3.2). We hypothesize that T5-large can learn the complexities of our source tasks and represent their transferabilities- supported by how both our T5-large transfers and T5-3B expert models in Jang et al. (2023) found CosmosQA and Soc to be great source tasks.
is
Evaluation setup. We use the same set of evalu- ation tasks used by Jang et al. (2023). For ANLI- R R1/R2 which are not included in TASK WEB, we apply the subset of tasks chosen for ANLI-R3 for the upper baseline. Meanwhile, for the Story Cloze task which is not included in TASK due to its lack of training set, use subset of five tasks with the best transfer scores for the upper baseline. For each target task, we perform the evaluation in a leave-one-out setup by removing the target task from TASK WEB along with its scores. This is to maximize the number of available source tasks
5.3 Discussion
The results of our experiments indicate that single- task transfer metrics can help improve multi-task transfers. We perform further experiments sup- port this hypothesis and address three questions.
How many source tasks do need? We inves- tigate whether different numbers of source tasks in the training set affect target task performance. To this end, we train T5-3B on training sets with top-1, 3, 10 and 21 source tasks in addition to five tasks.

---

Method
ANLI-RI
ANLI-R2
ANLI-R3
CB
COPA
Hellasw.
RTE
StoryC
WiC
Winogr.
WSC
Mean
Top-1
40.83
34.53
38.08
75.0
80.08
28.56
70.49
89.68
50.74
52.6
36.54
54.28
Top-3
41.78
36.54
40.86
79,46
86,16
45.54
70.54
89.66
51.32
52.61
54.81
59.03
Top-5
42.86
36.15
41.41
84.52
86.08
41.94
76.73
94.04
51.49
53.0
59.4
60.69
Top-10
40.58
35.17
38.88
75.6
84,92
42.24
78.65
93.99
51.41
52.54
58.97
59.36
Top-21
41.49
35.32
39.61
79.96
82.08
39.73
74.95
91.93
52.93
57.35
44.44
58.16
Table 4: Results of choosing different numbers of source tasks for multi-task learning with TASKSHOP ROB. For each target task, the highest scoring setup is bolded. Results for top-5 are taken from THSKSH in Table 3.
anli
copa
hellaswag
overall
90
80
70
accuracy
75
40
60
40
70
70
30
50
60
65
helpful >> unhelpful replacements
Figure 4: Variations in the zero-shot target performance the top-5 source tasks for each target incrementally replaced by the bottom-5 source tasks according to TASKWEB while maintaining the size of the training set.
Method
ANLI COPA
Hellasw. Mean
Table 5 summarizes the results. A random set of source tasks underperforms the TO-3B baseline, and the bottom-5 tasks from TASKSHOP further ob- serves decreases in 3.4 accuracy points on average. Finally, the bottom-5 tasks based on TASK W EB re- sults in similarly low performances. These results indicate that negative pairwise transfers between source and target tasks impact multi-task learning.
Random
35.25
72.58
29.64
50.51
Bottom-5 w/ TASKSHOP
34.41
55.92
25.01
47.13
Bottom-5 w/ TASKWEB
34.72
52.92
25.37
46.25
Table 5: Results of choosing random and worst sets of tasks according to TASKSHOP and TASK WBB for three example target tasks, as well as the mean over all target tasks. Table 9 in the appendix provides the full results.
What happens if we mix helpful and unhelpful source tasks? While grouping helpful sources improves target performance and vice versa, it is unclear what happens in between. To address this, experiment with different proportions of helpful tasks and measure the target task performance. We repeat this process over four target tasks in our eval- uation setup-ANLI (R3), COPA, HellaSwag and RTE. For each task, we start with the top-5 tasks according to TASK WFB and replace a task with a bottom-5 task until all top-5 tasks are replaced. We perform the same evaluations as Tables 3, 4 and 5. Figure 4 visualizes the results. As each helpful source task is replaced with an unhelpful source task, the target performance decreases across all four tasks. However, there are several instances where such replacement increases performance, can be seen from 0->1 in HellaSwag and 4-+5 in ANLI. These results indicate that while pairwise transferability between the source and target heav- ily impacts target performance during multi-task learning, other factors such as negative interference between the source tasks may also be involved, which is an interesting direction for future work.
Table 4 shows the results. We observe that most target tasks achieve performance improvements from training on to 5 source tasks. Using five source tasks results in the best overall performance and ranks first or second across most targets. Mean- while, using ten source tasks results in a worse overall performance. The performance drops con- siderably when 21 tasks are used. According to our results, most targets only require a careful selection of three to five source tasks except several tasks such Winogrande. Our findings differ from pre- vious work which finds performance to scale with the number of tasks (Sanh al., 2022: Wei al., 2022a; Wang et al., 2022b) because while they add tasks in a targer-agnostic manner, we add helpful source tasks based on the target task.
Do our methods identify both helpful and un- helpful source tasks? We demonstrate that our methods can also identify unhelpful tasks in multi- task settings. To this end, we pick the bottom-5 source tasks for cach target with THSKSHOp and TASK WEB, as well as five random source tasks.

---

6 Conclusion
et al., 2022a, 2023). Recently, Wang al. 2023 report that large language models finetuned on spe- cific instruction datasets perform better on related target tasks, which is closely related to our findings. Future work could extend our approach to setups without clear boundaries between tasks and explore ways to perform target-specific instruction tuning. Considering these limitations, we encourage the NLP community contribute to quantifying the transferabilities between different language tasks.
In this work, we investigate how using prior knowl- edge of task relationships quantified via pairwise task transfer aids selecting helpful source tasks for multi-task NLP. We build TASK W EB, benchmark and repository of pairwise task transfers across dif- ferent tasks, models and adaptation methods in NLP. Based on our analysis of TASK WEB, we pro- pose SHOK our method for selecting helpful source tasks for a new target task. We show that TASKSHOP outperforms existing methods in choos- ing helpful source tasks for different target tasks. Moreover, we use TASKSHOP and TASK WEB to build small multi-task training sets and outperform other methods that use much larger training sets.
Ethics Statement
TASKWEB is based on a set of representative NLP tasks that have widely been used in the NLP community. While this work explores pairwise task transfer and multi-task finetuning using non- harmful datasets, an adversary could potentially misuse approach to build another version of TASK W EB containing harmful tasks and quickly train models specifically for malicious target tasks. Hence we emphasize the importance of monitoring the content of tasks newly added to TASK WEB.
7 Limitations
Our work contains several limitations. First, our set of tasks does not constitute the entirety of NLP tasks. While we use 22 NLP tasks that are repre- sentative enough to cover various types of reason- ing, we do not include long-form tasks (e.g., sum- marization, LFQA) or domain- speccffc tasks (e.g., law, medicine) to facilitate experiments across vari- ous model architectures such as encoder-only mod- els. In order to add entirely new forms of task to TASKWEB, one would have to compute pairwise transfer scores between the new task and other tasks in TASK If the model is known beforchand, this would require ||T|| iterations of fine-tuning with 1,000 examples where T is the set of tasks in TASKWER. On the other hand, if the model is not known beforchand, this would require | [M] x |[T'] iterations where M is the set of models used in TASK WEB.
Acknowledgements
We thank members of the H2Lab and UW NLP for their discussion and constructive feedback. This work was funded in part by the DARPA MCS pro- gram through NIWC Pacific (N66001-19-2-4031,, NSF IIS- IS-2004,60, and gifts from A12. Joongwon Kim is supported by the National Science Foun- dation Graduate Research Fellowship under Grant No. DGE-21 140004 Akari Asai is funded by the IBM PhD Fellowship.
Moreover, our datasets are in English and we do not incorporate multilinguality in our experi- ments. Second, our work focuses models with at most three billion parameters. Our finding may not be directly applicable to models with orders of magnitude more parameters considering factors such as emergence (Wei et al., 2022b), which can be explored in future work. Third, we perform our multi-task finetuning experiments by uniformly sampling 2,000 examples from each source task fol- lowing the style of Wang et al. (2022b). Therefore, different behavior may arise when other sampling strategies are used. Finally, recent work shows the effectiveness of using diverse instruction-output pairs which do not necessarily have clear bound- aries as our tasks do (Ouyang et al., 2022; Wang
References
Alessandro Achille, Michael Lam, Rahul Tewari. Avinash Ravichandran, Subhransu Maji, Charless C. Fowikes, Stefano Soatto, and Pietro Perona. 2019. Task2vec: Task embedding for meta-learning. In 2019 IEFF/CVP International Conference on Com- puter Vision, ICCV 2019, Seoul, Korea (South), Octo- ber 27 - November 2, 2019, pages 6429 6438. IEEE.
Armen Aghajanyan, Anchit Gupta. Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with fonettntnning In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 5799-5811. Association for Computational Linguistics.

---

Alon Albalak, Yi-Lin Tuan, Pegah Jandaghi, Connor Pryor, Luke Yoffe, Deepak Ramachandran, Lise Getoor, Jay Pujara, and William Yang Wang. 2022. FETA: A benchmark for few-sample task transfer in open-domain dialogue. In Proceedings of the 2022 Conference on Empirical Methody in Natural Lan- guage Processing, EMNIP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 10936- 10953, Association for Computational Linguistics.
Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432- 7439. AAAT Press.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large an- notated corpus for learning natural language infer- ence. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, EMNLP 2015, Lisbon, Portugal, September 17-21, 2015, pages 632-642. The Association for Computa- tional Linguistics.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang. Vinh Q Tran, Dara Bahri, Jianmo Ni. Jai Prakash Gupta, Kai Hui, Sehastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi- task scaling for transfer learning. In The Tenth In- lernational Conference on Learning Representations. ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net.
Joaquin Quiñonero Candela, Ido Dagan, Bernardo Magnini, and Florence d' Alché-Buc, editors. 2006. Machine Learning Challenges, Evaluating Predic- tive Uncertainty, Visual Object Classification and Recognizing Textual Entailment, First PASCAL Ma- chine Learning Challenges Workshop, MLCW 2005, Southampton, UK, April 11-13, 2005, Revised Se- lected Papers, volume 3944 of Lecture Notes in Com- puter Science. Springer.
Akari Asai, Mohammadreza Salehi, Matthew E. Pe- ters, and Hannanch Hajishirzi. 2022. ATTEMPT:
parameter-effiiien multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Lan- guage Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates. December 7-11, 2022, pages 6655- 6672. Association for Computational Linguisties.
Daniel M. Cer, Mona T. Diab, Eneko Agirre, Lñigo and Lucia Specia. 2017. Semeval- 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings the 11th International Workshop on Semantic Eval- uation, SemEval @ @ACL 2017, Vancouver Canada, August 3-4, 2017, pages 1-14. Association for Com- putational Linguistics.
Stephen H. Bach, Victor Sanh, Zheng Xin Yong, Al- bert Webson, Colin Raffel, Nihal V. Nayak, Ab- heesht Sharma, Taewoon Kim, M. Saiful Bari, Thibault Févry. Zaid Alyafeai, Manan Dey, An- drea Santilli, Zhiqing Sun, Srulik Ben-David, Can- wen Xu, Gunjan Chhablani, Han Wang, Jason Alan Fries, Maged Saeed AlShaibani, Shanya Sharma, Ur- mish Thakker. Khalid Almubarak, Xiangru Tang. Dragomir R. Radev, Mike Tian-Jian Jiang, and Alexander M. Rush. 2022. Promptsource: An in- tegrated development environment and repository for natural language prompts. In Proceedings of the 60th Annual Meeting of the Association for Com- putational Linguistics, ACL 2022- System Demon- strations, Dublin, Ireland, May 22-27, 2022, pages 90-104. Association for Computational Linguistics.
Jun Shern Chan, Michael Pieler, Jonathan Jao, Jérémy Scheurer, and Ethan Perez. 2022. Few-shot adap- tation works with unpredictable data. CoRR, 0tss/208.000099
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Eric Li, Xuezhi Wang. Mostafa Dehghani, Siddhartha Brahma, Albert Web- son, Shixiang Shane Gu, Zhuyun Dai, Mirac Suz- gun, Xinyun Chen, Aakanksha Chowdhery, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y. Zhao, Yanping Huang, Andrew M. Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Jacob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le, and Jason Wei. 2022. Scaling instruction finetuned language models. CORR, abs/2210.114..
Roy Bar- Bar-Haim. Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising tex- tual entailment challenge.
Christopher Clark, Kenton Lee, Ming-Wei Wei Chang. Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume (Long and Short Papers). pages 2924 2936.993... Associa- tion for Computational Linguistics.
Luisa Bentivogli. Bernardo Magnini, Ido Dagan, Hoa Trang Dang, and Danilo Giampiccolo. 2009. The fifth PASCAL recognizing textual entailment challenge. In Proceedings of the Second Text Analy- sis Conference, TAC 2009. Gaithersburg, Maryland. USA, November 16-17, 2009. NIST.
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The Thirty-Fourih AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference. IAAI 2020, The Tenth AAAI Symposium on Educational
Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmenthank: Investi- gating projection in naturally occurring discourse.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets

---

into natural language inference datasets. CoRR, abs/1809.02922.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord. Peter Clark, and Hannaneh Hajishirzi. 2020. Unifiedqa: Crossing format bound- aries with a single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online Event, 16-20 November 2020, volume EMNLP 2020 of Findings of ACI., pages 1896-1907. Association for Computational Linguistics.
Jesse Dodge, Gabriel Ilharco. Roy Schwartz, Ali Farhadi, Hannanch Hajishirzi, and Noah A. Smith. 2020. Fine-tuning pretrained language models: Weight initializations, data orders, and early stop- ping. CoRR, abs/2002.06305.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Proceedings of the Thirty- Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Arti- ficial Intelligence (IAAI-18), and the 8th AAAI Sym- posium on Educational Advances in Artificial Intel- ligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 5189-5197. AAAT Press.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, /WP@ IJCNLP 2005, Jeju Island, Korea, October 2005, 2005, Asian Federation of Nat- ural Language Processing.
Danilo Giampicoolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Repub- lic, June 28-29, 2007, pages 1-9, Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 3045- 3059. Association for Computational Linguistics.
the ACL-PASCAL@ACE
Andrew S. Gordon, Zornitsa Kozareva, and Melissa Roemmele. 2012. Semeval-2012 task 7: Choice of plausible alternatives: An evaluation of com- monsense causal reasoning. In Proceedings of the 6th International Workshop on Semantic Evaluation, SemEval SemEval@@NAALL NAACL-HLT 2012, Montréal, Canada, June 7-8, 2012, pages 394-398. The Association for Computer Linguisties.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Principles of Knowledge Representation and Rea- soning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012. AAAT Press.
Bill Yuchen Lin, Kangmin Tan, Chris Miller, Beiwen Tian, and Xiang Ren. 2022. Unsupervised cross- task generalization via retrieval augmentation. In NeurIPS.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficien transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfci Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907. 11692.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Nat- 2019,
Shayne Longpre, Le Hou, Tu Vu, Albert Webson, Hyung Won Chung, Yi Tay, Denny Zhou, Quoc V. Le, Barret Zoph, Jason Wei, and Adam Roberts. 2023. The flan collection: Designing data and methods for effective instruction tuning. CoRR, abs/2301. 136888
Andrew I.. Maas, Raymond F. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In The 49th Annual Meeting of the Association for Computational Linguistics: Human Language Tech- nologies, Proceedings of the Conference. 19-24 June, 2011, Portland, Oregon, USA, pages 142-150. The Association for Computer Linguistics.
ural Language Processing, EMNL.P-IJNN Hong Kong, China, November 3-7, 2079, pages 2391- 2401. Association for Computational Linguisties.
Hamish Ivison, Noah A. Smith, Hannaneh Hajishirzi, and Pradeep Dasigi. 2022. Data-efficient finetun- ing using cross-task nearest neighbors. CoRR. abs/2212.00196.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- naneh Hajishirzi. 2022. Metaicl: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Tech- nologies, NAACL 2022, Seattle, WA, United States,
Joel Jang, Seungone Kim, Seonghyeon Ye, Doyoung Kim, Lajamugen Logeswaran, Moontae Lee, Kyung- jue Lee, and Minjoon Seo. 2023. Exploring the bene- fits of training expert language models over instruc- tion tuning. CoRR, abs/2302.03202.

---

July 10-15, 2022, pages 2791-2809. Association for Computational Linguistics.
American Chapter of the Association for Computa-
Human Language Technologies, NAACI-HIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume I (Long and Short Papers), pages 1267- 1273. Association for Computational Linguistics.
tional Linguistics:
Swaroop Mishra, Daniel Khashabi, Chitta Baral, and Hannaneh Hajishirzi. 2022. Cross-task generaliza- tion via natural language crowdsourcing instructions. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics ( (Volume Long Papers), ACI. 2022, Dublin, Treland, May 22-27, 2022, pages 3470-3480 Association for Com- putational Linguisties.
Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event Punna Cana, Dominican Republic, 7-11 November, 2021, pages 10585-10605. Association for Computa- tional Linguistics.
Nasrin Mostafazadeb, Michael Roth, Annie Louis, Nathanael Chambers, and James F. Allen. 2017. Ls- dsem 2017 shared task: The story cloze test. In Proceedings of the 2nd Workshop on Linking Models of Lexical. Sentential and Discourse-level Seman- rics, LSDSem @ EACL 2017, Valencia, Spain, April 3, 2017, pages 46-51, Association for Computational Linguistics.
Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediatc-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020. Online, July 5-10, 2020, pages 5231-5247. Association for Computational Linguisties.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language under- standing. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguistics. ACL 2020, Online, July 5-10, 2020. pages 4885-4901. Association for Computational Linguistics.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll L. Wainwright, Pamela Mishkin, Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welin- der, Paul F. Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instruc- tions with human feedback. In NeurlPS.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. J. Mach. Learn. Res., 21:140::-140:67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018 Know what you don't It know: Unanswerable questions for squad. In Proceedings of the 56th Annual Meet- ing of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 789. Association for Computational Linguistics.
Vishakh Padmakumar, Leonard Lausen, Miguel Balles- teros, Sheng Zha, He He, and George Karypis. 2022. Exploring the role of task transferability in large- scale multi-task learning. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACI. 2022, Seartle, WA, United States, July 10-15, 2022, pages 2542-2552 Association for Computational Linguistics.
Nils Reimers and Iryna Gurevych. 2019. Sentence- bert: Sentence embeddings using siamese bert-networks. In Proceedings of the 2019 Conference on Empiri- cal Methods in Natural Language Processing and the 9th International Joint Conference Natural Language Processing, EMNLP-IJCNL 2019, Hong Kong, China, November 3-7, 2019, pages 3980-3990. Association for Computational Linguistics.
Bo Pang, Lillian Lee, and Shivakumar Vaithyanathan. 2002. Thumbs up? sentiment classification using machine learning techniques. In Proceedings of the 2002 Conference on Empirical Methods in Natural Language Processing, EMNLP 2002, Philadelphia. PA, USA, July 6-7, 2002, pages 79-86.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavat- ula, and Yejin Choi. 2020. Winogrande: An adver- sarial winograd schema challenge at scale. In The Thirty-Fourth AAAI Conference on Artificial Intelli- gence, AAAI 2020, The -ecoot Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8732- 8740. AAAI Press
Bhargavi Paranjape, Scott M. Lundberg, Sameer Singh, Hannaneh Hajishirzi, Luke Zettlemoyer, and Marco Túlio Ribeiro. 2023. ART: automatic multi- step reasoning and tool-use for large language mod- els. CoRR, abs/2303.09014.
Mohammad Taher Pilehvar and José Camacho-Colladda 2019. Wie: the word-in-context dataset for evaluat- ing context-sensiticn meaning representations. In Proceedings of the 2019 Conference of the North
Victor Sanh, Albert Webson, Colin Raffel, Stephen H. Bach, Lintang Sutawika, Zaid Alyafeai, Antoine

---

Chaffin, Arnadd Stiegler, Arun Raja, Manan Dey, M. Saiful Bari, Canwen Xu, Urmish Thakker, Shanya Sharma Sharma, Eliza Szezechla, Taewoon Kim, Gunjan Chhablami, Nihal V. Nayak, Debajyoti Datta, Jonathan Chang, Mike Tian-Jian Jiang, Han Wang. Matteo Manica, Sheng Shen, Zheng Xin Yong. Harshit Pandey, Rachel Bawden, Thomas Wang. Tr- ishala Neeraj, Jos Rozen, Abheesht Sharma, An- drea Santilli, Thibault Févry, Jason Alan Fries, Ryan Techan, Teven Le Scao. Stella Biderman, Leo Gao, Thomas Wolf, and Alexander M. Rush. 2022. Multi- task prompted training enables zero-shot task gener- alization. In The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net.
Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, and Hannanch Hajishirzi. 2023. How far can camels go? exploring the state of instruction tuning on open resources. CORR, abs/2306.00755.
Yizhong Wang, Yeganch Kordi, Swaroop Mishra, Al- isa Liu, Noah A. Smith, Daniel Khashabi, and Han- naneh Hajishirzi. 2022a. Self-instruct: Aligning lan- guage model with self generated instructions. CoRR. abs/2212.10560.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormo- labashi. Yeganch Kordi, Amirreza Mirzaei, Atharva Naik, Arjun Ashok, Arut Selvan Dhanasckaran, An- jana Arunkumar, David Stap, Eshaan Pathak, Gi- annis Karamanolakis, Haizhi Gary Lai, Ishan Puro- hit, Ishani Mondal, Jacob Anderson, Kirby Kuz- nia, Krima Doshi, Kuntal Kumar Pal, Maitreya Pa- tel, Mehrad Moradshahi, Mihir Parmar, Mirali Puro- hit, Neeraj Varshney, Phani Rohitha Kaza, Pulkit Verma, Ravsehaj Singh Puri, Rushang Karia, Savan Doshi, Shailaja Keyur Sampat, Siddhartha Mishra, Sujan Reddy A. Sumanta Patro, Tanay Dixit, and Xudong Shen. 2022b. Super- anarrartnnttalinstuuttoonsss Generalivation via declarative instructions on 1600+ NLP tasks. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Process- ing. EMNLP 2022, Abu Dhabi, United Arab Emirates, December 7-11, 2022, pages 5085-5109 Association for Computational Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social iqa: Common- sense reasoning about social interactions. In Proceed- ings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th Inter- national Joint Conference on Natural Language Pro- cessing, EMNLP-IJCNEL 2019, Hong Kong. China, November 3-7, 2019, pages 4462 4472. Association for Computational Linguistics.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu. Peng Li. Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguisties: Hu- man Language Technologies, NAACI. 2022, Seattle, WA, United States, July 10-15, 2022, pages 3949 3969. Association for Computational Linguisties.
Zhiguo Wang, Wacl Hamza, and Radu Florian. 2017. Bilateral
matching for natural lan- guage sentences. In Proceedings of the Twenty-Sixth International Joint Conference on Artificial Intelli- gence, IJCAI 2017, Melbourne, Australia, August 19-25. 2017, pages 4144 ijeai.org.
Oyvind Tafjord, Matt Gardner, Kevin Lin, and Peter Clark. 2019. Quartz: An open-domain dataset of qualitative relationship questions. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNL---JCNN 2019, Hong Kong, China, Novem- ber 3-7, 2019. pages 5940-5944 5945. Association for Computational Linguistics.
Jason Wei, Maarten Bosma, Vincent Y. Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, An- drew M. Dai, and Quoe V. Le. 2022a. Finetuned language models are zero-shot Icarners. In The Tenth International Conference on Learning Representa- tions, ICLR 2022, Virtual Event, April 25-29, 2022. ReenRevie
Tu Vu. Brian Lester, Noah Constant, Rami Al-Rfou', and Daniel Cer. 2022. Spot: Better frozen model adaptation through soft prompt transfer. In Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics ( (Volume I: Long Pa- pers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5039-5059, Association for Computational Linguistics.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022b. Emer- gent abilities of large language models. Trans. Mach. Learn. Res., 2022.
Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessan- dro Sordoni. Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transferabiliity across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing, EMNLP 2020, Online, November 16-20, 2020, pages 7882-7926. Association for Computational Linguis- tics.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. 2021. Crossfit: A few-shot learning challenge for cross- task generalization in NLP. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 7163-7189, Association for Computa- tional Linguistics.

---

Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine- tunettnnng for transformer-based masked language-models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics ( (Volume 2: Shori Pupers), ACI. 2022, Dublin, Treland, May 22- 27, 2022, pages 1-9. Association for Computational Linguistics.
Amir Zamir, Alexander Sax, William B. Shen, Leonidas J. Guibas, Jitendra Malik, and Silvio Savarese, 2019. Taskonomy: Disentangling task transfer learning. In Proceedings of the Twenty- Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10- 16, 2019, pages 6241-6245. ijeai.org.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. 2019. Hellaswag: Can machine really finish your sentence? Proceedings of the 57th Conference of the Association for Compu- tational Linguistics, ACL 2019, Florence, Italy, July 28- August 2019, Volume I: Long Papers, pages 4791-4800. Association for Computational Linguis- tics.

---

A Appendix
Target
Selected Tasks
A.1 More Experimental Details
ANLI
RTE, CB, SNLI CsmsQA, Soc.IQA
CB
ANLI, CsmsQA, Soc.IQA, WSC, SNLI
Full list of the datasets. Table 5 presents the complete list of the 22 tasks studied in TASK WEB, along with references to the original papers.
COPA
CsmsQA, Soc.QQA, Winogr., Hellasw. PIQA
Hellasw.
PIQA, CsmsQA, Soc.QQ, Winogr, COPA
RTE
ANLI QNLI, Soc.IQA, MRPC. SQuADv2
StoryC.
CsmsQA, COPA, Soc.IQA, Hellasw, Winogr.
Pairwise Task Transfer Metric For source and target evaluation function p. model my tuned on and model Ma-1 tuned from 8 to t,
WIC
PIQA, MRPC, ANLI, Hellasw. Soc.QAA
Winogr
Soc. IeA, CsmsQA, PIQA, COPA, WSC
WSC
Winogr,, ANLI, Soc. IQA, WIC, RTF.
p(m, 1²) p(m))
Table 6: Top-5 source tasks selected using TASKSHOP.
PC(s,
1) x
MEM
p(mss
()( x (p(m)+1)) > p(m))
Target
Selected Tasks
mEM
ANLI
CsmsQA, BoolQ, SNLI, Rot.Tom, RTE
PC refers to the average percentage change of the model performance across all random seeds, and PM refers to the proportion of models that resulted in positive transfer across all random seeds.
CB
ANLI, BoolQ, SNLI, Rot. SciTail
COPA
CsmsQA, Winogr,, SciTail, PIQA, Soc.IQA
Hellasw.
CsmsQA, Soc.QQ,, PIQA. RTE, Rot.Tom
RTR
ANLI, CsmsQA, Winogr,, SQuADv2, Soc.QQ
StoryC.
CsmsQA, SocIQQ, PIQA, Winogr., Tom
Implementation Details of Task Selection. For Sffrieval-of-Experre we use a similar implementa- tion by taking 100 examples of the source task and 32 examples of the target task and computing the similarity between text embeddings of the prompts. We use PromptSource (Bach et al., 2022) extract prompts and Sentence Transformers (Reimers and Gurevych, 2019) obtain text embeddings. For LLM-similarity, we write prompt that con- tains several pairs of tasks not used in our setup, where each pair has 1) an example of each task, and 2) an answer noting whether the two tasks are similar or not. Then, for each source-target pair, we pass the prompt prepended to source and target examples to text-davinci- 000 (Ouyang et al., 2022). We the ratio of the log probabilities of the an- swers "yes" and "no" to assign a score between the source and target tasks.
WiC
QNLI, MRPC, SNLL, RTE, ANLI
Winogr.
SQuADv2. Soc.IQ, CsmsQA, ANLI, Quartz
WSC
ANLI, QNLI, QQP, Soc.QQA, SNLI
Table 7: Top-5 source tasks selected using TASKWER
A.2 More Pair-wise Transfer Results
Full results. Figure 5 displays pairwise transfer scores for all tasks in TASKWEB averaged over training setups. Scores for individual setups are shown in Figure 6 (T5-large finetune), Figure 7 (T5- base finetune), Figure 8 (RoBERTa-base finetune), Figure 9 (GPT2-medium finetune), Figure 10 (T5- base Adapters), Figure 11 (T5-base BitFit) and Figure 12 (T5-small finetune).
Commutativity results. Figure 13 shows the commutativity experiment results.
Transitivity results. Figure 14 shows the exper- imental results of the transitivity analysis for all setups in our experiments.
Multi-Task Finetuning Details. We construct our multi-task training set by randomly select- ing 2,000 examples with prompts from each task. For our T5-3B + all tasks baseline we choose all 21 tasks in TASK WEB apart from the target task, resulting in 42,000 examples. For all other methods Sittrieval-of-Expertte LLM-similarity, TASKSHOPROR, TASK WEB), we choose the five highest-scoring tasks according to each method, resulting in 10,000 examples. Then, we fully fine- tune LM-adapted T5-3B on our training for five epochs, with an Adam Optimizer using a learning rate le-4 and batch sizes ranging from to 16 depending on the maximum length of each dataset.
A.3 More Multi-Task Selection Results
Tasks chosen for the multi-task setup. Tables and 7 list the top-5 (left to right) source tasks chosen for our multi-task setup using TASKSHOP and TASK WEB, respectively.
Bottom-5 and random-5 full results. Table 9 presents the evaluation results for the bottom- source tasks selected with TASkSHop and TASKWED as summarized in Table 5, as well as five random source tasks.

---

datasets used in our experiments
ANII (Nic et al., 2020), BoolQ (Clark et al., 2019), CB (de Marneffe et al., 2019), COPA (Gordon et al., 2012), CosmosQA (Huang et al., 2019), HellaSwag (Zellers et al., 2019), IMDB (Maas et al., 2011), MRPC (Dolan and Brockett, 2005), PIQA (Bisk et al., 2020). QNLI (Demszky et al.. 2018). QQP (Wang et al., 2017). QuaRTz (Tafjord et al., 2019), Rotten Tomatoes (Pang et al.. 2002), RTE (Candela et al., 2006; Bar-Haint et al., 2006; Giampiccolo et al., 2007: Bentivogli et al., 2009), SciTail (Khot et al., 2018). SNLI (Bowman et al., 2015). SocialIQA (Sap et al., 2019). SQuAD2.0 (Rajpurkar ct al., 2018), Story Cloze (Mostafazadeh et al., 2017), STSB (Cer ct al., 2017), WIC (Pilchvar and Camacho-Collados, 2019). Winogrande (Sakaguchi et al., 2020). WSC (Levesque et al., 2012)
Table 8: References for datasets used in experiments.
anli boolq
0.12 0.0133.001000
0.01 -0.0
0.01
0.0
0.0 0:03 0.01 0. 0.06 0.03 0.010.12 0.02 0.14 0 0.04 0.07 0.02 0.01
0.07
0.03-0.02
02-0 04.0
00-0.03-0000...00000000000000000 020
0100000000000000 010000100000000101 0.03 8.0 01 02 0.01 40.0 0..0000000.0000000000 040 02 00011 0.0 0.03 0.01
cb
0.00 0.000.00 04
0-0 0400 000.00000000000 011.0000000000001111100000000001
0.0 - 02.0020000000 02 0.0 -0 01-0 01 0.0 00.01 0 02 40 0.000.00
copa
0.0-0.01.0.0
0. 0.020.01 0.03
0.0
0.01
0.0 - 010 01 0.020 02 0.00 10.00000222200 0.02 0.03 0.01-0.01
cosmqa
0.09
0.03
hellasw
0:01 -0.0 0.02 0.04
0.02
0.05
0.0
0 0.05 0.01 0.02
0.03
0.01
0.02-0.07
imdb
0.020. 00-00000,00000.. 040 05.00000000
0.04
000000000000000
0700 0.03-0 0.-000000 01.000000000000000 0..00000000000
01
0.030.01
0.0
-0.02
mrpc
0.0 a. 02 0.0 0. 0.01-0 02
0.0 4.01
0.01
0.04
0
0.02
0.0
-0
0.011 01 0.03
0.02
05
0.07
0.05
0.02
0.01
source tasks
piqa
0.03 0.02 0.05 0.000.04 04 0.01
0.05
0 0.01
01
0.02 00.0 0 0.09 0.02
0.0 0.03
0.04
0.06
0.03
0.03
0.0
qnli
0.0 0.04.0.000000
qqp
0.030.0
0.06 0.0.0000000000000.. 0.120.00...0000
0.040.44
0.03 0.03 040.03
0.0
0.050.0000000
0.05 0.02.0.2
quartz
0.020.
0 03 0.0 01-0
0300
0240 001
0100
02.0..1
0.0 0.00
0.0
0.0
00.00010.0000
020
0.088.01
0.03 0.00
rtomato
0.01 0.05
0.01
-0.0
0-0000000000 03 0.0 00..4.0.11.000
0.0
-0000.000
00..0.000.0
0.05
9.0 3.01
02
040
03
0.0
0.03
0.030.04-0000
rte
0.04-0.08 2.4-0.02-0.02 0.0 -0.0 0.02 0.0.0.0001011.
0.0 0. 02
0.06
0.0 0.01
0.05
0.080.00.00
0.04
0.02
0.01
scitail
0:01
0.0 0.030.0.222.
0.06
0.02
0.03
0.0 0.01
snli
-0.1
0.02-0.02
0.030.03
0.07
sociqa
0.01 0.04 0.05 0.00 0.09 0.09 0.07 0. 0.08 0.02 0.02 0.15
0.00
0.1
0.02
0.08
0.03
0.04
0.05
squad2
0.010 08 0 0.02
0.00000000000000000 0.06-0
0240 022
02 0.02 0.08 b 03
0.100.144400 10-14.00.10
04
0.0
04
0.440.000
0.06
0.0660000
09
0.05
0.01
0.05 .0.0
stsb
0.01 0.03 0 0.w.0.000000
0. 00-00000000000 020.020003300
0.000 0. 0.01 0.04 0.05 0 0.01 0.05 0.06 04
0.07 0.04
0.04
wic
0.020.
0.05
0. 0.02
0.0.00030000000
winogrd
0.0 08.0080 0.01
0 0.0 0.0 0.0 0.04 0.01 0.12 0.01 0.08 0.02 0.06
0 0.03 0.02 0.04
wsc
0.01 -0.0 0 00003 03 0.0 -0 00-000000000 01-0 000.00000000000000000 0.00 00.000 040 01-0 0 0.000 0000000000000000000000000000000 - 0200 0. - 0.03 0.0 -0. 01
anli
boold
cb copa cosmga hellasw imdb
mroc
piqa qnli
qqp guartz itomato
rte scitail
snli sociga stsb
winogrd WSC
wic
target tasks
Figure 5: Visualization of pairwise transfer between 22 different NLP tasks, averaged over our training setups. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.
Method
ANLI-RI ANLI-R2 ANLI-R3
CB
COPA Hellasw. RTE StoryC WiC Winogr:
WSC
Mean
Random
34.35
35.29
36.12
65.67
72.58
29.64
73.69
55.84
49.53
51.03
51.92
50.51
Bottom-5 w/ TASKSHOP
33.39
34,21
35.63
67.76
$5.92
25.01
62.57
59.42
50.33
50.45
43.69
47.13
Bottom-5 w/ TASK WF
34.33
33.56
36.28
47.02
52.92
25.37
67.2
57.3
50.05
50.1
54.59
46.25
Table Results of choosing random and worst sets of tasks according to TASKSHOP and TESKWEB. Refer to the third row in Table 4 for target task performances with the top-5 source tasks selected by TASKS11OP.

---

anli
0.176.79 of Dib orno ans.0s 00100000.11
boolq
1:1
0.72
077077801 00 6.8 0.1 1018.00 88 0.0011
cb
copa
0000.05
0.0-5 095.000
01001 20-50.0000 0...0....0....00 c0-.0.000--0
cosmqa
12.034...30
82
2.
2.00..0..000 0. 5.00 5.00 o. a. 10000000 or 5.07..n
hellasw
imdb
b.2
mrpc
00 0.000.15
0.000000000000000
6..00....
5.00
01 030.0000.77
8.05 00000000 0.0 0.
source tasks
piqa
0.130.36 80 0.160.16
5:00
0.244
0. 6.01000
qnli
00000
2.150.0000000
0000000000000 0.45
0.00
0.0000 oz
qqp
quartz
0.130.16
800 0.22 0.00
5.00 mm
9.19 03
0. 00 60000 os s.
rtomato
0.12..21
0.00000000000
rte
81 0.08 0.2
0.00
a. 0.22 3.0 0.06 0.0000000000 0.0.0....0
is 0.000. aasis.o
scitail
A1 0.330.19
07 6.77.00
010.0 0.00-. o5a.
snli
0.00002 0..0000.....0....
0.000000000000000000000000000000
0.120 140.0 6.00
sociqa
10 0.340.35
0.0 o.e 0. 2.00 0.0966.
9.00..0
squad2
0...3...11
22.0. 220.00000 b.r 8.00
2.00/0
0.000.00
01
0s0
30..0.0...00000 2..0..0
stsb
000.116.17
05
0.78
0.010.00
136
0.0
wic
140.000.55
8.020 0z 0. 0.0 0000000000
00000000000000000000000000000000
000000000077
6.0
4.8
5.02 5.0
winogrd
10 0.1. ooz 0.1 04 03 0 8:b 0.00 0.000.00 08 03 050007
Wsc
anli
boolg
cb
copa chellaww imdb
mroe
piga qnli
aqp
quartz rtomato
rte scitail
snli sociga stsb
wic winogrd WSC
target tasks
Figure Visualization of pairwise transfer between 22 different NLP tasks for T5-large (Raffel et al., 2020) finetune. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.
anli boolq
0.16
oo d10 0.00 o.aaa 0.1
0.00
3:0
3.424..0
dadd a1d os aa. vold 2.06 2.00 0.060.0100
0.06 oz as 0 2.000.00
cb
0000000
8.03 0.
8.0 0.010
0600006
copa
02.0.1100
001-3.0
0.89 04002.0 0.000.011
04..220.08 02 00000
cosmqa
0..0.6
0.0
01
5.00
0.ce 5.06
0.000.44 ooo.c.o o oaao earaa p.12 0z o.ss 0.00
hellasw
6.00 000008
6.
9.00 0. 0000 0. w ono lo on 0.2 2.000.
imdb
0000020000:3
00700 8:81 0.00..14
0.000
o.05
mrpc
200 0300
03 00 s.0
90 3.000.000
51-0 03- a bas a.000..0
source tasks
piqa
m.o.--omaa a 0.0.0.00000000 0.00 am so
4n.-0 on a..0 0..02 0 aes
qnli
hore
qqp
aww. 0300 030.000 ou 000-0.33..00
0.240.500000 00000000...0000 cr..2 0. 0. oeeee. aaaas.s
quartz
0.1
4.22000000 a1 0.0.010 2o0.00
rtomato
0000.08
aoc 8.0 01 0.00.330
0.000.0...3
84
0040.0
3.00
rte
a.00 a.o o 240.. 0330050.04
5.000000000000
0.0 5.02
scitail
004 0.1 0..00.000 04
os 2.0 0.00
8525.04
1.09
0.00
snli
000007
sociqa
10.0000 00000000100000000
oo4 as
0000.
0.13
0200.7
0030007
8.00
squad2
0.7.120000000000 0.000.1000 a o a.00 2.00.0 000.19
0.12000
9 0:00
stsb
wic
1000 w.w.
a.
winogrd
000 0.1 8:240 02 a a 8.0 5.080.00
a.10..0 9.00 0.00 0000000
WSC
anli
boold
cb
copa costia imdb
mrpe
piga
qnlli
aqp
quartz
rtomato
rte scitall
snli sociga
stsb wic winogrd
WSC
target tasks
Figure 7: Visualization pairwise transfer between 22 different NI.P tasks for T5-hase finetune. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.

---

anli
0.170210:1 0.0000000000
5.16 8.000 0.11 8:1:000000
boolq
0.001154.11 123.885.00
185.080
nes 60. 060 and 5.or
cb
03-003--00
copa
0050 020.03
0.0.4
00.00
0.06-p 00:5
cosmqa
11 1:110000
0.4
s.n 2. 3.1.8.8...00 or 0.03 5.
hellasw
imdb
00:0000000
mrpc
00 0...0...00 0400070111
0,063.00
0.000000000000000000000000000000
0.060.33 03.,0000000000 0.00
0.1 0.00 no0000 0.0 5.00000
source tasks
piqa
0.000
0.00000
0.
6.040.24 06 0.100.0005 0.00
qnli
040.04
00000000.00000000000000
qqp
00 07 0...0.0-0115..0 115511152200 200.00
015 nee orh or
quartz
1000 42.. 0000 ot a.0 amm.m 6.00.000
91:000000000 01 05 8.00
rtomato
800.000.00 0.00000330 6.108
001000
rte
80.0.066.0 10-0.11....0000000000 0.00-01555 2.4 o00 0.
scitail
8.000.000
2.000
9:00
0100000000000 00000.00 0.05 co5.00
snli
9.
1o3. w.w. 0.00000000
0.00
0.05
sociqa
80 9.8 0.33 0280
280220000 220...0..0000........
003
0.000.0 0.08..000 5.00
squad2
0000.32..0
5:11 0....1
2..0
10000.000 2.0.
stsb
010.18
0.1 0000110.11
wic
No w.0...0 0.00
2. 8.2..1 0. 02 oes0.
0.0.0.000000000000 0.0000000000
0.000.4
winogrd
0.00 0 8.00 940 9 0...9
0.299.83 8.3
WSC
anli
boolg
db
copa chellaww imdb
mrpe
piga
qnli
abp
quartz
rtomato
rte scitail
snli sociga
stsb
wic winogrd
WSC
target tasks
Figure 8: Visualization pairwise transfer between 22 different NLP tasks for RoBERTa-base (Liu et al., 2019)
finetune. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.
anli
0:2 04-2.000 2..00..00
0.11
boolq
3:00
2.34
0.000....00 bas ooo 03 a.sa c.0e apo s.00
0000000000 0.00.22
0.13
cb
00000000:00
8.00
a20.00
0300030.00
copa
02 a.ee
0400000
0.0
cosmqa
0e 40.00000000 00..0.00000000
s.8
2.06
0.3
0.0.0.0
0.23
0:20
080.11
hellasw
00 0:0001 0000000000
8.0002
9.06
0.00 om.00.
0:15000015
0.0.9
imdb
0060
044000000000
0000000
8.00 0.00
0.0000
mrpc
8a 0.0--0 00 at-4
5.0022
source tasks
piqa
17 h00 a
0. 9.06
0.05
5.07
w...1.......... 3.0024
qnli
qqp
00.......1 01.00............. 133.120.000 0..00.0.:000000
0.060.20.08
01100006.110 000.000000000 00000 0...0060..00
quartz
08 ooe cava 26
0.
4.23
0.01 5.004 5.0024
rtomato
005000 9.34
00400
000.00
0.00.35
005000
0250.0
0.00.1111
rte
oosaoss 8.000 0.se 2.1o
2.00
scitail
00s o.
o.ro 0.0- a 5.03..0
3.09
0.200.000
2250.1200
5.00.05
snli
000
sociqa
03 0.00 0000000
0.00 an 8.008.0 1o.0 a.0 0.02 060.0.0..7
0040.77
00:012000
6.13
squad2
00
0.040.00 0400 04..0.0...00000 0.900.00000
a.ss 5.20.00000 0.000.555000000 02000
0. 0.11
380133.00..2
stsb
wic
one mee
winogrd
4oa0o. 2 12.. 68..8.0.. a. 2.00 0.0.9.4..2 a 29.0.4 0.1 3.07 0000111
WSC
anli
boold
cb
copa Cosmna
imdb
mrpc
piga
anlli
aqp
quartz
itomato
rte scitall
snli sociga stsb wic winogrd
WSC
target tasks
Figure 9: Visualization of pairwise transfer between different NL.P tasks for GPT-2 medium (Radford al.,
2019) finetune. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.

---

anli
0.00 0.16 mo..mi..o aos
oo..mmm..00 0.000.
0. satss
boolq
000
2010000 m00.100 conamasa 6.00 5.09 1.00..0 3.00 0.0-0.0
a0-5-0 0..0..5 5.0 5.0 6.5r, ns
cb
13007
copa
6.0-5.22
010
cosmqa
0 w.t........om 0:1100000
s.ar 3. o.cea. 2666.77.00 2.000 0.
hellasw
801002000000
imdb
22 8.02 220000000 220.222000 a.0-0.0-0000 8-0.00 22-0000000 5.00
mrpc
0070 020 an 0.0001000 5.00
5.08
0.08
03
0.00
oo soca 00 00.00.0 6.31 5.00
source tasks
piqa
00:00:1000000 2.0000
8.07
0.3 0.000:04 o.
qnli
10:00220000
0.s 100. s. 0.20.4
2.9 8.09 09 0.09
qqp
0
quartz
0000.tt 64 honooooo ses a osa 00000.07777 a.
rtomato
5.09
rte
hape a os 1.0. ans..00 oreat 5.55
000000000.000000
scitail
1o...7...00 w.w.aa.................. 1.0. aona 5 5.120000
9.24 00000.000000
8.465.0
snli
0..4
5.000000.1
sociqa
000.0.0.000000 00500s 0o p.nn a.r..vr a.
0.29
0.00
0.07
0.03 9.0 6.040.00000
squad2
0.1
000050000007
0.
06.0.00
stsb
0000
01
0. 0.000.0
6.25.00
wic
1ep00ssee ao 9.19 www.ass.oooo..o 0.000
9.
9.06 00000000000000000000000000000000
5.00
000
020000
5:00 5.05
winogrd
02240030.16 00 a 029 01-4 4 2... 9. 0.27 w.r.8.8...... 9.00-001 6.0 0.02
Wsc
anli
boolg
db
copa chllasw
imdb mrpe
piga qnli
aqp
quartz rtomato
rte scitail
snli sociga stsb
wic winogrd
WSC
target tasks
Figure 10: Visualization of pairwise transfer between 22 different NLP tasks for TS-base adapters (Houlsby et al.,
2019). We display the actual transfer scores, with positive transfers in blue and negative transfers in red.
anli
0.21-5.000000
5.08
00.00
00.00000000
0900000000
0.2
boolq
-3255- 0200 op0 apd..aa. www.aa......
05-4.5
03-0 20000000000000
cb
0:000:00 000
040000 0.000000000 0.080.00
0.050.0000
copa
0:0-0-0
0043..5000 0.00000
0.12000
0.000
0.14.01
0.0.0.000.0
cosmqa
00
0.26-2.00000 0. ue wor say sare 9. 0.290.00.000
0.11
0. 0.15
0.0.000.000000 9.06 5.94
hellasw
349.00 4.000mm
0.0.0.0
imdb
05.0000.00
000000001
0.000.00.00
0.57
0.1
8.076.00
mrpc
00 0:1 -11--.000010
0 0
2..45.00
9..7
0.1
0.00..0
8.00
022
5.00 or
source tasks
piqa
0.00 no 0.00000
n.071
qnli
qqp
000 0.1 0.0000000000000
1230012000
0.00 0.130.3.377.000
0.240.00
080.12 0.122.00
quartz
020.000 000 26-0066-..0000
20.00000
3:30
5.16
0.360.0000
rtomato
0110.110.00
0..0.0.00000
0.00 003 0.000.03
0.00.:2
8.17
0.00112
rte
012 1220000.00 www.0000000000 0.0
0. wos 0.000.0..0000000
0.063.0000
0.000000000000
0.17
0.0000
scitail
0.3.4... 03.0000000000
0020.120a
0.2
9.00020
020
009.0.0000000 5.000.0
snli
1330100000
0.
sociqa
06000000000 020000
0.1
0.000.00
0
6.079
0.220.080.13
0020.14
0:17
0.30 04
squad2
0.144:0 0.110.110000
0.07
a.1 0.060.00 0.000. 0.000.4...
0.37..0
stsb
wic
0.0.00
8.1 360 09.0
m an 979.1 9...6600000000 08 1:60000
winogrd
1700.09..0o 05-0 0e0 0.1
0
WSC
anli
boold
cb
copa Postia
imdb
mrpe
piga
anll
aqp
quartz
itomato
rte scitall
snli sociga
stsb
wic winogrd
WSC
target tasks
Figure II: Visualization of pairwise transfer between 22 different NI.P tasks for TS-base BitFit (Zaken al., 2022).
We display the actual transfer scores, with positive transfers in blue and negative transfers in red.

---

anli
o.cho sa........ o4-0.11o.0 1..00.000 5.00
boolq
100
0. 02- an. n.o in o..s ano en a a 05 5.75 03 o 00-- 6.555.... st
cb
copa
8.
cosmqa
1000.30..00000 0.0
01 20..000..00000 0...0.0o..c0 a........... a. 020.000000000000000000000 0z 0.03 5.045.0
hellasw
00040.00
imdb
009.000000
so 20002
30.00 4000000 07..0/.......................... wtw..........0........ 00300000000000000000000000000000 00 05000000
mrpc
0.0
8:300 s:: 0. 0.36 36.000 0023 011000000000
00077 0.700.02 0.32
source tasks
piqa
1010.330.0 0.0 020000 9.0
00 0060.66 08 0010.02
qnli
oos 0040040.
qqp
100...oa.co a...nn....nnoo...... a and and a 126.0
quartz
0000.970.00 oa ono oes as o50 the
9.22 0050055
rtomato
0000
rte
01-0 oto oro. asa ao0 ass 23.. 295.050 2..00m0... 4244
005-5073 0.0
scitail
na n.0.0re. coln the ore osn on 010.2 6.0.0...22..
5.000.00
snli
00.0050.000 00000 48.00000000000000000 00000.0000 000.0000..0 2.00000..00000
5.05
sociqa
0o0.a220.co 00000 00000000.. o.oce us or www....................... 00004
0.00 0000:1
squad2
030.330.02 os o.00 os oa caso 223 030002-00000 0e0
00..00
stsb
0.000000.000
sos o..0..
6056.00
wic
10M0069..0 0100010000 o a. 0.219 01100 02001
4. 0.7
5.0 3.00
winogrd
has s 0000000 9. 0202022..2
Wsc
anli boolg
db
copa chellaww
imdb mrpe
piga
qnli
aqp
quartz rtomato
rte scitail snli sociga stsb wic winogrd
WSC
target tasks
Figure 12: Visualization of pairwise transfer between 22 different NLP tasks for T5-small finetune. We display the actual transfer scores, with positive transfers in blue and negative transfers in red.
transfer commutativity
anli
boolq
cb
copa
cosmqa
hellasw
imdb
mrpc
source tasks
piqa
qnli
qqp
quartz
rtomato
scitail
snli
sociga
stsb
wic
winogrd
wsc
anli
boolg
tb
copa
cosmga
hellasw
imdb
mrrc
piga qnll
gap
quartz
itomato
Tte
scitail
snli
sociga
stsb
winogrd
WSC
target tasks
Figure Visualization of commutativity between all tasks in our pairwise transfer setup. The white color indicates that transfers in both directions share the same signs, and the orange color indicates opposite signs.

---

0.95
classification accuracy
0.90
0.85
0.80
0.75
0.00
0.01
0.02
0.03
0.04
0.05
transfer threshold
training setup
tSl-finetune
1Sb-finetune
1Sb-adapter
roberta-fneture
gpt2m-fnnetune
15a-bifitt
overall
Figure 14: Results for Figure 3 (right) but for all setups in TASKWEB, with the probability of identifying positive source -> target transfers the minimum threshold for (source -> pivot, pivot ~> target) transfers increased.


Reviews:
[{"review_id": "MXOpZyIOTB", "reviewer": "Reviewer_jiqJ", "summary": "", "questions": "A. Why does T have to be directional while F does not? It seems like both the text-embedding method and the LLM method are symmetric when acting as F.\n\nB. Line 442: for ANLI-R1/R2, we apply the subset of tasks chosen for ANLI-R3 for the upper baseline. Can you explain this in more detail? Which baseline is the \"upper baseline\"? Besides, further explanations may also needed for the story cloze task.\n\nC. In the multi-task transfer setting (table 3), if it is a zero-shot setting, where do the 32 examples come from? Are they from the training set with labels removed? If so, can it be called zero-shot?", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We would like to thank the reviewer for their thorough evaluation of the paper, and their helpful and insightful comments. More specifically, we appreciate how the reviewer pointed out that 1) we integrate pairwise transfer scores between 22 popular NLP tasks into TaskWeb, 2) we use TaskWeb to investigate properties such as commutativity and transitivity of pairwise transfer, 3) we propose an effective method to approximate transferability to a new target task using its examples, and 4) we demonstrate the effectiveness of our method in both single-task and multi-task settings.\n\nBelow we first provide our answers to the two weakness points proposed by the reviewer.\n\n**1. About the cost of adding a new task to TaskWeb (Reasons to Reject 1)**\n\n> Although the research method is insightful, I can hardly imagine a situation where future research does not fall into the categories of TaskWeb … It is hard to first compute pair-wise transferability of any given set of tasks.\n\nOur findings in this paper demonstrate that it is possible to select helpful source tasks for new target tasks without having to compute pairwise transfer scores. More specifically, refer to the TaskShop row in Table 3 of the paper, which removes access to pairwise transfer scores for the target task. Our experiments indicate that for new target tasks that are sufficiently similar to tasks within TaskWeb, we can select helpful source tasks without prior transferability scores for the new task. One example of such a target task could be a scientific QA task which involves reasoning and question answering skills that are well addressed by TaskWeb. On the other hand, a machine translation task would be less similar to tasks in TaskWeb. That being said, the results of our experiments indicate that computing pairwise transfer scores for a new target task is not needed if the task shares sufficiently similar traits with tasks in TaskWeb.\n\nMeanwhile, we acknowledge that as language model capabilities improve over time, the scope of problems addressed by these models evolves and entirely new tasks are introduced (e.g., long-form question answering). For target tasks that considerably differ from tasks in TaskWeb, we believe it is worthwhile to compute pairwise transfer scores for new tasks that are of interest to the research community. Even if initially computing the pairwise transfer scores takes a non-trivial amount of compute, we would like to point out that the cost can be amortized over subsequent usage of the scores for different purposes. Moreover, since TaskWeb is designed to cover diverse task categories, it becomes easier to add new tasks as more tasks are added. We thank the reviewer for raising this question - we will add these explanations to the discussion and limitation sections during subsequent iterations of our draft.\n\n**2. About a suggested baseline trained on the tasks with the same category (Reasons to Reject 2)**\n\n> It would be better to have a baseline which just uses tasks within the same category as source tasks (i.e., some heuristic baselines).\n\nWe thank the reviewer for the helpful suggestion. In fact, we already have results for a related experiment (which was not included in the submission draft due to space limitation) where we paired each target task with the most similar source task in terms of pairwise transferability and followed the multi-task setup presented in Section 5.2 and Table 3 of our paper. Here, 9 out of 11 target tasks use source tasks from the same task category as suggested by the reviewer, including CosmosQA to HellaSwag, CosmosQA to Story Cloze, SocialIQA to Winogrande and Winogrande to COPA for commonsense and SNLI to ANLI, ANLI to CB and ANLI to RTE for NLI/entailment. Our evaluation results indicate that this setup underperforms the top-5 multitask training sets chosen according to TaskShop or TaskWeb. For clarity, below are the results of our experiment - we will include these results into the final version of the paper if deemed necessary.\n\n| Method           | ANLI-R1 | ANLI-R2 | ANLI-R3 | CB    | COPA  | HellaSw | RTE   | Story Cloze | WiC   | Winogr. | WSC   | Mean  |\n|------------------|---------|---------|---------|-------|-------|---------|-------|-------------|-------|---------|-------|-------|\n| Top-1 Similarity | 40.83   | 34.53   | 38.08   | 79.17 | 88.17 | 42.04   | 74.56 | 93.97       | 50.38 | 52.6    | 36.54 | 57.35 |\n| Top-5 TaskShop   | 42.86   | 36.15   | 41.41   | 84.52 | 86.08 | 41.94   | 76.73 | 94.04       | 51.49 | 53.0    | 59.4  | 60.69 |\n| Top-5 TaskWeb    | 40.16   | 36.15   | 42.15   | 82.24 | 85.25 | 43.73   | 77.71 | 92.69       | 50.75 | 55.84   | 62.82 | 60.86 |\n\n- *For top-1 similarity, we randomly sample 10,000 examples from the source task to set the number of training examples equal to our other multi-task training sets.*\n- *The results for the Top-5 TaskShop and TaskWeb rows are taken from Table 3 in our paper.*\n\nNext we provide our responses to the set of questions by the reviewer, which we will clarify in the next iteration of our draft.\n\n> Q: Why does T have to be directional while F does not? It seems like both the text-embedding method and the LLM method are symmetric when acting as F?\n\nIn our description of TaskShop, T refers to the pairwise transfer scores between a pair of tasks provided in TaskWeb and F refers to the off-the-shelf task selection method. To clarify our vocabulary, directional means that the pairwise transfer relies upon the ordering of the source and target tasks and therefore is asymmetric, and vice versa. Our pairwise transfer scores (T) are directional as they are based on the (source -> target) transfer - refer to our commutativity analyses in line 258 and Figure 13. On the other hand, our off-the-shelf task selection methods (F) are invariant to the (source, target) orderings and are symmetric (e.g., embedding-based method), which means that they are not directional.\n\n> Q: Line 442: for ANLI-R1/R2, we apply the subset of tasks chosen for ANLI-R3 for the upper baseline. Can you explain this in more detail? Which baseline is the \"upper baseline\"? Besides, further explanations may also needed for the story cloze task.\n\nOur upper baseline refers to the last row in Table 3 for the multi-task experiments, which consists of choosing the top-5 source tasks for each target task according to pairwise (source->target) transfer scores in TaskWeb. As discussed in our first response (1. Cost of adding a new task), computing pairwise transfer scores among all tasks is expensive, and in this experiment, we explore the efficacy of estimating transferability in multi-task settings without relying on pairwise transfer scores. Naturally, selecting source tasks based on pairwise transfer scores becomes our upper baseline. As ANLI-R1 and ANLI-R2 are not included in TaskWeb, we approximate the performance for choosing the source tasks using TaskWeb by using the source tasks chosen for ANLI-R3 instead of omitting evaluation results for ANLI-R1 and ANLI-R2.\n\n> Q: In the multi-task transfer setting (table 3), if it is a zero-shot setting, where do the 32 examples come from? Are they from the training set with labels removed? If so, can it be called zero-shot?\n\nWe acknowledge that for our evaluation setup, we assume access to 32 labeled examples of the target task for using task selection methods between the source and target tasks. When we refer to zero-shot, we mean that we do not fine-tune the model parameters to the specific target task - we will make sure to further clarify this in subsequent refinements of the draft. Our reason for performing evaluation on the target task without any further finetuning is to remove confounding factors that may occur from finetuning the multi-task model on the target task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The TaskWeb benchmark shows pair-wise transferability across 22 popular NLP tasks.\n2. The authors summarized useful conclusions from TaskWeb: commutativity does not hold for task transfer but transitivity usually holds.\n3. The authors proposed an effective method for selecting source tasks for an unseen target task. The research method of using both existing pair-wise transfer results and example similarity is insightful and could benefit future research with similar situations.\n4. The authors showed that by using their task selection method, the selected source tasks are more helpful than pure similarity-based selection baselines or simply using all available source tasks in training. This conclusion holds for both single-task transfer or multi-task transfer.", "reasons_to_reject": "1. The scalability of this work. Although the research method is insightful, I can hardly imagine a situation where future research does not fall into the categories of TaskWeb (paraphrasing, NLI, sentiment, commonsense QA, reading comprehension, semantic similarity). It is hard to first compute pair-wise transferability of any given set of tasks.\n2. It would be better to have a baseline which just uses tasks within the same category as source tasks (i.e., some heuristic baselines).", "questions_to_authors": "A. Why does T have to be directional while F does not? It seems like both the text-embedding method and the LLM method are symmetric when acting as F.\n\nB. Line 442: for ANLI-R1/R2, we apply the subset of tasks chosen for ANLI-R3 for the upper baseline. Can you explain this in more detail? Which baseline is the \"upper baseline\"? Besides, further explanations may also needed for the story cloze task.\n\nC. In the multi-task transfer setting (table 3), if it is a zero-shot setting, where do the 32 examples come from? Are they from the training set with labels removed? If so, can it be called zero-shot?", "paper_topic_and_main_contributions": "This paper provides insight on source task selection during transfer learning. Contributions are:\n1. The authors investigated pair-wise transferability across 22 popular NLP tasks into a TaskWeb benchmark.\n2. They proposed to leverage this benchmark to search source tasks for unseen target tasks. They estimate the transferability of a source task to a target task by (1) the available transfer results between a source task and other tasks in TaskWeb (2) the example similarity between other tasks in TaskWeb and the target task.\n3. Through experiments, the authors proved the effectiveness of their source task selection method in both single-task transfer and multi-task transfer settings (the difference is whether the source task is a single one or multiple ones), compared to selection methods that are purely based on example similarity.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "Hf6QprensT", "reviewer": "Reviewer_yFyo", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are excited and thankful to the reviewer for providing us encouraging comments on our submission. More specifically, we appreciate that the reviewer 1) found the idea of quantifying the relationships between different NLP tasks novel and interesting, 2) enjoyed the organization of the paper and 3) considered the experimental design to be comprehensive. As for the statistical significance, we plan to include the associated results in further iterations of the draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The idea of quantifying the relationship between different NLP tasks is novel and interesting.\n2. The paper is well-organized and easy to follow.\n3. The experiment design is comprehensive, demonstrating a solid workload.", "reasons_to_reject": "There have been no tests to determine the statistical significance of the results.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper first constructs a large-scale benchmark for pairwise task transfers for 22NLP tasks. Then, it introduces a method that predicts the transferability from a source task to a target task. The experimental results are based on single-task and multi-task settings, demonstrating the proposed model's effectiveness.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "WEPP6VMA1Y", "reviewer": "Reviewer_5RiK", "summary": "", "questions": "A. When performing multi-task learning, how did you decide the training order? Did you shuffle training data from all tasks or you train with a task first and another?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly thank the reviewer for the positive feedback on our submission. More specifically, we appreciate how the reviewer found our idea to be clear and interesting, the experiments to be comprehensive, our observations regarding pairwise transfer results to be insightful, and our experiments to be empirically rigorous.\n\nRegarding the question:\n\n> Q: When performing multi-task learning, how did you decide the training order? Did you shuffle training data from all tasks or you train with a task first and another?\n\nWe randomly shuffled training data from all tasks that we chose for each multi-task learning experiment. We will make sure to clarify this point in subsequent iterations of our draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper conducts in-depth investigation on the relationship of source and target tasks, and provide insights on task selection for transfer learning with comprehensive empirical results. The idea is clear and interesting. The authors propose a benchmark TaskWeb for pairwise task transfer, which includes pairwise transfer between all pairs of 22 tasks. Experiments are conducted with three model architectures including encoder-based, decoder-based and encoder decoder-based, which is comprehensive. The observations on the commutativity and transitivity are insightful and valuable to the community. Leveraging the observation that transitivity exists, they propose TaskShop for estimating the source task usefulness. The proposed method shows effectiveness over baselines though experiments on a variety of tasks, demonstrating the significance of the methodology. Lastly, the paper is well-written and easy to follow, which is very appreciable. The claims are clearly stated, supported with empirical evidence.", "reasons_to_reject": "Overall, the paper is of high quality and the discussion section answers most of my questions. I have one minor concern and put them in the question section.", "questions_to_authors": "A. When performing multi-task learning, how did you decide the training order? Did you shuffle training data from all tasks or you train with a task first and another?", "paper_topic_and_main_contributions": "This paper focuses on source task selection in transfer learning. Specifically, the paper study whether knowing the pairwise task relationship helps selecting better source tasks. The study comes along with a large scale benchmark and a new method to estimate the usefulness of a source task.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper addresses source task selection for transfer learning. In particular, the authors construct a large-scale benchmark for pairwise task transfers for 22 NLP tasks, and introduce a method that predicts the transferability from a source task to a target task. The experimental results show the effectiveness in single-task and multi-task transfer settings.

Pros / Strengths:
- Research questions and proposed method are sound and well-designed
- Paper is well written
- Experiments are comprehensive and show the effectiveness of the proposed approach
- Observations on the commutativity and transitivity are insightful and potentially valuable to the community

Con / Weaknesses:
- Scalability: The authors acknowledged in their response that a new target task needs to share "sufficiently similar traits" with the tasks in TaskWeb for their method to work. While this is also briefly mentioned in the limitation section, the authors should elaborate more on which steps are necessary (and how computationally intense those steps might be) if a target task is not sufficiently similar to the tasks in TaskWeb
- Simple baselines missings

Action items for improved version of paper:
- Add simple heuristic baselines
- Clearly define limitations (see above)
- Clearly define "zero-shot" setup (having access to labeled examples of a target task is typically not refered to as zero-shot)
- Add statistical significance tests
Decision: Accept-Main