Learning to Predict Task Transferability via Soft Prompt
Authors: Lingyun Feng
Venue: EMNLP 2023 Main (2023)
Keywords: transfer learning
prompt tuning

Abstract: Fine-tuning pretrained language models on helpful intermediate tasks often greatly improves the performance of target tasks. However, how to efficiently find the source tasks that can successfully transfer still remains under-explored. In this work, we propose to learn an affinity scoring function to predict transferability between tasks. Specifically, we conduct prompt tuning and regard soft prompts as task embeddings that summarize task-specific information. Then we randomly sample task pairs to train an affinity scoring function. The goal is to predict the transfer gain (i.e., affinity) between a task pair, by conditioning on their task embeddings. Once the scoring function is trained, given a novel target task, we use it to predict the most transferable source tasks, without a brute-force search for all possible source-target pairs. Experimental results across 50 tasks show that our method efficiently identifies beneficial tasks for transfer learning.

Learning to Predict Task Transferability via Soft Prompt
Lingyun Feng
China Mobile Information Technology Center fenglingyuni techinamobilee com
Abstract
the most helpful well investigated. How effectively predict the transferability between tasks still challenging.
Fine-tuning pretrained language models on helpful intermediate tasks often greatly im- proves the performance of target tasks. How- ever, how to efficiently find the source tasks that successfully transfer still remains under- explored. In this work, we propose to learn an affinity scoring function to predict transfer- ability between tasks. Specifically, duct prompt tuning and regard soft prompts as task embeddings that summarize saec-spec information. Then randomly sample task pairs to train an affinity scoring function. The goal predict the transfer gain (i.e., affin- ity) between a task pair, by conditioning their task embeddings. Once the scoring func- tion is trained, given novel target task, use it to predict the most transferable source tasks, without a brute-force search for all pos- sible source-taie -trget pairs. Experimental results across 50 tasks show that our method efficiently identifies beneficial tasks for transfer learning.
To shed light on the task relationship, this paper, we conduct comprehensive study of the transferability between 50 tasks, covering wide range of text genres and degrees of difficulty. As fine-tuning and deploying separate instance of the entire large model for each downstream task is prohibitively expensive, we conduct prompt tuning (Liu et al., 2021b,a; Lester et al.. 2021) on a diverse set of source tasks. Prompt tuning is a parameter-efficient approach which freezes all parameters of PLMs and merely tunes soft prompts (Lester et al., 2021). As prompts effec- tively simulate the knowledge of large language model (Su al., 2022), we interpret soft prompts as task embeddings following Vu al. (2021) and store task prompts in a prompt pool. Then we randomly sample task prompt pairs from the pool, along with their transfer gain (i.e., the per- formance difference between "with transfer" and "without transfer") to learn an affinity scoring func- tion. Once the scoring function is trained, given novel target task, beneficial source task in the prompt pool. Using the affinity scoring function predict the most help- ful source task costs substantially less than using a brute-force search over all possible source-target pairs. Experimental results show that the proposed affinity scoring function efficiently identifies bene- ficial tasks for transfer learning and obtains larger transfer gain the target task than baseline meth- ods. Moreover, our method effectively predicts task transferability with much fewer supervision demands.
con-
1 Introduction
Fine-tuning pretrained language models (PLMs), such as BERT (Devlin et al., 2018) and RoBERTa (Liu et al., 2019b), achieves remarkable performance various natural language process- ing tasks. In addition, fine-tuning on intermediate source tasks yields further gains (Phang et al., 2018; Vu et al., 2020: Pruksachatkun al., 2020).
it predict the most
While incorporating intermediate stages of knowledge transfer has shown compelling bene- fits on various tasks, choosing inappropriate source tasks results in negative transfer performance on the target tasks (Pruksachatkun et al., 2020; Bingel and Søgaard, 2017; Chang and Lu, 2021). Previous work use data size, task complexity, or cosine simi- larity between tasks as well as domains (Vu et al., 2020; Poth et al., 2021; Vu et al., 2021) to model the relationship between datasets. However, the condi- tions for successful transfer remain unclear (Phang et al., 2018). Given a target task, which source task
Our contributions are as follows:
We propose an affinity scoring function learn to predict task transferability and demon- strate that the affinity scoring function can measure task relationship efficiently and ef-

---

fectively.
3 Methodology
As shown in Figure I, we first construct prompt pool which consists of wide range of source task prompts. Then, randomly sample task pairs from the pool train scoring function that learns affinity scores between tasks. Once the affinity scor- ing function is learned, given a novel target task, we use it to effectively predict the most transferable source task from the pool. Then we initialize the target prompt encoder with the best source prompt and conduct prompt tuning on the target task.
2) We conduct an extensive study of the transfer- ability between 50 NLP tasks. We store small learned task-specific prompts a prompt pool while enabling the reuse of a single frozen pretrained model for all tasks, yielding low computational and storage costs.
3) Experimental results show that the proposed method efficiently identifies beneficial source tasks and predicts task transferability better than existing methods.
3.1 Prompt Pool Construction
Given n source tasks, denoted by S S5.,, we first train a task-specific model for each task in S. Since updating and stor- ing the entire model for each specific task is ex- tremely expensive and the knowledge in large pretrained language model can be elicited with prompts (Liu et al., 2021a; Li and Liang, 2021; Lester et al., 2021; Ding et al., 2021), we conduct prompt tuning, a lightweight alternative to fine- tuning, on each task and store each task prompt in prompt pool. Specifically, we first randomly ini- tialize the prompt encoder and learn a task-specific model for each task in S using Equation (1). We choose the best prompts (according to the valida- tion results) and store them in the prompt pool. Since the task prompts encode task seec-specific knowl- edge which is used to reason about the nature those tasks and their relations (Vu et al., 2021), we interpret the output of the prompt encoder as the task embeddings to construct semantic space of tasks.
2 Background: Prompt Tuning
Soft prompts are trainable continuous embeddings prepended the input. During training, we keep the large pretrained language model frozen and only update the parameters of the prompt. We use soft prompts in our work since discrete prompts need carefully handcrafted design (Gao et al., 2020; Sanh et al., 2021; Bach et al., 2022). Even if signifi- cant effort is invested, discrete prompts are likely to be suboptimal (Zhao et al., 2021). It is difficult and time-consuming finding proper discrete prompts for each task (Liu et al., 2021a; Vu et al., 2021).
Given series of K input tokens, x
[w,,.., wk] }. where w, denotes i-th token and K denotes the input length. We first utilize PLM to embed the tokens, forming a matrix X C RK*H where H is the dimension of the embed- ding space. Following reparametrization meth- ods (Li and Liang, 2021; Liu et al., 2021b) which can lead to more stable optimization (Li and Liang, 2021), we use two-layer feed-forward neural network with parameter 0 as our prompt encoder. Then we use the prompt encoder to embed the ran- domly initialized prompt tokens, forming matrix P E R4xff where J. denotes the length of the prompt. We concatenate it to the embedded input [P: IR (L++))X which then passes through the downstream model as normal while only pa- rameters of the prompt encoder, i.e., 0, are updated. Given the input embedding X and its label y, the model is updated by maximizing the likelihood:
In contrast to full fine-tuning, which updates all transformer parameters and thus requires storing the entire model for each task, we only optimize the prompt parameters and store the prompt en- coder (with ~ 0.1% parameters) for each task in the prompt pool, significantly reducing computing and storage costs.
3.2 Task Affinity Scoring Function
We use an affinity scoring function, denoted by s. to estimate the transfer gains between tasks, so that we can choose the most helpful source prompt from the constructed pool.
max plu XI;
(1)
Let Pl, P2, ..., P,, represent task embeddings in the prompt pool. We define the transfer gain aij (from task i to task j) as follows: First, we conduct prompt tuning with a randomly initialized
Notice that we do not use verbalizers and prompt engineering in our experiments.

---

Prompt Pool
Task A Affinity Scoring
The Most
Transferable Source Prompt
s(
P
P
Pa
s(
Pt
P2
P3
P4
s(
P
P3
s(
P4
Initialize from Pa
PLM
PLM
PLM
PLM
P
P2
Prompt Tuning on the Target Task
Novel Taryet Task
PLM
PLM
P3
P
(a) Prompt Pool Construction
(b) Evaluate on Novel Tasks
Figure 1: Overview of the proposed method. We first conduct prompt tuning on a diverse set of source tasks and store each task prompt the prompt pool. Then, we randomly sample task pairs from the pool train an affinity scoring function to estimate the transfer gains between tasks. When the affinity scoring function is well learned. given a novel target task, we first compute its task embedding, then use the affinity scoring function s to select the best source prompt from the prompt pool, and it to initialize the prompt encoder for the target prompt tuning.
soft prompt to obtain the performance 71 on the task j. Second, we employ the task embedding P; to initialize the prompt for the task j, denoting the performance as The transfer gain Oiy - T1, i..., the performance difference on the task j between "with transfer" and "without transfer". We randomly sample M task pairs < i,j > ((ig) E [1, i * j) from the source pool to train the scoring function s. Labels are their trans- fer gains C:y. The backbone network of the affinity scoring function s is initialized from the pretrained model, language models have demonstrated great power in learning semantic relationship of input representations (Qiu et al., 2020; Zhang et al., 2019). For tasks i, concatenate their task embeddings as [P:: P:] as the input of the scoring function s and feed into the scoring function by =(PP; [[PP;PP) P.)) in order to predict the transfer gain aij.
loss Baevski et al. (2022) to make training less sen- sitive to outliers. Once the affinity scoring function is learned, we use it to effectively predict the transfer gain between tasks.
3.3 Evaluation on Novel Target Tasks
Given novel target task, to find the best source task from the pool to transfer, it is infeasible to con- duct exhaustive search over all possible <source, target> task pairs. In contrast, give each task in
the afore-
mentioned scoring function the prompt pool a task affinity score and take the task with the highest score as the best source task to transfer.
Specifically, given novel target task / E T, we first learn target task embeddings P. using Equa- tion (1). Then we concatenate P. with each task embedding in the prompt pool P. (i nl) [P; Pt). We use the affinity scoring function give each source task task affinity score s(P;: P.]) ci n]). We choose the source task that has the highest prediction score, denoted by p*, as the best source task to transfer. The process is formulated as follows:
The training objective is:
min Σ
Maais ((P):P.))
12
(2)
[u (lu-v)?3 -u- <B
f(u, v)
- 11 1/8 otherwise
p' Pk
where 3 specifies the threshold at which to switch between LI and L2 loss. We choose smooth
where arg max P.)), [1,n] (3)

---

Then conduct prompt tuning on the target task t with the prompt encoder initialized by pt. Formally, given the label y and input embedding of the target task f, the model is updated by maximizing the likelihood of the ground-truth y:
in Equation is be Due the sets for tasks in GLUE (Wang et al., 2018) and Super- GLUE (Wang al., 2019) not publicly available, we evaluate our models on target validation sets. For source and target tasks, the maximum number of samples is set to 50k for the training set and 10k for dev set in consideration of training efficiency.
max p(yl[P";]]
0
4.3 Baselines
where only the parameters of the prompt encoder, i.e., 0, are updated.
We compare with the following baselines:
4
Experiments
Brute Force: We use a brute-force search to identify the best source prompt from the prompt pool to initialize the prompt encoder for the target task.
4.1 Source and Target Tasks
We experiment diverse set of 50 tasks, cov- cring wide range of text genres and degrees of difficulty. We include natural language infer- ence (NLI) tasks (Williams et al., 2017; Dagan et al., 2005; Haim et al., 2006; Giampiccolo et al., 2007: Bentivogli et al., 2009), paraphrase iden- tification (PI) tasks (Dolan and Brockett, 2005), semantic similarity (Cer et al., 2017), linguistic acceptability (Warstadt et al., 2019), text classifica- tion tasks (Socher et al., 2013; Clark et al., 2019) in GLUE (Wang et al., 2018) and SuperGLUE (Wang al., 2019) benchmarks and other classification tasks (McAuley and Leskovee, 2013; Zhang et al., 2015; De Gibert et al., 2018; Saravia et al., 2018). We also include question answering (QA) tasks (Ra- jpurkar et al.. 2016, 2018; Saha et al., 2018), com- monsense reasoning (Bhagavatula et al., 2019), se- quence labeling tasks (Pradhan et al., 2012) such entity recognition (Sang and De Meulder, 2003; Carreras and Mårquez) and chunking (Sang and Buchholz, 2000). We divide them into 44 source tasks and target tasks. Detailed statisties about the tasks are presented in Appendix A.
Random Selection: We randomly choose source task prompts from the prompt pool to initialize the prompt encoder for target tasks.
We also compare with different similarity mea- surements for task embeddings estimate the re- lationship between tasks. Let p Pt denote the source and target task embeddings respectively, p/ denote the respective prompt tokens, L de- notes the length of the prompt, we measure task similarity follows:
D mah Manhattan distance of the average representations between source and target task embeddings, i.e., D (P= Pt)
++ E E
Dmah: tan and
Per-token average Manhat- distance between the source task embeddings, i.e., Dman PMx pt)
target
22 Σ. m -55
Deue (Su et al., 2021): Euclidean dis- tance of the average representations between source and target task embeddings, i.c.. Dene [P pt)
4.2 Implementation Details
We experiment with RoBERTa-large (Liu et al.. 2019b) as the pretrained model and prompt length to be 50. The prompt encoder is a two- layer feed-forward network and the hidden size is 200. The batch size is set to be 16 and the learn- ing rate is tuned in { 1ee-4.e--4] We use Adam optimizer (Kingma and Ba, 2014), with a linear warmup for the first 6% of steps. For the affinity scoring function s, also model and the learning rate is tuned in (1e-5,2e- 5.3e-5). The task pool has 44 source tasks in total and the number task pairs for affinity scoring function learning is 400. The hyperparameter /
E E,
Deue (Su et al., 2021): Per-token average Fu- clidean distance between the source and tar- get task embeddings, i.e., Deue (PM, , Pl)
1+ E, 1P]1
RoBERTa-large
Dens (Vu et al.. 2021): Cosine similarity of the average representations between source and target task embeddings, i.e., Deos (Ps, pt) of + Σ,e +f E

---

Deos (Vu al., 2021): Per-token average co- sine similarity between the source and tar- get task embeddings, i.e., Deos Pt) =
that using Manhattan distance and Euclidean dis- tance for task similarity compulation always have the same transfer gain on the target task, as these methods tend to predict the same source task to transfer. However, these heuristic methods can not measure the task similarity well, and even degrade the performance on the target tasks (e.g.. -2.35% average for maah). Using cosine similarity measure task relationship performs better but still achieves sub-optimal performance on the target task.
pept
+ E
We use the aforementioned different methods to measure the similarity between task embeddings. The source prompt whose associated task embed- ding has the highest similarity to the target embed- ding is chosen as the best source prompt. Then we use the best source prompt to initialize the target prompt encoder and conduct prompt tuning on the target task. For fair comparisons, experiment set- tings except the source prompt choice set to be the same.
Compared with other similarity measurements. the affinity scoring function measures the task trans- ferability more effectively and achieves the highest average transfer gains on the target task. im- proves the performance on the target task by 3.78% on average, close to the upper bound results with brute-force search which is 4.29%.
4.4 Results
We compare different methods of predicting the most beneficial source task given a novel target task and report their performance on each target task in Table 1. We first conduct a brute force search with all source and target task combinations. Experimental results demonstrate that the choice of intermediate source tasks can heavily affect target task performance e.g., improve the performance of MRPC target task by 8.83% and degrade the performance on the SST-2 target task by almost 10%. On average, a brute-force search for the best source task has 4.29% accuracy improvements over no transfer baseline.
Besides, we also evaluate the correlation be- tween task transferability and task similarity mea- sured by the aforementioned methods. Given a target task, we rank all the source tasks in the prompt pool by different similarity measurements. The ranking is evaluated using three metrics: Pear- son correlation, Spearman correlation, and Re- gret@k (Renggli et al., 2020) which measures the relative performance difference between the Top-k selected source tasks and the optimal one following Poth et al. (2021). Results are shown in Table 2, using Manhattan distance and Euclidean distance achieves similar but sub-optimal performance. Us- ing per-token average cosine similarity between the source and target task embeddings performs better than calculating the cosine similarity of the average representations to measure task similarity. However, none of them can effectively learn the task relationship. The proposed affinity scoring function correlates better with task transferability and provides higher transfer gain on the target task than existing methods. It also demonstrates that the soft prompt effectively encodes task-specific char- acteristics and serves good means to explore task transferability.
In general, tasks with fewer training samples benefit the most from transfer learning, c.g., 8.28% accuracy improvements on CoNLL2004 and 8.83% accuracy improvements on MRPC task. It shows the importance and necessity of task transferability prediction. Transferring from useful intermediate source tasks can provide significant gain on the target task.
We also observe that positive transfer can occur when the source task has a small size or different task type, e.g., question answering tasks such as SQuAD produce high transfer gain semantic similarity task QQP, CoNLL2000 task which has a small amount of data improves the performance on MRPC task by a large margin. It indicates that tra- ditional methods such as using data size or domain similarity fail to estimate task affinity precisely.
4.5 Ablation Studies
Effect of 3. We study the effect of 3 in Equa- tion (2) which controls the transition from squared loss an LI loss. We vary 8 (0.5, 1, 1,5, 2, 2.5) and report the Spearman corre- lation between the prediction the affinity scoring function and the ground-trun -ttnt transfer gains. As
When adopting different transferability metrics to choose source tasks, experimental results show
'The detailed results of all source and target combinations and transfer gains on each target task with different source tasks in the pool are presented in Appendix B

---

SST-2
QQP
MRPC IMDB
SciTail
CoNLL22004
MEAN(A)
Upper bound results of task transfer with brute-force search
Brute Force
96.22
86.56
90.69
94.78
97.09
95.31
4.29
Baseline results without transfer or using random source task for transfer
No Transfer
95.07
83.94
81.86
93.90
93.10
87.03
Random Selection
84.29
84.79
79.90
93.68
95.78
84.78
-1.95
Use transferability meirics choose source tasks
Dman
94.15
83.35
78.19
93.76
85.97
85.37
-2.35
Denc (Su et al., 2021)
94.15
83.35
78.19
93.76
89.80
85.37
-1.71
Dees (Vu et al., 2021)
96.22
85.83
78.19
94.58
94.87
95.31
1.68
Dian
94.15
83.35
78.19
93.76
89.80
85.37
-1.71
Deue (Su et al., 2021)
94.15
83.35
78.19
93.76
89.80
85.37
-1.71
Deom (Vu et al., 2021)
95,41
85.58
78.19
94.58
96.70
95.31
1.81
Affinity Scoring Function
95.53
86.25
89.46
94.48
97.09
94.75
3.78
Table I: Accuracy on the target tasks. All values are scaled by 100. "No Transfer" shows the baseline performance when training on the target task with randomly initialized prompt without transfer. We also report the average absolute transfer gain in the rightmost column of the table. A positive delta indicates successful transfer.
S. (
(f)
R@1 1 (L)
R@3 (L)
0.7
Dman
0.45
0.39
1.55
1.17
0.6
Deue
0.45
0.40
1.40
1.17
Spearman
0.5
Deas
0.26
0.22
0.61
0.61
0.4
Dman
0.33
0.29
1.40
1.17
0.3
Deuc
0.33
0.30
1.40
1.17
0.2
Dess
0.39
0.51
0.1
Affinity Scoring
Function 0.55
0.45
0.12
0.06
0.5
1.5 B
2.5
Table 2: Evaluation of intermediate source task rank- ings produced by different methods. S. and P. are short for Spearman correlation and Pearson correlation, re- spectively. R@k is short for Regret@k, For Spearman and Pearson correlation, higher is better; for Regret@k, Jower better.
Figure 2: Effect of 8 on affinity scoring function learn- ing. The y-coordinate denotes the Spearman correlation between the prediction of the affinity scoring function and the ground-truth transfer gains.
baselines: Deos and Deos in Vu et al. (2021).
shown in Fig 2, a larger f improves the affinity scoring function, while too large values tend to harm results. The best performance is achieved when 3
Results are shown in Figure 3. Though using cosine similarity as task similarity measurement improves the performance of the model on the tar- get task, the affinity scoring function consistently performs better than them by large margin. Even when the number of task pairs to train the affin- ity scoring function 100, which is much fewer than the total task pairs (44 x 44 1936 task pair combinations in total), the affinity scoring function still outperforms the baseline methods by 1.54% on average for best of Top-1 and 1.75% for best of Top-3 setting, As the number of the training task pairs increases, the affinity scoring function performs better. When there are only 400 training task pairs, the affinity scoring function achieves 4.01 % gain in accuracy with the best of Top-3 source prompt while the oracle selection of source
4.6
Analysis
4.6.1
Efficiency
To demonstrate the efficiency of the proposed affin- ity scoring function, we vary the number of selected task pairs M E {100, 200, 300, 300,400) 400} and use them to train the affinity scoring function respectively. Then use the affinity scoring function trained with the respective amount of supervision signal to access tasks in the prompt pool and select best of Top-k (k E (1,3)) source prompt initialize the target prompt encoder. We compare the proposed affinity scoring function with the most competitive

---

5.0
50
4,5
4.0
Transfer Gain (%)
40
Gain
3.5
3.0
30
2.5
20
2.0
10
100
200
300
400
Affinity Sooring Function
Dess
Dess
2000
5000
10000
20000
Num
(a) Best of Top-1
Figure 4: Absolute transfer gain on the target task with different target datasize.
5.0
Transfer Gain (%)
L
L
L
4.6.3 Low-Resource Setting
To investigate how the affinity scoring function per- forms when the target task in the low-resource scenario, we take SST-2 task as the target task and subsample the full datasets to obtain small sets of size {2000, 5000, 10000, 20000]. We use the pro- posed affinity scoring function to select the most beneficial source task, i.e., MNLI task, initialize the target prompt encoder and conduct prompt tun- ing on SST-2 in the settings of different data sizes. We report the absolute transfer gain on SST-2 tar- get task for each data size Figure 4. We observe that the proposed affinity scoring function yields larger transfer learning improvement when the tar- get data is in low-data regime. Though the model achieves low accuracy for data-constrained target tasks, the performance is significantly improved with the help of transfer learning (38.99% with 2000 training samples). It also echoes that effec- tive task transferability prediction greatly reduces the supervision cost on the target task, which can be particularly valuable when there is insufficient data to train or finetune the model.
100
200
300
400
Affinity Scoring Function
Dos
Des
(b) Best Top-3
Figure 3: Average absolute transfer gain the target tasks with the Best Top-I and Top-3 source prompt chosen by different methods.
prompts (exhaustively brute force fine-tuning on all possible source-target pairs) is 4.29%. It indicates that with randomly selected few task pairs from the prompt pool, the proposed affinity scoring function is able effectively predict the task transferability, rather than enumerate every possible task combi- nation pair. Given a novel target task, the affinity scoring function efficiently predicts the most ben- eficial source tasks with much fewer supervision demands.
4.6.2 Generalization
To investigate the generalization of the affinity scor- ing function on new task types 2, we remove the sequence labeling tasks (including named entity recognition, word segmentation, chunking, and part of speech) from the pool, using the remaining tasks train the affinity scoring function and test its pre- diction performance on the new task CoNLL2004 (named entity recognition task). Experimental re- sults show that the affinity scoring function still able to select beneficial source tasks from the pool and achieve positive transfer gain on the target task: the largest transfer gain on the CoNLL2004 task is 5.71% in accuracy. It suggests that the affinity scoring function is generalizable different task types.
4.6.4 Intermediate- Task Full Fine-Tuning Setting
The task relationship predicted by the affinity scor- ing function is helpful only for prompt transfer learning but also for intermediate-tast full fine- tuning setting. To demonstrate this, we first ran- domly select 20 source tasks from the pool and test their transfer performance on the target in-pool task (e.g., RTE task) and out-of-poot task (e.g., MRPC task) respectively with different fine-tuning methods. We find that the transfer gain obtained by prompt tuning is positively correlated with the transfer gains achieved by full fine-tuning, the Spearman correlation 0.13 for MRPC task and 0.54 for RTE task. Furthermore, we choose the most transferable source task predicted by the affin-
"Detilled task division by type is presented in Appendix

---

Method
SST-2
MRPC
QQP
SciTail
IMDB
CoNLL22004
No Tranfer
96.10
88.97
88.43
97.32
94.96
95.57
Intermediate Fine-Tuning
96.22
91.91
88.76
97.39
95.06
96.20
A
+0.12
+2.94
+0.33
+0.07
+0.10
+0.63
Table Accuracy on the target tasks with vanilla fine-tuning and intermediate fine-tuning. All values are scaled by 100. We also calculate the absolute transfer gain obtained by transfer learning these target tasks.
ity scoring function and conduct full fine-tuning on the six target tasks respectively, following sequen- tial fine-tuning setup in Phang et al. (2018), i.e.. first fine-tune on the chosen intermediate source task, then fine-tune the model on the specific tar- get task. Table shows the absolute transfer gain on the target tasks with intermediate source task fine-tuning. We find that the source tasks selected by the affinity scoring function achieve consistent gains on the target tasks in full fine-tume tie-tunnng setting. The transfer gain is more obvious for low-resource target tasks, c.g., it achieves 2.94% gain in accuracy on the MRPC task. It indicates that the proposed affinity scoring function is capable of choosing beneficial source tasks for the given target task not only in the parameter-efficient setting but also in full model fine-tuning.
source tasks given the target task. Compared with them, we adopt an affinity scoring function which is more effective and efficient to predict the task transfer gain. Besides, the proposed method is not only helpful for prompt transfer learning, but also can be generalized to intermediate- ttsk transfer well.
Prompt tuning is a parameter-efficient method that only tunes "soft prompts" with frozen lan- guage models. In contrast to discrete prompts with carefully handcrafted design (Brown et al.. 2020; Schick and Schütze, 2020; Jiang et al., 2020; Sanh et al., 2021; Bach et al., 2022), soft prompts are trainable continuous embeddings the original sequence of input word embeddings which can be learned through back-propagation (Liu et al., 2021b; Wang et al., 2021; Lester et al., 2021; Schick and Schütze, 2020; Wu and Shi, 2022; Asai et al., 2022a). Lester et al. (2021) learns task- specific soft prompts and shows that prompt tuning can be comparable to fine-tuning when the model exceeds billions of parameters. Gu al. (2021) explore the effectiveness of prompt pre-training. Vu et al. (2021) cast each task into a unified text- to-text format and propose a prompt-based trans- fer learning approach with T5 model. Wang et al. (2022) learns to dynamically prompt pre-trained model to learn tasks sequentially for computer vi- sion tasks. Liu et al. (2021a) a) adapt prefix- -uning (Li and Liang. 2021) for natural language understand- ing. They apply continuous prompts for every layer of pretrained models, showing that prompt tun- ing can be comparable to fine-tuning universally across scales and tasks. Asai et al. (2022b) pro- poses parameter-efficient multi-task tuning via at- tentional mixtures of soft prompts, which requires high-resource source datasets and multiple target tasks. By contrast, in our work, the target can be from one or more tasks and the source data can be less sufficient. Besides, the proposed method is helpful not only for prompt transfer learning, but also for intermediate task transfer.
5 Related work
Transfer between tasks has been investigated in many research arcas, such NLP (Poth et al., 2021; Bingel and Søgaard, 2017; Pruksachatkun al., 2020; Vu al., 2020; Liu et al., 2019a; Li et al., 2022) and computer vision (Zamir et al., 2018; Achille et al., 2019; Wang et al., 2022). Li et al. (2022) propose prompt-basee based method in a transferable setting for text generation tasks while we focus on natural language understanding (NLU) tasks. Phang et al. (2018) explore whether fine- tuning intermediate source tasks further improve the performance on the target tasks for text classification. Talmor and Berant (2019) con- duct an empirical investigation of generalization and transfer in reading comprehension. Vu et al. (2020) compute a task embedding based on the model's gradients with respect to the task-specific loss while Poth et al. (2021) focus on adapter- based intermediate fine-tuning. To investigate task transferability in more parameter-efficient way, Vu et al. (2021) propose a prompt-based transfer learning approach and cast task prompts as task em- beddings which further reduce the computational costs. They use cosine similarity to select the best

---

6 Conclusions
2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 0655-6672, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
In this paper, we conduct an extensive study of the transferability between 50 tasks and propose an affinity scoring function to predict transferability between tasks. We interpret soft prompts task embeddings, and store task prompts in a prompt pool. The affinity scoring function is trained with task pairs sampled from the pool to Iearn to esti- mate task affinity. Then given a novel target task, we use the affinity scoring function to predict help- ful source tasks from the pool. Experimental results show that the proposed method outperforms the baseline methods on task affinity prediction. More- over, our method effectively predicts task transfer- ability with much fewer supervision demands.
Akari Asai, Mohammadreza Salehi, Matthew E Pe- ters, and Hannaneh Hajishirzi. 2022b. Attempt: Parameter-efficien multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022
Conference on Empirical Methods in Natural Lan- guage Processing, pages 6655-6672.
Stephen H Bach, Victor Sanh, Zheng-Xin Sing--iin Yong, Al- bert Webson, Colin Raffel, Nihal V Nayak, Abheesht Sharma, Taewoon Kim, M Saiful Bari, Thibault Fevry, et al. 2022. Promptsource: An integrated development environment and repository for natural language prompts. arXiv preprint arXiv:2202.01278
Alexei Baevski, Wei-Ning Hsu, Qiantong Xu, Arun Babu, Jiatao Gu, and Michael Auli. 2022. Data2vec: A general framework for self-supervised learning in speech, vision and language. arXiv preprint 200iv:2202.03555.
Limitations
Given a novel target task, we predict the most ben- eficial source task and show that fine-tuning on a helpful intermediate source task can greatly im- prove the performance on the target task. In this paper, we use source task prompt initialize the prompt encoder for the target task. We con- duct single intermediate-task training to study the affinity between intermediate source tasks and tar- get tasks. It is possible that the performance on the target task can be further improved by properly combining multiple source tasks. We will explore mixing up prompts from multiple source tasks for intermediate-task transfer learning for future work.
Luisa Bentivogli, Peter Clark, Ido Dagan, and Danilo Giampiccolo. 2009. The fifth pascal recognizing textual entailment challenge. In TAC.
Chandra Bhagavatula, Ronan Le Bras, Chaitanya Malaviya, Keisuke Sakaguchi, Ari Holtzman, Han- nah Rashkin, Doug Downey, Scott Wen-tau Yih, and Yejin Choi. 2019. Abductive commonsense reason- ing. arXiv preprint arXiv:1908.05739
Joachim Bingel and Anders Søgaard. 2017. Identi- fying beneficial task relations for multi-task learn- ing in deep neural networks. arXiv preprint 1rXiv:1702.08300
Samuel R Bowman, Gabor Angeli, Christopher Potts. and Christopher Manning. 2015, A large annotated corpus for learning natural language inference. arXiv preprint arXiv: 1508.0536..
Acknowledgements
This work is supported by Big Data Key Soft- ware System Research and Development Project (R23100LX)..
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33: 1877-1999..
References
Lasha Abzianidze and Johan Bos. 2017. wards universal semantic tagging. arXiv preprint (tt::1770.1088...
To-
Xavier Carreras and Lluís Mårquez. Introduction to the CoNLL-2004 shared task: Semantic role labeling. Proceedings of the Eighth Conference on Computa- tional Natural Language Learning (CoNLL-2004) at HLT-NAACL 2004, pages 89-97.
Alessandro Achille, Michael Lam, Rahul Tewari, Avinash Ravichandran, Subhransu Maji, Charless C Fowlkes, Stefano Soatto, and Pietro Perona. 2019. Task2vec: Task embedding for meta-learning. In Proceedings of the IEEE/CVF International Confer- ence on Computer Vision, pages 6430-6439.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez- Gazpio, and Lucia Specia. 2017. Semeval-2017 task 1: Semantic textual similarity-muttinngaal and cross-lingual focused evaluation. arXiv preprint arXiv:1708.00055.
Akari Asai, Mohammadreza Salehi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022a. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the
Ting- Yun Chang and Chi-Jen Lu. 2021. Rethinking why intermediate-task line-tuning works. arXiv preprint arXiv:2108.11696

---

Ankush Chatterjee, Kedhar Nath Narahari, Meghana Joshi, and Puneet Agrawal. 2019. Semeval-2019 task 3: Emocontext contextual emotion detection in text. In Proceedings of the 13th international workshop on semantic evaluation, pages 39-48.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2020. Making pre-trained language models better few-shot learners. arXiv preprint arXiv:2012. 20722333......3.
Danilo Giampiccolo. Bernardo Magnini. Ido Dagan, and William B Dolan. 2007. The third pascal recognizing textual entailment challenge. In Proceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, pages 1-9.
Christopher Clark, Kenton Lee, Ming-W Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905. 10044.
Andrew Gordon, Zomitsa Kozareva, and Melissa Roem- mele. 2012. Semeval- 2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In SEM 2012: The First Joint Conference on Lexical and Computational Semantics- Volume 1: Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixh International Workshop on Semantic Evalua- lion (SemEval 2012), pages 394-398.
Arman Cohan, Waleed Ammar, Madeleine Van Zuylen, and Field Cady, 2019. Structural scaffolds for cita- tion intent classification in scientific publications. In NAACI.
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2005. The pascal recognising textual entailment chal- lenge. Machine Learning Challenges Workshop, pages 177-190. Springer.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang, 2021. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint arXiv:2109.04332
Thomas Davidson, Dana Warmsley, Michael Macy, and Ingmar Weber. 2017. Automated hate speech de- tection and the problem of offensive language. In Proceedings of the International AAAI Conference on Web and Social Media, pages 512-515.
R Bar Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo. Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising textual entail- ment challenge. In Proceedings of the Second PAS- CAL Challenges Workshop on Recognising Textual Entailment, volume 7.
Ona Gibert, Naiara Perez, Aitor Garcia-Pablos, and Montse Cuadros. 2018. Hate speech dataset from white supremacy forum. arXiv preprint 1XXiv:880904444
Charles T Hemphill, John Godfrey, and George R Doddington. 1990. The atis spoken language sys- tems pilot corpus. In Speech and Natural Language: Proceedings of a Workshop Held at Hidden Valley, Pennsylvania, June 24-27, 1990.
Marie-Catherine De Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Inves- tigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pages 107-124.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos qa: Machine reading com- prehension with contextual commonsense reasoning. arXiv preprint arXiv:1909.000
Leon Derczynski, Eric Nichols, Marieke van Erp, and Nut Limsopatham. 2017. Results of the w shared task on novel and emerging entity recognition. In Proceedings of the 3rd Workshop on Noisy User- generated Text, pages 140-147.
Zhengbao Jiang, Frank F Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguisties, $3422-438.
Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprini 1xXiv:1810.04000
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018 Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Pu- pers). pages 252-262.
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu. Hai-Tao Zheng, and Maosong Sun. 2021. Openprompt: An open-source framework for prompt-learning. arXiv prepriat arXiv:2111.07990
Xiaowen Ding, Bing Liu, and Philip S Yu. 2008. A holistic axiicon-based approach to opinion mining. In Proceedings of the 2008 international conference on web search and data mining, pages 231-240.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In Thirty-Second AAAI Confer- ence on Artificial Intelligence.
Bill Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Third International Workshop on Paraphrasing (IWP2005).
Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint 4rXiv:1412.6900

---

Jens Lehmann, Robert Isele, Max Jakob, Anja Jentzsch, Dimitris Kontokostas, Pablo N Mendes, Sebastian Hellmann, Mohamed Morsey, Patrick Van Kleef, Sören Auer, al. 2015. Dbpedia-a large-scale, mul- tilingual knowledge base extracted from wikipedia. Semantic web, 6(2):167-195.
Maria Pontiki, Dimitrios Galanis, Haris Papageorgiou. lon Androutsopoulos, Suresh Manandhar, Moham- mad Al-Smadi, Mahmoud Al-Ayyoub, Yanyan Zhao, Bing Qin, Orphée De Clereq, et al. 2016. Semeval- 2016 task 5: Aspect based sentiment analysis. In In- ternational workshop on semantic evaluation, pages 19-30.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-effiient prompt tuning. arXiv preprini urXiv:2104.08662
Clifton Poth, Jonas Pfeiffer, Andreas Rücklé, and Iryna Gurevych. 2021. What to pre-train on? ef- ficient intermediate task selection. arXiv preprint arXiv:2104.08244
Junyi Li, Tianyi Tang, Jian- Nie, Ji-Rong Wen, and Wayne Xin Zhao. 2022. Learning to trans- fer prompts for text generation. arXiv preprint arXiv:2205.01543.
Sameer Pradhan, Alessandro Moschitti, Nianwen Xue, Olga Uryupina, and Yuchen Zhang. 2012. Conll- 2012 shared task: Modeling multilingual unrestricted coreference in ontonotes. In Joint Conference on EMNIP and CoNNI-Shhrred Task, pages 1-40.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wixix:2101000111
Nelson F Liu, Matt Gardner, Yonatan Belinkov, Matthew E Peters, and Noah A Smith. 2019a. Lin- guistic knowledge and transferability of contextual representations. arXiv preprint arXiv: 08855.
Yada Pruksachatkun. Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang. Clara Vania, Katharina Kann, and Samuel R Bow- man. 2020. Intermediate-tast transfer learning with pretrained models for natural language understand- ing: When and why does work? arXiv preprint arXiv:2005.000 2062.0068..
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang. and Jie Tang. 2021a. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:21
Xipeng Qiu, Tianxiang Sun, Yige Xu, Yunfan Shao, Ning Dai, and Xuanjing Huang. 2020. Pre-trained models for natural language processing: survey. Science China Technological Sciences, 63(1)):1872- 1897.
Xiao Liu, Yanan Zheng, Zhengxiao Du Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. Gpt understands, too. arXiv preprint arXiv:2103. 10385.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't 't know: Unanswerable questions for squad. arXiv preprint
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 16692.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. arXiv preprint arXiv: 16066052500
Andrew Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th annual meeting of the associ- ation for computational linguisties: Human language technologies, pages 142-150.
Marek Rei and Helen Yannakoudakis, 2016. Composi- tional sequence labeling models for error detection in learner writing. arXiv preprint arXiv: 1607,06 06153.
Cedric Renggli, André Susano Pinto, Luka Rimanic, Joan Puigcerver. Carlos Riquelme, Ce Zhang, and Mario Lucie. 2020. Which model transfer? finding the needle in the growing haystack. arXiv preprint arXiv:2010.06402.
Julian McAuley and Jure Leskovec. 2013. Hidden fac- tors and hidden topics: understanding rating dimen- sions with review text. In Proceedings of the 7th ACM conference on Recommender systems, pages 165-172.
Bo Pang and Lillian Lee. 2005. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. arXiv preprint cx/0506075.
Amrita Saha, Rahul Aralikatte, Mitesh Khapra, and Karthik Sankaranarayanan. 2018. Duore: To- wards complex language understanding with para- phrased reading comprehension. arXiv preprint orXiv:1804.0070 1804.07927.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv: N0T1:1881.00000 1.01088.
Erik F Sang and Sabine Buchholz. 2000. Introduction to the conll-200x shared task: Chunking. arXiv preprint cs/0009008.
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2018. Wie: the word-in-context dataset for evaluat- ing context-sensitive meaning representations. arXiv preprint arXiv: 180::1808.009 0911..
Erik F Sang and Fien De Meulder. 2003. Introduction to the conll-2003 shared task: Language-independer named entity recognition. arXiv preprini cx/306050.

---

Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, et al. 2021. Multitask prompted training en- ables zero-shot task generalization. arXiv preprint arXiv:2110.082007
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07460
Chengyu Wang, Jianing Wang. Minghui Qiu, Jun Huang, and Ming Gao. 2021. Transprompt: Towards automatic transferable prompting framework for few-shot text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 2792-2802.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang. Junlin Wu, and Yi-Shin Chen. 2018. Carer: Con- textualized affect representations for emotion recog- nition. In Proceedings of the 2018 conference on empirical methods natural language processing, pages 3687-3697.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee, Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer Dy. and Tomas Pfister. 2022. Learning to prompt for continual learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pottern Recognition, pages 139-149.
Timo Schick and Hinrich Schütze. 2020. It's not just size that matters: Small language models are also few-shot learners. arXiv preprint Xxxv:2009.071188
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang. Christopher D Manning, Andrew Y Ng. and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empiri- cal methods in natural language processing, pages 1631-1642.
Alex Warstadt, Amanpreet Singh, and Samuel R Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-641.
Ralph Weischedel, Martha Palmer, Mitchell Marcus, Ed- uard Hovy, Sameer Pradhan, Lance Ramshaw, Nian- wen Xue, Ann Taylor, Jeff Kaufman, Michelle Fran- chini, et al. 2013. Ontonotes release 5.0 1 Linguistic Data Consortium, Philadelphia, PA, 23.
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, et al. 2021. On transferability of prompt uning for natural language understanding. arXiv preprint arXiv:21 211011679.9..
Adina Williams, Nikita Nangia, and Samuel R Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint urXiv:1704.05::
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. 2022. On transferabil- ity prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 3949-3969.
Hui Wu and Xiaodong Shi. 2022. Adversarial soft prompt tuiigg for cross-domain sentiment analysis. In Proceedings of the 60th Annual Meeting of the Association for Compulational Linguistics ( (Volume 1: Long Papers), pages 2438-2447.
Alon Talmor and Jonathan Berant. 2019. Multiqa: An empirical investigation of generalization and transfer in reading comprehension. arXiv preprint 190.:/9555.355..
Amir Zamir, Alexander Sax, William Shen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2018. Taskonomy: Disentangling task transfer learning. In Proceedings of the IEEE con- ference on computer vision and pattern recognition, pages 3712-3722.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adap- tation through soft prompt transfer. arXiv preprint 10.09:21
Rowan Zellers, Yonatan Bisk, Roy Schwartz, and Yejin Choi. 2018. Swag: A large-scale adversarial dataset for grounded commonsense inference. arXiv preprint
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessan- dro Sordoni. Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transfcrability across nlp tasks. arXiv preprint 2005
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019, Bertscore: Eval- uating text generation with bert. arXiv preprint
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019. Superglue: A stick- ier benchmark for general-purpose language under- standing systems. Advances in nearal information processing systems, 32.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classi- fication. Advances neural information processing systems, 28.

---

Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shol performance of language models. In In- ternationnal Conference Machine Learning, pages 12697-12706. PMLR.
Appendix
A Datasets
Detailed statistics of source and target tasks are presented in Table 4.
B Transfer Results
Detailed transfer results on target tasks with dif- ferent source tasks are presented in Table 5. We further group source tasks by different task types and show the transfer gains on each target task in Fig. 5.

---

Task
MTain
Devl
Task Type
Metries
Cosmos QA (Huang et al., 2019)
25.3k
3k
Commonsense Reasoning
ACC
SWAG (Zellers et al., 2018)
50k
10k
Commonsense Reasoning
ACC
DuoRC-s (Saha et al., 2018)
50k
QA
DuoRC-p (Saha al., 2018)
50k
10k
QA
Fl
SICK (?)
4.4k
0.5k
NLI
ACC
TREC (?)
4.4k
1k
Question Classification
ACC
SciCite (Cohan et al., 2019)
8.2k
1k
Citation Intent Classification
ACC
CoLA (Warstadt et al., 2019)
8.6k
1k
Linguistic Acceptability
MCC
Emotion (Saravia et al., 2018)
16k
2k
Emotion Classification
ACC
IMDB (Maas et al., 2011)
20k
5k
Sentiment Classification
ACC
Rotten Tomatoes (Pang and Lee, 2005)
8.5k
1.1k
Sentiment Classification
ACC
STS-B (Cer et al., 2017)
5.7k
1.5k
Semantic Textual Similarity
Pearson
Yelp Polarity (Zhang et al., 2015)
50k
10k
Sentiment Classification
ACC
SNLI (Bowman al., 2015)
50k
9.8k
NLI
ACC
MNLI (Williams et al., 2017)
50k
9.8k
NLI
ACC
QQP 3
50k
10k
Semantic Textual Similarity
ACC
QNLJ (Wang et al., 2018)
50k
5.5k
NLI
ACC
SST-2 (Sacher et al., 2013)
50k
0.9k
Sentiment Classification
ACC
SciTail (Khot ct al., 2018)
27k
1.3k
NLI
ACC
SQuAD 1.0 (Rajpurkar et al., 2016)
50k
10k
QA
SQuAD 2.0 (Rajpurkar al., 2018)
52.2k
10.6k
QA
FI
NER- (Derczynski et al., 2017)
3.4k
lk
NER
ACC
Chunk-CoNLLL2000 (Sang and Buchholz, 2000)
7.1k
1.8k
Chunking
ACC
POS-CONLL200S (Sang and Meulder, 2003)
14k
3.3k
POS
ACC
NER-CoNLL2003 (Sang and De Meulder, 2003)
14k
3.3k
NER
ACC
ST-PMB (Abzianidze and Bos, 2017)
50k
10k
Semantic Tagging
ACC
NER-MIT Movie
6.3k
1.6k
NER
ACC
FCE-eroor-dttection (Rei and Yannakoudakis, 2016)
2.9k
2.2k
Error Detection
ACC
BoolQ (Clark et al., 2019)
9.4k
3.3k
QA
ACC
CB (De Marneffe et al., 2019)
250
57
NLI
ACC
COPA (Gordon et al., 2012)
400
100
Commonsense Reasoning
ACC
MultiRC (Khashabi et al., 2018)
27k
4.8k
QA
FI
RTE (Dagan et al., 2005)
2.5k
277
NLI
ACC
WiC (Pilehvar and Camacho-Collados, 2018)
5.4k
WSD
ACC
MRPC (Dolan and Brockett, 2005)
3.7k
408
Semantic Textual Similarity
ACC
CoNLL2004 (Carreras and Mårquez)
922
231
NER
ACC
AG news (Zhang et al., 2015)
50k
10k
Topic Classification
ACC
Amazon Polarity (McAuley and Leskovec, 2013)
50k
10k
Sentiment Classification
ACC
Hate-speech18 (De Gibert et al., 2018)
8.8k
2189
hatespeech identification
ACC
Hate-speech-offensiiee (Davidson et al., 2017)
19.8k
5k
Hatespeech Identification
ACC
CoNLL2012 (Pradhan al.. 2012)
50k
POS
ACC
ATIS (Hemphill et al., 1990)
3982
996
Intent Classification
ACC
CR (Ding et al., 2008)
2.9k
719
Sentiment Classification
ACC
ontonotes (Weischedel al., 2013)
50k
8.5k
NER
ACC
DBpedial4 (Lehmann al., 2015)
50k
10k
Ontology Classification
ACC
L4ptop14 (Pontiki et al., 2016)
1.7k
432
Sentiment Classification
ACC
Emo (Chatterjee et al., 2019)
24k
6k
Emotion Classification
ACC
ART (Bhagavatula et al., 2019)
50k
1.5k
Commonsense Reasoning
ACC
Table 4: Detailed statistics about the source and target tasks. WSD stands for Word Sense Disambiguation task. QA
stands for question answering task. NLI stands for natural language inference task. NER stands for Named Entity
Recognition task. POS stands for part-of-speect task. MCC short for Matthews Correlation Coefficient and ACC
stands for accuracy.

---

SST-2
MRPC
QQP
SciTail
IMDB
CoNLL2004
No transfer
95.07
81.86
83.94
93.1
93.9
87.03
AG News
95.18
81.86
83.80
93.25
93.96
92.06
Amazon Polarity
95.41
86.76
84.79
94.63
94.30
90.73
APP Reviews
84.29
79.17
77.43
93.48
86.70
84.01
ART
95.18
87.99
84.19
95.78
93.08
84.36
ATTS
92.43
82.11
84.07
92.79
92.90
89.09
BoolQ
94.84
71.81
78.17
71.55
89.38
87.26
CB
86.47
79.41
82.17
87.27
87.88
89.02
COLA
94.72
82.84
83.55
93.87
93.50
84.27
CoNLL2000
95.30
90.69
85.63
96.63
94.56
94.75
NER-CONLL2003
95.30
87.25
85.67
96.17
94.40
95.31
NOS-CONNLL2003
95.07
83.58
85.33
95.78
94.56
93.09
CoNLL2012
94.50
88.73
85.70
96.09
94.06
94.22
COPA
94.15
78.19
83.35
89.80
93.76
85.37
Cosmos QA
95.30
89.22
85.58
97.09
94.78
92.74
CR
95.07
82.84
84.36
95.55
93.68
88.30
DBpedia_ 14
94.72
86.27
85.30
95.71
94.30
91.35
DuoRC_p P
95.99
90.20
86.10
96.17
94.48
94.29
DuoRC_s
95.64
89.46
86.16
96.63
94.64
93.32
Emo
95.07
83.82
85.42
92.95
94.12
88.27
Emotion
93.81
78.19
84.55
93.63
87.36
88.65
cretectoor-ttt
95.76
86.27
85.09
95.63
94.50
91.66
Hate_ site_speech18
83.03
78.43
84.20
91.79
85.54
86.23
hate sate__peech____ offeensive
88.76
83.82
83.36
88.11
91.22
84.78
Laptop14
93.58
81.37
83.15
92.48
87.30
89.80
MIT movie
96.10
88.24
85.57
96.01
93.72
94.51
MNLI
95.53
89.22
86.32
94.87
94.24
90.26
MultirRC
94.38
72.30
82.59
85.97
93.68
84.84
ontonotes
95.07
86.76
85.85
96.47
94.34
95.05
ST-PMB
95.76
72.06
85.35
96.47
94.72
93.71
QNLI
95.41
83.82
85.83
93.72
93.84
88.15
QuAIL 93.81
80.64
83.61
93.02
93.88
84.93
Rotten Tomatoes
96.22
83.33
85.35
96.32
94.28
88.57
RTE
90.71
79.17
83.42
90.72
85.28
81.87
SciCite
95.18
84.56
85.22
93.49
94.12
90.30
SICK
96.10
87.25
85.52
93.72
93.98
86.41
SNLI
96.21
88.97
85.85
94.10
93.88
90.75
SQuAD
95.99
89.71
86.25
95.94
94.58
93.55
SQuAD_vy v2
95.64
90.44
86.56
96.70
94.54
92.56
STS-B
94.61
85.05
84.55
94.71
86.58
88.59
SWAG
93.35
73.04
84.19
93.25
93.40
89.76
TREC
95.18
82.35
85.89
93.41
94.02
89.93
WiC
95.64
79.90
84.55
94.79
93.28
88.86
NER-WNUTI7
95.64
77.70
85,52
94.56
94.58
94.75
Yclp Polarity 95.41
84.56
85.49
95.78
94.58
87.29
Table 5: Accuracy on each target task with different source tasks. All values are scaled by 100.

---

15
10
Transfer Gain (%)
10
15
SST-2
MRPC
QQP
SciTail
IMDB
CoNLL2004
Task
NLI
QA
Word Sense Disambiguation
Text Classification
Common Sense Reasoning
P
Sequence Labeling
Figure 5: Transfer gains on each target task with different source tasks. Each violin represents one target task. Each point inside violin represents an individual source task. The point color denotes task type, and the y-coordinate denotes the absolute transfer gains on the specific target task with source tasks. Here we group named entity recognition, word segmentation, chunking, and part of speech into sequence labeling tasks; sentiment classification, emotion classification, hate speech identification, question classification, intent classification, grammatical judgment, and topic classification into text classification tasks.


Reviews:
[{"review_id": "mFzrC3VInf", "reviewer": "Reviewer_uUvq", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your suggestion! Besides the OOD experiments conducted in 4.6.2, we also remove the NLI tasks from the prompt pool and use the remaining tasks to train the affinity scoring function. Then we test its prediction performance on the SciTail task (NLI task). Experimental results show that the largest transfer gain on the SciTail task is 3.16% in accuracy, which suggests the generalization of the affinity scoring function on new task types. We will add more such OOD experiments in the revised version.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The method is sound and provides a moderate improvement over existing methods for predicting transferrability using soft prompts. The extension to full fine-tuning is also novel (to my knowledge). The experiments could be more extensive, but large answer the given research question.", "reasons_to_reject": "My primary concern is regarding the set of target tasks, which is 1) small (only 6) and 2) not greatly separate from the training tasks (e.g. Yelp Polarity is a sentiment classification task and among the training tasks, while the SST-2 is also a sentiment task and in the target tasks). The authors do address generalization more concretely in 4.6.2 on a small experiment. While the results point in the right direction, the task-type overlap for the remaining experiments do mean that their results need to be taken with a grain of salt. I would be willing to raise the score with more such OOD experiments.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method for predicting task transferrability by learning a scoring function based on soft prompts. They show that the transferrability identified applies both to transfer learning with soft prompts as well as to full intermediate fine-tuning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Vu et al. 2021 is cited out of place for prompt tuning (L 056).", "Typos_Grammar_Style_and_Presentation_Improvements": "I would emphasize up front that this work studies the same problem as Vu et al., 2021., and that the key novelty is using a learned scoring function."}, "extra_text": {}}, {"review_id": "rcZwagwENG", "reviewer": "Reviewer_23Bc", "summary": "", "questions": "A) Line 57: The phrase \"simulate the knowledge of a large language model\" is unclear. Do you mean that prompts effectively enable LLMs to access their knowledge? (This is what I would expect.)\n\nB) Section 2: The prompt tuning procedure is slightly unclear. What do you mean by \"randomly-initialized prompt tokens\" (lines 109-110)? Does this mean that the initial prompt for each task consists of random tokens? Why not start with a prompt that would seem good from a human perspective, even though it's not engineered/optimized?\n\nC) Lines 134-135: What kind of architecture is being used as the task-specific model here? Is it an LLM?\n\nD) Lines 343-350: These two sentences seem to contradict each other. The first sentence (\"We also observe that positive transfer...\") states that positive transfer can occur when the source task is small, which implies that small task size is a potential indicator of good transferability. But the final sentence (\"It indicates that...\") seems to say that data size can't be used as a predictor of task affinity. Didn't the observations indicate otherwise?\n\nE) Are the scores in Table 1 the average of multiple runs, or only from single runs?", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions! \n\nManually-written prompts need carefully handcrafted design. Finding proper prompts for each task is difficult and time-consuming. Even if significant effort is invested, manual prompts are likely to be suboptimal[1][2]. So we use soft prompt in the experiments. We will add the comparison of soft prompts and manually-written prompts in the revised version.\n\nWe answer your questions as follows.\n\nA）\tYes, It means that prompts effectively enable LLMs to access their knowledge. I will make it more clear in the revised vesion.\n\nB）\tYes, the initial prompt for each task consists of random tokens. Finding proper prompts for each task is difficult and time-consuming. Even if significant effort is invested, manual prompts are likely to be suboptimal[1][2]. So we initialize prompt for each task with random tokens. \n\nC）\tYes, it is an LLM. Specifically, we use RoBERTa-large as the task-specific model as mentioned in Section 4.2 (Line 250).\n\nD）\tTraditional methods use data size and task type to predict task transferability. They take tasks with large data size or the same task type as the best source tasks to transfer. However, experimental results show that positive transfer can occur when the source task has a small size or a different task type. So it is not precise to use data size or domain similarity to estimate task affinity.\n\nE）\tWe report the average score of multiple runs in Table 1.\n\n[1] P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. Liu et al.,2021\n\n[2] Making Pre-trained Language Models Better Few-shot Learners. Gao et al.,2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The task of determining the best source task for transfer is important and nontrivial. This paper proposes an effective method for doing so that requires less supervision than previous methods. The paper's theory is adequately explained by equations, and the experiments provide an extensive comparison with 2 baseline selection methods, as well as 6 baseline similarity measures. This provides a good justification of the effectiveness of the scoring function, and in fact the scoring function outperforms all baselines, save for brute force search (which is expensive). Finally, an ablation study is performed to test the optimal parameter values for the scoring function.", "reasons_to_reject": "While the paper's explicit focus is on soft prompts and prompt tuning, the experiments do not demonstrate the behavior of the affinity scoring function when using manually-written/engineered prompts in the prompt pool. The paper also does not justify why soft prompts are preferable to manual prompts in this particular setting. (For example, is there a fundamental incompatibility, or are soft prompts simply the authors' preference here?) Therefore, it is unclear how beneficial the affinity scoring function would be when given manually-written task prompts in the prompt pool, or a possible mix of manual and soft prompts. This is an issue that should be discussed in some way, for the sake of completeness.", "questions_to_authors": "A) Line 57: The phrase \"simulate the knowledge of a large language model\" is unclear. Do you mean that prompts effectively enable LLMs to access their knowledge? (This is what I would expect.)\n\nB) Section 2: The prompt tuning procedure is slightly unclear. What do you mean by \"randomly-initialized prompt tokens\" (lines 109-110)? Does this mean that the initial prompt for each task consists of random tokens? Why not start with a prompt that would seem good from a human perspective, even though it's not engineered/optimized?\n\nC) Lines 134-135: What kind of architecture is being used as the task-specific model here? Is it an LLM?\n\nD) Lines 343-350: These two sentences seem to contradict each other. The first sentence (\"We also observe that positive transfer...\") states that positive transfer can occur when the source task is small, which implies that small task size is a potential indicator of good transferability. But the final sentence (\"It indicates that...\") seems to say that data size can't be used as a predictor of task affinity. Didn't the observations indicate otherwise?\n\nE) Are the scores in Table 1 the average of multiple runs, or only from single runs?", "paper_topic_and_main_contributions": "This paper deals with the topic of choosing the optimal source task when conducting transfer learning to a target task. While it is well known that knowledge transfer benefits pretrained language models, choosing a sub-optimal source task will either not benefit or may even harm performance on the target task. However, methods of finding optimal source tasks remain underexplored, and brute-force search over tasks is inefficient. Thus, it is relevant to investigate efficient methods of finding source tasks that will result in performance gains (i.e. tasks that are transferable).\n\nThis paper provides a method of predicting the transferability of tasks by employing an affinity-scoring function based on soft prompts. First, source task prompts are collected into a prompt pool. Next, an affinity scoring function is learned, which represents the transfer gain (gain in performance after transfer) for each source task with the given target task. The source task with the highest affinity score is chosen for transfer. The performance is evaluated against several baseline methods, including brute-force search, random selection, and various similarity measures. It is found that the proposed affinity scoring function method outperforms all baselines in average absolute transfer gain save for brute-force search.\n\nConcretely, the paper makes the following contributions:\n\n1) A new affinity scoring function based on soft prompts that determines which source task is the most beneficial for transfer to a target task.\n2) An ablation study and experiments that demonstrate the effectiveness of the proposed scoring function.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Line 237: benchmark --> benchmarks\n\nLine 279-280: measurement --> measurements\n\nLine 338: for --> of\n\nLine 449: There's a space between \"types\" and the superscript 2.\n\nLines 605-613: It seems like the Ethics Statement section was left in by accident? The text is from the EMNLP template.\n\nLine 1004: Missing a period after \"Table 4\"."}}, {"review_id": "GzRis55Dj8", "reviewer": "Reviewer_fsyc", "summary": "", "questions": "1. In Section 3.2, you claimed that the task embedding P_i is used to initialize the prompt for task j. In this way, P_i will be trained using the training data of task j \"again\" after learning P_j. Therefore, every calculation of transfer gain involves two times of training, which will lead to high computation costs. Can you provide the total training time in your experiments?\n2. You are missing some important baselines including GPT-3 and ChatGPT. Since today's researches are mostly focused on large language models using discrete prompts (or instructions), you should compare with them and discuss your advantages on them. \n3. For table 1, you should list or present the performance of Affinity Scoring Function is based on which source task, which will make your analysis in Section 4.4 more clear.  \n4. I understood that you only use the prompt of the best source task. So in Section 4.6.1, is there any difference for k=1 or 3?\n5. This work is very similar to previous work [1,2,3], but you miss explicit and insightful comparison with them.\n[1] Learning to transfer prompts for text generation;\n[2] Learning to prompt for continual learning;\n[3] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts.", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your valuable suggestions!\n\n1. Every calculation of transfer gain only cost a few minutes with one GPU(V100).  Rather than enumerating every task combination pair, we randomly selected 400 task pairs from the prompt pool to calculate the transfer gain. Thus, the total training time is around 16 hours with 8 V100 GPUs. We will add the total training time in the revised version.\n\n2. Discrete prompts need carefully handcrafted design. Even if significant effort is invested, discrete prompts are likely to be suboptimal[4]. It is difficult and time-consuming to finding proper prompts for each task[5][6]. So we use soft prompt in the training process. We will add the comparison of soft prompts and discrete prompts in the revised version.\n\n3. Thanks for your suggestion! We will list the source task which the performance of Affinity Scoring Function is based on in the revised version.\n\n4. Given a novel target task, the affinity function gives each task in the prompt pool a task affinity score. For k=1, we take the task with the highest score as the best source task to transfer. For k=3, we select the source tasks with top-3 scores, conduct prompt tuning using these three source tasks respectively and present the best performance on the the target task (i.e., selecting the best of top-k requires prompt tuning k times on the target task). We report the average absolute transfer gain on the target tasks with the Best of Top-1 and Top-3 source prompt in Fig.3. \n\n5. [2] learns to dynamically prompt a pre-trained model to learn tasks sequentially for computer vision tasks, [1] learns to transfer prompts for text generation tasks, while we focus on natural language understanding (NLU) tasks. [3] proposes parameter-efficient multi-task tuning via attentional mixtures of soft prompts. Our work differs from theirs as follows:\n\n      (1) They conduct multi-task learning over a group of target tasks by sharing the attention module. Different from [3] which requires high-resource source datasets and multiple target tasks, in our work, the target can be from one or more tasks and the source data can be less sufficient. \n\n      (2) They focus on classification and QA tasks while our method can be generalizable to more diverse sets of NLU tasks such as sequence labeling tasks. \n\n      (3) The proposed method is not only helpful for prompt transfer learning, but also can be generalized to intermediate-task transfer as well.\n\n[1] Learning to transfer prompts for text generation. Li et al. 2022.\n\n[2] Learning to prompt for continual learning. Wang et al. 2022\n\n[3] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts. Asai et al.2022.\n\n[4] Calibrate before use: Improving few-shot performance of language models. Zhao et al., 2021.\n\n[5] P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks. Liu et al.,2021\n\n[6] SPoT: Better Frozen Model Adaptation through Soft Prompt Transfer. Vu et al.,2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novelty in method. This work is the first to leverage an affinity scoring function to measure the task transferability.\n2. Extensive experiments. This work conducts many related experiments.\n3. The writing is clear to understand this paper.", "reasons_to_reject": "1. There are some drawbacks of this method such as training complexity.\n2. Missing some important baselines and not clear experimental analysis.\n3. Missing insightful comparison with missing important references.\n", "questions_to_authors": "1. In Section 3.2, you claimed that the task embedding P_i is used to initialize the prompt for task j. In this way, P_i will be trained using the training data of task j \"again\" after learning P_j. Therefore, every calculation of transfer gain involves two times of training, which will lead to high computation costs. Can you provide the total training time in your experiments?\n2. You are missing some important baselines including GPT-3 and ChatGPT. Since today's researches are mostly focused on large language models using discrete prompts (or instructions), you should compare with them and discuss your advantages on them. \n3. For table 1, you should list or present the performance of Affinity Scoring Function is based on which source task, which will make your analysis in Section 4.4 more clear.  \n4. I understood that you only use the prompt of the best source task. So in Section 4.6.1, is there any difference for k=1 or 3?\n5. This work is very similar to previous work [1,2,3], but you miss explicit and insightful comparison with them.\n[1] Learning to transfer prompts for text generation;\n[2] Learning to prompt for continual learning;\n[3] ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts.", "paper_topic_and_main_contributions": "This paper studies a method to predict the transferability between tasks through learning an affinity scoring function. \n\nThe contributions of this paper are twofold. First, this work proposes to learn an affinity scoring function to predict the transferability between 50 tasks. Second, the extensive experiments validate the effectiveness of this method to some extent.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a novel method for predicting task transferability in the context of natural language understanding tasks. The approach involves learning an affinity scoring function based on soft prompts, allowing efficient selection of source tasks for transfer learning. Extensive experiments demonstrate the method's effectiveness compared to baselines, although some additional comparisons and clarifications are needed. The paper also highlights the practicality of using soft prompts over manually-crafted prompts and emphasizes its applicability to various NLU tasks.

The authors commit to including a comparison of soft prompts and manually-written prompts, as well as more OOD experiments in the revised version. While they promise to look into LLMs like ChatGPT, I don't think that's necessary given that direct access to the weights is not given, making the approach difficult to assess from a research perspective.
Decision: Accept-Main