Exploring the Effectiveness and Consistency of Task Selection in Intermediate-Task Transfer Learning (2024)

Abstract: Identifying beneficial tasks to transfer from is a critical step toward successful intermediatetask transfer learning. In this work, we experiment with 130 source-target task combinations and demonstrate that the transfer performance exhibits severe variance across different source tasks and training seeds, highlighting the crucial role of intermediate-task selection in a broader context. We compare four representative task selection methods in a unified setup, focusing on their effectiveness and consistency. Compared to embedding-free methods and text embeddings, task embeddings constructed from fine-tuned weights can better estimate task transferability by improving task prediction scores from 2.59% to 3.96%. Despite their strong performance, we observe that the task embeddings do not consistently demonstrate superiority for tasks requiring reasoning abilities. Furthermore, we introduce a novel method that measures pairwise token similarity using maximum inner product search, leading to the highest performance in task prediction. Our findings suggest that token-wise similarity is better predictive for predicting transferability compared to averaging weights.1

heading: 1 Introduction
text: Pre-trained language models (PLMs) have become foundational in the transfer learning paradigm of natural language processing (NLP) (Devlin et al., 2019; Brown et al., 2020; Chowdhery et al., 2023). Intermediate-task transfer learning aims to improve model performance further by introducing an intermediate stage of supervised training on data-rich tasks before fine-tuning the target downstream task (Phang et al., 2018; Pruksachatkun et al., 2020; Vu et al., 2020). The paradigm has shown to be particularly useful for improving performance in resource-constrained scenarios where annotated
1We release the code publicly at https://github.com/udslsv/intermediate-task-selection.
training data is often limited (Prasad et al., 2021; Vu et al., 2022b).
A crucial aspect of intermediate-task transfer learning is to select beneficial tasks to transfer from. However, the costs of searching for the optimal intermediate-task, especially with the growing array of available NLP tasks and the exhaustive process of model fine-tuning (Pruksachatkun et al., 2020; Vu et al., 2020), are prohibitive. Research on intermediate-task selection mainly predicts task transferability using task-specific embeddings, which condense the task information of a given target task into a single vector representation. For example, some works construct task embedding from fine-tuned weights (Vu et al., 2022b; Zhou et al., 2022) or leverage text embedding (Poth et al., 2021). More specifically, Poth et al. (2021) use sentence transformers to encode dataset examples as text embeddings. The more recent approach by Vu et al. (2022b) constructs task embeddings from the weights of soft prompts, which have been effectively applied in large-scale studies.
Despite their promising results, a systematic study of the consistency of these task selection methods is still missing. Specifically, it remains unclear how consistent these approaches are at predicting the best source task to transfer from.
264
To address this gap, we perform a comprehensive evaluation of existing task selection methods in intermediate-task transfer learning. Our research questions are: (1) Do intermediate-task selection approaches exhibit consistent performance across downstream tasks? (2) What are the key ingredients that result in accurate transferability predictions?
To answer these questions, we perform experiments across 130 intermediate and downstream task combinations derived from 13 source and 10 target tasks. Our results show that intermediatetask transfer exhibits significant performance variance across tasks. Comparing four representative task selection methods, we find that task embeddings based on fine-tuned weights (Vu et al., 2022b) generally outperform embedding-free and text embedding methods (Poth et al., 2021). However, we also observe that such task embeddings do not consistently perform well on tasks requiring highlevel reasoning abilities. Exploring this further, we revisit the task embedding design and propose a new construction method based on pairwise token similarity (see Figure 1), which yields the highest average task prediction performance of 82.5%. Our main contributions are as follows:
1. We systematically investigate intermediatetask transfer learning across 130 intermediate and downstream task combinations.
2. We examine four representative task selection methods in a unified setup, including both embedding-free and embedding-based methods.
3. We introduce a novel task embedding construction approach based on pairwise token similarity, which achieves the highest task prediction performance of 82.5% in nDCG score.
4. We provide an in-depth analysis of the impact of task type and training seed, along with an exploration into embedding distributions.

heading: 2 Related Work
text: Identifying a beneficial task from a broader set of source tasks is a crucial step in intermediate-task transfer learning. Various studies have proposed methods to estimate task transferability based on task embeddings.
A foundational approach is Task2Vec (Achille et al., 2019; Vu et al., 2020), which involves computing the Fisher information matrix and enables to
measure semantic and taxonomic relationships between tasks. In contrast, Poth et al. (2021) demonstrate the effectiveness of text embeddings based on sentence encoders. The landscape of task selection approaches has further evolved with the introduction of parameter-efficient fine-tuning (PEFT) techniques. For instance, Vu et al. (2022b) use soft prompts to generate task embeddings, demonstrating the effectiveness of prompt-based embeddings. Expanding on this, Zhou et al. (2022) investigate other PEFT methods, including P-tuning (Liu et al., 2022a,b), fine-tuning only bias terms (Ben Zaken et al., 2022), and LoRA (Hu et al., 2022). They construct task embeddings based on the fine-tuned weights.
Task selection based on neuron activations provides another perspective by focusing on the patterns of activations within models. Su et al. (2022) propose model stimulation similarity to identify beneficial source tasks through the overlap rate of activations. More recently, Xi et al. (2023) introduce connectivity patterns as task embeddings, identifying task-specific patterns in deep neural networks that best represent the tasks.
Our work differs from previous studies by contributing a comparison of existing task selection methods in a unified setup, specifically focusing on the effectiveness and consistency of these approaches.

heading: 3 Background
text: In the following, we introduce the intermediatetask transfer learning paradigm and motivate our focus on parameter-efficient fine-tuning.

heading: 3.1 Intermediate-Task Transfer Learning
text: As depicted in Figure 2, intermediate-task training involves sequentially fine-tuning on a source task followed by fine-tuning on a target task. By incorporating an intermediate stage of supervision (typically on data-rich tasks), intermediate-task transfer learning enables knowledge transfer across tasks, thereby enhancing performance on low-resource target tasks (Vu et al., 2022b).
More formally, the intermediate-task transfer learning paradigm can be divided into two stages: (1) training a PLM fθ on a given source task T s to obtain the intermediate model fθ′ ; (2) training the intermediate model fθ′ on the target task T t. The objective function with a cross-entropy loss L of
the first stage is defined as follows:
θ ′ = argmin
θ LT s(fθ). (1)
Here, the source task T s is selected based on a selection criterion using metadata of datasets, domain similarity, or task similarity. Subsequently, the intermediate model is trained on the target task:
θ∗ = argmin θ ′ LT t(fθ′ ) (2)
Note that in Equation 2 the intermediate model f is parameterized with θ ′ , representing the parameters of the model trained on source task T s.

heading: 3.2 Parameter-Efficient Fine-Tuning via Soft Prompts
text: Modern language models often contain billions of parameters, making sequential fine-tuning and experimenting with a large number of source and target task combinations impractical. Recent studies have explored parameter-efficient fine-tuning approach through prompt tuning, which involves learning task-specific soft prompts that allow a frozen language model to efficiently perform specific downstream tasks (Lester et al., 2021; Li and Liang, 2021; Liu et al., 2022a). Unlike discrete prompts, soft prompts consist of a set of learnable prompt tokens that are learned through backpropagation and can be applied to various downstream tasks. This approach has been successfully used to efficiently adapt large language models in various scenarios (Qin and Eisner, 2021; Vu et al., 2022a; Asai et al., 2022).
More recently, researchers have focused on intermediate-task transfer learning using prompt tuning, specifically Soft Prompt Transfer (SPoT) (Vu et al., 2022b). SPoT employs a series of soft prompt tokens to adapt frozen models to specific
downstream tasks, making it highly parameterefficient for intermediate-task transfer learning. In this transfer learning procedure, a pre-trained model is adapted to each task by conditioning on a set of learnable prompt tokens. Moreover, the resulting prompts can directly serve as task embeddings to assess task transferability.

heading: 4 Intermediate-Task Selection Methods
text: Intermediate-task transfer can improve the performance of the target downstream task, but it is computationally infeasible to try out all possible task combinations, making choosing a beneficial source task an important problem.
Intermediate-task selection aims to predict task transferability and retrieve the most beneficial task from a broad set of available source tasks. This eliminates the need for exhaustive training and is more feasible in resource-constrained scenar-
ios. Here, we compare existing intermediate-task selection methods which can be categorized into two groups: embedding-free and embedding-based methods (see Table 1).

heading: 4.1 Embedding-Free Methods
text: The first group of methods operates without accessing any model. They estimate task transferability based on certain criteria, such as data size, or simply perform random selection. These methods serve as baseline approaches in Poth et al. (2021).
Random selection (RANDOM) This method selects the intermediate-tasks randomly without using any specific information for the tasks and models.
Data size (SIZE) This method predicts the task transferability based on the data size, assuming that larger datasets indicate higher transferability to model performance.

heading: 4.2 Embedding Methods
text: The second group of methods constructs embeddings either using a pre-trained sentence encoder model or an intermediate model fθ′ . We consider two such methods:
Sentence embeddings (SEMB) It represents the text embedding obtained by averaging all sentence representations on the whole dataset (Poth et al., 2021). Each sentence representation, denoted as hxi , is encoded by the encoder model for a given example xi. These sentence representations are averaged over the entire dataset:
∑ xi∼D hxi |D| . This
method captures linguistic properties of the input text x for both the source and target tasks, independent of the intermediate-task training algorithm.
Prompt similarity (FEATURE) It measures task similarity based on the similarity between their task-specific prompts and employs solely finetuned weights to create task embeddings (Vu et al., 2022b). Let the prompt weights be denoted as [e1, e2, ...eN ] ∈ RN×d, consisting of N soft prompt tokens with d feature dimensions. The prompt similarity score between two tasks, t1 and t2, is defined as the cosine similarity of the average representations of prompt tokens:
sim(t1, t2) = cos( 1
N
N∑
i=1
e1i , 1
N
N∑
j=1
e2j ) (3)
where e1i and e 2 j represent the prompt token representations of the tasks t1 and t2, and cos denotes the
cosine similarity. This method computes the task embedding, represented as a vector in Rd, by averaging the feature values across all prompt tokens. We refer to this method as FEATURE to emphasize its focus on capturing task-specific features.

heading: 5 Systematic Evaluation of Task Selection Methods

heading: 5.1 Experimental Setup
text: Datasets. We consider 13 source tasks of various types, including question answering (QA), natural language inference (NLI), and sentiment analysis, among others. We evaluate the transfer performance on 10 target tasks, following the setting in Vu et al. (2022b), as presented in Table 2. More details on the datasets are provided in Appendix A.1.
Models. For all experiments, we adopt T5 BASE (Raffel et al., 2020) as our PLM. The pre-trained weights remain frozen, and only the weights of the soft prompt tokens are updated. After training,
these fine-tuned weights are then used to construct task embeddings and perform soft prompt transfer.
Implementation details. We closely follow the training configurations outlined in Lester et al. (2021). We train soft prompts for 30K steps, using three random seeds (42, 150, 386). We use N = 100 prompt tokens and initialize the weights of the prompt tokens from the embeddings of the top 5K most frequent tokens in the pre-training data. We use the AdaFactor optimizer (Shazeer and Stern, 2018) with a linear scheduler. After conducting prompt tuning, we select the best-performing checkpoint for prompt transfer. The prompt transfer experiment is conducted with another set of training seeds (112, 28, 52).
We evaluate the effectiveness of prompt transfer using a relative transfer performance metric, calculated as follows: Ms→t−MtMt . Here, the Mt indicates the model performance with no-transfer prompt tuning, and Ms→t represents the transfer performance. The evaluation metric for the model performance varies according to individual tasks.

heading: 5.2 Task Selection Methods and Evaluation
text: Embedding-based methods. For text embeddings, we follow the model choice in Poth et al. (2021). We use the off-the-shelf encoder models to derive sentence representations for both source and target tasks. Specifically, we adopt Sentence-BERT and Sentence-RoBERTa (Reimers and Gurevych, 2019) as encoders for SEMB-B and SEMB-R, respectively.
Selection criterion. We rank the order of beneficial tasks based on quantitative values from embedding-free methods. For embedding-based methods on tasks t1 and t2, we employ cosine similarity using the mapping function h(·) to construct the task embedding or text embedding for a given intermediate task. To get the ranking order, we sort the source tasks based on the score sim(t1, t2) = cos(h(t1), h(t2)) between the source and target tasks. The ground-truth ranking is obtained by transferring source tasks to the downstream task and sorting them based on transfer performance.
Evaluation. We use two metrics2 to evaluate the effectiveness of task selection methods: (1) Normalized Discounted Cumulative Gain (nDCG) (Järvelin and Kekäläinen, 2002), a widely accepted information retrieval measure that evaluates the
2See formal definitions in Appendix A.2.
overall quality of a ranking, emphasizing the entire order rather than merely focusing on the rank of the best source task. The nDCG score ranges from 0 to 1, where 1 presents the exact match with the ideal order and lower values indicate a lower quality of ranking. (2) Regret@k (Renggli et al., 2022), a metric for computational regret, quantifying the relative performance between the expected performance of the top-k selected intermediate-tasks and the optimal intermediate-task. Lower regret signifies a more effective selection strategy among the k intermediate models. For each target task, we evaluate the overall ranking prediction of the 13 source tasks against the ground-truth ranking using nDCG score. We evaluate the efficacy of the top-k selected source tasks compared to the ground-truth selection using Regret@k.

heading: 5.3 Results
text: Intermediate-task transfer exhibits highperformance variance across tasks. Figure 3 illustrates the relative transfer performance across 10 target tasks, sorted by their training data sizes 3. We find that relative transfer performance through intermediate-task training exhibits significant variance across tasks, especially for the downstream tasks COLA, RTE, COPA, and CB. This observation aligns with previous studies showing significant performance variation across source tasks (Pruksachatkun et al., 2020; Jiang et al.,
3The detailed transfer performances are presented in Appendix C.
2023). Additionally, we find that this phenomenon is particularly pronounced in downstream tasks with extremely limited labeled data, such as COPA and CB. In contrast, the relative transfer performance is more consistent for downstream tasks that have sufficient training data, like BOOLQ and STS-B. In Appendix B, we show that there exists a correlation between transfer gains and training data sizes. These results highlight the importance of carefully selecting beneficial tasks to enhance transfer gains, especially in low-resource scenarios.
Embedding-based selection methods outperform embedding-free methods, but the transfer gains are limited. Table 3 presents results for the four task selection methods. Embedding-based approaches show higher task prediction performance over embedding-free methods, indicating richer information is obtained from encoded representations for predicting task transferability. Specifically, FEATURE outperforms all other task selection methods on average. Despite its strong performance, FEATURE falls short of the simple SIZE approach in Regret@1 for multiple choice (M. CHOICE) and question answering (QA) tasks. This highlights the need to further improve task embeddings, especially for tasks that require reasoning abilities.
In Table 4, we show the effectiveness of task selection methods on prompt transfer performance. RANDOM and SIZE select the source task with the highest task transferability score. SEMB-R and FEATURE select top-k tasks that exhibit the largest value of the transferability scores. Compared to the notransfer baseline, these task selection methods show average absolute performance improvements ranging from 0.38% to 0.91%. With an increase of the selection pool (k=1 to k=3), the improvements by SEMB-R and FEATURE further increase to 0.78% and
1.03%, respectively. However, the overall transfer gains remain marginal, indicating that the effectiveness of intermediate-task selection is still limited across diverse tasks.

heading: 5.4 Effect of Task Type and Training Seed
text: To dissect the impact of task type and training seed, Table 5 presents the top-3 beneficial intermediatetasks for COPA and CB. Results for all other tasks are shown in Appendix D.
Task type is not a reliable transferability predictor. While it is intuitive to assume that similar tasks should transfer well to the downstream task, our results reveal that the top-performing source tasks for a given target task can vary widely in task type. We find that task types are generally uncorrelated with transfer performances. For example,
the most performant source tasks for COPA and CB often come from different task types when various training seeds are used. Based on three separate runs, the most beneficial source tasks for COPA (QA) are from other task types, such as CXC (semantic similarity) and QQP (paraphrase detection). Similarly, many of the beneficial tasks for CB (NLI) originated from non-NLI tasks.
Random seed significantly impacts the transfer performance. For COPA, using different training seeds leads to 7.69% to 26.78% relative performance improvements. Similarly, the relative improvements for CB range from 4.11% to 7.60%. This emphasizes the crucial role of seed choice in determining transfer performance. We observe similar variations across seeds in other downstream tasks as well, such as COLA, WIC, and RTE. This can be attributed to the instability in finetuning introduced by different random seeds during prompt transfer (Mosbach et al., 2021; Chen et al., 2022), which can largely affect the robustness of intermediate-task selection.

heading: 6 Revisiting the Construction of Task Embeddings
text: Despite task embeddings from fine-tuned weights demonstrating superior performance in task prediction compared to other selection methods, the effectiveness of various task embedding constructions remains underexplored. In this section, we investigate different construction methods of task embeddings. In addition to FEATURE, we explore two more types of task embeddings as follows.

heading: 6.1 Construction Methods
text: Token-wise mean (UNIGRAM) In FEATURE, we compute the mean of token representations to obtain a task embedding in Rd. To explore an alternative approach, we compute the task embeddings
from another axis, resulting in a task embedding in RN . Specifically, the task embedding for a task t denotes as ht = 1d [ ∑ d e1, ∑ d e2, ..., ∑ d eN ]. The similarity between tasks t1 and t2 is defined as: sim(t1, t2) = cos(ht1 , ht2). We refer to this method as UNIGRAM to emphasize that task-specific information is aggregated from the token-wise dimension.
Maximum inner product search (MAX) We propose a novel task embedding method, referred to as MAX, based on the maximum token-to-token similarity scores. Given the source task t1 and the target task t2, for each prompt token in t2, we obtain the highest token representation similarity score across all tokens in t1. The task similarity is then defined as the mean of these maximum similarity scores:
sim(t1, t2) = 1
N
N∑
j=1
max i
cos(e1i , e 2 j ) (4)

heading: 6.2 Results and Analysis
text: MAX achieves the highest task transferability prediction. Figure 5 presents three types of task embeddings, each derived from prompt checkpoints trained for different numbers of steps. All three methods show improved performance with longer training steps, suggesting that longer training improves task transferability predictions. Notably, MAX achieves the highest nDCG score of 82.5% at the 20K step, indicating that token-wise similarity captures richer task information than FEATURE and UNIGRAM, leading to more accurate task predictions.
Prompt tokens from beneficial tasks are distributed closer to the target prompt tokens. To better understand the prompt token distribution and different levels of transfer performance, we project prompt tokens of the best, 2nd-best,
and worst-performing intermediate-tasks onto lowdimensional spaces using t-SNE (van der Maaten and Hinton, 2008). Figure 4 illustrates that the prompt tokens from no-transfer prompt tuning (red), are close to the tokens from their beneficial intermediate-tasks (green, blue). Furthermore, we observe a considerable overlap in these beneficial source tasks, such as MULTIRC and DROP, for downstream task COPA. This suggests that beneficial tasks tend to be distributed closer to the target prompt tokens and share similar characteristics in low dimensions. For COPA and CB, the worst-performing intermediate-task (brown) deviates from the no-transfer prompt tokens. Future research can further explore a clearer correlation between intermediate-task token distribution and transfer performance.

heading: 7 Conclusion
text: In this work, we conduct a systematic study on intermediate-task selection across a wide range of tasks. Our results show that task embeddings based on fine-tuned weights outperform random
selection, data size, and text embeddings with improvements of +3.96%, +3.85%, and +2.59% in nDCG scores, underscoring the importance of a task-specific approach. Nevertheless, we find that task embeddings do not excel in all scenarios, particularly in multiple choice and QA tasks. By revisiting the task embedding construction, we propose a novel method based on pairwise token similarity, which achieves the highest performance of 82.5% in task transferability prediction, suggesting that token-wise similarity is better predictive in task transferability prediction.
Limitation
Despite our proposed method being effective in many scenarios, we observe that it falls short in predicting task transferability for tasks requiring reasoning abilities, which needs to be further explored. We also face a challenge in precisely evaluating how the parameter configurations of soft prompt tuning impact transfer performance, as prompt tuning is highly sensitive to hyperparameter selection. Moreover, our evaluation of task selection is limited to one specific model architecture and focused on soft prompt tuning. Evaluating on different model architectures, model scales, and fine-tuning methods would provide a more comprehensive understanding of the robustness of intermediate-task selection.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their constructive feedback. This work was funded by the Deutsche Forschungsgemeinschaft
(DFG, German Research Foundation) – project-id 232722074 – SFB 1102.

heading: A More Details to Datasets and Evaluation Metrics
text: A.1 Datasets
We select the datasets drawn from different NLP benchmarks and families of tasks, including natural language inference (NLI), paraphrase detection, semantic similarity, sentiment analysis, question answering (QA), commonsense reasoning, and grammatical acceptability. In total, we consider 13 source and 10 target tasks. The distinguishing between high-resource and low-resource tasks follows conventional notions respect with to the training split size. Table 6 summarizes the statistics of 23 tasks and the evaluation metrics. All data was sourced from HuggingFace Datasets (Lhoest et al., 2021).
A.2 Evaluation Metircs
nDCG This metric is built on the concept of Discounted Cumulative Gain (DCG), a measure of the relevance score for a list of items, each discounted by its position in the ranking.
DCG(R) =
p∑
i=1
2reli − 1 log2(i+ 1)
(5)
where R represents the ranking of source tasks, where the relevance reli of the source task with rank i is set to the averaged target performance, i.e., reli ∈ [0, 100]. The ranking position ρ corresponds to the size of the selection budget.
The nDCG is computed as follows:
nDCG(Rpred, Rtrue) = DCG(Rpred)
DCG(Rtrue) (6)
While nDCG generally considers the overall ranking and the difference between predicted transfer performance and actual performance, realistic applications often prioritize the top-1 transfer performance. In this study, our focus is on metrics that accurately quantify the accuracy of top-1 predictions.
Regret@k The Regret@k metric is crucial for evaluating how well the task embeddings retrieve the beneficial task for top-1 prompt transfer performance. Its formula is as follows:
Regret@k = maxs∈S E[T (s,t)]−maxs̃∈Sk E[T (s̃,t)]
O(S) (7)
Now, let’s simplify the equation by understanding each term: T (s, t) represents the performance achieved on the target task t when knowledge is transferred from the source task s. In simpler terms, it measures how effective insights from task s are in improving performance on task t. Moving on to O(S, t), this term signifies the expected performance on the target task t under the optimal selection strategy. It establishes a performance benchmark achievable with the most advantageous source task selection. Finally, consider Mk(S, t), which takes into account the highest performance observed on task t among the k top-ranked source tasks. This aspect evaluates the potential of the selected set of source tasks in contributing to superior performance on the target task t.

heading: B Transfer Gains with Varying Training Data Sizes
text: We further explore how the training data size influences the relative performance. Figure 6 illustrates the correlation between the training split size and the level of transfer gains and losses. The plot shows 39 runs for each target task. Remarkably, tasks with extremely low resources (fewer than 1K training samples) exhibit a broad range of transfer gains and losses. Specifically, Tasks like COPA and CB with minimal training samples (400 and 250, respectively) show transfer gains varying from +25% to -15% in relative performance.
On the other hand, tasks with smaller variance in transfer gains, such as WSC and RTE, tend to have
fewer instances of positive transfer. This is influenced by a substantial number of runs achieving similar performance to baselines, leading to fewer positive transfers. Additionally, our prompt tuning settings, optimized for near-optimal performance, result in less pronounced benefits from prompt training.
The mean slope emphasizes trends, highlighting a strong correlation between the number of positive gains and the training sample sizes across most downstream tasks. Notably, the extent of performance improvement is more significant for tasks with smaller training sample sizes. However, despite high variance in relative performance, transfer gains tend to converge to zero when the dataset size reaches around 5K.
Prompt transfer’s success is intricately tied to the data size of downstream tasks. Smaller training examples are more likely to exhibit positive transfer. While prompt transfer brings benefits, the presence of negative transfer underscores associated risks.

heading: C Prompt Transfer Performance
text: Table 7 presents the mean performance across three runs on low-resource tasks, utilizing the bestperforming soft prompt as the initialization point. As seen in previous studies, the prompt transfer results indicate improvements over the no-transfer baselines.
In particular, our most successful transfer results exhibit significant enhancements, surpassing the no-transfer outcomes on tasks such as COPA and CB by considerable margins, with improvements of +8% and +3.46%, respectively. However, it’s noteworthy that the mean performance improvements for other tasks are relatively minor. This can be attributed to the extensive hyperparameter search conducted for the strong baseline (PROMPT-TEXT), contrasting with the suboptimal nature of the weak baseline (PROMPT-ABSTRACT). This underlines the significance of optimization in the prompt tuning process.
Our exploration of prompt transfer performance sheds light on the nuanced dynamics at play, emphasizing the need for strategic optimization strategies in achieving robust and notable improvements,
especially in the context of low-resource tasks.

heading: D More Results on the Effect of Task Type and Training Seed
text: Table 8 presents the top three prompt transfer results on eight downstream target tasks, along with their respective task types. These results reflect the most significant improvements in prompt transfer across three random seeds. On tasks with limited annotations, such as COPA and CB, different random seeds lead to substantial variance in transfer performance. Similarly, tasks like COLA, WIC, and RTE also exhibit high variance. For WSC †, we observed that most prompt transfer performances either present identical transfer gain or show no improvement in performance. This phenomenon is likely attributed to the unique task type of WSC compared to other downstream tasks. Specifically, the knowledge of source tasks has limited influence on performing the tasks.

heading: E More Results on the Construction of Task Embeddings
text: Figure 7 analyzes how training steps for prompt tuning affect ranking prediction across various task embedding constructions, MAX, FEATURE and UNIGRAM. We examined the prompt weights trained at intervals of 5K, up to 30K, using nDCG for ranking prediction. Three construction methods of task embeddings were compared across ten downstream
tasks, indexed alphabetically from BOOLQ (a) to CB (j).
Tasks with very limited data exhibit low nDCG scores. We found that the three methods performed well on five tasks, showing high nDCG scores. For instance, in BOOLQ, STS-B, CR, MRPC, and WSC, all three methods demonstrated similar performance with relatively flat performance curves.
We further observed the significant variability in task prediction performance across four tasks: COLA, RTE, COPA, and CB. Notably, COPA and CB presented considerable challenges due to their limited availability of labeled data. As a result, the computed nDCG scores for these tasks were notably lower compared to other downstream tasks, underscoring the difficulty in identifying effective intermediate tasks.
MAX yields superior performances in task prediction. Across 10 downstream tasks, we observed that MAX generally yields superior nDCG scores. On COLA, RTE, and COPA, nDCG surpasses FEATURE after 15K training steps. For CB, MAX excels in capturing the essence between intermediate tasks during continual prompt tuning on challenging low-resource tasks. This highlights the importance of measuring token-wise similarity between source and target prompts for improved performance. Our analysis suggests that MAX method tends to perform better in certain scenarios, em-
phasizing its effectiveness in ranking prediction compared to other methods.
Longer training leads to better performance. Furthermore, MAX achieves higher task prediction performance with longer training steps. Furthermore, MAX achieves higher task prediction performance with longer training steps. For example, in tasks such as COLA, WIC, and RTE, MAX shows marked improvements in the ranking prediction with extended training durations.

heading: Future Work
text: Future research can further explore a clearer correlation between intermediate-task token distribution and transfer performance