Venue: NAACL 2022

# On Transferability of Prompt Tuning for Natural Language Processing

Yusheng Su\({}^{1,3*}\), Xiaozhi Wang\({}^{1,3*}\), Yujia Qin\({}^{1,3}\), Chi-Min Chan\({}^{1,3}\), Yankai Lin\({}^{4}\), Huadong Wang\({}^{1,3}\), Kaiyue Wen\({}^{1,3}\), Zhiyuan Liu\({}^{1,3+}\), Peng Li\({}^{2}\), Juanzi Li\({}^{1,3}\), Lei Hou\({}^{1,3}\), Maosong Sun\({}^{1,3+}\), Jie Zhou\({}^{4}\)

###### Abstract

Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate training and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts _stimulate_ PLMs is essential. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code can be obtained from [https://github.com/thunlp/Prompt-Transferability](https://github.com/thunlp/Prompt-Transferability).

+
Footnote †: Partly done while P.Li was working at Tencent.

+
Footnote †: Partly done while P.Li was working at Tencent.

+
Footnote †: Partly done while P.Li was working at Tencent.

## 1 Introduction

Pre-trained language models (PLMs), such as BERT Devlin et al. (2019) and GPT Radford et al. (2018) have achieved great performance on various natural language processing (NLP) tasks Han et al. (2021). Recently, people have found that extremely large PLMs can achieve remarkable improvements, and various large PLMs are continually developed Brown et al. (2020); Raffel et al. (2020); Zhang et al. (2021); Zeng et al. (2021); Wei et al. (2021); Sun et al. (2021), which contain up to hundreds of billions of parameters.

Considering the extremely large scale of these state-of-the-art PLMs, conventional full-parameter fine-tuning methods become extremely expensive. Hence, various parameter-efficient tuning methods Houlsby et al. (2019); Ben Zaken et al. (2021); Lester et al. (2021); Li and Liang (2021); Liu et al. (2021) are explored, among which prompt tuning (PT) has attracted broad research attention. PT prepends some _soft prompts_, which are essentially learnable virtual tokens, into the input sequences and only trains them while keeping all the PLM's parameters fixed. The training objective is to generate desired outputs in the same way as the pretraining tasks. PT can match the downstream task performance of fine-tuning with only thousands of tunable parameters Lester et al. (2021) when the PLM has billions of parameters.

Although PT is an effective approach to utilizing extremely large PLMs, it requires much more

Figure 1: We explore prompt transferring across different tasks (cross-task) and PLMs (cross-model) with directly reusing prompts and initializing prompt tuning.



training time than fine-tuning to reach the convergence as shown in Figure 2; hence, it is worthwhile to explore how to improve the efficiency of PT. In this work, we attempt to improve PT via **prompt transfer** across different tasks and models. Knowledge transfer across tasks Vu et al. (2020) and models Qin et al. (2021) have been widely used to improve the efficiency and effectiveness of NLP systems. Intuitively, soft prompts are the only tuned parameters in PT and thus shall concentrate the knowledge required to solve tasks conditioned on PLMs. Thus transferring the trained prompts is promising to accelerate PT.

As shown in Figure 1, we empirically analyze the transferability of prompts across different tasks (_cross-task transfer_ setting) and PLMs (_cross-model transfer_ setting) in this paper. The empirical analysis is conducted on \(17\) NLP tasks of \(6\) types and two representative PLM series: RoBERTa Liu et al. (2019) and T5 Raffel et al. (2020). In cross-task transfer, the prompt transfer can be done by directly reusing the trained prompts of the source task on the target task. However, in cross-model transfer, directly reusing prompts is intractable since the semantic spaces of different PLMs are inconsistent; hence, we develop various **prompt projectors** to project the soft prompts trained on the source PLM to the semantic space of the target PLM. We conduct two lines of experiments: (1) We investigate the **zero-shot transfer performance** and find that the transferability of prompts is influenced by task types. In cross-task transfer, the soft prompts can directly transfer to same-type tasks and achieve non-trivial performance, but poorly transfer to different-type tasks requiring different language skills. In cross-model transfer, we can successfully train a prompt projector with PT on a task, but the trained projector also only well generalizes to the same-type tasks of the projector-training task. (2) To accelerate PT, we propose to **transfer prompts with initialization**. In cross-task transfer, we start PT with the trained soft prompts of similar tasks as initialization. While in cross-model transfer, the initialization is the projected prompts of the same task trained on the source PLM. The two methods are dubbed as \({\rm TPT}_{\textsc{tasks}}\) and \({\rm TPT}_{\textsc{MODEL}}\), which are short for transferable prompt tuning. Experiments show that they can both accelerate PT to some extent and also achieve a certain performance improvement.

Furthermore, we explore why can the prompts transfer and what decides their transferability. To this end, we design various prompt similarity metrics from different perspectives and examine how well they can serve as **transferability indicators**, i.e., how well they correlate with prompt transfer performance. Experiments find that our novel method of measuring prompt similarity via model activations in feed-forward layers is better correlated with prompt transferability than prompt embedding distance-based metrics. This suggests the prompts are essentially stimulating PLM's inner ability distributing among neurons to do specific NLP tasks, and future prompt transfer works should focus more on how the PLMs respond to different prompts' stimulation rather than the prompts' embedding properties.

To summarize, our contributions are three-fold: (1) We thoroughly analyze the transferability of prompts across different tasks and models, and show that improving PT with prompt transfer is possible and promising. (2) We propose to transfer prompts with initialization, which enhances both PT's efficiency and effectiveness. (3) We explore the effectiveness of various prompt similarity metrics serving as transferability indicators and demonstrate how the prompts stimulate PLMs to decide the transferability, which may facilitate further transferable PT research.

## 2 Related Work

Prompt TuningGPT-3 Brown et al. (2020) demonstrates remarkable few-shot performance by prepending textual prompts before the inputs and thus helps the PLM to generate desired outputs of NLP tasks directly. Motivated by this, many works have tried to improve various NLP tasks by creating manually-crafted Schick and Schutze (2021); Mishra et al. (2021) or automatically-searched Jiang et al. (2020); Shin et al. (2020); Gao et al. (2021)_hard prompts_, which are discrete to

Figure 2: Validation accuracies against training time of fine-tuning and PT for RoBERTaLARGE on MNLI. PT takes much more training time.

 

kens but not necessarily human-readable. Furthermore, _soft prompts_Hambardzumyan et al. (2021); Qin and Eisner (2021); Zhong et al. (2021); Liu et al. (2021) are proposed, which are tuneable embeddings rather than tokens in the vocabularies and can be directly trained with task-specific supervision. Lester et al. (2021) demonstrate that prompt tuning (PT) method can match the performance of full-parameter fine-tuning when the PLM has billions of parameters. This suggests that PT is promising to utilize extremely large PLMs. However, the much more training time needed to reach the convergence makes PT inefficient. In this work, we show that prompt transfer can improve the effectiveness to some extent with knowledge transfer, and empirically analyze the transferability of prompts across tasks and PLMs.

Knowledge TransferCross-task knowledge transfer Ruder (2017) has been a long-standing way to improve the effectiveness and efficiency of NLP systems. In the PLM era, some works propose to tune the PLMs on intermediate tasks Phang et al. (2018); Pruksachatkun et al. (2020); Gururangan et al. (2020); Wang et al. (2019); Vu et al. (2020); Poth et al. (2021) before fine-tuning on specific target tasks to achieve certain benefits. Vu et al. (2020) empirically analyze the transferability between tasks in this setting.

These explorations are all for fine-tuning. Considering the potential of PT, we believe the transferability and knowledge transfer methods for PT are worth exploring. As a prior attempt, Lester et al. (2021) demonstrate that PT's cross-domain transferability is stronger than fine-tuning.

Similar to our work, recent work Vu et al. (2021) also explores the cross-task transfer with prompt initialization and prompt similarity metrics based on cosine similarity. However, Vu et al. (2021) focus on improving the effectiveness of PT but we attempt to improve the efficiency. Additionally, we explore more transferability indicators, especially the overlapping rate of activated neurons, and also investigate cross-model transfer, which is inspired by previous cross-model knowledge transfer works such as Net2Net Chen et al. (2016), knowledge distillation Hinton et al. (2015) and knowledge inheritance Qin et al. (2021).

## 3 Preliminary

Here we introduce the basic knowledge about PT (SS 3.1) as well as the downstream tasks (SS 3.2) and models (SS 3.3) investigated in experiments.

### Prompt Tuning

In this work, we study the PT method that is capable of tuning large PLMs Lester et al. (2021); Liu et al. (2021), i.e., we only explore the PT method freezing PLM parameters. PT prepends some virtual tokens, i.e., the _soft prompts_, into the inputs of the PLM to provide knowledge about downstream tasks. The soft prompts are essentially tunable embedding vectors, which are trained with the objective enforcing the PLM to generate desired outputs of the downstream task in the same way of the pre-training objective.

Formally, given an input sequence with \(n\) tokens \(X=\{x_{1},x_{2},\ldots,x_{n}\}\), we first prepend \(l\) randomly initialized soft prompts \(P=\{\mathbf{p}_{1},\mathbf{p}_{2},\ldots,\mathbf{p}_{l}\}\) before them, where \(\mathbf{p}_{i}\in\mathbb{R}^{d}\) is an embedding vector, and \(d\) is the input dimension of the PLM. The training objective is to maximize the likelihood of decoding the desired output \(y\):

\[\mathcal{L}=p(y|P,x_{1},\ldots,x_{n}), \tag{1}\]

where only \(P\) is learnable. For the language understanding tasks, \(y\) is the label token corresponding to the label of \(X\). For the conditional generation tasks, \(y\) is a sequence. Especially, for the models pre-trained with the masked language modeling objective like RoBERTa, we additionally prepend a special [MASK] token before the prompts and train the prompts to let the PLM fill \(y\) into it.

### Investigated NLP Tasks

To comprehensively study the prompt transferability across various NLP tasks, we involve \(17\) diverse tasks, which can be divided into \(6\) types: (1) **Sentiment Analysis (SA)**, including IMDB Maas et al. (2011), SST-2 Socher et al. (2013), laptop Pontiki et al. (2014), restaurant Pontiki et al. (2014), Movie Rationales Movie Zaidan et al. (2008) and TweetEval Tweet) Barbieri et al. (2020); (2) **Natural Language Inference (NLI)**, including MNLI Williams et al. (2018), QNLI Wang et al. (2019) and SNLI Bowman et al. (2015); (3) **Ethical Judgment (EJ)**, including deontology Hendrycks et al. (2021) and justice Hendrycks et al. (2021); (4) **Paraphrase Identification (PI)**, including QQP Sharma et al. (2019) and MRPC Dolan and Brockett (2005); (5) **Question Answering (QA)**, including SQuAD Rajpurkar et al. (2016) and NQ-Open 

(Lee et al., 2019); (6) Summarization (SUM), including Multi-News (Fabbri et al., 2019) and SAMSum Gliwa et al. (2019). Details for these tasks, evaluation metrics, label tokens, implementations are in appendix A.

### Investigated Models

We investigate prompt transferability for two series of PLMs: RoBERTa Liu et al. (2019) and T5Raffel et al. (2020), which represent two mainstream pre-training types: masked language modeling and sequence-to-sequence pre-training. Considering RoBERTa can only predict a single token (or a fixed length of tokens) under prompt tuning paradigm, for the conditional generation tasks (QA and SUM) that output multiple tokens, we only investigate T5. We mainly report results for the two largest versions of PLMs, i.e., RoBERTa\({}_{\text{LARGE}}\) and T5\({}_{\text{XXLL}}\). The more detailed results for the other sizes are attached in appendix.

## 4 Cross-Task Transfer

We empirically study the cross-task transferability of soft prompts (SS 4.1) and try to improve the effectiveness and efficiency of PT with transfer (SS 4.2).

### Zero-shot Transfer Performance

To study the cross-task transferability, we first examine PT's zero-shot transfer performance, i.e., we conduct PT on a source task, then directly reuse the trained prompts on other target tasks and evaluate their performance. The results are shown in Figure 31, from which we can observe that: (1) For the tasks within the same type, transferring soft prompts between them can generally perform well and may even outperform vanilla PT on the target task, especially when the source task has more data (the case of transferring from IMDB to Movie in Figure 3 (a) and transferring from restaurant to laptop in Figure 3 (b)), which demonstrates that it is promising to improve PT's effectiveness and efficiency with knowledge transfer from similar tasks. (2) For the tasks of different types, the transferability of soft prompts among them is generally poor, and transferring soft prompts often achieve similar performance to randomly initialized prompts.

Footnote 1: More results on other PLMs are left in appendix B.1.

(3) However, some tasks can transfer to different-type tasks to some extent, such as the QA and SUM tasks to SA tasks in Figure 3 (b). To understand this, it is worthwhile to explore what controls the transferability between prompts, and we do some preliminary study in SS 6.

### Transfer with Initialization

To improve the effectiveness and efficiency of PT with cross-task transfer, we explore a cross-task transferable prompt tuning (TPT\({}_{\text{TASK}}\)) method, which initializes soft prompts with well-trained prompts of the most similar task and then starts PT.

For a target task, we start TPT\({}_{\text{TASK}}\) with trained prompts of the source task achieving the best zero-shot transfer performance in Figure 3. From the results of the performance and training time comparisons2 in Table 1, we can see TPT\({}_{\text{TASK}}\) can mostly achieve better or comparable performance to vanilla PT starting from random initialization, and TPT\({}_{\text{TASK}}\) generally takes less training time.

Footnote 2: Training time comparisons are left in appendix B.3.

## 5 Cross-Model Transfer

We further study the cross-model transferability of soft prompts. Intuitively, cross-model transfer allows us to train prompts on a small and computationally efficient PLM and use them on a massive and computationally expensive PLM, which will be much more efficient and environmentally. We investigate the feasibility of cross-model transfer on transferring from a source PLM (RoBERTa\({}_{\text{LARGE}}\)) to a larger and heterogeneous target PLM (T5\({}_{\text{XXL}}\)), which shall be the most difficult setting. Appendix C shows the experimental results of other settings. Directly reusing trained soft prompts between different PLMs is infeasible since their embedding spaces are different. Hence, we investigate how to do cross-model prompt projection (SS 5.1) and see the transfer performance (SS 5.2). Furthermore, we explore to improve PT with cross-model transfer initialization (SS 5.3).

### Cross-Model Prompt Projection

To project the trained soft prompts of a PLM to the semantic space of a different PLM, we train projectors with various objectives and examine their effectiveness. A good way to train the cross-model projectors may need some task-specific supervisions, but the trained projector shall generalize to different tasks so that the efficiency for learning the new tasks on the target model could be improved.

Formally, given the prompt of the source PLM \(P^{s}=\{\mathbf{p}_{1}^{s},\dots,\mathbf{p}_{l}^{s}\}\), we concatenate the \(l\) virtual 

tokens into a unified vector \({\bf P}^{s}\in\mathbb{R}^{ld_{s}}\). The projector \({\bf Proj}(\cdot)\) is to project it to \(\widetilde{\bf P}^{s}\in\mathbb{R}^{ld_{t}}\) in the semantic space of the target PLM, where \(d_{s}\) and \(d_{t}\) are the input embedding dimensions of the source and target PLM, respectively. We parameterize the projector with a two-layer perceptron as follows:

\[\widetilde{\bf P}^{s}\!=\!{\bf Proj}({\bf P}^{s})\!=\!{\bf W}_{2}(\sigma({\bf P} ^{s}{\bf W}_{1}\!+\!{\bf b}_{1}))\!+\!{\bf b}_{2}, \tag{2}\]

where \({\bf W}_{1}\in\mathbb{R}^{d_{h}\times lds},{\bf W}_{2}\in\mathbb{R}^{ld_{k} \times d_{h}}\) are trainable matrices, \({\bf b}_{1}\in\mathbb{R}^{dh_{h}},{\bf b}_{2}\in\mathbb{R}^{ld_{t}}\) are biases, \(\sigma\) is a non-linear activation function. We investigate two learning objectives to train the projector3:

Footnote 3: More projector-training details are left in appendix C.1.

Distance MinimizingWe firstly try to learn cross-model projections by minimizing the distance between the projected prompt and the parallel prompt \({\bf P}^{t}\) originally trained on the target PLM with the same task, i.e., the training objective is to minimize their \(L_{2}\)-distance \(\|{\bf Proj}({\bf P}^{s})-{\bf P}^{t}\|_{2}\).

Task TuningWe then try to train the cross-model projector with task-specific supervision signals on

Figure 3: Relative zero-shot transfer performance (zero-shot transfer performance / original PT performance) (%) on the target tasks (columns) of the soft prompts trained on the source tasks (rows) for RoBERTa\({}_{\text{LABGE}}\) and T5XXl-Colors of the task names indicate task types. Blue: SA. Green: NLI. Brown: EJ. Orange: PI. Purple: QA. Gray: SUM. _Random Prompt_ of the last row means the soft prompts are randomly generated without any training.

 

the target PLM. Specifically, we directly tune the projected prompts on some tasks and back propagate the supervision signals to train the projector weights, so that the projector can learn how to stimulate the target PLM and thus may generalize to transfer the prompts of other tasks.

These methods rely on some tasks (parallel trained soft prompts or training data) to train the projector. The projector learning methods are agnostic to the specific training tasks used, and we choose laptop and MNLI in experiments.

### Zero-shot Transfer Performance

The zero-shot transfer performance of various projector-learning methods are shown in Table 24 (a). We can observe that: (1) **Distance Minimizing** works well to transfer the prompts of the projector-training task, but falls back to random performance on the other unseen tasks, which is not practically usable. This is consistent with our findings in SS 6 that the embedding distances do not strongly correlate to prompt transferability. (2) **Task Tuning** performs better and successfully generalizes to same-type unseen tasks of the projector-training tasks (e.g. NLI tasks for the projectors trained with MNLI), which proves the feasibility of practical cross-model prompt transfer. (3) The projectors trained with **Task Tuning** still cannot work for different-type tasks, which may be limited by the cross-task prompt transferability investigated in SS 4.1. This urges further attention to developing universal cross-model projections.

Footnote 4: More results on other PLMs are left in appendix C.2.

### Transfer with Initialization

Similar to SS 4.2, we further study whether the projected soft prompts can initialize PT on the target PLM and accelerate training as well as improve performance. We propose cross-model transferable prompt tuning, TPT\({}_{\text{MODEL}}\), which adopts the **Task Tuning** projectors to project the soft prompts trained on the source PLM into the target PLM and initialize PT with the projected prompts.

The performance and speedup are shown in Table 2 (b). We can see that, for the tasks within the same type of the projector-training task, compared to vanilla PT, TPT\({}_{\text{MODEL}}\) can mostly achieve comparable or better performance with much less training time, which demonstrates that practical cross-model prompt transfer is promising for improving the efficiency and effectiveness of PT.

## 6 Exploring Transferability Indicator

Based on the positive results in cross-task and cross-model transfer, we explore why the soft prompts can transfer across tasks and what decides the transferability between them, which may shed light on the mechanisms behind PT and help to design transferable PT methods. We explore various **prompt similarity metrics** and examine how well do they align with the zero-shot transfer performance. If a similarity metric can well indicate transferability, it suggests the factors considered in designing

\begin{table}
\begin{tabular}{l|c c c c c|c c c|c c c|c c} \hline \multicolumn{1}{c|}{\multirow{2}{*}{**Method**}} & \multicolumn{5}{c|}{**SA**} & \multicolumn{5}{c|}{**NLI**} & \multicolumn{5}{c}{**FJ**} & \multicolumn{2}{c}{**PI**} \\ \cline{2-17}  & IMDB & SST-2 & laptop & restaurant & Movie & Tweet & MNLI & QNLI & SNLI & deontology & justice & QQP & MRPC \\ \hline PT on T5\({}_{\text{XXL}}\) & 96.5 & 97.4 & 76.6 & 88.1 & 97.9 & 72.5 & 90.5 & 95.2 & 93.4 & 87.0 & 92.5 & 90.0 & 86.3 \\ \hline Random Prompt & 49.7 & 49.0 & 19.8 & 17.0 & 51.6 & 15.5 & 31.8 & 49.3 & 31.9 & 51.3 & 50.0 & 36.4 & 67.0 \\ \hline \multicolumn{1}{c|}{\multirow{2}{*}{laptop}} & \multicolumn{5}{c}{(a) Zero-shot Transfer Performance (\%)} \\ \cline{2-17}  & Distance Minimizing & 49.6 & 49.0 & 76.6 & 17.5 & 51.5 & 14.4 & 31.8 & 48.1 & 32.8 & 53.3 & 49.9 & 36.8 & 66.6 \\ Task Tuning & **82.9** & **89.3** & **80.3** & **85.7** & **78.6** & **58.4** & 32.4 & 50.7 & 33.6 & 54.9 & 51.6 & 33.9 & 63.7 \\ \hline \multirow{2}{*}{MNLI} & Distance Minimizing & 49.6 & 50.1 & 19.8 & 18.3 & 51.2 & 15.0 & **90.5** & 49.0 & 32.9 & 50.3 & 49.0 & 36.8 & 65.6 \\ Task Tuning & 49.7 & 48.8 & 19.8 & 17.0 & 51.6 & 16.0 & 89.3 & **82.7** & **88.2** & 49.7 & 50.0 & 36.8 & 67.7 \\ \hline \multicolumn{1}{c|}{\multirow{2}{*}{laptop}} & \multicolumn{5}{c}{(b) Transfer with Initialization (TPNorm)} \\ \hline  & Performance (\%) & 96.5 & 97.4 & 82.9 & 90.3 & 97.4 & 74.4 & 91.0 & 95.4 & 93.4 & 92.5 & 92.5 & 90.0 & 87.9 \\  & Convergence Speedup & \(1.1\) & \(1.7\) & \(1.9\) & \(1.3\) & \(0.6\) & \(1.3\) & \(0.9\) & \(1.0\) & \(1.0\) & \(0.7\) & \(1.1\) & \(1.1\) \\  & Comparable-result Speedup & \(1.0\) & \(19.0\) & \(16.0\) & \(6.0\) & N/A & 2.2 & 3.6 & 1.1 & 6.0 & 6.0 & 0.9 & 1.8 & 3.4 \\ \hline \multirow{2}{*}{MRLI} & Performance (\%) & 96.5 & 97.4 & 82.7 & 88.5 & 95.8 & 74.7 & 91.2 & 95.9 & 93.5 & 94.6 & 92.5 & 90.0 & 87.7 \\  & Convergence Speedup & \(1.0\) & \(1.6\) & \(1.8\) & \(0.9\) & \(0.4\) & \(1.3\) & \(1.0\) & \(1.1\) & \(1.4\) & \(2.0\) & \(1.7\) & \(0.9\) & \(0.9\) \\  & Comparable-result Speedup & \(1.0\) & \(18.0\) & \(15.0\) & \(1.6\) & N/A & 1.5 & 18.0 & 20.0 & 30.0 & 7.5 & 5.0 & 1.5 & 1.9 \\ \hline \end{tabular}
\end{table}
Table 2: Cross-model prompt transfer (RoBERTALAGE to T5XXL) results, including non-transfer baselines (vanilla PT and randomly generated prompts), zero-shot transfer performance of various projectors, and TPT\({}_{\text{MODEL}}\) results (performance, convergence speedup, and comparable-result speedup similar to Table 1). TPT\({}_{\text{MODEL}}\) adopts the Task Tuning projectors to project the soft prompts.

 

this metric decide the prompt transferability. Moreover, the prompt similarity metrics can qualify task similarities using the trained soft prompts as task embeddings and may help in developing cross-task transfer methods. As a straightforward example, if we build a _prompt warehouse_ containing prompts of diverse tasks, we can retrieve prompts of similar tasks for a new task with a certain similarity metric and better improve PT with TPT\({}_{\text{TASK}}\).

### Prompt Similarity Metric

We explore the following two kinds of metrics:

Embedding SimilarityWe firstly regard the trained soft prompts as only embeddings in the vector space and calculate their _Euclidean similarity_ and _cosine similarity_, among which cosine similarity is also explored by Vu et al. (2021).

Given two groups of trained prompts containing \(l\) virtual tokens: \(P^{t_{1}}=\{\mathbf{p}_{1}^{t_{1}},\dots,\mathbf{p}_{l}^{t_{1}}\}\) and \(P^{t_{2}}=\{\mathbf{p}_{1}^{t_{2}},\dots,\mathbf{p}_{l}^{t_{2}}\}\), which correspond to tasks \(t_{1}\) and \(t_{2}\). Firstly, we concatenate the \(l\) virtual tokens for each group and get two concatenation embeddings \(\mathbf{P}^{t_{1}},\mathbf{P}^{t_{2}}\in\mathbb{R}^{ld}\), then we compute Euclidean similarity and cosine similarity of them:

\[\begin{split}\mathrm{E}_{\rm concat}(P^{t_{1}},P^{t_{2}})& =\frac{1}{1+\|\mathbf{P}^{t_{1}}-\mathbf{P}^{t_{2}}\|},\\ \mathrm{C}_{\rm concat}(P^{t_{1}},P^{t_{2}})&=\frac{ \mathbf{P}^{t_{1}}\cdot\mathbf{P}^{t_{2}}}{\|\mathbf{P}^{t_{1}}\|\|\mathbf{ P}^{t_{2}}\|}.\end{split} \tag{3}\]

We further explore a simple way to make the metrics invariant to token positions. We compute Euclidean distances and cosine similarities for every virtual token pairs in the two groups and use the averaged results in the final similarity metrics:

\[\begin{split}\mathrm{E}_{\rm average}(P^{t_{1}},P^{t_{2}})& =\frac{1}{1+\frac{1}{l}\sum\limits_{i=1}^{l}\sum\limits_{j=1}^{l} \|\mathbf{p}_{i}^{t_{1}}-\mathbf{p}_{j}^{t_{2}}\|},\\ \mathrm{C}_{\rm average}(P^{t_{1}},P^{t_{2}})&=\frac {1}{l^{2}}\sum\limits_{i=1}^{l}\sum\limits_{j=1}^{l}\frac{\mathbf{p}_{i}^{t_{ 1}}\cdot\mathbf{p}_{j}^{t_{2}}}{\|\mathbf{p}_{i}^{t_{1}}\|\|\mathbf{p}_{j}^{ t_{2}}\|}.\end{split} \tag{4}\]

Model Stimulation SimilarityIn the second way, we depict their similarities based on how they _stimulate the PLMs_, i.e., we examine the similarities between the responses of PLMs to the two soft prompts. Motivated by Geva et al. (2021) and Dai et al. (2021), which both find that the activation of the neurons in the feed-forward layers of Transformers Vaswani et al. (2017) corresponds to specific model behaviors, we propose to use the _overlapping rate of activated neurons_ as a similarity metric of prompts. Specifically, the feed-forward network \(\operatorname*{FFN}(\cdot)\) in a Transformer layer is:

\[\operatorname*{FFN}(\mathbf{x})=\max(\mathbf{xW}_{1}^{\top}+\mathbf{b}_{1}, \boldsymbol{0})\mathbf{W}_{2}+\mathbf{b}_{2}, \tag{5}\]

where \(\mathbf{x}\in\mathbb{R}^{d}\) is the input embedding, \(\mathbf{W}_{1},\mathbf{W}_{2}\in\mathbb{R}^{d_{m}\times d}\) are trainable matrices, and \(\mathbf{b}_{1},\mathbf{b}_{2}\) are bias vectors. The \(\max(\mathbf{xW}_{1}^{\top}+\mathbf{b}_{1},\boldsymbol{0})\) can be regarded as the non-negative activation values for \(d_{m}\) hidden neurons Geva et al. (2021). We then change all the positive elements of \(\max(\mathbf{xW}_{1}^{\top}+\mathbf{b}_{1},\boldsymbol{0})\) to \(1\) and get the one-hot activation state vector \(\mathbf{s}\).

We feed an input sequence \(\{P_{!}\texttt{
}\}\) into the PLMs, where \(\texttt{
}\) is the special token indicating the start of a sentence. For RoBERTa, a [MASK] is additional prepended. This sequence is in the format of PT inputs but without specific input sentences.

We use the activation states of the positions used to decode outputs, which shall be more task-specific. Specifically, for T5, we use the decoder module's activation states at the first position. For RoBERTa, we use the activation states of [MASK]. Finally, we concatenate the activation states of PLM's \(L\) layers to get the overall activation states:

\[\mathrm{AS}(P)=[\mathbf{s}_{1};\mathbf{s}_{2};...;\mathbf{s}_{L}]. \tag{6}\]

We can only retrieve the activation states of a part of layers in the similarity computation. In experiments, we find that the higher layers tend to be more task-specific, which is consistent with the probing results Liu et al. (2019). Hence we use the activation states of the top \(3\) layers5 in experiments below. We calculate the overlapping rate of activated neurons \(\mathrm{ON}(P^{t_{1}},P^{t_{2}})\) between

\begin{table}
\begin{tabular}{c|c|c c} \hline \hline \multirow{2}{*}{**Model**} & \multirow{2}{*}{**Metric**} & **Same** & **Different** \\  & & **Task** & **Tasks** \\ \hline \multirow{8}{*}{RoBERTa\({}_{\text{LLAGE}}\)} & \(\mathrm{E}_{\rm concat}\) & \(9.4\) & \(6.8\) \\  & \(\mathrm{E}_{\rm average}\) & \(41.6\) & \(37.6\) \\  & \(\mathrm{C}_{\rm concat}\) & \(47.6\) & \(31.7\) \\  & \(\mathrm{C}_{\rm averag}\) & \(1.7\) & \(1.1\) \\  & ON & \(39.4\) & \(21.4\) \\ \hline \multirow{8}{*}{T5\({}_{\text{XXL}}\)} & \(\mathrm{E}_{\rm concat}\) & \(0.5\) & \(0.3\) \\  & \(\mathrm{E}_{\rm average}\) & \(4.0\) & \(3.4\) \\ \cline{1-1}  & \(\mathrm{C}_{\rm concat}\) & \(29.4\) & \(3.4\) \\ \cline{1-1}  & \(\mathrm{C}_{\rm average}\) & \(4.0\) & \(2.1\) \\ \cline{1-1}  & ON & \(62.0\) & \(46.1\) \\ \hline \hline \end{tabular}
\end{table}
Table 3: The average values (%) of the \(5\) similarity metrics for prompt pairs of the same task (trained with \(3\) different random seeds) and different tasks.

 

the trained soft prompts of task \(t_{1}\) and \(t_{2}\) with the cosine similarity:

\[\mathrm{ON}(P^{t_{1}},P^{t_{2}})=\frac{\mathrm{AS}(P^{t_{1}})\cdot\mathrm{AS}( P^{t_{2}})}{||\mathrm{AS}(P^{t_{1}})|\lVert\mathrm{AS}(P^{t_{2}})\rVert}. \tag{7}\]

### Experimental Results

To evaluate the effectiveness of the above similarity metrics of soft prompts, we (i) test whether the similarity metrics can distinguish the trained prompts of the same tasks and different tasks, and (ii) examine whether these metrics align with the zero-shot transfer performance.

Regarding (i), we compare the similarities of the investigated metrics for two trained prompts within the same task (trained with different random seeds) and between different tasks in Table 3. From the results, we can observe that all the metrics work well to distinguish the prompts of the same task and different tasks. This suggests that the trained soft prompts of different tasks form distinguishable clusters in the embedding space and also stimulate different abilities within the PLM.

Moreover, to evaluate (ii), how well the similarity metrics align with the cross-task transfer performance, we quantify the correlations between the similarities and zero-shot transfer performance in Figure 3. Specifically, for each target task's prompt, we rank various source tasks' prompts with similarity scores and zero-shot transfer performance and then compute the Spearman's rank correlation Spearman (1987) between the two ranks generated by these two ways. The overall results are shown in Table 46. We can see that: (1) The _overlapping rate of activated neurons_ (\(\mathrm{ON}\)) metric works better than all the embedding similarities, which suggests that model stimulation is more important for prompt transferability than embedding distances. (2) \(\mathrm{ON}\) works much worse on T5XXL (\(11\)B parameters) than on RoBERTaLARGE (\(330\)M parameters). We guess this is because larger PLMs have higher redundancy Aghajanyan et al. (2021), which means prompts can activate different redundant neurons to do similar jobs and thus influence the sensitivity of \(\mathrm{ON}\) metric. This is supported by the experiments showing that the Spearman's correlation scores of \(\mathrm{ON}\) drop with the increase of PLM scales (Figure 4), from which we can see \(\mathrm{C_{average}}\) also exhibits a similar trend. We encourage future work to explore how to overcome the PLM redundancy for better transferable PT. As a preliminary trial, we find that by taking the intersection of activation states of \(3\) prompts trained with different random seeds, \(\mathrm{ON}\)'s correlation score on T5XXL raises from \(36.9\%\) to \(46.3\%\).

Footnote 6: The detailed results by task types are left in appendix D.2.

We further explore whether the prompt similarity metrics also work in the cross-model transfer setting by testing whether they work between the projected prompts and original prompts of the same task. In Table 5, we show the similarities of prompts projected with **Task Tuning** projectors by the two best metrics \(\mathrm{C_{average}}\) and \(\mathrm{ON}\). We can see: (1) ON metric shows that the projected prompts are highly similar to the original prompts within the same type of projector-training tasks but are not so similar to different-type tasks, which is quite consistent with the cross-m

\begin{table}
\begin{tabular}{l c r} \hline \hline
**Metric** & RoBERTaLARGE & T5XXL \\ \hline E\({}_{\text{concat}}\) & \(22.6\) & \(12.9\) \\ E\({}_{\text{average}}\) & \(2.8\) & \(-2.5\) \\ C\({}_{\text{concat}}\) & \(24.8\) & \(31.6\) \\ Ca\({}_{\text{average}}\) & \(44.7\) & \(33.5\) \\ ON & **49.7** & **36.9** \\ \hline \hline \end{tabular}
\end{table}
Table 4: The Spearman’s rank correlation scores (%) between various similarity metrics and cross-task zero-shot transfer performance change along with the parameter size of T5.

Figure 4: Spearman’s correlation scores of \(\mathrm{ON}\) and \(\mathrm{C_{average}}\) with cross-task zero-shot transfer performance change along with the parameter size of T5.

 

performance in Table 2. (2) However, \({\rm C}_{\rm average}\) cannot reflect this phenomenon, which shows that the perspective of model stimulation is more promising for understanding transferability again.

## 7 Conclusion

We empirically investigate the transferability of prompts in this paper. In the cross-task setting, we find that soft prompts can transfer to similar tasks without training. In the cross-model setting, we successfully project prompts into the space of other PLMs. Further, we utilize trained prompts of other tasks or other PLMs as initialization to significantly accelerate training and improve effectiveness. Moreover, we explore various prompt transferability indicators and show that how the prompts stimulate PLMs is important to transferability. We hope the empirical analyses and the _model stimulation_ idea can facilitate further research on transferable and efficient PT.

## Author Contributions

Yusheng Su and Xiaozhi Wang mainly initiated and organized this research. First, Xiaozhi Wang proposed the research idea. Yusheng Su, Xiaozhi Wang, Yujia Qin and Yankai Lin discussed most of the investigation settings. Yusheng Su, Chi-Min Chan, and Kaiyue Wen conducted the experiments. Xiaozhi Wang and Yusheng Su wrote the paper. Yankai Lin, Yujia Qin, Huadong Wang, Zhiyuan Liu, Peng Li, Juanzi Li, and Lei Hou revised the paper. Maosong Sun and Jie Zhou provided valuable advice and proofread this paper.

## Acknowledgment

This work is supported by the National Key Research and Development Program of China (No. 2020AAA0106500), Beijing Academy of Artificial Intelligence (BAAI), Institute Guo Qiang at Tsinghua University, and International Innovation Center of Tsinghua University, Shanghai China. We thank the anonymous reviewers for their valuable comments.

## References

* Agahajanyan et al. (2021) Armen Aghajanyan, Sonal Gupta, and Luke Zettlemoyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 7319-7328. External Links: Link Cited by: SS4.
* J. Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. ArXiv preprintabs/1607.06450. External Links: Link Cited by: SS4.
* F. Barbieri, J. Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves (2020)TweetEval: unified benchmark and comparative evaluation for tweet classification. In Findings of the Association for Computational Linguistics: EMNLP 2020, Online, pp. 1644-1650. External Links: Link Cited by: SS4.
* E. Ben Zaken, S. Ravfogel, and Y. Goldberg (2021)BitRT: simple parameter-efficient fine-tuning for transformer-based masked language models. ArXiv preprintabs/2106.10199. External Links: Link Cited by: SS4.
* S. R. Bowman, G. Angeli, Christopher Potts, and C. D. Manning (2015)A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, Lisbon, Portugal, pp. 632-642. External Links: Link Cited by: SS4.
* T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. Henighan, Rewon Child, Aditya Ramesh, M. Ziegler, Jeffrey Wu, Clemens Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, K. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei (2020)Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, H. Larochelle, W. Teh, X. Zhang, and Y. Jia (Eds.), Vol. , , pp. 1000-1006. External Links: Link Cited by: SS4.
* T. Chen, I. J. Goodfellow, and J. Shlens (2016)Net2net: accelerating learning via knowledge transfer. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings, External Links: Link Cited by: SS4.
* D. Dai, L. Dong, Y. Hao, Z. Sui, and F. Wei (2021)Knowledge neurons in pretrained transformers. ArXiv preprintabs/2104.08696. External Links: Link Cited by: SS4.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 4171-4186. External Links: Link Cited by: SS4.

 

William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005), Cited by: SS1.
* A. Fabbri, I. Li, T. She, S. Li, and D. Radev (2019)Multi-news: a large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 1074-1084. External Links: Link, Document Cited by: SS3.1.
* T. Gao, A. Fisch, and D. Chen (2021)Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 3816-3830. External Links: Link, Document Cited by: SS1.
* M. Geva, R. Schuster, J. Berant, and O. Levy (2021)Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 5484-5495. External Links: Link, Document Cited by: SS1.
* B. Gliwa, I. Mochol, M. Biesek, and A. Wawer (2019)SAMSum corpus: a human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, Hong Kong, China, pp. 70-79. External Links: Link, Document Cited by: SS1.
* S. Gururangan, A. Marasovic, S. Swayamb Gupta, K. Lo, I. Beltagy, D. Downey, and N. A. Smith (2020)Don't stop retraining: adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 8342-8360. External Links: Link, Document Cited by: SS1.
* K. Hambardzumyan, H. Khachatrian, and J. May (2021)WARP: word-level adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 4921-4933. External Links: Link, Document Cited by: SS1.
* X. Han, Z. Zhang, N. Ding, Y. Gu, X. Liu, Y. Huo, J. Qiu, L. Zhang, W. Han, M. Huang, et al. (2021)Pre-trained models: past, present and future. Proceedings of AI Openen. Cited by: SS1.
* D. Hendrycks, C. Burns, S. Basart, A. Critch, J. Li, D. Song, and J. Steinhardt (2021)Aligning ai with shared human values. In Proceedings of ICLR, Cited by: SS1.
* G. Hinton, O. Vinyals, and J. Dean (2015)Distilling the knowledge in a neural network. ArXiv preprintabs/1503.02531. External Links: Link, 1503.02531 Cited by: SS1.
* N. Houlsby, A. Giurgiu, Stanislaw Jastrzewski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly (2019)Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, Vol. 97 of Proceedings of Machine Learning Research, pp. 2790-2799. External Links: Link, Document Cited by: SS1.
* Z. Jiang, F. F. Xu, J. Araki, and G. Neubig (2020)How can we know what language models know?. Transactions of the Association for Computational Linguistics8, pp. 423-438. External Links: Link, ISSN 1558-1222, Link, Document Cited by: SS1.
* K. Lee, M. Chang, and K. Toutanova (2019)Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, Florence, Italy, pp. 6086-6096. External Links: Link, Document Cited by: SS1.
* B. Lester, R. Al-Rfou, and N. Constant (2021)The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 3045-3059. External Links: Link, Document Cited by: SS1.
* X. L. Li and P. Liang (2021)Prefix-tuning: optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), Online, pp. 4582-4597. External Links: Link, Document Cited by: SS1.
* C. Lin (2004)ROUGE: a package for automatic evaluation of summaries. In Text Summarization Branches Out, Barcelona, Spain, pp. 74-81. External Links: Link, ISBN 978-1-467-3861-1, Document Cited by: SS1.
* N. F. Liu, M. Gardner, Y. Belinkov, M. E. Peters, and N. A. Smith (2019)Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), Minneapolis, Minnesota, pp. 1073-1094. External Links: Link, Document Cited by: SS1.
* X. Liu, Y. Zheng, Z. Du, M. Ding, Y. Qian, Z. Yang, and J. Tang (2021)Gpt understands, too. ArXiv preprintabs/2103.10385. External Links: Link, 2103.10385 Cited by: SS1.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019) 

RoBERTa: A robustly optimized bert pretraining approach. In Proceedings of ICLR, Cited by: SS2.
* I. Loshchilov and F. Hutter (2019)Decoupled weight decay regularization. In International Conference on Learning Representations, External Links: Link Cited by: SS3.4.
* A. L. Maas, R. E. Daly, P. T. Pham, D. Huang, A. Y. Ng, and C. Potts (2011)Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, Portland, Oregon, USA, pp. 142-150. External Links: ISBN 978-1-649-3895-3 Cited by: SS2.
* S. Mishra, D. Khashabi, C. Baral, Y. Choi, and H. Hajishirzi (2021)Reframing instructional prompts to gptk's language. ArXiv preprintabs/2109.07830. Cited by: SS2.
* J. Phang, T. Fevry, and S. R. Bowman (2018)Sentence encoders on stilts: supplementary training on intermediate labeled-data tasks. ArXiv preprintabs/1811.01088. Cited by: SS2.
* M. Pontiki, D. Galanis, J. Pavlopoulos, H. Papageorgiou, I. Androutsopoulos, and S. Manandhar (2014)SemEval-2014 task 4: aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), Dublin, Ireland, pp. 27-35. External Links: ISBN 978-1-649-3895-3 Cited by: SS2.
* C. Poth, J. Pfeiffer, A. Ruckle, and I. Gurevych (2021)What to pre-train on? efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, Online and Punta Cana, Dominican Republic, pp. 10585-10605. External Links: Document, Link Cited by: SS2.
* Y. Pruksaachatkin, J. Phang, H. Liu, P. M. Hutt, X. Zhang, B. Yuanzhe Pang, C. Vania, K. Kann, and S. R. Bowman (2020)Intermediate-task transfer learning with pretrained language models: when and why does it work?. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, Online, pp. 5231-5247. External Links: Link Cited by: SS2.
* G. Qin and J. Eisner (2021)Learning how to ask: querying lms with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 5203-5212. External Links: Link Cited by: SS2.
* Y. Qin, Y. Lin, J. Yi, J. Zhang, X. Han, Z. Zhang, Y. Su, Z. Liu, P. Li, M. Sun, and J. Zhou (2021)Knowledge inheritance for pre-trained language models. arXiv preprintabs/2105.13880. Cited by: SS2.
* A. Radford, K. Narasimhan, T. Salimans, and I. Sutskever (2018)Improving language understanding by generative pre-training. arXiv e-prints. Cited by: SS2.
* C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P. J. Liu (2020)Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research21, pp. 1-67. External Links: ISSN 1093-690X, Link, Document Cited by: SS2.
* P. Rajpurkar, J. Zhang, K. Lopyrev, and P. Liang (2016)SOuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, Austin, Texas, pp. 2383-2392. External Links: Link, Document Cited by: SS2.
* S. Ruder (2017)An overview of multi-task learning in deep neural networks. ArXiv preprintabs/1706.05098. Cited by: SS2.
* T. Schick and Hinrich Schutze (2021)Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, Online, pp. 255-269. External Links: Link, Document Cited by: SS2.
* T. Schick and H. Schutze (2021)It's not just size that matters: small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Online, pp. 2339-2352. External Links: Link, Document Cited by: SS2.
* L. Sharma, L. Graesser, N. Nangia, and U. Evci (2019)Natural language understanding with the quora question pairs dataset. ArXiv preprintabs/1907.01041. Cited by: SS2.
* T. Shin, Y. Razeghi, R. L. Logan IV, E. Wallace, and S. Singh (2020)AutoPrompt: eliciting knowledge from language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online, pp. 4222-4235. External Links: Link, Document Cited by: SS2.
* R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Ng, and C. Potts (2013)Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, Seattle, Washington, USA, pp. 1631-1642. External Links: Link, Document Cited by: SS2.
* C. Spearman (1987)The proof and measurement of association between two things. In Proceedings of The American journal of psychology, Vol. 100, pp. 441-471.

 

Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibo Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. _ArXiv preprint_, abs/2107.02137.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA_, pages 5998-6008.
* Vu et al. (2021) Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. _ArXiv preprint_, abs/2110.07904.
* Vu et al. (2020) Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 7882-7926, Online. Association for Computational Linguistics.
* Wang et al. (2019a) Alex Wang, Jan Hula, Patrick Xia, Raghavendra Papagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherhi Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 4465-4476, Florence, Italy. Association for Computational Linguistics.
* Wang et al. (2019b) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In _7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019_. OpenReview.net.
* Wei et al. (2021) Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. _ArXiv preprint_, abs/2109.01652.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
* Xu et al. (2015) Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. _ArXiv preprint_, abs/1505.00853.
* Zaidan et al. (2008) Omar F. Zaidan, Jason Eisner, and Christine Piatko. 2008. Machine learning with annotator rationales to reduce annotation cost. In _Proceedings of NeurIPS (Workshop)_.
* Zeng et al. (2021) Wei Zeng, Xiaohe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Pangu\(\alpha\): Large-scale autoregressive pretrained chinese language models with auto-parallel computation. _ArXiv preprint_, abs/2104.12369.
* Zhang et al. (2021) Zhengyan Zhang, Yucain Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, et al. 2021. Cpm-2: Large-scale cost-effective pre-trained language models. _ArXiv preprint_, abs/2106.10715.
* Zhong et al. (2021) Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5017-5033, Online. Association for Computational Linguistics.

 

Basic Setup for Various Tasks

### Dataset and Task

Sentiment Analysis (SA)SA is the task of classifying sentiment polarities for a given sentence. We select IMDB Maas et al. (2011), SST-2 Socher et al. (2013), SemEval/laptop Pontiki et al. (2014), SemEval/restaurant Pontiki et al. (2014), Movie Rationales Movie Jaidan et al. (2008), and TweetEval Tweet Barbieri et al. (2020) for our experiments.

Natural Language Inference (NLI)NLI is the task of determining whether a hypothesis is entailed or contradicted by a given sentences (premise, hypothesis). We select MNLI Williams et al. (2018), QNLI Wang et al. (2019b), and SNLI Bowman et al. (2015) for our experiments.

Ethical Judgment (EJ)EI is the task of deciding whether a sentence is ethically acceptable. We select Ethics/deontology Hendrycks et al. (2021) and Ethics/justice Hendrycks et al. (2021) for our experiments.

Paraphrase Identification (PI)PI is the task of classifying whether a pair of sentences are semantically identical. We select QQP Sharma et al. (2019) and MRPC Dolan and Brockett (2005) for our experiments.

Question Answering (QA)QA is the task of answering a question. We choose SQuAD Rajpurkar et al. (2016) and NQ-Open Lee et al. (2019) to analyze. For SQuAD, a PLM captures the answer from the content. As for NQ-Open, a PLM directly generates the answer without the content.

Summarization (SUM)SUM is the task of summarizing a given article and generating the abstract. We select Multi-News Fabbri et al. (2019), and SAMSum Gliwa et al. (2019) for our experiments.

### Evaluation Metrics

For classification tasks (SA, NLI, EJ, and PI), we use accuracy (Acc.) as their evaluation metric. As for generation tasks (QA and SUM), we utilize F1 and ROUGE-L Lin (2004), respectively.

### Prompt Tuning Setting

In the experiments, for all the investigated tasks, we use AdamW Loshchilov and Hutter (2019) as the optimizer and set the learning rate as \(0.001\). We set the length of soft prompts \(l\) as \(100\). All the soft prompts are randomly initialized and optimized with Equation 1. In the inference stage, RoBERTa predicts the label tokens at the [MASK] position and T5 directly uses its decoder to do generation. For the classification tasks (SA, NLI, EJ and PI), we obtaining answers in a ranking manner, i.e., we rank the label tokens by their likelihoods and regard the PLMs as predict the label of the label token with highest likelihood. For the conditional generation tasks (QA and SUM), we directly take the outputs of PLMs as their answers.

### Label Tokens

The used label tokens for the classification tasks (SA, NLI, EJ, PI) are shown in Table 6. For generation tasks (QA, SUM), the desired output is just the annotated answers.

## Appendix B Cross-Task Transfer

### More Zero-shot transfer performance

In SS 4.1, we report the zero-shot transfer performance (relative performance) on RoBERTaLAGE and T5XXLL Here, we investigate the zero-shot transfer performance on other sizes of RoBERTa and T5, which are shown in Figure 5. According to these results, we can find that the transferability of soft prompts between the tasks of different types is generally poor, which is consistent with the conclusion in SS 4.1.

\begin{table}
\begin{tabular}{c|c} \hline \hline
**Task** & **Label Tokens** \\ \hline  & Sentiment Analysis (SA) \\ \hline IMDB & positive, negative \\ SST-2 & positive, negative \\ laptop & positive, moderate, negative \\ restaurant & positive, moderate, negative \\ Movie & positive, negative \\ Tweet & positive, moderate, negative \\ \hline  & Natural Language Inference (NLI) \\ \hline MNLI & yes, neutral, no \\ QNLI & yes, no \\ SNLI & yes, neutral, no \\ \hline  & Ethical Judgment (EJ) \\ \hline deontology & acceptable, un \\ justice & acceptable, un \\ \hline  & Paraphrase Identification (PI) \\ \hline QQP & true, false \\ MRPC & true, false \\ \hline \hline \end{tabular}
\end{table}
Table 6: Label tokens of classification tasks.

 

* [14] M. C. B. Krizhevsky, A. Vedaldi, and R. Girshick. Learning from image recognition using a large scale database. In _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pages 237-250, 2016.
* [15] J. L. Friedman, S. G. Hinton, D. P. Salakhutdinov, and I. Sutskever. Dropout: a simple way to prevent overfitting by reducing the number of parameters in the training set. In _Advances in Neural Information Processing Systems (IJCNN)_, pages 89-92, 2016.
* [16] T. H. Lee, Y. Bengio, and E. Rabinovich. Training deep neural networks for large data sets. In _NIPS_, volume 1, pages 1398-1406, 2016.
* [17] W. Li, X. Zhang, Z. Ren, and Q. Weinberger. Deep residual learning with convolutional neural networks. In _ICML_, pages 1398-1409, 2016.

[MISSING_PAGE_POST]

un. Deep residual learning with convolutional 



### Unifying Label Tokens

We hypothesize that the poor transferability between different task types may result from the fact that different-type tasks usually use different label tokens, e.g., yes and no are for NLI tasks while positive and negative are for SA tasks. To verify whether this factor influences the transferability, we unify the label tokens of different tasks into the same set of numbers (\(1\), \(2\), ...) and choose RoBERTaBASE for the experiments. In Figure 6, we can observe that the transferability between different-type tasks are generally not improved in this way. This indicates that different-type tasks surely require distinct abilities, which prohibits reusing prompts between them.

### Speedup Calculation

In this paper, we compute convergence speedup and comparable-result speedup as follows:

\[\begin{split}\text{Convergence Speedup}(\mathrm{x})=\frac{ \text{PT convergence time}}{\text{TPT convergence time}},\\ \text{Comparable-result Speedup}(\mathrm{x})=\\ \text{PT convergence time}\end{split} \tag{8}\]

time of TPT achieving comparable result to PT.

We calculate the training loss and the evaluation score per \(100\) steps during the training. When the training loss stops dropping and the evaluation score stops increasing for \(300\) steps, we set the point as the convergence point. For the convergence speedup in Equation 8, the PT convergence time is divided by the TPT convergence time. As for the comparable-result speedup in Equation 8, the PT convergence time are divided by the time of TPT achieving comparable performance to PT.

## Appendix C Cross-Model Transfer

### Implementation Details of Projector

As mentioned in SS 5.1, we give the prompt of the source PLM, \(P^{s}=\{\mathbf{p}_{1}^{s},\ldots,\mathbf{p}_{l}^{s}\}\), and concatenate its \(l\) virtual tokens into a unified vector \(\mathbf{P}^{s}\in\mathbb{R}^{ld_{s}}\), where \(d_{s}\) is the hidden size of the source PLM. To transfer \(\mathbf{P}^{s}\) to the target PLM whose hidden size is \(d_{t}\), we design a projection function \(\mathbf{Proj}(\cdot)\) parameterized by a two-layer perceptron as follows:

\[\tilde{\mathbf{P}}^{s}\!=\!\mathbf{Proj}(\mathbf{P}^{s})\!=\!\mathbf{W}_{2}( \sigma(\mathbf{P}^{s}\mathbf{W}_{1}\!+\!\mathbf{b}_{1}))\!+\!\mathbf{b}_{2}, \tag{9}\]

where \(\mathbf{W}_{1}\in\mathbb{R}^{d_{h}\times ld_{s}},\mathbf{W}_{2}\in\mathbb{R}^ {ld_{t}\times d_{h}}\) are trainable matrices, \(\mathbf{b}_{1}\in\mathbb{R}^{db_{h}},\mathbf{b}_{2}\in\mathbb{R}^{ld_{t}}\) are biases, \(\sigma\) is a non-linear activation function. For training configurations of projector, the optimizer is AdamW Loshchilov and Hutter (2019), the training batch size is \(16\), the learning rate is \(0.005\)

Figure 6: To exclude the poor transferability, which may result from the fact that different-type tasks use different label tokens, we unify the label tokens of different tasks into the same set of numbers (\(1\), \(2\), ...) and choose RoBERTaBASE for the experiments. From Figure (a) and (b), we observe that the transferability between different-type tasks are still generally not improved in this way. This indicates that different-type tasks surely require distinct abilities.



and the inner hidden size \(d_{h}\) is \(768\). In this paper, we investigate cross-model transfer among various PLMs including BERTBASE, RoBERTBABSI, RoBERTALARG, T5SMALL, T5Base, and T5XXL, whose hidden sizes are \(768\), \(768\), \(1024\), \(512\), \(768\), and \(1024\), respectively. Besides, for non-linear activation functions, we have tried tanh and LeakyReLUXu et al. (2015), and find their performance on various PLMs are similar. The reported results are based on LeakyReLU.

### More Zero-shot Transfer Performance

In SS 5.2, we showed the zero-shot transfer performance of various projector-learning methods in the setting of transferring from RoBERTALAGER to T5XXL. We explore more cross-model transfer settings here, which are transferring between various PLMs in different scales and heterogeneous frameworks, including from BERTBASE to RoBERTBASE, ROBERTBAESI, RoBERTBaASE to RoBERTLAARGE, and from T5BASE to T5XXL. We can find that the results in Table 7 are all consistent with SS 5.2.

### Technical Details of TPTMODEL

In SS 5.3, we demonstrate cross-model transferrable prompt tuning (TPT MODEL) can well improve performance and reduce training time.

However, when we apply TPTMODEL to more PLMs, we find that the projected prompts may have quite different \(L_{2}\) norm values with the original prompts, especially for the small-scale PLMs (e.g., from BERTB BASE to RoBERTBASE). Specifically, we obtain the projected prompts with the trained Task Tuning projector, and find that the projected prompts are hard to optimize in some tasks as shown in Figure 7[Without LayerNorm]. Thus, we attempt to add the layer normalization operation Ba et al. (2016)LayerNorm into the projectors to regularize the norm of the projected prompt as follows:

\[\mathbf{\tilde{P}^{s}}=\texttt{LayerNorm}(\mathbf{Proj}(\mathbf{P^{s}})). \tag{10}\]

By the LayerNorm, the projected prompts can work well on TPTMODEL and achieve better performance and speedup as shown in Figure 7[With LayerNorm]. Interestingly, although prompts projected by the projectors [Without LayerNorm] are hard to be trained in TPTMODEL, they can achieve similar zero-shot transfer performance with the prompts projected by the projectors [With LayerNorm] in Table 8.

 



## Appendix D Transferability Indicator

### Effectiveness of Similarity Metrics

We categorize all prompts into three groups: same tasks (prompts trained with different seeds on the same dataset), same-type tasks, and different-type tasks. Table 9 shows that all the similarity metrics successfully distinguish task types.

### Correlation Between Prompt

Transferability and Prompt Similarity

In SS 6, we provide the overall averaged Spearman's rank correlation scores (%) between various similarity metrics and zero-shot transfer performance

\begin{table}
\begin{tabular}{c|c c c c c|c c c|c c|c c c} \hline \multirow{2}{*}{**Method**} & \multicolumn{5}{c|}{**SA**} & \multicolumn{3}{c|}{**NLI**} & \multicolumn{3}{c|}{**EJ**} & \multicolumn{1}{c}{**PI**} \\ \cline{2-14}  & IMDB & SST-2 & laptop & restaurant & Movie & Tweet & MNLI & QNLI & SNLI & decontiology & justice & QQP & MRPC \\ \hline PT on RoBERTa\({}_{\text{BASE}}\) & 89.9 & 93.8 & 77.3 & 80.7 & 79.2 & 74.5 & 80.6 & 90.5 & 88.5 & 72.9 & 70.0 & 86.9 & 83.9 \\ \hline \multicolumn{14}{l}{[Without LayerNorm]} \\ \hline Task Tuning (IMDB, laptop) & **86.5** & **84.9** & **73.4** & **75.3** & **76.6** & 47.7 & 31.8 & 52.0 & 32.9 & 50.3 & 50.0 & 37.6 & 67.5 \\ Task Tuning (MNLI) & 66.6 & 70.4 & 53.0 & 43.8 & 57.8 & 47.9 & **82.4** & **74.9** & **78.1** & 50.4 & 49.9 & 45.3 & 70.1 \\ \hline \multicolumn{14}{l}{[With LayerNorm]} \\ \hline Task Tuning (IMDB, laptop) & **88.2** & **82.2** & **76.3** & **77.9** & **73.4** & 43.6 & 32.0 & 47.9 & 32.8 & 49.8 & 49.4 & 50.2 & 47.7 \\ Task Tuning (MNLI) & 50.9 & 52.0 & 11.9 & 13.1 & 45.8 & 18.2 & **80.0** & **74.9** & **80.0** & 50.4 & 49.9 & 36.8 & 68.1 \\ \hline \end{tabular}
\end{table}
Table 8: We find that the zero-shot performances of prompts projected by two Task Tuning projectors ([With LayerNorm] and [Without LayerNorm]) are close. **Bold** represents non-trivial performance.

Figure 7: The (—) represents vanilla PT on RoBERTa\({}_{\text{BASE}}\). As for (—), it utilizes projected prompts from BERT\({}_{\text{BASE}}\) as initializations to conduct PT on RoBERTa\({}_{\text{BASE}}\). The projected prompts respectively come from two different Task Tuning projectors: [Without LayerNorm] and [With LayerNorm].



of soft prompts for RoBERTaLARGE and T5XXL.

Here, we further show Spearman's rank correlation scores grouped by the task types on more PLMs. The results are shown in Table 10 and Table 11.

### PLMs' Redundancy Influence Indicators

From Table 10, we find that the correlation between prompt transferability and prompt similarity will drop with the increase of PLM size. We guess that this phenomena may result from PLMs' high redundancy Aghajanyan et al. (2021).

To try to overcome this, we simultaneously utilize the prompts trained with three random seeds on the same dataset and take their intersection of activation states as the activated neurons into the similarity (ON) computation. This similarity is called ON**I**. By using it, the correlation score of ON can significantly raise as shown in Table 10.

### Overlapping Rate of Activated Neurons in Different Layers

To further understand model stimulation in PLMs, we investigate ON in different layers of PLMs. Specifically, on RoBERTaBASE, we measure the similarity between different prompts with activation states of from 1 to 3 layers (Figure 8), from 4 to 6 layers (Figure 9), from 7 to 9 layers (Figure 10), from 10 to 12 layers (Figure 11), and all

\begin{table}
\begin{tabular}{l|r r r r r r|c} \hline \hline

- & **25.3** \\ ON\({}_{\textbf{I}}\) (Top 3) &
- & **46.3** \\ ON\({}_{\textbf{I}}\) (All 24) &
- & **40.0** \\ \hline \end{tabular}
\end{table}
Table 10: Spearman’s rank correlation scores (%) between various similarity metrics and zero-shot transfer performance of soft prompts for various scales of T5 and ON\({}_{\textbf{I}}\) as introduced in appendix D.3.

\begin{table}
\begin{tabular}{l|r|r|r} \hline \hline
**Metric** & Same & Same-type & Different-type \\  & Tasks & Tasks & Tasks \\ \hline \multicolumn{6}{c}{RoBERTaLARG} \\ \hline E\({}_{\text{concat}}\) & 9.4 & 9.4 & 6.8 \\ E\({}_{\text{average}}\) & 41.6 & 41.4 & 37.6 \\ C\({}_{\text{concat}}\) & 47.6 & 45.3 & 31.7 \\ Cav\({}_{\text{average}}\) & 1.7 & 1.3 & 1.1 \\ ON (Bottom 3) & 42.8 & 43.3 & 39.1 \\ ON (Top 3) & 39.4 & 28.2 & 21.4 \\ ON (All 24) & 40.0 & 35.8 & 29.6 \\ \hline \multicolumn{6}{c}{T5XXL (Decoder Module)} \\ \hline E\({}_{\text{concat}}\) & 0.5 & 0.5 & 0.3 \\ E\({}_{\text{average}}\) & 4.0 & 5.1 & 3.4 \\ C\({}_{\text{concat}}\) & 29.4 & 2.8 & 2.4 \\ C\({}_{\text{average}}\) & 4.0 & 2.6 & 2.1 \\ ON (Bottom 3) & 80.3 & 75.4 & 76.3 \\ ON (Top 3) & 62.0 & 52.7 & 46.1 \\ ON (All 24) & 60.8 & 54.0 & 49.2 \\ \hline \end{tabular}
\end{table}
Table 9: The average values (%) of the \(5\) similarity metrics for prompt pairs within the same task (trained with \(3\) different random seeds) and between different tasks (of the same type and different types) on RoBERTaLARGE and T5XXL.

 

We find that the activated neurons are common in the bottom layers but tend to be more task-specific in top layers, which is consistent with the findings of previous works (Liu et al., 2019).

 

Figure 8: ON in 1 - 3 layers of RoBERTa\({}_{\text{BASE}}\).

Figure 9: ON in 4 - 6 layers of RoBERTa\({}_{\text{BASE}}\).

 

Figure 12: ON in all 12 layers of RoBERTa\({}_{\text{Base}}\).

Figure 11: ON in 10 - 12 layers of RoBERTa\({}_{\text{BASE}}\).