SlowGuess/NAACL_2022_OCR huggingface dataset:
Yusheng Su $^{1,3*}$ , Xiaozhi Wang $^{1,3*}$ , Yujia Qin $^{1,3}$ , Chi-Min Chan $^{1,3}$ , Yankai Lin $^{4}$ , Huadong Wang $^{1,3}$ , Kaiyue Wen $^{1,3}$ , Zhiyuan Liu $^{1,3†}$ , Peng Li $^{2‡}$ , Juanzi Li $^{1,3}$ , Lei Hou $^{1,3}$ , Maosong Sun $^{1,3+}$ , Jie Zhou $^{4}$
$^{1}$ Department of Computer Science and Technology, BNRist;
$^{2}$ Institute for AI Industry Research (AIR);
$^{3}$ Institute for Artificial Intelligence, Tsinghua University, Beijing, 100084, China
$^{4}$ Pattern Recognition Center, WeChat AI, Tencent Inc, China.
{suys19, wangxz20}@mails.tsinghua.edu.cn
Prompt tuning (PT) is a promising parameter-efficient method to utilize extremely large pre-trained language models (PLMs), which can achieve comparable performance to full-parameter fine-tuning by only tuning a few soft prompts. However, PT requires much more training time than fine-tuning. Intuitively, knowledge transfer can help to improve the efficiency. To explore whether we can improve PT via prompt transfer, we empirically investigate the transferability of soft prompts across different downstream tasks and PLMs in this work. We find that (1) in zero-shot setting, trained soft prompts can effectively transfer to similar tasks on the same PLM and also to other PLMs with a cross-model projector trained on similar tasks; (2) when used as initialization, trained soft prompts of similar tasks and projected prompts of other PLMs can significantly accelerate training and also improve the performance of PT. Moreover, to explore what decides prompt transferability, we investigate various transferability indicators and find that the overlapping rate of activated neurons strongly reflects the transferability, which suggests how the prompts stimulate PLMs is essential. Our findings show that prompt transfer is promising for improving PT, and further research shall focus more on prompts' stimulation to PLMs. The source code can be obtained from https://github.com/thunlp/Prompt-Transferability.
Pre-trained language models (PLMs), such as BERT (Devlin et al., 2019) and GPT (Radford et al., 2018) have achieved great performance on various natural language processing (NLP) tasks (Han et al., 2021). Recently, people have found that
Cross-Task Transfer
Cross-Model Transfer
Figure 1: We explore prompt transferring across different tasks (cross-task) and PLMs (cross-model) with directly reusing prompts and initializing prompt tuning.
extremely large PLMs can achieve remarkable improvements, and various large PLMs are continually developed (Brown et al., 2020; Raffel et al., 2020; Zhang et al., 2021; Zeng et al., 2021; Wei et al., 2021; Sun et al., 2021), which contain up to hundreds of billions of parameters.
Considering the extremely large scale of these state-of-the-art PLMs, conventional full-parameter fine-tuning methods become extremely expensive. Hence, various parameter-efficient tuning methods (Houlsby et al., 2019; Ben Zaken et al., 2021; Lester et al., 2021; Li and Liang, 2021; Liu et al., 2021) are explored, among which prompt tuning (PT) has attracted broad research attention. PT prepends some soft prompts, which are essentially learnable virtual tokens, into the input sequences and only trains them while keeping all the PLM's parameters fixed. The training objective is to generate desired outputs in the same way as the pretraining tasks. PT can match the downstream task performance of fine-tuning with only thousands of tunable parameters (Lester et al., 2021) when the PLM has billions of parameters.
Although PT is an effective approach to utilizing extremely large PLMs, it requires much more
† Corresponding author: Z.Liu and M.Sun.
Partly done while P.Li was working at Tencent.
Human Language Technologies, pages 3949 - 3969
Figure 2: Validation accuracies against training time of fine-tuning and PT for RoBERTa $_{\text{LARGE}}$  on MNLI. PT takes much more training time.
training time than fine-tuning to reach the convergence as shown in Figure 2; hence, it is worthwhile to explore how to improve the efficiency of PT. In this work, we attempt to improve PT via prompt transfer across different tasks and models. Knowledge transfer across tasks (Vu et al., 2020) and models (Qin et al., 2021) have been widely used to improve the efficiency and effectiveness of NLP systems. Intuitively, soft prompts are the only tuned parameters in PT and thus shall concentrate the knowledge required to solve tasks conditioned on PLMs. Thus transferring the trained prompts is promising to accelerate PT.
As shown in Figure 1, we empirically analyze the transferability of prompts across different tasks (cross-task transfer setting) and PLMs (cross-model transfer setting) in this paper. The empirical analysis is conducted on 17 NLP tasks of 6 types and two representative PLM series: RoBERTa (Liu et al., 2019b) and T5 (Raffel et al., 2020). In cross-task transfer, the prompt transfer can be done by directly reusing the trained prompts of the source task on the target task. However, in cross-model transfer, directly reusing prompts is intractable since the semantic spaces of different PLMs are inconsistent; hence, we develop various prompt projectors to project the soft prompts trained on the source PLM to the semantic space of the target PLM. We conduct two lines of experiments: (1) We investigate the zero-shot transfer performance and find that the transferability of prompts is influenced by task types. In cross-task transfer, the soft prompts can directly transfer to sametype tasks and achieve non-trivial performance, but poorly transfer to different-type tasks requiring different language skills. In cross-model transfer, we can successfully train a prompt projector with PT on a task, but the trained projector also only well generalizes to the same-type tasks of the projector-training task. (2) To accelerate PT,
we propose to transfer prompts with initialization. In cross-task transfer, we start PT with the trained soft prompts of similar tasks as initialization. While in cross-model transfer, the initialization is the projected prompts of the same task trained on the source PLM. The two methods are dubbed as  $\mathrm{TPT}_{\mathrm{TASK}}$  and  $\mathrm{TPT}_{\mathrm{MODEL}}$ , which are short for transferable prompt tuning. Experiments show that they can both accelerate PT to some extent and also achieve a certain performance improvement.
Furthermore, we explore why can the prompts transfer and what decides their transferability. To this end, we design various prompt similarity metrics from different perspectives and examine how well they can serve as **transferability indicators**, i.e., how well they correlate with prompt transfer performance. Experiments find that our novel method of measuring prompt similarity via model activations in feed-forward layers is better correlated with prompt transferability than prompt embedding distance-based metrics. This suggests the prompts are essentially stimulating PLM's inner ability distributing among neurons to do specific NLP tasks, and future prompt transfer works should focus more on how the PLMs respond to different prompts' stimulation rather than the prompts' embedding properties.
To summarize, our contributions are three-fold: (1) We thoroughly analyze the transferability of prompts across different tasks and models, and show that improving PT with prompt transfer is possible and promising. (2) We propose to transfer prompts with initialization, which enhances both PT's efficiency and effectiveness. (3) We explore the effectiveness of various prompt similarity metrics serving as transferability indicators and demonstrate how the prompts stimulate PLMs to decide the transferability, which may facilitate further transferrable PT research.
Prompt Tuning GPT-3 (Brown et al., 2020) demonstrates remarkable few-shot performance by prepending textual prompts before the inputs and thus helps the PLM to generate desired outputs of NLP tasks directly. Motivated by this, many works have tried to improve various NLP tasks by creating manually-crafted (Schick and Schütze, 2021a,b; Mishra et al., 2021) or automatically-searched (Jiang et al., 2020; Shin et al., 2020; Gao et al., 2021) hard prompts, which are discrete to
kens but not necessarily human-readable. Furthermore, soft prompts (Hambardzumyan et al., 2021; Qin and Eisner, 2021; Zhong et al., 2021; Liu et al., 2021) are proposed, which are tuneable embeddings rather than tokens in the vocabularies and can be directly trained with task-specific supervision. Lester et al. (2021) demonstrate that prompt tuning (PT) method can match the performance of full-parameter fine-tuning when the PLM has billions of parameters. This suggests that PT is promising to utilize extremely large PLMs. However, the much more training time needed to reach the convergence makes PT inefficient. In this work, we show that prompt transfer can improve the effectiveness to some extent with knowledge transfer, and empirically analyze the transferability of prompts across tasks and PLMs.
Knowledge Transfer Cross-task knowledge transfer (Ruder, 2017) has been a long-standing way to improve the effectiveness and efficiency of NLP systems. In the PLM era, some works propose to tune the PLMs on intermediate tasks (Phang et al., 2018; Pruksachatkun et al., 2020; Gururan-gan et al., 2020; Wang et al., 2019a; Vu et al., 2020; Poth et al., 2021) before fine-tuning on specific target tasks to achieve certain benefits. Vu et al. (2020) empirically analyze the transferability between tasks in this setting.
These explorations are all for fine-tuning. Considering the potential of PT, we believe the transferability and knowledge transfer methods for PT are worth exploring. As a prior attempt, Lester et al. (2021) demonstrate that PT's cross-domain transferability is stronger than fine-tuning.
Similar to our work, recent work (Vu et al., 2021) also explores the cross-task transfer with prompt initialization and prompt similarity metrics based on cosine similarity. However, Vu et al. (2021) focus on improving the effectiveness of PT but we attempt to improve the efficiency. Additionally, we explore more transferability indicators, especially the overlapping rate of activated neurons, and also investigate cross-model transfer, which is inspired by previous cross-model knowledge transfer works such as Net2Net (Chen et al., 2016), knowledge distillation (Hinton et al., 2015) and knowledge inheritance (Qin et al., 2021).
Here we introduce the basic knowledge about PT (§ 3.1) as well as the downstream tasks (§ 3.2) and
models (§ 3.3) investigated in experiments.
3.1 Prompt Tuning
In this work, we study the PT method that is capable of tuning large PLMs (Lester et al., 2021; Liu et al., 2021), i.e., we only explore the PT method freezing PLM parameters. PT prepends some virtual tokens, i.e., the soft prompts, into the inputs of the PLM to provide knowledge about downstream tasks. The soft prompts are essentially tunable embedding vectors, which are trained with the objective enforcing the PLM to generate desired outputs of the downstream task in the same way of the pre-training objective.
Formally, given an input sequence with  $n$  tokens  $X = \{x_{1},x_{2},\ldots ,x_{n}\}$ , we first prepend  $l$  randomly initialized soft prompts  $P = \{\mathbf{p}_1,\mathbf{p}_2,\dots ,\mathbf{p}_l\}$  before them, where  $\mathbf{p}_i\in \mathbb{R}^d$  is an embedding vector, and  $d$  is the input dimension of the PLM. The training objective is to maximize the likelihood of decoding the desired output  $y$ :
$$
\mathcal {L} = p (y | P, x _ {1}, \dots , x _ {n}), \tag {1}
$$
where only  $P$  is learnable. For the language understanding tasks,  $y$  is the label token corresponding to the label of  $X$ . For the conditional generation tasks,  $y$  is a sequence. Especially, for the models pre-trained with the masked language modeling objective like RoBERTa, we additionallyptides a special [MASK] token before the prompts and train the prompts to let the PLM fill  $y$  into it.
3.2 Investigated NLP Tasks
To comprehensively study the prompt transferability across various NLP tasks, we involve 17 diverse tasks, which can be divided into 6 types: (1) Sentiment Analysis (SA), including IMDB (Maas et al., 2011), SST-2 (Socher et al., 2013), laptop (Pontiki et al., 2014), restaurant (Pontiki et al., 2014), Movie Rationales (Movie) (Zaidan et al., 2008) and TweetEval (Tweet) (Barbieri et al., 2020); (2) Natural Language Inference (NLI), including MNLI (Williams et al., 2018), QNLI (Wang et al., 2019b) and SNLI (Bowman et al., 2015); (3) Ethical Judgment (EJ), including deontology (Hendrycks et al., 2021) and justice (Hendrycks et al., 2021); (4) Paraphrase Identification (PI), including QQP (Sharma et al., 2019) and MRPC (Dolan and Brockett, 2005); (5) Question Answering (QA), including SQuAD (Rajpurkar et al., 2016) and NQ-Open
(Lee et al., 2019); (6) Summarization (SUM), including Multi-News (Fabbri et al., 2019) and SAMSum (Gliwa et al., 2019). Details for these tasks, evaluation metrics, label tokens, implementations are in appendix A.
3.3 Investigated Models
We investigate prompt transferability for two series of PLMs: RoBERTa (Liu et al., 2019b) and T5 (Raffel et al., 2020), which represent two mainstream pre-training types: masked language modeling and sequence-to-sequence pre-training. Considering RoBERTa can only predict a single token (or a fixed length of tokens) under prompt tuning paradigm, for the conditional generation tasks (QA and SUM) that output multiple tokens, we only investigate T5. We mainly report results for the two largest versions of PLMs, i.e., RoBERTaLarge and  $\mathrm{T}5_{\mathrm{XXL}}$ . The more detailed results for the other sizes are attached in appendix.
4 Cross-Task Transfer
We empirically study the cross-task transferability of soft prompts (§ 4.1) and try to improve the effectiveness and efficiency of PT with transfer (§ 4.2).
4.1 Zero-shot Transfer Performance
To study the cross-task transferability, we first examine PT's zero-shot transfer performance, i.e., we conduct PT on a source task, then directly reuse the trained prompts on other target tasks and evaluate their performance. The results are shown in Figure  $3^{1}$ , from which we can observe that: (1) For the tasks within the same type, transferring soft prompts between them can generally perform well and may even outperform vanilla PT on the target task, especially when the source task has more data (the case of transferring from IMDB to Movie in Figure 3 (a) and transferring from restaurant to laptop in Figure 3 (b)), which demonstrates that it is promising to improve PT's effectiveness and efficiency with knowledge transfer from similar tasks. (2) For the tasks of different types, the transferability of soft prompts among them is generally poor, and transferring soft prompts often achieve similar performance to randomly initialized prompts.
(3) However, some tasks can transfer to different-type tasks to some extent, such as the QA and SUM tasks to SA tasks in Figure 3 (b). To understand
this, it is worthwhile to explore what controls the transferability between prompts, and we do some preliminary study in § 6.
4.2 Transfer with Initialization
To improve the effectiveness and efficiency of PT with cross-task transfer, we explore a cross-task transferable prompt tuning  $(\mathrm{TPT}_{\mathrm{TASK}})$  method, which initializes soft prompts with well-trained prompts of the most similar task and then starts PT.
For a target task, we start  $\mathrm{TPT}_{\mathrm{TASK}}$  with trained prompts of the source task achieving the best zero-shot transfer performance in Figure 3. From the results of the performance and training time comparisons in Table 1, we can see  $\mathrm{TPT}_{\mathrm{TASK}}$  can mostly achieve better or comparable performance to vanilla PT starting from random initialization, and  $\mathrm{TPT}_{\mathrm{TASK}}$  generally takes less training time.
5 Cross-Model Transfer
We further study the cross-model transferability of soft prompts. Intuitively, cross-model transfer allows us to train prompts on a small and computationally efficient PLM and use them on a massive and computationally expensive PLM, which will be much more efficient and environment-friendly. We investigate the feasibility of cross-model transfer on transferring from a source PLM (RoBERTa $_{\text{LARGE}}$ ) to a larger and heterogeneous target PLM (T5XXL), which shall be the most difficult setting. Appendix C shows the experimental results of other settings. Directly reusing trained soft prompts between different PLMs is infeasible since their embedding spaces are different. Hence, we investigate how to do cross-model prompt projection ( $\S$  5.1) and see the transfer performance ( $\S$  5.2). Furthermore, we explore to improve PT with cross-model transfer initialization ( $\S$  5.3).
5.1 Cross-Model Prompt Projection
To project the trained soft prompts of a PLM to the semantic space of a different PLM, we train projectors with various objectives and examine their effectiveness. A good way to train the cross-model projectors may need some task-specific supervisions, but the trained projector shall generalize to different tasks so that the efficiency for learning the new tasks on the target model could be improved.
Formally, given the prompt of the source PLM  $P^{s} = \{\mathbf{p}_{1}^{s},\dots ,\mathbf{p}_{l}^{s}\}$ , we concatenate the  $l$  virtual
1 More results on other PLMs are left in appendix B.1.
$^{2}$ Training time comparisons are left in appendix B.3.
(a) RoBERTaLARGE
Figure 3: Relative zero-shot transfer performance (zero-shot transfer performance / original PT performance) (\%) on the target tasks (columns) of the soft prompts trained on the source tasks (rows) for RoBERTaLARGE and T5XXL. Colors of the task names indicate task types. Blue: SA. Green: NLI. Brown: EJ. Orange: PI. Purple: QA. Gray: SUM. Random Prompt of the last row means the soft prompts are randomly generated without any training.
(b) T5XXL
Task Type	SA	NLI	EJ	PI	QA	SUM
Task	IMDB	SST-2	laptop restaurant	Movie	Tweet	MNLI	QNLI	SNLI	deontology justice	QQP	MRPC	SQuADNQ-Open	Multi-News	SAMSum
Metric	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	Acc.	F1	F1	ROUGE-L	ROUGE-L
RoBERTaLARGE
Performance (PT) (%)	92.2	96.1	76.4	83.7	84.9	76.1	87.3	92.4	91.9	85.6	81.0	88.9	81.2	N/A	N/A	N/A	N/A
Performance (TPTASK) (%)	92.4	96.3	79.1	85.8	85.1	76.1	87.9	93.1	91.9	85.6	78.2	86.1	79.2	N/A	N/A	N/A	N/A
Convergence Speedup	1.7	1.1	1.0	1.9	1.2	0.9	1.2	1.2	1.3	0.9	0.7	0.8	0.9	N/A	N/A	N/A	N/A
Comparable-result Speedup	2.5	2.4	1.0	3.8	1.5	1.3	1.1	2.3	1.0	0.9	N/A	N/A	N/A	N/A	N/A	N/A	N/A
T5XXL
Performance (PT) (%)	96.5	97.4	76.6	90.1	97.9	76.2	90.5	95.2	93.4	87.0	92.5	90.0	86.3	86.3	20.8	29.2	45.8
Performance (TPTASK) (%)	96.6	97.8	84.2	88.6	97.5	77.0	92.0	96.2	94.0	95.3	90.7	90.9	89.0	85.9	21.3	29.3	46.8
Convergence Speedup	1.2	49.7	2.2	1.1	3.9	1.4	12.5	24.9	49.9	29.8	1.5	1.0	3.3	1.1	1.0	2.0	2.0
Comparable-result Speedup	1.2	48.9	219.8	N/A	N/A	1.5	12.5	29.9	49.9	29.9	N/A	1.0	5.0	N/A	1.0	2.0	2.5
Table 1: Performance on 17 NLP tasks of vanilla prompt tuning (PT) and prompt tuning with transferring initialization  $\left(\mathrm{TPT}_{\mathrm{TASK}}\right)$ , which initialize PT with the one performing best in zero-shot transfer, as well as the convergence speedup (the quotient of the training steps of PT by the training time of  $\mathrm{TPT}_{\mathrm{TASK}}$  reaching convergence) and comparable-result speedup (the quotient of the training time of PT by the training time of  $\mathrm{TPT}_{\mathrm{TASK}}$  achieving comparable performance to PT). N/A represents the tasks that RoBERTaLarge cannot conduct, or we fail to speed up training with  $\mathrm{TPT}_{\mathrm{TASK}}$ .
tokens into a unified vector  $\mathbf{P}^s\in \mathbb{R}^{ld_s}$ . The projector  $\mathbf{Proj}(\cdot)$  is to project it to  $\tilde{\mathbf{P}}^s\in \mathbb{R}^{ld_t}$  in the semantic space of the target PLM, where  $d_{s}$  and  $d_{t}$  are the input embedding dimensions of the source and target PLM, respectively. We parameterize the projector with a two-layer perceptron as follows:
$$
\tilde {\mathbf {P}} ^ {s} = \mathbf {P r o j} \left(\mathbf {P} ^ {s}\right) = \mathbf {W} _ {2} \left(\sigma \left(\mathbf {P} ^ {s} \mathbf {W} _ {1} + \mathbf {b} _ {1}\right)\right) + \mathbf {b} _ {2}, \tag {2}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d_h\times ld_s}$ $\mathbf{W}_2\in \mathbb{R}^{ld_t\times d_h}$  are trainable matrices,  $\mathbf{b}_1\in \mathbb{R}^{d_h}$ $\mathbf{b}_2\in \mathbb{R}^{ld_t}$  are biases,  $\sigma$  is a non-linear activation function. We investigate
two learning objectives to train the projector<sup>3</sup>:
Distance Minimizing We firstly try to learn cross-model projections by minimizing the distance between the projected prompt and the parallel prompt  $\mathbf{P}^t$  originally trained on the target PLM with the same task, i.e., the training objective is to minimize their  $L_{2}$ -distance  $\| \mathbf{Proj}(\mathbf{P}^{s}) - \mathbf{P}^{t}\|_{2}$ .
Task Tuning We then try to train the cross-model projector with task-specific supervision signals on
3More projector-training details are left in appendix C.1.
Method	SA	NLI	EJ	PI
IMDB	SST-2	laptop	restaurant	Movie	Tweet	MNLI	QNLI	SNLI	deontology	justice	QQP	MRPC
PT on T5XXL	96.5	97.4	76.6	88.1	97.9	72.5	90.5	95.2	93.4	87.0	92.5	90.0	86.3
Random Prompt	49.7	49.0	19.8	17.0	51.6	15.5	31.8	49.3	31.9	51.3	50.0	36.4	67.0
(a) Zero-shot Transfer Performance  $(\%)$
laptop	Distance Minimizing	49.6	49.0	76.6	17.5	51.5	14.4	31.8	48.1	32.8	53.3	49.9	36.8	66.6
Task Tuning	82.9	89.3	80.3	85.7	78.6	58.4	32.4	50.7	33.6	54.9	51.6	33.9	63.7
MNLI	Distance Minimizing	49.6	50.1	19.8	18.3	51.2	15.0	90.5	49.0	32.9	50.3	49.0	36.8	65.6
Task Tuning	49.7	48.8	19.8	17.0	51.6	16.0	89.8	82.7	88.2	49.7	50.0	36.8	67.7
(b) Transfer with Initialization  $(\mathrm{TPT}_{\mathrm{MODEL}})$
laptop	Performance (%)	96.5	97.4	82.9	90.3	97.4	74.4	91.0	95.4	93.4	92.5	92.5	90.0	87.9
Convergence Speedup	1.1	1.7	1.9	1.3	0.6	1.3	0.9	0.9	1.0	1.0	0.7	1.1	1.1
Comparable-result Speedup	1.0	19.0	16.0	6.0	N/A	2.2	3.6	1.1	6.0	6.0	0.9	1.8	3.4
MNLI	Performance (%)	96.5	97.4	82.7	88.5	95.8	74.7	91.2	95.9	93.5	94.6	92.5	90.0	87.7
Convergence Speedup	1.0	1.6	1.8	0.9	0.4	1.3	1.0	1.1	1.4	2.0	1.7	0.9	0.9
Comparable-result Speedup	1.0	18.0	15.0	1.6	N/A	1.5	18.0	20.0	30.0	7.5	5.0	1.5	1.9
Table 2: Cross-model prompt transfer (RoBERTaLARGE to T5XXL) results, including non-transfer baselines (vanilla PT and randomly generated prompts), zero-shot transfer performance of various projectors, and  $\mathrm{TPT}_{\mathrm{MODEL}}$  results (performance, convergence speedup, and comparable-result speedup similar to Table 1).  $\mathrm{TPT}_{\mathrm{MODEL}}$  adopts the Task Tuning projectors to project the soft prompts.
the target PLM. Specifically, we directly tune the projected prompts on some tasks and back propagate the supervision signals to train the projector weights, so that the projector can learn how to stimulate the target PLM and thus may generalize to transfer the prompts of other tasks.
These methods rely on some tasks (parallel trained soft prompts or training data) to train the projector. The projector learning methods are ag- nostic to the specific training tasks used, and we choose laptop and MNLI in experiments.
5.2 Zero-shot Transfer Performance
The zero-shot transfer performance of various projector-learning methods are shown in Table  $2^{4}$  (a). We can observe that: (1) Distance Minimizing works well to transfer the prompts of the projectortraining task, but falls back to random performance on the other unseen tasks, which is not practically usable. This is consistent with our findings in  $\S 6$  that the embedding distances do not strongly correlate to prompt transferability. (2) Task Tuning performs better and successfully generalizes to same-type unseen tasks of the projector-training tasks (e.g. NLI tasks for the projectors trained with MNLI), which proves the feasibility of practical cross-model prompt transfer. (3) The projectors trained with Task Tuning still cannot work for different-type tasks, which may be limited by the cross-task prompt transferability investigated
in  $\S 4.1$ . This urges further attention to developing universal cross-model projections.