SlowGuess/NAACL_2022_OCR huggingface dataset:
5.3 Transfer with Initialization
Similar to § 4.2, we further study whether the projected soft prompts can initialize PT on the target PLM and accelerate training as well as improve performance. We propose cross-model transferable prompt tuning,  $\mathrm{TPT}_{\mathrm{MODEL}}$ , which adopts the Task Tuning projectors to project the soft prompts trained on the source PLM into the target PLM and initialize PT with the projected prompts.
The performance and speedup are shown in Table 2 (b). We can see that, for the tasks within the same type of the projector-training task, compared to vanilla PT,  $\mathrm{TPT}_{\mathrm{MODEL}}$  can mostly achieve comparable or better performance with much less training time, which demonstrates that practical cross-model prompt transfer is promising for improving the efficiency and effectiveness of PT.
6 Exploring Transferability Indicator
Based on the positive results in cross-task and cross-model transfer, we explore why the soft prompts can transfer across tasks and what decides the transferability between them, which may shed light on the mechanisms behind PT and help to design transferable PT methods. We explore various prompt similarity metrics and examine how well do they align with the zero-shot transfer performance. If a similarity metric can well indicate transferability, it suggests the factors considered in designing
4More results on other PLMs are left in appendix C.2.
this metric decide the prompt transferability. Moreover, the prompt similarity metrics can qualify task similarities using the trained soft prompts as task embeddings and may help in developing cross-task transfer methods. As a straightforward example, if we build a prompt warehouse containing prompts of diverse tasks, we can retrieve prompts of similar tasks for a new task with a certain similarity metric and better improve PT with  $\mathrm{TPT}_{\mathrm{TASK}}$ .
6.1 Prompt Similarity Metric
We explore the following two kinds of metrics:
Embedding Similarity We firstly regard the trained soft prompts as only embeddings in the vector space and calculate their Euclidean similarity and cosine similarity, among which cosine similarity is also explored by Vu et al. (2021).
Given two groups of trained prompts containing  $l$  virtual tokens:  $P^{t_1} = \{\mathbf{p}_1^{t_1},\dots ,\mathbf{p}_l^{t_1}\}$  and  $P^{t_2} = \{\mathbf{p}_1^{t_2},\dots ,\mathbf{p}_l^{t_2}\}$ , which correspond to tasks  $t_1$  and  $t_2$ . Firstly, we concatenate the  $l$  virtual tokens for each group and get two concatenation embeddings  $\mathbf{P}^{t_1},\mathbf{P}^{t_2}\in \mathbb{R}^{ld}$ , then we compute Euclidean similarity and cosine similarity of them:
$$
\operatorname {E} _ {\text {c o n c a t}} \left(P ^ {t _ {1}}, P ^ {t _ {2}}\right) = \frac {1}{1 + \left\| \mathbf {P} ^ {t _ {1}} - \mathbf {P} ^ {t _ {2}} \right\|}, \tag {3}
$$
$$
\mathrm {C} _ {\text {c o n c a t}} \left(P ^ {t _ {1}}, P ^ {t _ {2}}\right) = \frac {\mathbf {P} ^ {t _ {1}} \cdot \mathbf {P} ^ {t _ {2}}}{\| \mathbf {P} ^ {t _ {1}} \| \| \mathbf {P} ^ {t _ {2}} \|}.
$$
We further explore a simple way to make the metrics invariant to token positions. We compute Euclidean distances and cosine similarities for every virtual token pairs in the two groups and use the averaged results in the final similarity metrics:
$$
\operatorname {E} _ {\text {a v e r a g e}} \left(P ^ {t _ {1}}, P ^ {t _ {2}}\right) = \frac {1}{1 + \frac {1}{l ^ {2}} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} \left\| \mathbf {p} _ {i} ^ {t _ {1}} - \mathbf {p} _ {j} ^ {t _ {2}} \right\|}, \tag {4}
$$
$$
\mathrm {C} _ {\mathrm {a v e r a g e}} (P ^ {t _ {1}}, P ^ {t _ {2}}) = \frac {1}{l ^ {2}} \sum_ {i = 1} ^ {l} \sum_ {j = 1} ^ {l} \frac {\mathbf {p} _ {i} ^ {t _ {1}} \cdot \mathbf {p} _ {j} ^ {t _ {2}}}{\| \mathbf {p} _ {i} ^ {t _ {1}} \| \| \mathbf {p} _ {j} ^ {t _ {2}} \|}.
$$
Model Stimulation Similarity In the second way, we depict their similarities based on how they stimulate the PLMs, i.e., we examine the similarities between the responses of PLMs to the two soft prompts. Motivated by Geva et al. (2021) and Dai et al. (2021), which both find that the activation of the neurons in the feed-forward layers of Transformers (Vaswani et al., 2017) corresponds to specific model behaviors, we propose to use the overlapping rate of activated neurons as a similarity metric of prompts. Specifically, the feedforward network  $\mathrm{FFN}(\cdot)$  in a Transformer layer is:
Model	Metric	Same Task	Different Tasks
RoBERTaLARGE	Econcat	9.4	6.8
Eaverage	41.6	37.6
Cconcat	47.6	31.7
Caverage	1.7	1.1
ON	39.4	21.4
T5XXL	Econcat	0.5	0.3
Eaverage	4.0	3.4
Cconcat	29.4	3.4
Caverage	4.0	2.1
ON	62.0	46.1
Table 3: The average values  $(\%)$  of the 5 similarity metrics for prompt pairs of the same task (trained with 3 different random seeds) and different tasks.
$$
\operatorname {F F N} (\mathbf {x}) = \max  \left(\mathbf {x} \mathbf {W} _ {1} ^ {\top} + \mathbf {b} _ {1}, \mathbf {0}\right) \mathbf {W} _ {2} + \mathbf {b} _ {2}, \tag {5}
$$
where  $\mathbf{x} \in \mathbb{R}^d$  is the input embedding,  $\mathbf{W}_1, \mathbf{W}_2 \in \mathbb{R}^{d_m \times d}$  are trainable matrices, and  $\mathbf{b}_1, \mathbf{b}_2$  are bias vectors. The  $\max(\mathbf{xW}_1^\top + \mathbf{b}_1, \mathbf{0})$  can be regarded as the non-negative activation values for  $d_m$  hidden neurons (Geva et al., 2021). We then change all the positive elements of  $\max(\mathbf{xW}_1^\top + \mathbf{b}_1, \mathbf{0})$  to 1 and get the one-hot activation state vector  $\mathbf{s}$ .
We feed an input sequence  $\{P, < s > \}$  into the PLMs, where  $< s >$  is the special token indicating the start of a sentence. For RoBERTa, a [MASK] is additional prepended. This sequence is in the format of PT inputs but without specific input sentences.
We use the activation states of the positions used to decode outputs, which shall be more task-specific. Specifically, for T5, we use the decoder module's activation states at the first position. For RoBERTa, we use the activation states of [MASK]. Finally, we concatenate the activation states of PLM's  $L$  layers to get the overall activation states:
$$
\operatorname {A S} (P) = \left[ \mathbf {s} _ {1}; \mathbf {s} _ {2}; \dots ; \mathbf {s} _ {L} \right]. \tag {6}
$$
We can only retrieve the activation states of a part of layers in the similarity computation. In experiments, we find that the higher layers tend to be more task-specific, which is consistent with the probing results (Liu et al., 2019a). Hence we use the activation states of the top 3 layers $^5$  in experiments below. We calculate the overlapping rate of activated neurons  $\mathrm{ON}(P^{t_1}, P^{t_2})$  between
5More results about the different layers's performance are left in appendix D.4.
Metric	RoBERTaLARGE	T5XXL
Econcat	22.6	12.9
Eaverage	2.8	-2.5
Cconcat	24.8	31.6
Caverage	44.7	33.5
ON	49.7	36.9
Figure 4: Spearman's correlation scores of ON and  $C_{\text{average}}$  with cross-task zero-shot transfer performance change along with the parameter size of T5.
the trained soft prompts of task  $t_1$  and  $t_2$  with the cosine similarity:
$$
\mathrm {O N} \left(P ^ {t _ {1}}, P ^ {t _ {2}}\right) = \frac {\mathrm {A S} \left(P ^ {t _ {1}}\right) \cdot \mathrm {A S} \left(P ^ {t _ {2}}\right)}{\| \mathrm {A S} \left(P ^ {t _ {1}}\right) \| \| \mathrm {A S} \left(P ^ {t _ {2}}\right) \|}. \tag {7}
$$
To evaluate the effectiveness of the above similarity metrics of soft prompts, we (i) test whether the similarity metrics can distinguish the trained prompts of the same tasks and different tasks, and (ii) examine whether these metrics align with the zero-shot transfer performance.
Regarding (i), we compare the similarities of the investigated metrics for two trained prompts within the same task (trained with different random seeds) and between different tasks in Table 3. From the results, we can observe that all the metrics work well to distinguish the prompts of the same task and different tasks. This suggests that the trained soft prompts of different tasks form distinguishable clusters in the embedding space and also stimulate different abilities within the PLM.
Moreover, to evaluate (ii), how well the similarity metrics align with the cross-task transfer performance, we quantify the correlations between the similarities and zero-shot transfer performance in Figure 3. Specifically, for each target task's prompt, we rank various source tasks' prompts
Table 4: The Spearman's rank correlation scores  $(\%)$  between various similarity metrics and cross-task zero-shot transfer performance of soft prompts.
Projector	Task	C-average	ON
Task Tuning (laptop)	laptop	3.8	52.4
Same-Type Tasks	4.1	51.0
Different-Type Tasks	3.4	46.0
Task Tuning (MNLI)	MNLI	2.7	70.7
Same-Type Tasks	2.7	56.7
Different-Type Tasks	4.1	53.4
Table 5: Similarities (%) between the prompts projected with Task Tuning projector and the original prompts trained on T5XXL.
with similarity scores and zero-shot transfer performance and then compute the Spearman's rank correlation (Spearman, 1987) between the two ranks generated by these two ways. The overall results are shown in Table  $4^{6}$ . We can see that: (1) The overlapping rate of activated neurons (ON) metric works better than all the embedding similarities, which suggests that model stimulation is more important for prompt transferability than embedding distances. (2) ON works much worse on  $\mathrm{T5}_{\mathrm{XXL}}$  (11B parameters) than on  $\mathrm{RoBERTa}_{\mathrm{LARGE}}$  (330M parameters). We guess this is because larger PLMs have higher redundancy (Aghajanyan et al., 2021), which means prompts can activate different redundant neurons to do similar jobs and thus influence the sensitivity of ON metric. This is supported by the experiments showing that the Spearman's correlation scores of ON drop with the increase of PLM scales (Figure 4), from which we can see  $\mathrm{C}_{\mathrm{average}}$  also exhibits a similar trend. We encourage future work to explore how to overcome the PLM redundancy for better transferrable PT. As a preliminary trial, we find that by taking the intersection of activation states of 3 prompts trained with different random seeds, ON's correlation score on  $\mathrm{T5}_{\mathrm{XXL}}$  raises from  $36.9\%$  to  $46.3\%$ .
We further explore whether the prompt similarity metrics also work in the cross-model transfer setting by testing whether they work between the projected prompts and original prompts of the same task. In Table 5, we show the similarities of prompts projected with Task Tuning projectors by the two best metrics  $C_{\text{average}}$  and ON. We can see: (1) ON metric shows that the projected prompts are highly similar to the original prompts within the same type of projector-training tasks but are not so similar to different-type tasks, which is quite consistent with the cross-model zero-shot transfer
The detailed results by task types are left in appendix D.2.
performance in Table 2. (2) However,  $C_{\text{average}}$  cannot reflect this phenomenon, which shows that the perspective of model stimulation is more promising for understanding transferability again.
We empirically investigate the transferability of prompts in this paper. In the cross-task setting, we find that soft prompts can transfer to similar tasks without training. In the cross-model setting, we successfully project prompts into the space of other PLMs. Further, we utilize trained prompts of other tasks or other PLMs as initialization to significantly accelerate training and improve effectiveness. Moreover, we explore various prompt transferability indicators and show that how the prompts stimulate PLMs is important to transferability. We hope the empirical analyses and the model stimulation idea can facilitate further research on transferable and efficient PT.
Yusheng Su and Xiaozhi Wang mainly initiated and organized this research. First, Xiaozhi Wang proposed the research idea. Yusheng Su, Xiaozhi Wang, Yujia Qin and Yankai Lin discussed most of the investigation settings. Yusheng Su, Chi-Min Chan, and Kaiyue Wen conducted the experiments. Xiaozhi Wang and Yusheng Su wrote the paper. Yankai Lin, Yujia Qin, Huadong Wang, Zhiyuan Liu, Peng Li, Juanzi Li, and Lei Hou revised the paper. Maosong Sun and Jie Zhou provided valuable advice and proofread this paper.
This work is supported by the National Key Research and Development Program of China (No. 2020AAA0106500), Beijing Academy of Artificial Intelligence (BAAI), Institute Guo Qiang at Tsinghua University, and International Innovation Center of Tsinghua University, Shanghai China. We thank the anonymous reviewers for their valuable comments.
Armen Aghajanyan, Sonal Gupta, and Luke Zettle-moyer. 2021. Intrinsic dimensionality explains the effectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language
Processing (Volume 1: Long Papers), pages 7319-7328, Online. Association for Computational Linguistics.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. 2016. Layer normalization. *ArXiv preprint*, abs/1607.06450.
Francesco Barbieri, Jose Camacho-Collados, Luis Espinosa Anke, and Leonardo Neves. 2020. TweetEval: Unified benchmark and comparative evaluation for tweet classification. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1644–1650, Online. Association for Computational Linguistics.
Elad Ben Zaken, Shauli Ravfogel, and Yoav Goldberg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models. ArXiv preprint, abs/2106.10199.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Tianqi Chen, Ian J. Goodfellow, and Jonathon Shlens. 2016. Net2net: Accelerating learning via knowledge transfer. In 4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Damai Dai, Li Dong, Yaru Hao, Zhifang Sui, and Furu Wei. 2021. Knowledge neurons in pretrained transformers. ArXiv preprint, abs/2104.08696.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
William B. Dolan and Chris Brockett. 2005. Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Alexander Fabbri, Irene Li, Tianwei She, Suyi Li, and Dragomir Radev. 2019. Multi-news: A large-scale multi-document summarization dataset and abstractive hierarchical model. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1074-1084, Florence, Italy. Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 3816-3830, Online. Association for Computational Linguistics.
Mor Geva, Roei Schuster, Jonathan Berant, and Omer Levy. 2021. Transformer feed-forward layers are key-value memories. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5484-5495, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Bogdan Gliwa, Iwona Mochol, Maciej Biesek, and Aleksander Wawer. 2019. SAMSum corpus: A human-annotated dialogue dataset for abstractive summarization. In Proceedings of the 2nd Workshop on New Frontiers in Summarization, pages 70-79, Hong Kong, China. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Karen Hambardzumyan, Hrant Khachatrian, and Jonathan May. 2021. WARP: Word-level Adversarial ReProgramming. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4921-4933, Online. Association for Computational Linguistics.
Xu Han, Zhengyan Zhang, Ning Ding, Yuxian Gu, Xiao Liu, Yuqi Huo, Jiezhong Qiu, Liang Zhang, Wentao Han, Minlie Huang, et al. 2021. Pre-trained models: Past, present and future. Proceedings of AI Open.
Dan Hendrycks, Collin Burns, Steven Basart, Andrew Critch, Jerry Li, Dawn Song, and Jacob Steinhardt. 2021. Aligning ai with shared human values. In Proceedings of ICLR.
Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. 2015. Distilling the knowledge in a neural network. ArXiv preprint, abs/1503.02531.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Zhengbao Jiang, Frank F. Xu, Jun Araki, and Graham Neubig. 2020. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-438.
Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6086–6096, Florence, Italy. Association for Computational Linguistics.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online. Association for Computational Linguistics.
Chin-Yew Lin. 2004. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74-81, Barcelona, Spain. Association for Computational Linguistics.
Nelson F. Liu, Matt Gardner, Yonatan Belinkov, Matthew E. Peters, and Noah A. Smith. 2019a. Linguistic knowledge and transferability of contextual representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1073-1094, Minneapolis, Minnesota. Association for Computational Linguistics.
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021. Gpt understands, too. ArXiv preprint, abs/2103.10385.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019b.
RoBERTa: A robustly optimized bert pretraining approach. In Proceedings of ICLR.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In International Conference on Learning Representations.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Swaroop Mishra, Daniel Khashabi, Chitta Baral, Yejin Choi, and Hannaneh Hajishirzi. 2021. Reframing instructional prompts to gptk's language. ArXiv preprint, abs/2109.07830.
Jason Phang, Thibault Févry, and Samuel R. Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. ArXiv preprint, abs/1811.01088.
Maria Pontiki, Dimitris Galanis, John Pavlopoulos, Harris Papageorgiou, Ion Androutsopoulos, and Suresh Manandhar. 2014. SemEval-2014 task 4: Aspect based sentiment analysis. In Proceedings of the 8th International Workshop on Semantic Evaluation (SemEval 2014), pages 27-35, Dublin, Ireland. Association for Computational Linguistics.
Clifton Poth, Jonas Pfeiffer, Andreas Rückle, and Iryna Gurevych. 2021. What to pre-train on? Efficient intermediate task selection. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 10585-10605, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Guanghui Qin and Jason Eisner. 2021. Learning how to ask: Querying LMs with mixtures of soft prompts. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5203-5212, Online.
Yujia Qin, Yankai Lin, Jing Yi, Jiajie Zhang, Xu Han, Zhengyan Zhang, Yusheng Su, Zhiyuan Liu, Peng Li, Maosong Sun, and Jie Zhou. 2021. Knowledge inheritance for pre-trained language models. ArXiv preprint, abs/2105.13880.