SlowGuess/NAACL_2022_OCR huggingface dataset:
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. arXiv e-prints.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21:1-67.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Sebastian Ruder. 2017. An overview of multi-task learning in deep neural networks. *ArXiv preprint*, abs/1706.05098.
Timo Schick and Hinrich Schütze. 2021a. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 255–269, Online. Association for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021b. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 2339-2352, Online. Association for Computational Linguistics.
Lakshay Sharma, Laura Graesser, Nikita Nangia, and Utku Evci. 2019. Natural language understanding with the quora question pairs dataset. ArXiv preprint, abs/1907.01041.
Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Prompts. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 4222-4235, Online. Association for Computational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Charles Spearman. 1987. The proof and measurement of association between two things. In Proceedings of The American journal of psychology, volume 100, pages 441-471.
Yu Sun, Shuohuan Wang, Shikun Feng, Siyu Ding, Chao Pang, Junyuan Shang, Jiaxiang Liu, Xuyi Chen, Yanbin Zhao, Yuxiang Lu, Weixin Liu, Zhihua Wu, Weibao Gong, Jianzhong Liang, Zhizhou Shang, Peng Sun, Wei Liu, Xuan Ouyang, Dianhai Yu, Hao Tian, Hua Wu, and Haifeng Wang. 2021. Ernie 3.0: Large-scale knowledge enhanced pre-training for language understanding and generation. ArXiv preprint, abs/2107.02137.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. ArXiv preprint, abs/2110.07904.
Tu Vu, Tong Wang, Tsendvarsen Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit Iyyer. 2020. Exploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 7882-7926, Online. Association for Computational Linguistics.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappagari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019a. Can you tell me how to get past sesame street? sentence-level pretraining beyond language modeling. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465-4476, Florence, Italy. Association for Computational Linguistics.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoc V Le. 2021. Finetuned language models are zero-shot learners. ArXiv preprint, abs/2109.01652.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans,
Louisiana. Association for Computational Linguistics.
Bing Xu, Naiyan Wang, Tianqi Chen, and Mu Li. 2015. Empirical evaluation of rectified activations in convolutional network. ArXiv preprint, abs/1505.00853.
Omar F. Zaidan, Jason Eisner, and Christine Piatko. 2008. Machine learning with annotator rationales to reduce annotation cost. In Proceedings of NeurIPS (Workshop).
Wei Zeng, Xiaozhe Ren, Teng Su, Hui Wang, Yi Liao, Zhiwei Wang, Xin Jiang, ZhenZhang Yang, Kaisheng Wang, Xiaoda Zhang, et al. 2021. Panguα: Large-scale autoregressive pretrained chinese language models with auto-parallel computation. ArXiv preprint, abs/2104.12369.
Zhengyan Zhang, Yuxian Gu, Xu Han, Shengqi Chen, Chaojun Xiao, Zhenbo Sun, Yuan Yao, Fanchao Qi, Jian Guan, Pei Ke, et al. 2021. Cpm-2: Large-scale cost-effective pre-trained language models. ArXiv preprint, abs/2106.10715.
Zexuan Zhong, Dan Friedman, and Danqi Chen. 2021. Factual probing is [MASK]: Learning vs. learning to recall. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5017-5033, Online. Association for Computational Linguistics.
A Basic Setup for Various Tasks
A.1 Dataset and Task
Sentiment Analysis (SA) SA is the task of classifying sentiment polarities for a given sentence. We select IMDB (Maas et al., 2011), SST-2 (Socher et al., 2013), SemEval/laptop (Pontiki et al., 2014), SemEval/restaurant (Pontiki et al., 2014), Movie Rationales (Movie) (Zaidan et al., 2008), and TweetEval (Tweet) (Barbieri et al., 2020) for our experiments.
Natural Language Inference (NLI) NLI is the task of determining whether a hypothesis is entailed or contradicted by a given sentences (premise, hypothesis). We select MNLI (Williams et al., 2018), QNLI (Wang et al., 2019b), and SNLI (Bowman et al., 2015) for our experiments.
Ethical Judgment (EJ) EJ is the task of deciding whether a sentence is ethically acceptable. We select Ethics/deontology (Hendrycks et al., 2021) and Ethics/justice (Hendrycks et al., 2021) for our experiments.
Paraphrase Identification (PI) PI is the task of classifying whether a pair of sentences are semantically identical. We select QQP (Sharma et al., 2019) and MRPC (Dolan and Brockett, 2005) for our experiments.
Question Answering (QA) QA is the task of answering a question. We choose SQuAD (Rajpurkar et al., 2016) and NQ-Open (Lee et al., 2019) to analyze. For SQuAD, a PLM captures the answer from the content. As for NQ-Open, a PLM directly generates the answer without the content.
Summarization (SUM) SUM is the task of summarizing a given article and generating the abstract. We select Multi-News (Fabbri et al., 2019), and SAMSum (Gliwa et al., 2019) for our experiments.
For classification tasks (SA, NLI, EJ, and PI), we use accuracy (Acc.) as their evaluation metric. As for generation tasks (QA and SUM), we utilize F1 and ROUGE-L (Lin, 2004), respectively.
A.3 Prompt Tuning Setting
In the experiments, for all the investigated tasks, we use AdamW (Loshchilov and Hutter, 2019) as the optimizer and set the learning rate as 0.001. We set the length of soft prompts  $l$  as 100. All the soft
prompts are randomly initialized and optimized with Equation 1. In the inference stage, RoBERTa predicts the label tokens at the [MASK] position and T5 directly uses its decoder to do generation. For the classification tasks (SA, NLI, EJ and PI), we obtaining answers in a ranking manner, i.e., we rank the label tokens by their likelihoods and regard the PLMs as predict the label of the label token with highest likelihood. For the conditional generation tasks (QA and SUM), we directly take the outputs of PLMs as their answers.
A.4 Label Tokens
The used label tokens for the classification tasks (SA, NLI, EJ, PI) are shown in Table 6. For generation tasks (QA, SUM), the desired output is just the annotated answers.
Task	Label Tokens
Sentiment Analysis (SA)
IMDB	positive, negative
SST-2	positive, negative
laptop	positive, moderate, negative
restaurant	positive, moderate, negative
Movie	positive, negative
Tweet	positive, moderate, negative
Natural Language Inference (NLI)
MNLI	yes, neutral, no
QNLI	yes, no
SNLI	yes, neutral, no
Ethical Judgment (EJ)
deontology	acceptable, un
justice	acceptable, un
Paraphrase Identification (PI)
QQP	true, false
MRPC	true, false
Table 6: Label tokens of classification tasks.
B Cross-Task Transfer
B.1 More Zero-shot transfer performance
In § 4.1, we report the zero-shot transfer performance (relative performance) on RoBERTaLARGE and T5XXL. Here, we investigate the zero-shot transfer performance on other sizes of RoBERTa and T5, which are shown in Figure 5. According to these results, we can find that the transferability of soft prompts between the tasks of different types is generally poor, which is consistent with the conclusion in § 4.1.
(a) T5SMALL
(b) T5BASE
(c) RoBERTaBASE
Figure 5: Relative performance (transferring zero-shot performance / original PT performance) (\%) on the target tasks (columns) of the soft prompts trained on the source tasks (rows), both of which demonstrate the relative performance for zero-shot transfer of prompts of RoBERTa and T5. Colors of the tasks names indicate the task types. Blue: sentiment analysis (SA). Green: natural language inference (NLI). Brown: ethical judgment (EJ). Orange: paraphrase identification (PI). Purple: question answering (QA). Gray: summarization (SUM). Random Prompt of the last row means the soft prompts are randomly generated without any training.
(d) RoBERTaLARGE
(a) Directly transferring  $(\mathrm{RoBERTa}_{\mathrm{BASE}})$
Figure 6: To exclude the poor transferability, which may result from the fact that different-type tasks use different label tokens, we unify the label tokens of different tasks into the same set of numbers (1, 2, ...) and choose  $\mathrm{RoBERTa_{BASE}}$  for the experiments. From Figure (a) and (b), we observe that the transferability between different-type tasks are still generally not improved in this way. This indicates that different-type tasks surely require distinct abilities.
(b) Unifying the label tokens (RoBERTa  $\mathrm{BASE}$ )
B.2 Unifying Label Tokens
We hypothesize that the poor transferability between different task types may result from the fact that different-type tasks usually use different label tokens, e.g., yes and no are for NLI tasks while positive and negative are for SA tasks. To verify whether this factor influences the transferability, we unify the label tokens of different tasks into the same set of numbers  $(1,2,\ldots)$  and choose RoBERTa<sub>BASE</sub> for the experiments. In Figure 6, we can observe that the transferability between different-type tasks are generally not improved in this way. This indicates that different-type tasks surely require distinct abilities, which prohibits reusing prompts between them.
B.3 Speedup Calculation
In this paper, we compute convergence speedup and comparable-result speedup as follows:
$$
\text {C o n v e r g e n c e S p e e d u p} (\mathrm {x}) = \frac {\text {P T c o n v e r g e n c e t i m e}}{\text {T P T c o n v e r g e n c e t i m e}},
$$
$$
\text {C o m p a r a b l e - r e s u l t S p e e d u n} (x) - \tag {8}
$$
$$
\frac {\text {P T c o n v e r g e n c e t i m e}}{\text {t i m e o f T P T a c h e i v i n g c o m p a r a b l e r e s u l t t o P T}}.
$$
We calculate the training loss and the evaluation score per 100 steps during the training. When
the training loss stops dropping and the evaluation score stops increasing for 300 steps, we set the point as the convergence point. For the convergence speedup in Equation 8, the PT convergence time is divided by the TPT convergence time. As for the comparable-result speedup in Equation 8, the PT convergence time are divided by the time of TPT achieving comparable performance to PT.
C Cross-Model Transfer
C.1 Implementation Details of Projector
As mentioned in § 5.1, we give the prompt of the source PLM,  $P^{s} = \{\mathbf{p}_{1}^{s},\dots ,\mathbf{p}_{l}^{s}\}$ , and concatenate its  $l$  virtual tokens into a unified vector  $\mathbf{P}^s\in \mathbb{R}^{ld_s}$  where  $d_{s}$  is the hidden size of the source PLM. To transfer  $\mathbf{P}^s$  to the target PLM whose hidden size is  $d_{t}$ , we design a projection function  $\mathbf{Proj}(\cdot)$  parameterized by a two-layer perceptron as follows:
$$
\tilde {\mathbf {P}} ^ {s} = \operatorname {P r o j} \left(\mathbf {P} ^ {s}\right) = \mathbf {W} _ {2} \left(\sigma \left(\mathbf {P} ^ {s} \mathbf {W} _ {1} + \mathbf {b} _ {1}\right)\right) + \mathbf {b} _ {2}, \tag {9}
$$
where  $\mathbf{W}_1\in \mathbb{R}^{d_h\times ld_s}$ $\mathbf{W}_2\in \mathbb{R}^{ld_t\times d_h}$  are trainable matrices,  $\mathbf{b}_1\in \mathbb{R}^{d_h}$ $\mathbf{b}_2\in \mathbb{R}^{ld_t}$  are biases,  $\sigma$  is a non-linear activation function. For training configurations of projector, the optimizer is AdamW (Loshchilov and Hutter, 2019), the training batch size is 16, the learning rate is 0.005,
Method	SA	NLI	EJ	PI
IMDB	SST-2	laptop	restaurant	Movie	Tweet	MNLI	QNLI	SNLI	deontology	justice	QQP	MRPC
From BERTBASE to RoBERTaBASE
PT on RoBERTaBASE	89.9	93.8	77.3	80.7	79.2	74.5	80.6	90.5	88.5	72.9	70.0	86.9	83.9
Random Prompt	50.6	50.8	2.3	1.2	50.5	40.5	32.8	50.5	33.3	50.4	50.2	36.8	68.0
IMDB, laptop	Distance Minimizing	89.7	53.1	75.6	18.3	54.2	24.0	31.2	50.0	33.3	50.6	50.0	36.8	67.2
Task Tuning	88.2	82.2	76.3	77.9	73.4	43.6	32.0	47.9	32.8	49.8	49.4	50.2	47.7
MNLI	Distance Minimizing	55.6	51.0	2.5	1.4	53.1	41.1	80.0	50.6	33.3	50.6	50.0	48.3	68.0
Task Tuning	50.9	52.0	11.9	13.1	45.8	18.2	80.0	74.9	80.0	50.4	49.9	36.8	68.1
From RoBERTaBASE to RoBERTaLARGE
PT on RoBERTaLARGE	91.8	96.0	78.1	81.7	81.7	76.6	88.5	93.4	90.7	85.6	81.1	89.0	82.7
Random Prompt	50.1	50.2	2.0	2.0	49.5	40.5	32.7	51.0	33.3	50.3	49.9	40.6	61.2
IMDB, laptop	Distance Minimizing	92.1	50.1	77.0	1.4	51.0	37.6	33.1	50.2	32.8	50.4	50.0	62.3	38.3
Task Tuning	90.4	76.2	64.2	69.5	79.7	45.0	33.3	50.5	33.1	50.3	50.0	38.5	79.7
MNLI	Distance Minimizing	50.3	51.2	5.2	5.9	51.0	40.6	88.5	49.1	33.2	50.3	50.0	45.1	66.4
Task Tuning	67.7	76.1	28.9	43.7	60.4	49.1	87.1	79.4	84.5	49.7	50.0	36.8	68.5
From T5BASE to T5XXL
PT on T5XXL	96.5	97.4	76.6	88.1	97.9	72.5	90.5	95.2	93.4	87.0	92.5	90.0	86.3
Random Prompt	49.7	49.0	19.8	17.0	51.6	15.5	31.8	49.3	31.9	51.3	50.0	36.4	67.0
laptop	Distance Minimizing	49.0	49.7	76.6	17.0	52.3	16.3	31.8	48.7	33.3	54.1	49.0	36.7	67.7
Task Tuning	77.2	86.2	80.3	83.5	64.6	55.2	31.9	49.9	32.9	48.7	52.8	50.7	53.1
MNLI	Distance Minimizing	49.7	49.0	19.8	17.1	51.6	15.5	90.5	49.3	34.8	52.3	50.0	36.8	67.7
Task Tuning	54.9	70.0	60.8	74.1	3.6	41.4	89.7	84.8	90.8	49.7	50.0	37.2	66.4
Table 7: We conduct experiments between various PLMs in different scales and heterogeneous frameworks: from  $\mathrm{BERT}_{\mathrm{BASE}}$  to  $\mathrm{RoBERTa}_{\mathrm{BASE}}$ , from  $\mathrm{RoBERTa}_{\mathrm{BASE}}$  to  $\mathrm{RoBERTa}_{\mathrm{LARGE}}$ , and from  $\mathrm{T5}_{\mathrm{BASE}}$  to  $\mathrm{T5}_{\mathrm{XXL}}$ . Besides, we highlight the non-trivial zero-shot performance (\%) of the cross-model setting with **bold**.
and the inner hidden size  $d_h$  is 768. In this paper, we investigate cross-model transfer among various PLMs including BERT<sub>BASE</sub>, RoBERTa<sub>BASE</sub>, RoBERTa<sub>LARGE</sub>, T5<sub>SMALL</sub>, T5<sub>BASE</sub>, and T5<sub>XXL</sub>, whose hidden sizes are 768, 768, 1024, 512, 768, and 1024, respectively. Besides, for non-linear activation functions, we have tried tanh and LeakyReLU (Xu et al., 2015), and find their performance on various PLMs are similar. The reported results are based on LeakyReLU.
C.2 More Zero-shot Transfer Performance
In § 5.2, we showed the zero-shot transfer performance of various projector-learning methods in the setting of transferring from RoBERTaLARGE to T5XXL. We explore more cross-model transfer settings here, which are transferring between various PLMs in different scales and heterogeneous frameworks, including from BERTBASE to RoBERTaBASE, from RoBERTaBASE to RoBERTaLARGE, and from T5BASE to T5XXL. We can find that the results in Table 7 are all consistent with § 5.2.
C.3 Technical Details of TPTMODEL
In § 5.3, we demonstrate cross-model transferrable prompt tuning  $(\mathrm{TPT}_{\mathrm{MODEL}})$  can well improve per
formance and reduce training time.
However, when we apply  $\mathrm{TPT}_{\mathrm{MODEL}}$  to more PLMs, we find that the projected prompts may have quite different  $L_{2}$  norm values with the original prompts, especially for the small-scale PLMs (e.g., from  $\mathrm{BERT}_{\mathrm{BASE}}$  to  $\mathrm{RoBERTa}_{\mathrm{BASE}}$ ). Specifically, we obtain the projected prompts with the trained Task Tuning projector, and find that the projected prompts are hard to optimize in some tasks as shown in Figure 7 [Without LayerNorm]. Thus, we attempt to add the layer normalization operation (Ba et al., 2016) LayerNorm into the projectors to regularize the norm of the projected prompt as follows:
$$
\tilde {\mathbf {P}} ^ {s} = \text {L a y e r N o r m} (\mathbf {P r o j} (\mathbf {P} ^ {s})). \tag {10}
$$
By the LayerNorm, the projected prompts can work well on  $\mathrm{TPT}_{\mathrm{MODEL}}$  and achieve better performance and speedup as shown in Figure 7 [With LayerNorm]. Interestingly, although prompts projected by the projectors [Without LayerNorm] are hard to be trained in  $\mathrm{TPT}_{\mathrm{MODEL}}$ , they can achieve similar zero-shot transfer performance with the prompts projected by the projectors [With LayerNorm] in Table 8.
(a) MNLI training loss. [Without LayerNorm]
(b) MNLI evaluation accuracy. [Without LayerNorm]
(c) MNLI training loss. [With LayerNorm]
(d) MNLI evaluation accuracy. [With LayerNorm]
(e)IMDB training loss. [Without LayerNorm]
(f)IMDB evaluation accuracy.[Without LayerNorm]
(g)IMDB training loss. [With LayerNorm]
(h) IMDB evaluation accuracy. [With LayerNorm]
(i) restaurant training loss. [Without LayerNorm]
(j) restaurant evaluation accuracy. [Without LayerNorm]
(k) restaurant training loss. [With LayerNorm]
(1) restaurant evaluation accuracy. [With LayerNorm]
Figure 7: The  $(—)$  represents vanilla PT on RoBERTa<sub>BASE</sub>. As for  $(—)$ , it utilizes projected prompts from BERT<sub>BASE</sub> as initializations to conduct PT on RoBERTa<sub>BASE</sub>. The projected prompts respectively come from two different Task Tuning projectors: [Without LayerNorm] and [With LayerNorm].
Method	SA	NLI	EJ	PI
IMDB	SST-2	laptop	restaurant	Movie	Tweet	MNLI	QNLI	SNLI	deontology	justice	QQP	MRPC
PT on RoBERTaBASE	89.9	93.8	77.3	80.7	79.2	74.5	80.6	90.5	88.5	72.9	70.0	86.9	83.9
[Without LayerNorm]
Task Tuning (IMDB, laptop)	86.5	84.9	73.4	75.3	76.6	47.7	31.8	52.0	32.9	50.3	50.0	37.6	67.5
Task Tuning (MNLI)	66.6	70.4	53.0	43.8	57.8	47.9	82.4	74.9	78.1	50.4	49.9	45.3	70.1
[With LayerNorm]
Task Tuning (IMDB, laptop)	88.2	82.2	76.3	77.9	73.4	43.6	32.0	47.9	32.8	49.8	49.4	50.2	47.7
Task Tuning (MNLI)	50.9	52.0	11.9	13.1	45.8	18.2	80.0	74.9	80.0	50.4	49.9	36.8	68.1
Table 8: We find that the zero-shot performances of prompts projected by two Task Tuning projectors ([With LayerNorm] and [Without LayerNorm]) are close. Bold represents non-trivial performance.
D Transferability Indicator
D.1 Effectiveness of Similarity Metrics
We categorize all prompts into three groups: same tasks (prompts trained with different seeds on the same dataset), same-type tasks, and different-type tasks. Table 9 shows that all the similarity metrics
successfully distinguish task types.
D.2 Correlation Between Prompt Transferability and Prompt Similarity
In § 6, we provide the overall averaged Spearman's rank correlation scores (\%) between various similarity metrics and zero-shot transfer performance
Metric	Same Tasks	Same-type Tasks	Different-type Tasks
RoBERTaLARGE
Econcat	9.4	9.4	6.8
Eaverage	41.6	41.4	37.6
Cconcat	47.6	45.3	31.7
Caverage	1.7	1.3	1.1
ON (Bottom 3)	42.8	43.3	39.1
ON (Top 3)	39.4	28.2	21.4
ON (All 24)	40.0	35.8	29.6
T5XXL (Decoder Module)
Econcat	0.5	0.5	0.3
Eaverage	4.0	5.1	3.4
Cconcat	29.4	2.8	2.4
Caverage	4.0	2.6	2.1
ON (Bottom 3)	80.3	75.4	76.3
ON (Top 3)	62.0	52.7	46.1
ON (All 24)	60.8	54.0	49.2
Table 9: The average values  $(\%)$  of the 5 similarity metrics for prompt pairs within the same task (trained with 3 different random seeds) and between different tasks (of the same type and different types) on RoBERTaLARGE and T5XXL.
Metric	SA	NLI	EJ	PI	QA	SUM	All
T5SMALL (Decoder Module)
Econcat	10.1	19.6	31.3	5.3	27.3	38.0	21.9
Eaverage	-6.8	-28.0	18.7	-2.6	29.1	42.9	8.9
Cconcat	34.6	63.6	26.6	19.3	-2.1	12.5	25.7
Caverage	64.3	65.1	30.7	15.7	27.7	19.2	37.1
ON (Bottom 3)	32.9	72.6	41.8	14.2	45.5	52.8	43.3
ON (Top 3)	50.6	74.8	51.4	2.6	60.3	78.8	52.5
ON (All 24)	44.8	79.7	44.5	6.3	59.7	67.9	50.5
T5BASE (Decoder Module)
Econcat	55.2	-17.0	10.2	21.5	5.9	-1.1	20.8
Eaverage	53.4	-42.3	-10.7	7.5	-27.7	-10.8	9.0
Cconcat	57.2	25.2	35.1	37.0	30.2	-20.5	28.4
Caverage	47.6	70.0	30.4	48.0	34.9	16.8	42.4
ON (Bottom 3)	34.7	29.8	40.8	16.9	24.2	72.2	36.0
ON (Top 3)	53.8	24.3	50.6	46.1	54.7	79.1	49.1
ON (All 24)	46.1	25.0	42.6	39.7	56.7	72.3	43.4
T5XXL (Decoder Module)
Econcat	40.8	-13.4	19.3	11.4	-4.3	-19.5	12.9
Eaverage	32.2	-42.6	9.7	-2.0	-27.7	-34.0	-2.5
Cconcat	21.4	40.9	42.6	24.6	30.2	45.6	31.6
Caverage	23.3	44.8	33.3	29.3	34.9	49.9	33.5
ON (Bottom 3)	9.1	20.7	14.8	18.3	24.2	-9.9	12.4
ON (Top 3)	42.7	33.6	39.1	30.3	54.7	11.1	36.9
ON (All 24)	31.0	23.6	37.7	34.2	56.7	15.4	32.0
ONI (Bottom 3)	--	--	--	--	--	--	25.3
ONI (Top 3)	--	--	--	--	--	--	46.3
ONI (All 24)	--	--	--	--	--	--	40.0
Table 10: Spearman's rank correlation scores  $(\%)$  between various similarity metrics and zero-shot transfer performance of soft prompts for various scales of T5 and  $\mathrm{ON_I}$  as introduced in appendix D.3.
Metric	SA	NLI	EJ	PI	All
RoBERTaBASE
Econcat	31.1	-5.9	30.5	16.2	20.2
Eaverage	17.2	-52.4	12.1	-13.5	-4.4
Cconcat	51.6	8.8	38.5	29.7	36.3
Caverage	65.8	55.9	26.1	28.9	51.7
ON (Bottom 3)	56.2	64.3	17.9	21.2	46.8
ON (Top 3)	77.9	74.2	43.4	32.7	64.8
ON (All 24)	71.2	70.5	33.6	25.0	58.1
RoBERTaLARGE
Econcat	42.5	-16.3	21.4	22.8	22.6
Eaverage	34.5	-55.1	-5.8	3.6	2.8
Cconcat	44.5	-11.7	23.6	22.0	24.8
Caverage	38.2	77.1	12.4	47.8	44.7
ON (Bottom 3)	32.0	34.8	44.5	30.3	34.3
ON (Top 3)	70.9	45.6	13.5	28.9	49.7
ON (All 24)	62.7	40.6	16.0	31.1	45.6
Table 11: Spearman's rank correlation scores  $(\%)$  between various similarity metrics and zero-shot transfer performance of soft prompts for various scales of RoBERTa.
of soft prompts for RoBERTaLARGE and T5XXL.
Here, we further show Spearman's rank correlation scores grouped by the task types on more PLMs. The results are shown in Table 10 and Table 11.
D.3 PLMs' Redundancy Influence Indicators
From Table 10, we find that the correlation between prompt transferability and prompt similarity will drop with the increase of PLM size. We guess that this phenomena may result from PLMs' high redundancy (Aghajanyan et al., 2021).