Venue: NAACL 2024

# Is Prompt Transfer Always Effective? An Empirical Study of Prompt Transfer for Question Answering

Minji Jung\({}^{1,3\dagger\lx@sectionsign}\)

Soyeon Park\({}^{2\ddagger}\)

Jeewoo Sul\({}^{2,3\dagger\lx@sectionsign}\)

Yong Suk Choi\({}^{1,2*}\)

###### Abstract

Prompt tuning, which freezes all parameters of a pre-trained model and only trains a soft prompt, has emerged as a parameter-efficient approach. For the reason that the prompt initialization becomes sensitive when the model size is small, the prompt transfer that uses the trained prompt as an initialization for the target task has recently been introduced. Since previous works have compared tasks in large categories (e.g., summarization, sentiment analysis), the factors that influence prompt transfer have not been sufficiently explored. In this paper, we characterize the question answering task based on features such as answer format and empirically investigate the transferability of soft prompts for the first time. We analyze the impact of initialization during prompt transfer and find that the train dataset size of source and target tasks have the influence significantly. Furthermore, we propose a novel approach for measuring catastrophic forgetting and investigate how it occurs in terms of the amount of evidence. Our findings can help deeply understand transfer learning in prompt tuning1.

Footnote 1: Equal contributions. Alphabetical order.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote †: Corresponding author.

+
Footnote

we performed gradient updates to the following log-likelihood objective: \({\max_{\Theta}\sum^{N}_{i}\log p_{\Theta}(y_{i}|x_{i})}\) where \(x_{i}\) is the input text, and \(y_{i}\) is the output sequence.

\[\underset{\theta_{\mathbf{P}}}{\mathrm{max}}\!\!\sum^{N}_{i}\!\!\log p_{\theta ,\theta_{\mathbf{P}}}(y_{i}|[\mathbf{P};x_{i}]) \tag{1}\]

The prompt tuning method proposed in Lester et al. (2021) is represented by Equation 1. The parameter of a pre-trained language model \(\theta\) is fixed, and only the prompt parameter \(\theta_{\mathbf{P}}\) of the soft prompt \(\mathbf{P}=[p_{1},p_{2},\ldots,p_{l}]\in\mathbb{R}^{l\times d}\) is learnable. We use the prompt length \(l=100\), and \(d\) is the input dimension of the model.

### Datasets

Following the two classification systems from Rogers et al. (2023), we show 16 QA datasets2 used in our analysis in Table 1. Detailed descriptions of each dataset are provided in Appendix A.

Footnote 2: In cases where only one of valid or test datasets was available such as Rajpurkar et al. (2016), we used it in the testing process. Additionally, we split the train datasets into a 9:1 ratio, and used it in the train and valid process, respectively. The number of datasets we used is shown in Table 1.

First, the amount of evidence is how much evidence is provided to answer the question. _Single Source_ indicates that the information required to answer the question is explicitly contained within a context. Partial Source means that although some evidence is available, it needs to be integrated with external knowledge to answer the question. **No Source** needs to find answers solely from implicit knowledge. The more evidence available to answer a question, the more explicit knowledge exists; conversely, the less evidence, the more implicit knowledge exists.

Second, the answer format is divided into four types. Extractive format refers to when the answer span can be found within the provided context. Categorical format denotes that the correct answer is in a pre-defined option, exclusively employing yes or no formats in our dataset. Multi-choice format indicates that answer options are given, and the answer is to be chosen from among them. Lastly, Freeform format refers to cases where the model generates answers without following a specific format.

## 3 Results and Analysis

To study the transferability of soft prompts, we used 16 QA datasets as the source and target tasks. The main terms referred to in this section are as follows: (1) vanilla prompt tuning (Vanilla PT), the result of training the prompt in Equation 1 after random initializing; (2) zero-shot performance, the result of solving the target task using the source prompt without additional training; and (3) prompt transfer (PoT), the result of initializing the target prompt with the selected source prompt and training it as shown in Equation 1. For our experiments, we used the T5BASE3 as our base LM. Further experimental details are in Appendix B.

Footnote 3: [https://huggingface.co/t5-base](https://huggingface.co/t5-base)

### Transferability with Initialization

Can transferability be interpreted as cosine similarity?As shown in Figure 1, we investigated the prompt transferability with cosine-similarity. We can observe that prompt embeddings with the

\begin{table}
\begin{tabular}{l|c c r r r} \hline \hline
**Dataset** & **Answer format** & **Amount of evidence** & **Train** & **Valid** & **Test** \\ \hline \hline DuoRC Saha et al. (2018) & Freeform & Partial & 60,094 & 12,845 & 12,415 \\ NQ-Open Lee et al. (2019) & Freeform & **No** & 79,132 & 8,793 & 3,610 \\ WO Berant et al. (2013) & Freedom & **No** & 3,400 & 378 & 2,032 \\ MRQA-NewSQA Trischler et al. (2017) & Extrative & _Single_ & 66,744 & 7,416 & 4,212 \\ SQuAD Rajpurkar et al. (2016) & Extractive & _Single_ & 78,839 & 8,760 & 10,570 \\ BoolQ Clark et al. (2019) & Categorical & _Single_ & 8,484 & 943 & 3,270 \\ MultiRC Khashabi et al. (2018) & Categorical & _Single_ & 24,518 & 2,725 & 4,848 \\ TBA Joshi et al. (2017) & Freeform & Partial & 78,859 & 8,763 & 11,313 \\ CosmosQA Huang et al. (2019) & Multi-choice & Partial & 22,735 & 2,527 & 2,985 \\ SIQA Sap et al. (2019) & Multi-choice & Partial & 30,069 & 3,341 & 1,954 \\ \hline SQuAD w/o ctx & Freeform & **No** & 78,839 & 8,760 & 10,570 \\ BoolQ w/o ctx & Categorical & **No** & 8,484 & 943 & 3,270 \\ MultiRec w/o ctx & Categorical & **No** & 24,518 & 2,725 & 4,848 \\ TOA w/o ctx & Freeform & **No** & 78,859 & 8,763 & 11,313 \\ CosmosQA w/o ctx & Multi-choice & **No** & 22,735 & 2,527 & 2,985 \\ SIQA w/o ctx & Multi-choice & **No** & 30,069 & 3,341 & 1,954 \\ \hline \hline \end{tabular}
\end{table}
Table 1: The details of QA datasets. ”w/o ctx” refers to the removal of context from the original dataset to evaluate the influence of the amount of evidence.

 

same answer formats are clustered together in Figure 1(b). However, Figure 1(a) demonstrates that the high similarity score between the source and target task does not necessarily result in positive transferability. For example, even though the transfer BoolQ Clark et al. (2019)\(\rightarrow\)MultiRC Khashabi et al. (2018) has the highest similarity score of \(0.9\), it yields a negative transferability of \(-2.8\%\). We note that the PoT performance varies significantly depending on the target task. Therefore, prompt initialization with high cosine-similarity does not guarantee performance improvement. As a result, we find that it is not suitable to interpret transferability through cosine-similarity in the QA task.

Can transferability be interpreted as zero-shot performance?To verify the effectiveness of selecting the best zero-shot prompt when used for initialization, we compare PoT performance between the best and worst zero-shot prompts in Table 2. When initialized with the best zero-shot prompt, it only outperforms the worst one in 7 out of 16

Figure 1: (a) Heatmap of our task transferability results. (b) Heatmap of the cosine similarities between the source prompt embeddings. The colors of the task names indicate the answer format type: Blue, Extractive; Green, Categorical; Brown, Freeform; Yellow, Multi-choice.

 

cases. The mean absolute error was \(1.58\), indicating that the performance difference is approximate. Additionally, Figure 5 and Figure 6 indicate that most cases converge to similar values as the epoch progresses, regardless of which source prompt is selected. It can therefore be seen that the method proposed in Su et al. (2022) cannot assure better or comparable transfer performance in the QA task.

Effect of Dataset SizeIn Table 4, the PoT performance varies considerably depending on the target task. Therefore, we applied min-max normalization4 to each target task to compare the correlation between the source and target tasks. We classified the QA datasets based on the number of train datasets into small, medium, and large (see Appendix D). Subsequently, we divided into three groups 5 founded on the difference in size between the source task and target task as follows: _Target > Source_, _Same_ and _Target < Source_.

Footnote 4: See the formula in Appendix C.

Footnote 5: For example, _Target > Source_, indicating the train dataset group of the target task is larger than the source task (_e.g._, target task: large, source task: small).

As shown in Figure 2, the normalized task transferability results are based on the difference in the dataset group size between the source task and the target task. Regarding the _Target < Source_ group, most cases show positive transferability. The median (Q2) of each box plot indicates a tendency to drop in the sequence of _Target < Source_, _Same_, and _Target > Source_. We demonstrate that the dataset size of the source and target tasks in the QA task is a key factor in transferability.

Investigating Catastrophic Forgetting FormulaCatastrophic forgetting (Kirkpatrick et al., 2017) is the tendency for previously learned task knowledge to be abruptly lost as information relevant to the current task is incorporated. However, there is still no clear method for measuring this phenomenon.

Therefore, we propose a novel metric for evaluating catastrophic forgetting:

\[\frac{(Zero\text{-}shot\ correct)\cap(PoT\ incorrect)}{Zero\text{-}shot\ correct} \tag{2}\]

where _Zero-shot correct_ is the case of correct responses in a zero-shot setting, and _PoT incorrect_ is the case of incorrect answers after prompt transfer in the target task. In a zero-shot setting, correct responses indicates that the trained prompt from the source task retains valuable information for the target task. On the other hand, incorrect answers after additional learning with the target task indicate forgetting of source task knowledge relevant to the target task.

We analyse catastrophic forgetting in terms of the amount of evidence in the QA datasets. _Single Source_ use the most explicit knowledge, followed by Partial Source, and **No Source**. The relationship between explicit and implicit knowledge is a trade-off. When comparing the quantity of explicit and implicit knowledge with the amount of evidence, Equation 2 is used for cases where the target task has a bigger, equal, or smaller amount of explicit knowledge than the source task.

Analyzing Catastrophic ForgettingAs illustrated in Figure 3, the results compare the extent of catastrophic forgetting based on the levels of explicit and implicit knowledge in each dataset. If the source task has more explicit knowledge or less implicit knowledge than the target task, catastrophic forgetting tends to occur. In the right side6 of Figure 3, _Partial-Single_, _No-Single_, and _No-Partial_ are displayed mixed together and the left also shows a similar trend. As a result, the existence of a knowledge gap between the source and target task is more influential in catastrophic forgetting than the extent of the knowledge gap.

Figure 2: Normalized transferability difference based on the train datasets size. The X-axis refers to each case where the target dataset size is bigger, equal, and smaller than the source dataset size. The Y-axis denotes the difference between the PoT performance and the vanilla prompt performance after normalization.

 



## 4 Conclusion

In this paper, we study PoT in the QA task. In particular, we empirically investigate prompt initialization, demonstrating that the difference of train dataset size between source and target tasks is affecting the transferability. We also define a novel method to measure catastrophic forgetting and show that there is a relationship between the amount of evidence in QA datasets and the tendency of catastrophic forgetting. Finally, our fine-grained analyses provide meaningful insights to help improve the performance of PoT.

### Limitations

Our paper has two limitations as follows: First, we only perform all experiments on the T5BASE model. We cannot serve results on various models and model sizes because of our limited computational resources. Secondly, although we show the type of occurrence for catastrophic forgetting by our proposed evaluation metric, we do not present an approach to mitigate them.

In further experiments, we observe the possibility that prompt transferability could be influenced by different model architectures, prefixes, or other factors. Therefore, in the future work, we will explore strategies to save the knowledge of source tasks related to target tasks and investigate the use of various backbone models.

## Acknowledgements

We thank anonymous reviewers for valuable feedback and helpful suggestions. This work was supported in part by the National Research Foundation of Korea (NRF) under Grant 2018R1A5A7059549 and Grant 2020R1A2C1014037; and in part by the Institute of Information and Communications Technology Planning and Evaluation (IITP), funded by the Korean Government [Ministry of Science and Information and Communication Technology (MSIT)] under Grant 2020-0-01373.

## References

* Asai et al. (2022) Akari Asai, Mohammadreza Salehi, Matthew Peters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In _Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing_, pages 6655-6672, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
* Berant et al. (2013) Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on Freebase from question-answer pairs. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 1533-1544, Seattle, Washington, USA. Association for Computational Linguistics.
* Brown et al. (2020) Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Prauw Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In _Advances in Neural Information Processing Systems_, volume 33, pages 1877-1901. Curran Associates, Inc.
* Clark et al. (2019) Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In _Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers)_, pages 2924-2936, Minneapolis, Minnesota. Association for Computational Linguistics.
* Devlin et al. (2018) Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep

Figure 3: Percentage of Catastrophic Forgetting based on the amount of evidence. The X-axis shows the differences between the target and source task according to the amount of evidence. The Y-axis represents the percentage of catastrophic forgetting. Each label indicates the amount of evidence type in Target-Source order.



bidirectional transformers for language understanding. _arXiv preprint arXiv:1810.04805_.
* Fisch et al. (2019) Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In _Proceedings of the 2nd Workshop on Machine Reading for Question Answering_, pages 1-13, Hong Kong, China. Association for Computational Linguistics.
* Huang et al. (2019) Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
* Joshi et al. (2017) Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
* Khashabi et al. (2018) Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers)_, pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
* Kirkpatrick et al. (2017) James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A Rusu, Kieran Milan, John Quan, Tiago Ramalho, Agnieszka Grabska-Barwinska, et al. 2017. Overcoming catastrophic forgetting in neural networks. _Proceedings of the national academy of sciences_, 114(13):3521-3526.
* Kwiatkowski et al. (2019) Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polouschulin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Lee et al. (2019) Kenton Lee, Ming-Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open domain question answering. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 6086-6096, Florence, Italy. Association for Computational Linguistics.
* Lester et al. (2021) Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In _Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing_, pages 3045-3059, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
* Li et al. (2022) Junyi Li, Tianyi Tang, Jian-Yun Nie, Ji-Rong Wen, and Xin Zhao. 2022. Learning to transfer prompts for text generation. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3506-3518, Seattle, United States. Association for Computational Linguistics.
* Patro and Sahhu (2015) SGOPAL Patro and Kishore Kumar Sahu. 2015. Normalization: A preprocessing stage. _arXiv preprint arXiv:1503.06462_.
* Raffel et al. (2020) Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. _The Journal of Machine Learning Research_, 21(1):5485-5551.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Rogers et al. (2023) Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2023. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading comprehension. _ACM Computing Surveys_, 55(10):1-45.
* Saha et al. (2018) Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1683-1693, Melbourne, Australia. Association for Computational Linguistics.
* Sap et al. (2019) Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Commonsense reasoning about social interactions. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JICNLP)_, pages 4463-4473, Hong Kong, China. Association for Computational Linguistics.
* Su et al. (2022) Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2022. On transferability of prompt tuning for natural language processing. In _Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 3949-3969, Seattle, United States. Association for Computational Linguistics 

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In _Proceedings of the 2nd Workshop on Representation Learning for NLP_, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
* Vaswani et al. (2017) Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30. Curran Associates, Inc.
* Vu et al. (2022) Tu Vu, Brian Lester, Noah Constant, Ramil Al-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In _Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
* Wang et al. (2023) Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogerio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. In _The Eleventh International Conference on Learning Representations_.
* Xie et al. (2023) Kaige Xie, Tong Yu, Haoliang Wang, Junda Wu, Handong Zhao, Ruiyi Zhang, Kanak Mahadik, Ani Nenkova, and Mark Riedl. 2023. Few-shot dialogue summarization via skeleton-assisted prompt transfer. _arXiv preprint arXiv:2305.12077_.
* Zhong et al. (2022) Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. _arXiv preprint arXiv:2208.10160_.

## Appendix A Datasets

Table 1 displays the datasets used in our experiments. More descriptions of each dataset are as follows:

* **DuoRC**Saha et al. (2018) is a reading comprehension dataset with low lexical overlap between questions and context. It has unique question-answer pairs generated from a movie plots collection. The original dataset included _no answer_, but we only used the data that has an answer. Background and common sense are required to derive the answer, surpassing the context's explicit knowledge. We used the dataset from [https://huggingface.co/datasets/duorc/viewer/SelfRC](https://huggingface.co/datasets/duorc/viewer/SelfRC).
* The original Natural Questions (NQ) dataset was introduced by Kwiatkowski et al. (2019). The **NQ-Open**, which removes the context from NQ, was introduced by Lee et al. (2019). We used the dataset from [https://huggingface.co/datasets/nq_open](https://huggingface.co/datasets/nq_open).
* The WebQuestions (**WQ**, Berant et al., 2013) dataset consists of questions that can be answered via Freebase. The dataset link we used, [https://huggingface.co/datasets/web_questions](https://huggingface.co/datasets/web_questions), only provides a freebase link, so we did not use a separate context.
* **MRQA-NewsQA**. Newsa (Trischler et al., 2017) is a machine comprehension dataset composed of CNN news articles. The answers consist of spans of texts from the article. We used the NewsQA dataset from [https://huggingface.co/datasets/mrqa](https://huggingface.co/datasets/mrqa), within the MRQA (Fisch et al., 2019) benchmark.
* The Stanford Question Answering Dataset (**SQuAD**, Rajpurkar et al., 2016) is a benchmark dataset in machine reading comprehension. It comprises Wikipedia articles accompanied by question-answer pairs formulated by human annotators. Answers in SQuAD are spans of text directly extracted from the provided context. We used the dataset from [https://huggingface.co/datasets/squad](https://huggingface.co/datasets/squad).
* **BoolQ**Clark et al. (2019) is a format of yes/no questions. A context is given along with question-answer pairs. We used the dataset from [https://huggingface.co/datasets/boolq](https://huggingface.co/datasets/boolq).
* The **MultiRC**Khashabi et al. (2018) dataset consists of a paragraph (context), question, and answer as well as a label to determine whether the answer to the question was correct. We used this dataset to solve a categorical answer format problem that determines whether the answer to a question is correct. We used the dataset from [https://huggingface.co/datasets/super_glue/viewer/multicr](https://huggingface.co/datasets/super_glue/viewer/multicr).
* TriviaQA (**TQA**, Joshi et al., 2017) is a reading comprehension dataset, which is more challenging than other QA datasets because the questions cannot be answered directly by span prediction and the context is much longer than other benchmarks. We used two versions of TQA: the _unfiltered_ version (TQA) 

and the _unfiltered.nocontext_ version (TQA w/o ctx). We used the dataset from [https://huggingface.co/datasets/trivia_qa](https://huggingface.co/datasets/trivia_qa).
* **CosmosQA**Huang et al. (2019) requires commonsense-based reading comprehension and consists of questions that require additional knowledge rather than extracting spans from the context. The answer is in the form of choosing one of four options. We used the dataset from [https://huggingface.co/datasets/cosmos_qa](https://huggingface.co/datasets/cosmos_qa).
* Social Interaction QA (**SIQA**, Sap et al., 2019) is a dataset for testing commonsense reasoning about social situations. The answer is to choose one of three options. We used the dataset from [https://huggingface.co/datasets/social_i_qa](https://huggingface.co/datasets/social_i_qa).

## Appendix B Training Details

In prompt tuning, we trained a soft prompt using a NVIDIA RTX A5000 single GPU with 23GB memory. We applied the AdamW optimizer with a learning rate 0.005, set batch size of 16, and used early stopping in three steps. We set the soft prompt length \(l\)=100, which is the same as most prompt transfer settings Vu et al. (2022); Asai et al. (2022); Su et al. (2022); Wang et al. (2023).

## Appendix C Min-Max Normalization

The PoT performance for each target task is normalized using a formula derived with reference to Patro and Sahu (2015). We remove the denominator from the formula because sometimes it becomes zero. The formula we used is as follows :

Normalized PoT score \[=\frac{(PoT~{}score)-(Vanilla~{}PT~{}score)}{\max(PoT~{}score)- \min(PoT~{}score)}\] (3)

## Appendix D Comparing Train Dataset Size

Figure 4 illustrates the categorization of QA datasets based on the size of train datasets.

## Appendix E Prompt Transfer Performance in Each Epoch

Figure 5 and Figure 6 show that as the epoch progresses, the influence of initialization gradually decreases. The red and blue lines denote the scores per epoch for the **Best Source Task** and **Worst Source Task** shown in Table 2. Specifically, even though some prompts have EM score of 0 in the zero-shot setting, they achieve better or comparable PoT performance than prompts with the best zero-shot performance.

## Appendix F Zero Shot Performance

The full results of zero-shot performance are shown in Table 3.

 

* [25] M. C. Bader, J. A. Krizhevsky, and D. V. Kingma, "Adam: a method for stochastic optimization with applications in deep learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, 2014, pp. 378-396.
* [26] S. Ghahramani, L. R. Bishop, and Y. Bengio, "Deep residual networks for image recognition," _IEEE Transactions on Image Processing Systems_**10**, no. 1, 2015, pp. 1-10.
* [27] T. H. Nguyen, P. N. Torr, and X. Zhang, "Image denoising by reducing the number of pixels," _IEEE transactions on pattern analysis and machine intelligence_**10**, no. 1, 2015, pp. 1-12.

[MISSING_PAGE_POST]

 

* [26] M. C. Bader, J. A. Krizhevsky, and D. V. Kingma, "Adam: a method for stochastic optimization with applications in deep learning," in _Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR)_, pp. 1-40, 2015.
* [27] S. Ghahramani, P. L. LeCun, and R. Fergus, "Deep residual networks for image recognition," _IEEE Transactions on Image Processing Systems_, vol. 38, no. 9, pp. 1043-1050, 2016.

[MISSING_PAGE_POST]

 

* [14] M. C. Bender, J. R. Friedman, and A. K. Sigmund, "Theoretical Population of the Milky Way: The Case of the Galaxy," _Astrophys. J._**672**, L35-L40, (2008)

[MISSING_PAGE_POST]

 

* [14] J.-P. Wang, M.-C. Zhang, and Y.-H. Chen, "A new method for the estimation of the unknown parameters in a two dimensional model," _Journal of Computational Physics: Series B_**206**(3), no. 5, pp. 798-803, 2000.
* [15] A. Zarembo, R.-M. Li, and S.-Y. Zhu, "The estimation of the unknown parameters using the generalized least squares method," _Computational Mathematics_**201**(1), no. 1, pp. 1-10, 2000.

[MISSING_PAGE_POST]