Few-shot Unified Question Answering: Tuning Models or Prompts?
Authors: Srijan Bansal
Semih Yavuz
Bo Pang
Meghana Moorthy Bhat
Yingbo Zhou
Venue: EMNLP 2023 Findings (2023)
Keywords: unified question answering
QA
universal QA
paramter efficient QA
model tuning . prompt tuning
TL;DR: Prompt tuning can achieve comparable performance to model tuning with a good initialization for unified question-answering in low-resource scenarios.

Abstract: Question-answering (QA) tasks often investigate specific question types, knowledge domains, or reasoning skills, leading to specialized models catering to specific categories of QA tasks. While recent research has explored the idea of unified QA models, such models are usually explored for high-resource scenarios and require re-training to extend their capabilities. To overcome these drawbacks, the paper explores the potential of two paradigms of tuning, model, and prompts, for unified QA under a low-resource setting. The paper provides an exhaustive analysis of their applicability using 16 QA datasets, revealing that prompt tuning can perform as well as model tuning in a few-shot setting with a good initialization. The study also shows that parameter-sharing results in superior few-shot performance, simple knowledge transfer techniques for prompt initialization can be effective, and prompt tuning achieves a significant performance boost from pre-training in a low-resource regime. The research offers insights into the advantages and limitations of prompt tuning for unified QA in a few-shot setting, contributing to the development of effective and efficient systems in low-resource scenarios.

Few-shot Unified Question Answering: Tuning Models or Prompts?
Srijan Bansal
Semih Yavuz" Bo Pang
Meghana Bhat
Yingbo Zhou'
Carnegie Mellon University, srijanbeandrew. cmu, edu, {syavuz, b. p.pa,g, meghana.. bhat shaattesalsfoorcccomm yingbo.zhou @ salesforce.com
Salesforce Research
Abstract
FT-MT
PT-C
PT-F
ATT-MT
Question-answering (QA) tasks often investi- gate specific question types, knowledge do- mains, reasoning skills, leading to special- ized models catering to specific categories of QA tasks. While recent research has explored the idea of unified QA models, such models are usually explored for high-resource scenar- ios and require re- re-training to extend their ca- pabilities. To overcome these drawbacks, the paper explores the potential of two paradigms model tuning and prompt tuning for uni- fied QA under low-resource setting. The paper provides an exhaustive analysis of their applicability using 16 QA datasets spanning across various domains, revealing that prompt tuning outperforms model tuning in a few- shot setting with good initialization for out-of- distribution target tasks. The study also shows that parameter-sharing results in superior few- shot performance, and simple knowledge trans- fer techniques for prompt initialization can be effective low-resource regime. The re- search offers insights into the advantages and limitations of prompt tuning for unified QA in a few-shot setting, contributing to the devel- opment of effective and efficient systems in low-resource scenarios.
55.00
50.00
Mean Performance
45.00
(Domain)
(Skil)
Figure 1: Comparison of Multi-Task Model-Tuning (FT- MT), Prompt-Tuuee Tuning with format based (PT-F) and format agnostic (PT-C), and ATTEMPT (ATT-MT) (Asai et al., 2022) (a complex prompt transfer learning approach) for Unified QA on target QA datasets in a 32-shot scenario using T5- the backbone model.
The results show that prompt-tuning with prior outper- forms multi-task full-model fine-tuning and ATTEMPT does not provide any additional advantage, especially for out-of of-domiin target tasks. Here IID has been used to refer In-Distribution while OOD refers to out-of- distribution.
1 Introduction
enabling seamless integration of new datasets. This would contribute to the development efficient QA models with minimal computational and stor- age costs, enhanced generalization capabilities, and greater practicality for real-world use cases.
Question answering (QA) pivotal area of re- search in NLP that evaluates the language under- standing and reasoning capabilities of language models. To this end, the NLP community has de- veloped numerous QA datasets that span various domains, question-answer formats, and reasoning skills (Rogers al., 2022). Consequently, there is an increasing demand for Unified QA system that can manage mixed batches of instances from differ- ent datasets and tasks during training and inference (Liu et al., 2022). Such a system would climinate the need for manual tuning or per-task adjustments,
The success of transformer-based models in text- to-text generation has led to growing interest in Unified QA systems. Khashabi et al. (2020) proposed Unified-Q22 -Afiied--,, single QA model pre- trained on diverse datasets that outperforms format- specialized models. While prompt-tuning methods (Lester et al., 2021; Vu et al., 2022) have emerged as a promising alternative to fine-tuning, (Zhong et al., 2022a) proposed to model the commonal-
work done during internship at Salesforce Research

---

ities and distinguish task differences through structurally designed prompt-based input schema. However, these approaches have limitations related to scalability, expensive pre- pre-training requirements, and the need for tens of thousands of training ex- amples for each task. Moreover, the performance of pre-trained QA models significantly degrades when only few question-answeriin examples are available (Ram et al., 2021). While Unified QA ap- proaches have shown success high-data scenar- ios, their efficacy in more practical scenarios with limited training examples remains unexplored.
for several downstream tasks in NLP (Devlin al., 2019; Liu et al., 2019; Raffel et al., 2022; Brown et al., 2020; He et al., 2021b; Lan et al., 2019; Yang et al., 2019). However, standard fine-tuning approaches update all the model parameters, which can often lead to deployment challenges. Recent research (Houlsby et al., 2019; He et al.. 2021c; Lester et al., 2021; Li and Liang, 2021a) has shown that similar performance by updating or adding a few trainable parameters while keeping pre-trained language model param- eters frozen. Several approaches have been pro- posed in this direction: Adapter-based methods (Houlsby et al., 2019; mahabadi et al., 2021; Rücklé et al., 2021) insert small trainable ferd-forward networks (modules) between layers of pre-trained language models while BitFit (Ben Zaken et al., 2022) updates only the language model biases. An- other computationally efficient approach is prompt- tuning (Lester et al., 2021) and prefix-tuning (Li and Liang, 2021a), which concatenate trainable continuous embeddings to the input. These train- able parameters, called soft prompts, can be used as plug ins with frozen LM to capture task-specific, Nomain-speciftt or language-specinn knowledge. He al. (2021a) presents unified view of differ- ent parameter-efffcient training (PET) approaches.
be obtained
This paper aims to explore the potential of two different paradigms of tuning, model, and prompts, for unified question answering under low resource setting. Despite the importance of this problem, there have been no previous studies investigating the effectiveness of these paradigms for this task. In response, we conduct an exhaustive analysis of the applicability of these two paradigms to a unified question ceestion-answeer answering system. To do so, we evaluate their promise, effectiveness, and trade-offs using a set of 16 QA datasets, covering diverse domains and wide range of skills and formats.
Our empirical study reveals several key find- ings, including (i) prompt tuning with good ini- tialization can outperform model tuning under a low resource regime for out-of -iitr---iitiittiion tasks, (ii) parameter-sharing results in superior few-shot performance, but the trends are reversed in the full- shot setting, (iii) simple knowledge transfer tech- niques for prompt initialization can be as effective as more complex methods in the few-shot setting, without introducing additional parameters, and (iv) prompt tuning achieves a significant performance boost from pre-training in a low resource regime while increasing model size does not significantly affect prompt tuning with initialization. In addition, we perform systematic quantitative and qualita- tive study to provide insights into the advantages and limitations of prompt tuning for unified QA with an emphasis on the behaviors in the few-shot setting. Overall, our research aims to contribute to the development of effective and efficient uni- fied question-answering systems in low-resource scenarios.
Multi-task transfer learning. Efficient task transferability in NLP has been extensively studied (Wang et al., 2019; Liu et al., 2021a; Vu et al., 2020, 2021). With T5 (Raffel et al., 2022) demonstrat- ing the capabilities of using existing downstream task datasets to learn new task, proposing effi- cient methodologies for unifying NL.P models has become a promising research paradigm in the com- munity. Following this, (Khashabi et al., 2020) pro- posed UnifiedQA, a single QA model pre-trained on datasets involving diverse formats and reason- ing skills. Transfer learning has been demonstrated to be effective from rich data sources (Phang al., 2018), 8), between similar target tasks eVu al., 2020), and for tasks that require similar reasoning skills (Pruksachatkun et al., 2020). However, this ap- proach would require updating/reeraining the model on a new task or a different domain, which could lead to catastrophic forgetting (Kirkpatrick et al., 2017). Moreover, Aghajanyan et al. (2021) showed approaches towards unifying NLP models suffer from negative interference to less represented tasks and between dissimilar tasks.
2 Related Work
Parametter-effiieett tuning. Large-scale pre-
trained language models fine-tuned on specific tar- get datasets have shown remarkable performance
Most recently. Liu et al. (2022) validates that

---

parameter-efficien tuning methods can perform well with mixed lask batches. Zhong al. (2022b) takes the first step towards building unified QA models utilizing structural prompt tuning. Along these lines, Vu et al. (2022); Asai et al. (2022) in- tegrates both the paradigms of parameter-efficient tuning and unifying NLP models to propose sin- gle pre-trained model for different downstream tasks by learning target task-specific prompts from the source task prompts. Asai et al. (2022) demon- strates transfer using the attention module, while Vu et al. (2022) facilitates prompt transfer by learn- ing the target prompt initialized from similar source prompts. These approaches require fewer than 0.1% of trainable LM parameters with little trade- off in performance.
Approach
Paradigm
SM
KT
NT
Fx
FT
Model-tuning
Limited
High
FT-MI
Model-tuning
J
High
Limited
PT-R
Prompt-tume tuningg
J
Limited
Limited
V
PT-F/PT-C
Prompt-tuning
High
Limited
ATT-HT
Prompt-tuning
High
High
Table I: Comparison of various model-tuning and prompt-tuning approaches concerning their effective- ness in fulfilling the desired properties of a unified QA model, such as Single Model (SM), Knowledge Transfer (KT), Minimal Negative Transfer (NT), and Extensibil- ity (Ex). The table uses checkmark symbol (V) to indicate "Yes" and "No" for each property, and the de- gree of the property is indicated as "Limited" "High" where applicable.
maintenance costs.
This study seeks assess the suitability of two prevalent training paradigms for NLP, namely model-tuning and prompt-tuning, as potential approaches for developing a unified question- answering (QA) model. Our investigation centers around four essential criteria we look for in an ef- fective unified QA model: (1) the ability to utilize single model to address range of different QA tasks, (2) effective knowledge transfer from multi- ple relevant tasks, (3) while minimizing the risk of negative interference, and (4) extensibility to new tasks without requiring expensive retraining. this study, our goal is to investigate the potential of soft prompt-tuning extensively and to better un- derstand its benefits and drawbacks in comparison with model-tuning--e based approaches for building unified QA system grounded the aforemen- tioned four principles. In particular, we further center the study around understanding these trade- offs in the few-shot learning scenarios, which a realistic and more practical challenge.
Few-shot question answering. Ram al. (2021) has identified a discrepancy between current pre- training objectives and QA, as standard models perform poorly when fine-tuned with few exam- plcs. They propose recurring span selection as pretraining scheme tailored for question answering, Chada and Natarajan (2021), on the other hand, proposes fine-tuning framework aligned with the pretraining framework.
However, there have been no studies focusing on the viability prompt tuning for unified QA under low-resource settings. To address this gap, we follow prior works, (Liu et al., 2022; Asai et al.. 2022; Khashabi et al., 2020), and extensively study the viability and trade-offss of prompt tuning and prompt-basee based transfer learning in comparison to approaches that involve full-model fine-tuning for few-shot unified QA. As result of our comprehen- sive experiments, we offer essential guidelines in the form of valuable insights into the advantages and limitations of prompt tuning with respect to model tuning for unified QA in both full and few- shot scenarios.
This paradigm involves the fine- tuning of all the parameters of language model to cater a specific task a set of tasks. Al- though fine-tuning (FT) on particular dataset is an effective strategy, is not suitable for unified QA because it requires specialized models for each dataset during inference, which counter-intuitin to the concept of unified QA model.
Model-tuning
3 Candidates for universal QA approach
Finetuming pre-trained language models (FT) on specific datasets yields specialized models that cater to individual tasks. However, more effi- cient approach is to build a unified QA model that perform multiple tasks without manual tun- ing or per-task adjustments. One of the signifi- cant advantages of such approaches is that they seamlessly support mixed-tast batch inference (Liu et al., 2022), where a single model can handle di- verse tasks, reducing computation, storage, and
In contrast, multi-task learning via fine-tuning (FT-MT) (Raffel et al., 2022; Aribandi al., 2021) involves the joint learning of single model on mul- tiple datasets by sharing all the trainable model pa- rameters across different tasks. By training on mul- tiple datasets, FT-MT allows for knowledge transfer

---

from relevant tasks during inference. However, sharing all the parameters often leads to negative transfer from unrelated tasks. Incorporating addi- tional tasks into existing models requires retraining the model with all previous tasks and the new ones, making them computationally expensive to scale and more prone to negative interference.
shared across tasks and trained in multi-task man- ner, and ATT-ST, where attention module parame- ters are not shared. While ATT-MT provides a sin- gle model for transferring knowledge from source prompts and is easily scalable to new target tasks, sharing attention modules across tasks may result in some negative transfer, compared to more straight- forward prompt-tuning methods.
Prompt-tuning This paradigm involves learning soft-prompt tokens added to the input while the backhone language model remains frozen. We fol- low the approach proposed by Lester al. (2021) to train soft prompts for each task, where prompts are initialized from random words in the vocabu- lary (PT-R). This vanilla prompt-tuning approach is parameter-efffcient and easy to scale. Since task- specific knowledge is captured in different set of parameters (i.e., the prompts), this approach avoids negative interference a great extent. With sin- gle backhone model, we can use these prompts for different tasks. However, this approach does not leverage knowledge from other tasks not already captured in the backbone model.
Datasets
In their recent study, Rogers et al. (2022) highlight a significant increase in the number of question- answering and reading comprehension datasets, spanning various domains, formats, and reason- ing abilities. This study aims to evaluate and fine- tune a range of models, leveraging collection of datasets referred to "source datasets" for pre- training, and a distinct set of datasets known as "target datasets" for evaluation. This paper in- cludes datasets that cover a wide range of reason- ing skills and complex linguistic phenomena, in- cluding conversational, temporal, causal, and co- reference reasoning. among others, enabling a more comprehensive evaluation of training paradigms on question answering datasets and facilitating anal- ysis of cross-skill transfer. This broader coverage across reasoning skills not only enables a more thorough evaluation of training paradigms on QA datasets but also facilitates analysis of cross-skill transfer. Table 3,4 presents an overview of the datasets employed in our study, detailing their size, domain, and associated primary reasoning skill.
Prompt initialization is technique that ad- dresses the issue of knowledge transfer from source tasks in vanilla prompt-tuning while retaining the benefits of single model, minimal negative trans- fer, and extensibility. Previous studies (Li and Liang, 2021b; Liu et al., 2023; Vu et al., 2022) have shown that prompt-tuning methods are often sensitive to initialization, particularly in low data settings. However, the impact of different initial- ization methods on QA datasets has not been well studied. Inspired by (Vu et al., 2022), we initialize the target prompt by taking the average of the top- 3 source task prompts most similar to the prompt trained on the target dataset. We employ two dis- tinct approaches to this initialization process: (i) selecting source task prompts with the same answer format as that of the target dataset (PT-F), and (ii) selecting source task prompts from the complete set of source prompts (PT-C).
Source Datasets. This study leverages source datasets for two primary purposes: pre-training models through model tuning and training source prompts via prompt- tuning approaches. The source datasets employed in our research comprise over 30,000 training instances. They aim to encompass essential reasoning skills such as reading compre- hension, conversational and commonsense reason- ing, as well as discrete and numerical reasoning necessary for question answering. Source datasets cover wide range of domains, including knowl- edge bases, news, web documents, and Wikipedia.
Apart from prompt initialization, another way to transfer knowledge from multiple tasks is through the composition of their corresponding prompts. To this end, Asai et al. (2022) proposes AT- TEMPT, transfer learning method that learns new task-specific target prompts by computing weighted combinations of source prompts using a sub-network-baase attention module trained on a single or set of tasks. We distinguish between two settings: ATT-MT, where attention modules are
Target Datasets. We employ target datasets to fine-tune models using the model-tuning paradigm, or to train target prompts for prompt-tuning ap- proaches. Target datasets are typically small in size, containing fewer training instances.
In- nr-Distribution Includes datasets that share the

---

Table 2: Question Answering (QA) datasets used as source and target datasets this study. For each dataset, the table provides details associated reasoning skills, domain, and the number of training examples available. RC stands for reading comprehension
Reasoning Skill
Dabret
Domiin
Fernut
Reasoning Skill
Dutnsct
Domzin
Format
Reading Congechension
SQuAD (Rijporkar et al., 2016)
Wikipedia
EXT
In-Distrbutium (II)
(RC)
Natural Questions (Kwistlowsi
SearchQAA
NewsQA (Trischer et al, 2017)
TriisQA (Joshi et al., 2017)
20177)
nows untickes
Wikipedia
Web. Wikipedin
Andivve
EXT
EXT
EXT
EXT
Comprehension
(RC)
Reading
DealQ (Clark 2019)
Mclest (Richardion et al. 2013)
URC (Fergusnn
stories
Wikspedia
Wikipedia
MCQ
MCQ
al. 2019)
ReClor (Ya 2020)
Wibh
MCQ
NQOpen (Les et al., 2019)
Wikipedia
ABS
Co-refenncce
Quaed (Dasigi al., 2019)
Wikipelin
EXT
RACE (Lai al., 2017)
Exams
Communscme
McTACO (Zhoo et al., 2019)
multiple
MCQ
DuoRC (Sha et al., 2018)
movie picts
Out-af-Disribation (OOD) Domain
PahMedQA (Jin al., 2019)
Puhmed
MCQ
(Long)
MurrativoQA (Kodisk)
books, movies
ABS
TweetQA (Kinng 2019)
Twitter
ABS
2018)
Commonense
COSMOSQA (Huang a al. 2019)
Spinnsel penconal narratives
MCQ
RC (Muttihop)
HarporQA (Yang et 2018)
Wikipedia
EXT
Conversatinnl
(Reddy al. 2019)
Wiki.WikiBooos Wikipedia
ABS
PIQA (Bisk at al., 2020)
News, Encyclope-
MCQ
QuAC (Choi et all, 2018)
EXT
dia
Comesoasense
ReCORD (Zhang ata..
CloeeOA
Commonsen=:QA (Talinox al.
Concept
MCQ
SIQA (Sap al., 2019)
knowkolge buse
MQQ
2019)
Discrese
DROP (Dux ol., 2019)
Wikipedia
ABS
Consenstional
SBARC (Sseldi et al., 2018)
law rule books
ABS
Table 3: Question Answering (QA) datasets used as source datasets in this study. For each dataset, the table provides details on associated reasoning skills, domain, and question format including Extractive (EXT), Ab- stractive (ABS) and Multi-choice (MCQ) questions.
Out-of Distof-Dittbbbtion (OOD)
Skill
ROPES (Lin et al., 2019)
Wiki
DBQA (Mihuyllvv et all, 2018)
wienee books
MCQ
Quakel (Tafjord al., 2018)
COPA (Gordon et al., 2012)
personal stories
MCQ
Table 4: Question Answering (QA) datasets used target datasets in this study, For each dataset, the table provides details on associated reasoning skills, domain, and question format including Extractive (EXT), Ab- stractive (ABS) and Multi-choice (MCQ) questions.
same domain and reasoning skills as one or more of the source datasets. Examples such datasets include MCTest and BoolQ, which cover generic domains and reasoning skills such as Wikipedia reading comprehension, respectively.
Experiments
We employ the T5-base model for all our exper- iments, unless stated otherwise. Source prompts are trained independently for each task, while the pre-trained language model (PrLM) and attention modules for ATTEMPT are trained jointly on all the source tasks. For target datasets, we randomly select a small number of instances for few-shot training and evaluation. The hyperparameters for training are presented in section 5.1. Table 7 de- tails the initialization used for different target tasks in both PT-F and PT-C. We select the best check- point based on the validation set performance, with FT-MT and ATT-MT using a single validation set comprising of all the target tasks, and PT-R, PT-F, and PT-C using a validation set for each target task individually. We evaluate the best checkpoint on the test set of each target dataset using Fl as the metric for extractive and abstractive QA datasets, and accuracy for MCQ and Yes/No QA datasets. In cases where test set is unavailable, we use the development set to report our model's perfor- mance and create a small subset from the training set for hyperparameter tuning and checkpoint se-
cut-of-Distribution Includes datasets that lack domain knowledge and reasoning skills found in one or more of the source datasets. This subset includes datasets that involve intricate and special- ized reasoning abilities such as temporal common- sense, causal reasoning, logical and inferential rea- soning, as well as datasets specific to domains like Twitter, TOEFL, law books, and personal narra- tives. These target datasets can benefit from generic source tasks.
In some contexts, certain tasks require multi- ple types of reasoning. For instance, the ShARC dataset necessitates a combination of conversa- tional and causal reasoning, while the COPA dataset entails the application of commonsense causal reasoning. Therefore, natural language pro- cessing models may face additional challenges in performing these tasks due to the integration of multiple reasoning skills. To assess the effective- ness of unified QA system, we perform experi- ments on the test set of the target datasets.

---

Backbone T5-Base
k-shot
FT
FT-MT
PT-R
PT-F
PT-C
ATT-ST
ATT-MT
16
45.12
45.62
38.92
44.61
45.55
45.70
45.23
32
46.36
47.00
42.76
45.75
46.88
47.68
47.40
48.08
49.12
44.34
49.33
48.85
48.56
49.08
128
50.14
52.27
44.41
50.31
50.83
48.76
50.31
256
52.96
55.39
45.77
53.15
52.31
50.21
52.68
512
56.60
59.54
47.03
54.91
55.34
51.20
54.70
Table 5: Comparison of Model-Tuning and Prompt-Tumin Paradigms in Few-Shot setting: Model-tuning approaches include FT and FT-MT, while PT-R represents vanilla prompt tuning and PT-F and PT-C correspond to prompt tuning with initialization. ATT-ST and ATT-MT are single-task task and multi-task variants of ATTEMPT, a prompt transfer learning approach. Bold values indicate best model with T5-base backbone for the k-shot scenario, while underline represents the second-best.
lection. We report the aggregate results of three sceds. Table summarizes the experimental results comparing the model-tuning and prompt-tuning paradigms for unified QA system. In the rest of this section, we share our key findings and insights that hopefully help guide which paradigm to prefer under which scenarios.
IID
OOD (Domain) OOD (Skill)
32 Examples
FT-MT
48.27
44.12 12
48.34
PT-C
44.14
45.08
51.95
PT-
43.27
45.56
48.89
ATT-MT
47.82
45.90
48.41
64 Examples
5.1 Hyper-parameters
FT-MT
50.62
46.15
50.28
After extensive tuning, we selected a learning rate of le-5 for the backbone model, along with a maxi- mum source length of 512, a gradient accumulation step of 2, and batch size of 16. During train- ing, we saved and evaluated checkpoints every 500 steps, and trained the model for 100K steps with pa- tience. For all experiments, the prompts consisted of k = 100 tokens with a hidden dimension of d 768.
PT-C
49.76
45.04
51.54
PT-F
51.52
46.30
49.75
ATT-MT
50.05
46.98
50.02
Table 6: Comparison of model-tuning and prompt- tuning paradigms in few-shot setting for different cate- gories of target tasks.
QuaRel, PIQA, OBQA, and COPA. It observed that prompt-based techniques significantly outper- form multi-task model tuning when dealing with dut-of-distribution target tasks. However, for in- distribution tasks, ATT-MT performs comparably to FT-MT. These results indicate the superiority of simple prompt initialization methods in terms of adaptability in few-shot scenarios compared to model tuning. Complex prompt transfer methods like ATTEMPT have demonstrated their usefulness in IID tasks exclusively. Importantly, PT-C outper- forms Pt-F for new skills, suggesting the transfer- ability of tasks beyond format boundaries.
Best Candidate for Unified QA FT-MT, PT-R, PT-F, PT-C, ATT-MT are potential candidates for the unified question answering task. low-resource scenarios, all candidates perform similarly, but PT-F and PT-C stands out due to its low num- ber of trainable parameters and ease of scaling to datasets. As the number of training instances increases, FT-MT outperforms other approaches, while prompt-tuning approaches remain competi- tive. Our findings suggest that simple approach like PT-F is on par with more sophisticated prompt- transfer learning approaches like ATT-MT.
Table 6 presents a comparison between model- tuning and prompt-tuning techniques in the 32-shot and 64-shot settings for various categories. The term IID refers in-distribution target tasks such BoolQ, IIRem MCTest, ReClor, MCTACO, and Quoref. On the other hand, 0OD (Domain) encom- passes TweetQA, COSMOSQA, CSQA, DREAM, and ShARC, while OOD (skill) includes ROPES,
Parameter-sharing results in superior few-shot performance; however, the trends are reversed in the full-shot setting. Multi-task fine-tuning (FT-MT) that employs the parameter-sharing tech- nique yields superior few-shot learning perfor- mance than traditional finetuning (FT). The extent of improvement increases with the number of train- ing examples and starts decreasing at a threshold

---

of approximately 512 examples on an aggregate level. However, this can vary across different target datasets. For instance, datasets such as TweetQA, PIQA, and ReClor exhibit this behavior beyond 512 examples, while OBQA, MCTest, and Com- M realize this at around 128. Increasing training examples from 16 to 512 leads to boost from A 0.50 to A 2.94 due to transfer learn- ing from other tasks. However, raising the number of examples to 1024 results drop in improve- ment to A 1.77. In the full-shot setting with unbalanced training samples, employing parameter sharing among all tasks can lead to negative inter- ference, resulting in a reversal of the trend where FT outperforms FT-MT. Similarly, sharing attention modules across all target tasks in multi-task prompt transfer learning approaches (ATT-MT) leads to comparable trend.
skill perspective, as depicted in Figure 10, indi- cated that PT-F performed better in Dialog reason- ing in the low range and in commonsense reasoning in the high range. On the other hand, PT-C was bet- ter suited for causal reasoning in the low range. More detailed information on our findings can be found in the appendix in Table 10.
Target
Formaat-asse (Pt-F)
Complete Set (PT-C)
Duinset
ropes
searchqu. newsqa, quuc
drop. siqu, quae
dream
record, race, siqu
scarchga, quac, siga
share
cuoore, cuqs, nar
quae, coga, nar qa
baokq
pubmed_ qa
race, newsqa, pubmeed qa
piga
record, siqu, race
quac; siqa, nar
quoref
quac, newsqa, li]
duorc, nar_qa, drop
cosmos_ qg
record, siqa, race
nar ua.. siga,
tweet_qu
ng. open, duore, nar
nq open, duore, qua
CQA
recerd, rice, siqu
newsqo, nar qa._ siya newsgo, race, siya duorc, siga. ngen duore, quae, nar qu nq. race, siga
obqa
record, race, skya record, race, siqu
reclor
quarel
record, race, siga
mittett
record, race, siqa
taco
pubmed qa
pubmed qu. siqp, nar qu searchua, race, siqu
Format-based prompt initialization achieves comparable performance to more complex prompt-transfer approaches. The Prompt- tuning paradigm has emerged as highly effective approach for fine-tuning pre-trained language models for specific tasks. However, it has been shown that the success of this paradigm can be highly sensitive to initialization. To address this issue, we drew inspiration from the work of (Vu et al., 2022) and explored the use of two different initialization techniques for the target prompt (PT-F and PT-C). Our results demonstrated that both initialization techniques outperformed random initialization by 6% with 32 examples, and this gap increased approximately 20% with 1024 examples. Notably, we found that the simpler format-based heuristic initialization was just as effective as the more complex cosine-basee search over the entire prompt pool. Furthermore, our results revealed that both prompt initialization approaches were competitive with the sophisticated mocentionnmmdule-baae approach ATT-MT.
pubmed
lire
botpotqu, newsqu, nq
newsqa, drop, nq
Table 7: Source Prompts most similar to target prompts for format-based and complete-set initialization corre- sponding PT-F and PT-C respectively. Bold indicates source tasks common in both partitions. Although some source prompts are shared across target tasks, Quoref and COPA have none common. The SIQA and RACE source prompts are typically used for initialization, but we found that lifting the constraint of choosing prompts from the same format allowed for successful cross- format initialization at the reasoning skill or domain level. For example, the DREAM dataset (MCQ) was ini- tialized with QuAC (ExIQA), which is reasonable since both involve conversational data. The IIRC dataset was also initialized with the most relevant source task, Hot- potQA. Yes/No questions strongly prefer PubmedQA as a format.
6 Discussion
6.1 Qualitative Analysis
poompt-taansfe learning
Do different models agree on their answers? Fig 7 shows the average agreement of different models on all the tasks across different few-shot scenarios. We find that PT-C and PT-F have the highest agree- ment scores. We partly attribute this to the high overlap of initialization prompts of format and log- ically similar tasks (PT-C, PT-C). However, the number of shots increases the overall agreement decreases across different modes. Furthermore, we investigate if different modes can be complemen- tary to each other by evaluating the union of their predictions across different shots. We find that fine-
Our analysis further revealed that the perfor- mance of PT-F and PT-C varied based on the skill or domain of the dataset (see Table 10). Evalu- ation on datasets from specific domains (Figure 9) reveals that in low-regime scenarios, PT-F out- performed PT-C in the Web+Social and domain- specific book domains, while PT-C was more effec- tive for Knowledge graphs and Wikipedia domains. However, in high-range scenarios, all models per- formed similarly. Furthermore, our analysis from

---

tuning (FT) and model tuning models (FT-MT) are complementary to each other at low resource set- tings whereas the gains from PT-R to other modes are minimum. For the complete results, refer to Ap- pendix (Figure 3). This might indicate that prompt tuning may not be practical without good initializa- tion for extremely low-resource QA scenarios. For further discussions around few shot analysis, refer Appendix 6.2.
7 Conclusion
In this work, we explore the viability of prompt- tuning as a solution to unified QA and conduct thorough analysis of its promise, effectivvnesss and trade-offs compared with the model-tuning paradigm
set of 16 QA datasets, focusing particularly on several few-shot scenarios. As result, we obtain several key findings and insights that hopefully will inform which paradigm to prefer under which scenarios. Prompt tuning is quite com- petitive with model-tuning in the lower extreme of the few-shot scenarios, given good initialization. While parameter-sharing leads to superior perfor- mance in the few-shot setting, the trends flip in the full-shot setting, A simple knowledge transfer approach (i.e., an average of relevant prompts) is effective as complex methods without introducing additional parameters. Pre-training the backbone model on the source tasks significantly benefits prompt tuning. While initializing from strong prior is very helpful for prompt tuming, its benefit is not as substantial when using a larger backbone model, especially when the number of training amples exceeds certain threshold.
A closer look at the task-level performance across different few-shot settings reveals counter-intuitive behaviors. We find that under low resource settings (< 256 shot) good initializa- tion helps significantly for target tasks that are sim- ilar to source tasks (e.g: OBQA, BoolQ, IIRC), and the performance gain decreases we increase the number of shots. As seen from Figure 8, for simi- lar tasks PT-C and model tuning FT-MT performed significantly better than PT-R. However, in cases where there is little domain overlap (ShaRC), ini- tializations do not contribute substantially to the overall performance of the model. Interestingly, in some cases, we find counter-intuitive results where performance remains flat (ShaRC) from Figure 8) zig-zag (Ropes) pattern is observed across dif- ferent shots. We point the reader to Appendix (Fig- ures 4, 5, 8) for performance across different modes against different shots.
Limitations
Our work has several limitations: (1) since few- shot experiments are prone to have considerable variance due to the randomly sampled few train- ing examples, we repeat all the experiments us- ing three randomness seeds for the T5-base back- bone. However, since the number of experiments per seed is more than 1500, we were able to run the same experiments with a T5-large backbone using only one seed and excluding specific few-shot set- tings due to computational limitations, especially given the latter model has 3.5 times more parame- ters. Although our comparisons of the two models are still presented in an entirely fair fashion us- ing the same single seed, it would have been more strongly conclusive to test our findings with a T5- base backbone on the larger model to the same extent. That is also the reason why the current version of our study does not include comparisons with even larger models such as T5-3b or T5- 1b. (2) We explore limited number of prompt-tuning methods both in terms of how the soft prompts are injected in the model architecture following (Lester et al., 2021) and how the knowledge from source tasks are used to inform target tasks following (Vu et al., 2022; Asai et al., 2022). For example, Liu
6.2 Qualitative Study
Table 8 presents a few qualitative examples across different shots and modes. We find prompt tuning with good initialization to leverage world knowl- edge better (e.g: Arctic Circle with cold weather) even in low resource settings while prompt tuming struggles in predicting local context-based reason- ing tasks (e.g: taking photos of home does not associate with new home).
Do the same model across different few-shot set- tings agree on its answers? Figure ?? presents the overall agreement of different models for sin- gle task under different shot settings. We observe patterns of high level agreement between adjacent shots that gradually decrease with an increase in the number of shots in fine-tuning and prompt tuning with initialization mode. However, prompt tuning with random initialization has an agreement per- centage of 50% across different shots and has no clear distinction of high agrecment between the adjacent shots as found in other settings.

---

(2022) proposes parameter-efficient fine-tuning alternative to soft prompt-tuning in recent work, while (Zhong et al., 2022a) shows the benefits of prompt-based pretraining. Although the key take- aways in the current version of our study are sup- ported by sufficient empirical evidence, incorporat- ing the aforementioned recent developments may prove even further promise and evidence for the prompt-based approaches towards few-shot unified QA. (3) Our study is currently limited to English- QA datasets, hindering our findings to be generally valid for cross-lingual and/or cross-model question- answering systems. Therefore, we need to consider how our findings would generalize to other lan- guages and modalities.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao. Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang. Vinh Q. Tran, Dara Bahri, Jianmo Ni, Jai Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2021. Ext5: Towards extreme multi-task scaling for transfer learning.
Akari Asai, Mohammadreza Salchi, Matthew Peters, and Hannaneh Hajishirzi. 2022. Attentional mix- tures of soft prompt tuning for parameter-efficient multi task knowledge sharing. arXiv preprint arXiv:205..19661
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-efficient fine-tuning for transformer- based masked language-modell models. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associa- tion for Computational Linguisties.
Ethical Statement
Yonatan Bisk, Rowan Zellers, Ronan Le Bras, Jianfeng Gao, and Yejin Choi. 2020. PIQA: reasoning about physical commonsense in natural language. In The ThirtyyFourrhh AAAI Conference on Artificial Intelli- gence, AAAI 2020, The Thirty-Second Innovative Ap- plications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 7432- 7439. AAAI Press.
We observe a preference for multiple-choice question (MCQ) answer formats across various question-answering (QA) datasets with varying levels of reasoning ability. Additionally, the ma- jority of the source datasets were sourced from Wikipedia, which may contain gender or political bias that could be further perpetuated by models. The T5 model, which was used for pre-training, may also have biases due to its pre-training data. However, the study did not conduct stress tests to identify potential biases, and users should be cautious when implementing the provided models. The current models' results may not align with the facts in input documents, potentially leading to the spread of false information online. This is com- mon issue in all current QA models, and further research is needed in this area. The study's s experi- ments were primarily conducted using A100 GPUs and consumed a significant amount of GPU time when repeated across random seeds. Nevertheless, our findings can benefit subsequent studies and applications by providing valuable insights, thus avoiding the need for extensive repetition of these comparisons.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler. Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz. Litwin, Scott Gray. Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, 1lya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Rakesh Chada and Pradeep Natarajan. 2021. Fewshotqa: A simple framework for few-shot learning of question answering tasks using pre-trained text-to-text models. ArXiv, abs/2109.011 109.01155.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen- tau Yih, Yejin Choi, Percy Liang, and Luke Zettle- moyer. 2018. QuAC: Question answering in context, In Proceedings of the 2018 Conference Empiri- cal Methods in Natural Language Processing, pages 2174-2184, Brussels, Belgium. Association for Com- putational Linguistics.
References
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. 2021. Muppet: Massive multi-task representations with pre- finetuning. In Proceedings of the 2021 Con- ference on Empirical Methods in Natural Lanuuage Processing, pages 5799-5811, Online and Punta Cana. Dominican Republic, Association for Com- putational Linguistics.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings

---

of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2924-2936, Minneapolis, Min- nesota. Association for Computational Linguisties.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and
Weizhu Chen. 2021b. (DEBERTA): (DECODING)- (enhanced) (bert] [with] { [attention). In International Conference on Learning Representa- lions.
Pradeep Dasigi, Nelson F. Liu, Ana Marasović, Noah A. Smith, and Matt Gardner. 2019. Quoref: A read- ing comprehension dataset with questions requir- ing coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natu- ral Language Processing and the 9th International Joint Conference on Natural Language Processing Hong Kong, China. Association for Computational Linguisties.
Ruidan He, Linlin Liu, Hai Ye, Qingyu Tan, Bosheng Ding, Liying Cheng, Jiawei Low, Lidong Bing, and Luo Si. 2021c. On the effectiveness of adapter-based tuning for pretrained language model adaptation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the I 11th International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pages 2208- 2222. Online. Association for Computational Lin- guistics.
(EMNLP-IJCNLP) pages
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter--efiiieen transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi. Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requir- ing discrete reasoning over paragraphs. In Proceed- ings of the 2019 Conference of the North American Chapter of the Association for Computational Lin- guistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2368-22 2378, Min- neapolis. Minnesota. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense rea- soning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLL pages 2391-2401, Hong Kong, China, Association for Com- putational Linguistics.
Qiao Jin, Bhuwan Dhingra, Zhengping Liu, William Cohen, and Xinghua Lu. 2019. PubMedQA: A dataset for biomedical research question answering. In Proceedings of the 2019 Conference Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- guage Processing (EMNLP-IJCNLP), pages 2567- 2577, Hong Kong. China. Association for Computa- tional Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with con- text from a search engine. ArXiv, abs/1704.05179.
James Ferguson, Matt Gardner, Hannanch Hajishirzi, Tushar Khot, and Pradeep Dasigi. 2020. IIRC: A dataset of incomplete information reading compre- hension questions. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1137-1147, Online. As- sociation for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver. Canada. Association for Computational Linguistics.
Andrew Gordon, Zornitsa Kozareva, and Melissa Roem- mele. 2012. SemEval- 2012 task 7: Choice of plau- sible alternatives: An evaluation of commonsense causal reasoning. In "SEM 2012: The First Joint Conference on Lexical and Computational Seman- tics Volume Proceedings of the main conference and the shared task, and Volume 2: Proceedings of the Sixth International Workshop on Semantic Eval- nation (SemEval 2012), pages 394-398, Montréal, Canada. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020. UNIFIEDQA: Crossing for- mat boundaries with a single QA system. In Find- ings of the Association for Computational Linguistics: EMNLP 2020, pages 1896-1907, Online, Association for Computational Linguistics.
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2021a. Towards unified view of parameter-efficiee transfer learning. CoRR, abs/21 10.04366.
James Kirkpatrick, Razvan Pascanu, Neil Rabinowitz, Joel Veness, Guillaume Desjardins, Andrei A. Rusu,

---

Kieran Milan, John Quan, Tiago Ramalho, Ag- nieszka Grabska- Barwinska, Demis Hassabis, Clau- dia Clopath, Dharshan Kumaran, and Raia Hadsell. 2017. Overcoming catastrophic forgetting in neural networks. Proceedings of the National Academy of Sciencex, 114((33:3521-3556
International Joint Conference on Natural Language Processing, ACL/UCNLP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2027, pages 4582- 4597. Association for Computational Linguistics.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gard- ner. 2019. Reasoning over paragraph effects in situ- ations. In Proceedings of the 2nd Workshop on Ma- chine Reading for Question Answering, pages 58-62, Hong Kong, China, Association for Computational Linguistics.
Tomáš Kočisky.Jonathan Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz, Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.
Evelyn Kai-Yan Liu. 2022. Low-resource neural ma- chine translation: A case study of Cantonese. In Proceedings of the Ninth Workshop on NLP for Simi- lar Languages, Varieties and Dialects, pages 28 28-40, Gyeongju, Republic of Korea. Association for Com- putational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wi Hing-Wei Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natu- ral questions: A benchmark for question answering research. Transactions of the Association for Compu- tational Linguistics. 7:452 466.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mo- hta, Tenghao Huang, Mohit Bansal, and Colin Raffel, 2022. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learning.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. RACE: Large-scale ReAd- ing comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785- 794, Copenhagen. Denmark. Association for Compu- tational I Linguistics.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021a. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprial arXiv:2107.13580
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Comput. Surv. 55(9).
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations.
Kenton Lee, Ming- Wei Chang, and Kristina Toutanova. 2019. Latent retrieval for weakly supervised open da- main question answering. In Proceedings of the 57th Conference of the Association for Computational Lin- guistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume I: Long Papers, pages 6086-6096 Association for Computational Linguistics.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. 2021b. P-tuning v2: Prompt tuning can be comparable to fine- fine-tuning universally across scales and tasks. ArXiv, abs/21 10.07602.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy. Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1977.199.....9..
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059, Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Rabech Karimi mahabadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems.
Xiang Lisa Li and Percy Liang. 202 Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582- 4597, Online. Association for Computational Lin- guistics.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018 Can a suit of armor conduct elec- tricity? new dataset for open book question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 2381-2391, Brussels, Belgium. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021b. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 1 1ith
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on stilts: Supplementary training on intermediate labeled-data tasks. arXiv preprimt arXiv: 1.00088.

---

Yada Pruksachatkun, Jason Phang. Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bow- man. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247 Online. Association for Computational Linguistics.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan 2018. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 1683- 1693, Melbourne, Australia. Association for Compu- tational Linguistics.
Maarten Sap, Hannah Rashkin, Derek Chen, Ronan Le Bras, and Yejin Choi. 2019. Social IQa: Com- monsense reasoning about social interactions. Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Lan- pages 4463-
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2022. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Much. Learn. Res., 21(1).
guage Processing (EMNLP-IJCNN) 4473, Hong Kong, China. Association for Computa- tional Linguistics.
Pranav Rajpurkar, Jian Zhang. Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natu- ral Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie. 2019. DREAM: A challenge data set and models for dialogue- based reading compre- hension. Transactions of the Association for Compu- tational Linguistics, 7:217-231.
Oni Ram, Yuval Kirstain, Jonathan Berant, Amir Glober- son, and Omer Levy. 2021. Few-shot question an- Proceed-
swering by pretraining span selection. ings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Juint Conference on Natural Language Processing (Volume 1: Long Papers), pages 10766-377, Online. Association for Computational Linguistics.
Oyvind Tafjord, Peter Clark, Matt Gardner, Wen tau Yih, and Ashish Sabharwal. 2018 Quarel: A dataset and models for answering questions about qualitative relationships. ArXiv, abs/181 1.08048.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQA: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume I (Long and Short Papers), pages 4 4149-4158, Minneapolis, Minnesota. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning. 2019. CoQA: A conversational question answering challenge. Transactions of the Association for Com- putational Linguistics, 7:249-266.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw, 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Pro- ceedings of the 2013 Conference on Empirical Meth- ods in Natural Language Processing, pages 193-203, Seattle. Washington, USA. Association for Computa- tional Linguistics.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehen- sion dataset. In Proceedings of the 2nd Workshop on Repressntaatinn Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2022. Qa dataset explosion: A taxonomy of nlp resources for question answering and reading com- prehension. ACM Computing Surveys (CSUR).
Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. Proceed- ings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Pa- pers), pages 5039-5059, Dublin, Ireland. Association for Computational Linguistics.
Andreas Rücklé Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer, Nils Reimers, and Iryna Gurevych. 2021. AdapterDrop: On the efficiency of adapters in transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7930-7946, Online and Punta Cana, Dominican Republic, Association for Computational Linguistics.
Tu Vu, Minh-Thang Luong, Quoc Le, Grady Simon, and Mohit lyyer. 2021. STraTA: Self-training with task au augmentation for better few-shot learning. In Pro- ceedings of the 2021 Conference on Empirical Meth- ods in Natural Language Processing, pages 5715- 5731, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Marzich Sacidi, Max Bartolo, Patrick Lewis. Sameer Singh, Tim Rocktäschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules conversational machine reading.

---

Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessan- dro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transferability across NLP tasks. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP),, pages 7882-7926, Online. Association for Computational Linguistics.
Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin. Zhiyuan Liu, Ming Zhou, Jiahai Wang, Jian Yin, and Nan Duan. 2022b. Proqa: Structural prompt-based pre-training for unified question answering.
Ben Zhou, Daniel Khashabi, Qiang Ning, and Dam Roth. 2019. "going on a vacation" takes longer than "go- ing for walk": A study of temporal commonsense understanding. In Proceedings of the 2019 Confer- ence on Empirical Methods in Natural Language Pro- cessing and the 9th International Joint Conference Natural Language Processing (EMNLP-IJCNLP), pages 3363-3369, Hong Kong, China, Association for Computational Linguistics.
Alex Wang, Jan Hula, Patrick Xia, Raghavendra Pappa- gari, R. Thomas McCoy, Roma Patel, Najoung Kim, Ian Tenney, Yinghui Huang, Katherin Yu, Shuning Jin, Berlin Chen, Benjamin Van Durme, Edouard Grave, Ellie Pavlick, and Samuel R. Bowman. 2019. Can you tell me how to get past sesame street? sentence-level pretraining beyond language model- ing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4465-4476, Florence, Italy. Association for Compu- tational Linguisties.
Wenhan Xiong. Jiawei Wu, Hong Wang. Vivek Kulka- mi, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang, 2019. TWEETQA: A social media focused question answering dataset. In Pro- ceedings of the 57th Annual Meeting of the Asso- ciation Computational Linguistics, pages 5020- 5031, Florence, Italy. Association for Computational Linguisties.
Zhilin Yang. Zihang Dai, Yiming Yang, Jaime Car- bonell, Russ R Salakhutdinov, and Quoe V Le. 2019. Xlnet: Generalized autoregressive pretraining for lan- guage understanding. Advances in Neural Infor- mation Processing Systems, volume 32. Curran Asso- ciates, Inc.
Zhilin Peng Qi. Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop -opp question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2380. Brussels, Belgium. Association for Com- putational Linguistics.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng. 2020. Reclor: A reading comprehension dataset re- quiring logical reasoning.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. 1. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. CoRR, abs/18118 10..1110.///0 12885.
Wanjun Zhong, Yifan Gao, Ning Ding, Yujia Qin, Zhiyuan Liu, Ming Zhou, Jiahai Wang. Jian Yin, and Nan Duan. 2022a. ProQA: Structural prompt- based pre-training for unified question answering. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 4230-4244 Seattle, United States. Association for Computational Linguistics.

---

A Appendix
A.1 Effect of Pre-training
Pre-training improves performance in few-shot scenarios, particularly in the lower range, with significant benefits observed in prompt-tuning. Following Unified-QA (Khashabi al., 2020), observe that pre-training the T5-base model on diverse source datasets with varying formats and skill requirements (as shown Table 2) can boost the performance of the pre-trained language model (PrLM) in both fine-tuning and prompt-tuning sce- narios. Our analysis reveals that pre-training can yield substantial performance gains through knowl- edge transfer from source tasks, especially when few training examples are available (refer to Fig- ure I). We further observe that prompt-tuning with pre-trained LM introduces inductive bias in prompts, resulting in much greater performance boost than FT-MT, with the difference becoming more pronounced as the number of instances in- creases (potentially due overfitting). Specifically, PT-R yields change in improvement from 36% to 24% the number of training instances increases from 16 1024, while improvement FT-MT dras- tically reduces from 27% to 7%. We note that ATT-MT follows similar pattern to that of Model Tuning (MT). Moreover, our findings indicate that datasets such as COSMOSQA, OBQA, DREAM, MCTest, IIRC, and BoolQ exhibit substantial per- formance gains through pre-training, likely due to their similarity to some of the source datasets. On the other hand, datasets such McTACO, QuaRel, ShARC, and PIQA, which are less closely related to the source datasets, do not exhibit significant improvements with pre-training.
PTRT
PTC
ATT-3I
462
52.9
522
31.3
53.3
52.1
50
P7-8
51.3
51.3
39.3
48.1
18.8
52.1
40
329
40.1
23.3
-4
22.4
53.5
(a) FT
F1
47.4
56.2
$3.0
56.3
54.8
56.0
57.
56
52.6
55.3
69.0
59.1
57.1
50.9
53.9
50.0
P7-
30.8
59.1
53.2
56.2
52.2
58.1
45.0
ATRAT
56.0
57.1
56.7
50.0
50.1
52.7
(b) PT-C
PATT
PS-
PFC
ATT-II
P1
35.1
58.6
60.
58.4
59.6
56.4
52.5
30.0
PPC
00.9
60.9
55.4
53.0
50.0
47.5
25
P7F
62.5
62.2
57.4
57.6
54.8
58.7
45.0
20
42.5
00.0
(c) PT-R
Figure 3: Heatmaps showing union matrix of different shots for each mode
Exemmpes
Figure 2: Improvement (%) of pre-trained backbone model (PrLM) over T5-base observed across different training approaches in few-shot setting.

---

Example
+ Few Shot
Label
FT-HT
PT-C
PT-R
Context: M: How long have you been teaching? W: To be trank, I'm tired of teaching the same textbook
though do being toacher. I'm considering
something probably going do!
Question: Waat's
128
Option: (A)
change
(C)
testbook.
Contert: Q2 Are you Natiee Americam America Inthan A: Yeu: just liniched ligh school
256
No
My parents wanted me to go college, but never applicd.
General Insinuction: who has been aecepted or enrulled acuredised degrce program,
the field of health camo, and
family membhrr
Question: Do qualify
benelit program?
Question: longeraiines
animals shiver most! Options: (A) Arotic Circle Sunatra Joon trogical minforest
513
Aretic Circle
Simutra
Aretic Cirele
Java
Context: Loe house painters finishd... While (oold not say they were not the greasest guy'...
they did nice job and the house looks much better. Hore are some photos
Question: What have color.
take photos of your house?
Options: (A]
house. wanted off
128
color. (D) None
Table 8: Table presenting qualitative examples showing model predictions across different shots for different tasks. Few shot column shows the shot until which predictions the table hold.
B Variation with model size
K-shot
size
FT
MI
PT (R)
PT (B)
ATT
16 Examples
Base
43.68
45.31
38.72
45,30
45.33
Large
52.19
46.34
50.21
50.96
Model Agroement aet true label %.
8,50
8.44
7,63
32 Exumples
Base
45.94
47.11
42.04
47.47
46.81
Large
55.96
56.86
48.09
50.35
51.16
A
10.02
9.75
6.04
2.88
4.34
128 Examples
Base
49.91
51.22
44.01
50.65
50.56
Lange
60.25
61.94
50.01
52.07
51.76
10.34
10.72
6.00
1.42
1,20
1024 Examples
57.00
59.51
COMTS.0 PAN commorserur,
sesmiin
2015
Large
69.71
69.87
52.37
58.06
57.15
A
10.16
9.19
6.23
1.06
-2.35
(a) 16 shot
Table 9: Comparison between model-tuning and prompt- tuning paradigms for different model sizes. Mean per- formance over 16 target datasets reported with T5 as pre pre-trained language model.
Stedal wgwemmenttttttmmmmmmmmmmmm. wet trus inbel
Recent studies have shown that the performance gap between prompt-tuning and fine-tuning reduces as the model size increases (Liu et al., 2021b). this work, we conduct experiments comparing the performance of base vs large variants of TS for a range of different fine-tuning methods shown in Table 9. Unless otherwise specified, we use the b5-base model for experimentation. We observe consistent improvement in performance with large language models. Specifically, model- tuning approaches achieve a consistent improve- ment of approximately 10 points across 32 to 1024 training instances, while prompt-tuning without ini- tialization achieves an improvement of roughly 6 points. However, prompt-tuning with initialization and ATTEMPT do not show significant improve- ment in performance with large models, and this improvement diminishes as the number of training instances increases. The limited performance gain from large models leads us to conclude that multi- task model model-tuning outperforms prompt-tume tupittttunggggg and
913 comPoreeoo
2013
(MOO)
(b) 64 shot
Wodel Agreement trae labal
Detasets (MOO)
(c) 256 shot
Figure 4: Graphs showing task level agreement across
different shots for different modes

---

(a) 16 shot
(b) 64 shot
(c) 256 shot
Figure 5: Graphs showing task level agreement across different shots for different modes
10248
CONTOLL
pipe necr
coAge
CLOPN
molest
book
0903
suorer
50
32
04
122
256
512
#shots
Figure 6: Graphs showing performance of PT-C on target datasets for different shots.
PRAT
PP4
PMT
PCC
61.0
65.7
100.0
38.6
58.8
100.0
50.4
50.0
100.0
000.0
100.0
45.5
16.1
652
32.8
#IC
02.9
50.4
00.5
100.0
212
50.9
38.0
47.8
56.2
100.0
03.7
48.7
P1C
50.6
45.7
52.8
100.0
57.0
47.4
05.7
53.3
64.0
72.2
100.0
53.7
58.8
48.7
56.7
63.7
200.0
48.0
mp
50.7
47.0
52.1
57.0
100.0
47.4
53.7
100.0
4044
44.7
43.5
48.7
100.0
43.7
(a) 16 shot
(b) 64 shot
(c) 256 shot
Figure 7: Heatmaps showing agreement matrix of different modes under different few shot settings
OBOA dalasel performance
Rapes performnnce
Shoro dalaset ce formance
PT4t PTC ITETT
IT PSC PTR ITAIT
PTR PTC. FTATT
fohols
Aptcca
(a) OBQA
(b) Dream
(c) ShaRC
Figure 8: The effect of prior and training data size on certain target tasks
ATTEMPT-MT in a few-shot setting. Nevertheless, prompt-tuning with initialization remains compa- rable to ATTEMPT-MT, and both methods signif- icantly outperform prompt-tuning without initial-
ization. Overall, these findings suggest that the effectiveness of different candidates for unified QA may depend on the size of the model and the num- ber of training instances available.

---

FI-NT
P1-F
PTC
AT--T
FT-T
PSFF
PT-C
ATT-I
46
44
50
40
40
04
1024
120 Exemples
domain-specific books
Knowledge graphs
65
54
60
Parfomance
52
40
35
32
64
128
512
1024
32
64
128
256
512
1024
Examptes
Examples
(c) Wikipedia
(d) Web & Social Domain
Figure Comparison of FT-MT, PT-F, PT-C and ATT-MT several few-shot scenarios using T5-Base the backbone model for different domains
Skill-based
Domain-hased
Machine Reading Comprehension
Web & Social Media
qü. mevest, boole, reclor
tweet ea, pica,
Commonsense Reasoning
Wikdpedia
cosmion qo, piga, commoneense, ga,
lire, myees, quevef, boolg
DHalng Reasoning
Knowledge Graph
share, dream, quoref
commonsense, qu, cosmose qj
Causal Reasoning
domuin specific book share, ohga, quarel
ropes, obga, quarel, capa
Table 10: Categorization of Target Datasets Based on Domain and Reasoning Skill

---

FTNT
PT-
AT
FT-MT
PTF
PT-C
ATT
65
60
55
47.5
Moan
50
45.0
42.5
45
40.0
64
128
512
1024
128
256
Excamptes
(a) Reading Comprehension
(b) Dialog reasoning
39
Meon Peremmnnc
55.0
52.5
144
50.0
47.5
50
32
64
128 Extemples
512
1024
32
128
236
512
1024
EExmmpess
(c) Causal reasoning
(d) Commonsense reasoning
Figure 10: Comparison of FT-MT, PT-F, PT-C and ATT-MT in several few-shot scenarios using T5- [S-BBse the backbone model for different reasoning skills

---

Iweel.ge
repes
OONmAN qpi
page
CQQA
dreum
olegs
shane
quarci
mudica
In Labk
baalq
capa
queeet
RGG
16 Examples
65.26
35.00
30.00
38.13
FEMT
60.77
48.76
35.90
54.08
39.78
20.33
31.00
24.27
57.62
48.32
50..3
65.48
54.93
99.33
42.17
33.72
45.62
PT-
39.97
40.22
25.93
52.39
35.68
35.92
25.90
25.80
23.10
47.48
53.50
66.13
37.90
51.33
38.26
23:23
38.92
PT-
68.15
37.04
35.10
36.09
43.35
40.67
34,53
26.13
39.68
47.24
58.17
65.36
37.83
53.00
42.34
27.00
44.61
PTCC
68.15
57.67
36.07
54.35
41.71
36.31
31.47
26.27
58.00
19.88
52.67
65.31
42.75
57.00
43.03
27. 18
45.55
40.57
32,93
27.20
39.17
66.20
51.12
40.49
45.70
ATT-MT
60.52
32.82
37.10
55.08
10,90
39.54
30.50
26.20
38.65
18.20
15,83
64:42
59.51
58.00
45.05
41.06
15.23
Examplex
FT
65.55
49.74
36,18
55.42
41.85
57.43
50.60
24.53
41.54
48.00
52.17
66.14
611.52
66.67
44.77
26.56
46.36
FT-IT
61.01
53.55
36.05
35.68
41.61
42.53
32.40
25.33
39.42
46.76
50.83
64.95
58.20
53.33
44.28
46.00
47.00
PER
35.72
39.50
32.23
53.01
35.76
36.00
27.33
25.27
30
46.52
51.53
66.06
61.80
53.67
41.25
30:3
42.76
PEF
68.61
46.41
34,80
56.22
42.97
41.44
40..0
27.077
40100
66.64
59.17
66,09
37,83
55.00
42.51
26.90
45.75
37.72
32:37
attss
63,80
50.97
34.52
55.11
41.88
36.88
33.20
27.20
44.18
49.64
53.50
66.31
53.96
64.67
46.46
40.69
47.68
ATE-MT
64.11
42.56
Examptes
FT
66.17
48.65
36.73
55.53
13.65
35.42
29.73
24.93
42..97
18.08
52.67
13
65.31
60.67
46.85
42.83
18.08
62.94
37.91
55.91
44.44
44.56
36.33
28.33
401.00
53.33
59.33
50.33
49.12
PER
57.15
40.45
31,65
53.12
37.51
36.57
27.13
25.95
401.1
47.44%
53,00
66.12
62.39
53.67
39.11
38.00
44.34
37.84
45.04
PTC
68.29
58.97
38:43
55.01
43.73
37.21
31.53
26.33
37. 56
49.65
$7.83
65.35
61.22
62. 63.33
43.00
44.5
48.0
63.26
37.45
19.114
63.67
46.96
48.25
ATEMT
63,86
55.02
41.68
35.46
39.97
40.51
34.20
27.67
48.91
48.08
52.83
64.60
63.47
57.33
41.01
50.773
49.08
125 Exampler
401.76
37.71
33.87
25.33
47.96
53.00
67.64
72.10
FT-IT
64.91
52.98
41.03
57.00
46.00
47.89
43.33
30.13
43.62
52.52
59.17
66.21
67.25
61.00
48.43
54.38
52.27
38.65
PTFF
68.79
47.59
39.03
56.58
44.25
43.99
42.33
25.13
41.26
46.25
66,00
65:49
61.90
57.67
42.74
52.66
50.31
68.79
43.13
40.02
artst
63:48
52.39
38.32
34.99
42.70
36.67
32.93
25.50
44.76
50.24
53.50
66.21
61.45
57.67
47.33
50.83
48.76
ATT-MT
65.34
46.84
42.14
$4,28
42.81
42.22
41.33
29.27
47.99
51,32
57.83
67.11
69,23
53.33
42.01
$1.91
50.31
Examplee
FT
67.64
4%.42
42.00
55.82
49.39
40.51
42.07
25,077
48.99
49.28
61,83
71.33
74.78
58.67
52.92
58.02
52,96
46.21
PTER
63.26
44:25
36.36
33.39
38.08
36.24
26.47
27.00
39.65
29.16
55.50
66.14
62.31
46.67
43.08
44.90
45.77
PTP
69.03
$1.30
42.27
57.40
46.98
50.96
57.93
25.47
45.46
46.04
67.00
66.51
67.34
55.33
45.08
54.77
53.15
PTC
69.03
50.37
42.21
56.89
44.55
41.36
28.73
43.58
68.67
66.36
71.21
60.33
45.54
53.96
52.31
62.79
51.77
39.79
35.91
16.25
37.34
36.67
26.10
46.30 76.70
49.10
51.67
66.09
65.73
63.00
47.57
53.34
50.21
ATT-T
66.68
41.43
43.86
29.20
49.3
53.12
61.17
67.55
72.75
67.00
52.68
512 Examples
$4.54
FEMT
67.98
55.25
42.75
38.74
51.35
56.23
50.00
34.33
52.002
58.99
72.50
74.99
75.46
65.67
57.36
66.02
59.54
65.44
38.74
33.77
40.16
49.77
PT-F
67.28
51.33
13.96
57.13
18.98
52.92
61.33
25.73
40.65
18.32
71.00
67.07
75.64
54.33
51.72
58.23
51.91
PT-
67.28
56.10
43.91
56.75
48.40
49.35
58.60
28.60
50.00
48.50
71,83
66.21
72.74
56. 53
53.90
56.63
55.34
27.73
57.67
51.20
ATT-MT
68.23
58.65
44.32
56.29
47.67
46.72
53.50
31.33
51.87
53.60
68.33
72.71
50.52
58.33
50.24
62.53
$1,70
7024 Esamples
FI'
68:90
52.90
46.21
56.46
54.57
52.17
60.00
27.93
56. 12
52.552
75.50
78.76
76.99
65.00
63.15
67.55
59.71
FTMT
70.59
58:57
42.91
58.74
54.38
59:22
64.90
35.80
55:87
61.15
76.33
76.84
76.27
63.67
61.28
67.26
61.48
64.58
35.88
52.97
40.87
36.03
33.60
26.53
42.41
46.64
53.67
66.54
62.15
46.67
52. 16.16
46.73
PEEP
67.39
55.21
42.66
56.40
52.33
$2.19
62.30
30.00
53.53
49.52
77.33
66.86
76.45
59.33
58.50
64.57
57.82
51.30
51.44
68.02
66.67
57.34
attst
69.69
57.27
10.22
55.77
50.01
47.60
55:27
26.87
53.74
19.64
62.50
66.59
75.02
60.67
50.83
59.28
55.06
ATE-M
51.35
75.41
61.33
Pull
FT
77.40
59.
69.82
67.63
62.74
66.32
74.20
47.20
67.
67.99
78.00
99.41
82.97
67.00
71.18
70.57
70.56
FT-MT
76.34
58.53
67.30
67.03
61.18
74.20
36.20
63.35
59.35
73.50
92.21
80.58
66.00
70.19
60:3 38
67.70
PER
75.63
55.32
55.58
60.94
59.38
60.49
68.40
20..60
58.72
62.23
74.0K)
95.95
HI.61
55.00
68.48
69.
65.03
57.24
76.87
PEC
75.23
52.13
58.60
60.93
58.07
64.11
71.40
41 16
58. 53
62.05
76.50
96.43
79.76
62.00
67.55
69. 55
65.97
55.31
62.62
64.00
AFEMT
74.37
57.02
58.63
61.53
59.89
61.67
67.70
00
60.11
57.19
27.00)
94.19
80.86
65.00
67.71
68.89
65.65
Table II: Complete set of results for comparison between model-tuning and prompt-tuning approaches 16 target QA datasets with T5-base as pre-trained language model.

---

Backbone T5-Base
Backbone PrLM
k-shot
FT
MT
PT-R
PT-F
PT-B
ATT-ST
ATT-MT
FT
MT
PT-R
ATT-MT
4.07
2.74
6.05
1.45
2.18
3.49
3.29
1.4
2.11
4.18
2.24
1.28
2.71
2.90
3.11
3,50
2.85
1.68
1.60
2.77
1.85
4.60
2.53
2.02
3.29
3.03
2.42
3.21
1.13
2.35
2.42
128
2.25
4.13
2.76
2.80
256
3.36
2.40
3.25
3.63
2.84
512
2,62
1.35
2,04
3.77
2.45
0.83
2.23
3.17
1024
1.73
1.71
4.31
2.62
2.21
3.95
2.93
1.35
2.09
2.78
Table 12: Table displays the aggregate standard deviation of target tasks with different seeds. Increasing training instances reduces standard deviation, improving model robustness and reducing sensitivity to minor variations. PrLM reduces standard deviation across all approaches, leading to stable performance and better generalization while addressing overfitting. Prompt tuning has higher deviation due to initialization sensitivity. Parameter-sharing and prompt initialization techniques reduce deviation, leveraging knowledge from other tasks stable performance, especially in low-resource scenarios, and mitigating overfitting.
tweel.ge
ippes
OONTON qpi
pip
CQA
dream
olugi
suare
quarci
midel
InC Tabk
baolq
capa
queeef
RgS
16 Examples
73.17
68.00
45.75
FT-MT
71.44
52.13
47.18
57.44
51.02
68.61
67.40
37.27
40.68
48.68
84.83
67.40
76.15
65 35
48.61
43.00
58.00
61.75
52.32
42.64
40
56.33
48.20
35.83
ATT-MT
69.37
69.36
47.07
36.09
52:47
63.78
56.60
36.40
41.61
50.12
85.83
67.14
71.87
67.67
47.09
45.18
57.47
32 Esumpler
42.45
50.59
FT-MT
71.46
$2.67
46.58
57.02
51.00
58.60
68.47
36,80
42.94
50,00
84.67
67.41
79.18
63.67
50.53
48:21
58.70
61.61
AFEMT
70089
50.39
48.94
55.79
51.73
66.23
67.67
35.80
38.81
48.48
86.17
66.29
72.30
65.00
51.26
46.89
57.63
Exmmpler
24.17
55.59
53.26
43:21
66.51
53.92
FEMT
72,06
52.55
48:15
56.78
52.17
68.82
67.73
37.87
40 10
49.16
84.83
66.70
80.10
64.00
$5.00
52.76
59.21
40.76
41.75
ATT-MT
71.61
51.48
18.14
55.98
52.33
66.06
68.33
36.07
40.15
50.36
85.67
63.57
77.32
66.33
$2.59
46.09
58.24
Examples
FT
74.00
51.63
50.73
56.51
54.79
68.30
59.00
35.80
43.39
48.44
86.33
67.71
81.70
66.67
56.16
61.00
61.02
FTMI
72.89
51.04
49.95
57.18
54.00
69.12
68.13
37,60
44.25
52.52
85.33
68.38
81.53
63.67
56.61
59.04
60.69
71.70
54.12
45.62
56.35
42.34
36.47
30.
83.83
72.94
58.
50.79
44.
56.24
AUEMT
71.76
52.32
48.39
56.18
51.05
66.34
68.30
34.73
401.3
50.36
85.33
62.20
78.50
59.00
54.49
52.64
58.31
256 Esamples
FT
74000
52.5%
52.25
56.20
55.88
69.26
69.67
37.53
46.86
.9..76
86.83
69.75
82.02
65.00
60.49
67.60
62.23
64:95
PTR
21.31
52.73
48.15
55.15
47.94
67.19
55.40
34.27
39.75
48.32
84.17
65.81
75.92
55.00
50.96
50.33
57.09
ATT-MT
71,11
52.20
49.65
56.33
52.06
66.21
69.73
37,73
44.07
52.52
86.33
67.96
78.50
62.67
55.05
56.64
59,94
Exumptees
FT
74.34
55.50
53:45
58.54
57.41
68.84
70.67
36.10
55.78
$1.08
86.67
75.79
82.14
65.33
63.96
69.1 18
61.07
53.94
PER
71.98
54.72
48.08
55.84
48.
66.67
69.93
34.60
37. :44
50.48
84.53
66.15
74.74
60.33
$1.66
47.00
$7.20
ATEMT
73.27
53.55
51.66
51.13
37,00
70.23
60.66
65.41
1024 Esumples
FT
73833
$7.80
57.20
58.29
59,
70.85
70.67
42.27
59.91
53.48
87.33
31124
82.20
66.00
66.28
69.36
65.90
73.87
56.21
58
79.16
PTR
71.79
52.84
48.59
56.13
51.11
66.63
68.53
34.27
38:71
49.40
85.33
65.70
77.19
61.33
55.15
49.38
58.26
ATE-MT
74.78
59.91
65.67
Pull
FT
79.05
65.52
71.06
67.85
62.16
73.24
72.20
54,40
66.99
73.74
85.00
99.40
82.75
69.00
72.78
72.19
72.96
FT-MT
27.16
61.79
69.82
67.25
61.75
73.33
72.00
42.60
65.94
69.12
83.00
93.37
82.91
69.00
71.61
70.62
70.74
PER
77.04
62.60
66.57
61.70
60.85
70.69
70.60
42,80
59.53
70.86
85.50
95.59
82.35
61.00
71.28
70.54
69.35
Table 13: Complete set results for comparison between model--uning and prompt-tuning approaches on 16 target QA datasets with PrLM as pre-trained language model trained source tasks.

---

Model
IweeL.a
ropes
cosmos,
piga
CQA
quarci
motest
mc papo
copa
lire
16 Exumplex
75.12
44.38
39.63
56,91
53.40
41.27
40.90
27.10
28.54
19.28
73.50
64.50
74.65
67.00
45.85
49.54
52.19
Frrnt
71.87
46.81
38.83
39.79
55.28
52.75
42.50
31,00
31.37
55,04
73.50
63.65
71.41
72.00
46.97
46.34
53.74
plonndom
74.79
54.52
37.09
55.60
48.32
41.13
31.50
22.60
8.46
48.55
56.90
66.10
59.54
58,00
51.04
27.46
46.34
pt-hest
7291
ath-nt
69.38
$6.38
36.72
55.82
52.91
39.51
29.60
26.80
41.19
16.76
65.00
66.14
65.76
65.00
54.52
43.83
50.96
Exumplex
75.56
$7.00
41.24
54.62
51.76
45.77
50.40
26.00
58,49
50.001
71.00
67.17
73.18
76.00
57.88
56.
55.96
Fomnt
72.84
52.79
41.81
38.65
55.61
58.53
52.00
31,20
39.16
53,95
71.50
65.99
70.64
77.00
54.47
53.63
56.86
plenandom
75.62
51.59
38.19
54.73
52.25
40.15
$2.00
22,40
14.93
47.44
58.00
69.79
52.55
28.
pt-hest
72.22
46.39
58.22
55.50
51.43
59.55
52.00
21.00
59
47.12
57.50
66.13
73.39
05.00
54.19
45,00
50.33
125 Examples
42.31
63.65
Fiimt
73.23
50.07
49.45
62.13
56.92
64.71
61.00
34,80
48.92
58.63
80.50
74,09
79.54
73.00
61.20
62:27
61.94
pl-tandom
75.23
60.79
36.35
56.37
48.73
41.13
32.50
21.40
40.40
19.64
59.50
66.17
68.87
62,00
$2.66
28. 25.20
50.01
pt-hest
73488
37.96
23.80
59.50
46.74
52.07
73.13
61.37
38.12
54.08
54.22
99.71
26.90 20
26.66
42.57
50.00
59.50
66.13
69.08
62.00
56.02
48
51,76
Examples
78.71
54.74
56.38
63.44
66.91
71.67
73.40
41.80
59.23
72.66
85.50
85.89
83.67
79.00
70.42
71.99
69.71
77.17
54.05
64.05
74.95
74.00
72,66
85.55
70.68
pterandom
73.38
56.78
38.39
54.35
54.46
40.59
32.40
24,40
47.05
52.88
60.50
66.08
73.39
63.00
44.37
52.37
pt-best
73.19
56.41
35.78
53.92
59.62
37.40
57.00
28.00
54
47.84
81.50
66.45
81.35
65.00
67.27
61.24
58.06
albent
75.57
39.30
50.53
66.13
79.82
67.00
57.15
Fuli
81.34
73.33
P-mnt
80.31
64.06
74.87
71.38
73.46
75.97
80.00
49.20
68.00
72.66
88.50
93.74
83.98
78.00
77.12
73.96
75.58
79.20
57.26
74.30
pt-best
80.08
58.00
67.57
67.36
68.55
77.40
78.20
42.80
63:33
72.30
86.50
98.28
83.79
73.00
75.34
73:43
72.87
sibend
78.88
54.88
57.35
63.87
68.14
67.45
74.60
33.40
59.79
62.59
84.50
85.12
84.59
69.00
74.62
71.33
68.13
Table 14: Complete set of results for comparison between model-tuning and prompt-tuning approaches on 16 target QA datasets with T5 L5-Large as rre-trained language model.
Example
Task
label
fl.mt
uqa_mi
init, random
init Format
init sqqud
att
how do you use cream? context: (A) skin, (B) get
(R)
(R)
(R)
(R)
(R)
(A)
(R)
spray it all
small arnonnt it, and spread it all over your skin wherever apply it until you can't see anymore.
piqu
d like to
How to keep cool outside.? context:
piqa
(B)
(B)
(8)
(8)
(B)
(B)
(A)
(A) Add some peppermint oil to spray water, spray self avoiding eyes. (B) Sit under shade mee.
How to grow plant.? contest: (A) Bury sand and cup water duily. (B) Bury seed in soil and add I cup of duily.
piqa
(B)
(B)
(B)
Which sumple will likely hiss? Context:
ropes
Sample B
Sample A
Sample B
Sample
Sample
Sample R
Sample B
WIll St. Louis have more or less acid rain than Seuttle? Context:...
ropes
less
less
less
less
... So recently installed coal-fired plant... When acid rain falls.
Which city will more likely have acid rain? Context... St. Louis recently installed a coal. fired power plant...
St.Lonis
St.Lous
So.
St.Louis
St.oous
Lo.is
St.is
When rain falls.
Tommy glided across marble with ease, but slipped and fell on the wet floor because more resistance.
quarel
wetfloor
wetffozr
wetfloor
wetfloor
marble floor
wetfloee
wetfloor
greater Which object will artruct fewer abjects to ir?
quarel
Mars


Reviews:
[{"review_id": "JxY4nYnxH7", "reviewer": "Reviewer_WoB3", "summary": "", "questions": "It is not clear the definition of a task. QA itself is a task I think this paper focuses on the dataset instead of task.", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for providing your valuable feedback. \n\n>Under the context of large language model, some of the motivation of this work is not valid anymore. I don't think we need to train a unified QA model by tuning the soft prompts -- the larger size of model can already answer different QA tasks well.\n\nAlthough LLMs have showcased promising adaptability across diverse tasks through in-context learning, prior research (as highlighted in https://arxiv.org/pdf/2302.06476.pdf and https://arxiv.org/pdf/2302.03494.pdf) has underscored their limitations in tackling intricate spatial, temporal, and causal common sense reasoning tasks—precisely the focus of this present study. It is apparent that when it comes to tasks involving symbolic reasoning, logical thinking, and question-answering, finetuned models outperform zero-shot LLM models by a significant margin. Notably, all of these multifaceted tasks can be framed as a question-answering task. The identified shortcomings emphasize the urgency of a model capable of effectively navigating numerous complex QA datasets, all while maintaining a competitive performance threshold within realistic computational constraints. Due to the substantial computational and memory demands associated with experimenting and deploying LLMs, T5 continues to be the favored option for real-world applications.\n\n>1) what can we expect if the LLM size goes to 13B, 65B or even larger? \n\nThe findings in Table 6 indicate that prompt initialization achieves similar results to more intricate prompt-transfer methods, even when using the T5-Large model. While this research focuses on an encoder-decoder model like T5, the approach can be applied to other models like 6B or larger. The goal of this study is to assess the effectiveness of PEFT-based techniques for unified QA. During the study, no open-source LLMs were available for prompt-tuning or fine-tuning, leaving room for future investigations in these areas.\n\n>2) If the model size goes larger, does the prompt tuning investigated in this work still useful?\n\nPrevious research (refer to https://arxiv.org/abs/2104.08691) indicates that fine-tuning prompts become increasingly effective as models surpass billions of parameters. Furthermore, the knowledge gained in this study from training medium-sized language models with fewer parameters can provide valuable insights for techniques such as LoRA-based finetuning, like Llama, which are well-known approaches for efficient parameter utilization.\n\n>It is not clear the definition of a task. QA itself is a task I think this paper focuses on the dataset instead of task.\n\nAlthough many of the existing NLP problems can be approached as question-answering tasks, we are addressing various datasets designed as question-answering tasks that emphasize different reasoning abilities, domains, and response formats. These distinct QA datasets are frequently denoted as separate tasks within the manuscript. We intend to provide further clarity on this matter in the final version of the camera-ready copy.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This analysis is in-depth and comprehensive", "reasons_to_reject": "1. Under the context of large language model, some of the motivation of this work is not valid anymore. I don't think we need to train a unified QA model by tuning the soft prompts -- the larger size of model can already answer different QA tasks well.\n2. The current investigation is based on a T5-base/large model. The investigated solution is limited and there are some uncovered questions: 1) what can we expect if the LLM size goes to 13B, 65B or even larger? 2) If the model size goes larger, does the prompt tuning investigated in this work still useful?", "questions_to_authors": "It is not clear the definition of a task. QA itself is a task I think this paper focuses on the dataset instead of task.", "paper_topic_and_main_contributions": "This paper investigates unified question-answering (QA) models for low-resource scenarios and explores the potential of two tuning paradigms, model tuning, and prompt tuning, to overcome the challenges of specialized models in specific QA tasks. The study analyzes their applicability using 16 QA datasets and finds that prompt tuning can achieve comparable performance to model tuning in a few-shot setting with a good initialization. ", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "yT8XRuXrmi", "reviewer": "Reviewer_6o8a", "summary": "", "questions": "None", "limitations": "", "rating": 4, "confidence": 3, "soundness": 5, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for valuing our work. \n\n>Only evaluated with T5.\n\nWhile this study demonstrates outcomes using an encoder-decoder model like T5, it can easily extend to other model architectures and larger sizes, such as 6B or beyond. Due to the high computational and memory requirements for experimenting and deploying with LLMs, T5 remains a popular choice in real-world applications.  This work aims to assess the feasibility of PEFT-based techniques for unified QA. Throughout this study, open-source LLMs for prompt-tuning or fine-tuning within a reasonable computational budget were unavailable. We will leave these explorations to future work. \n\nAdditionally, insights from parameter-efficient learning for medium-sized language models should guide approaches like LoRA-based fine-tuning (e.g., Llama), another widely used and parameter-efficient method. We will include the standard deviation in the core article of the camera-ready draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This is typically the kind of paper that the NLP community needs, instead of yet another method: to a great computation cost, authors propose a comprehensive comparison of PEFT against Full Fine tuning for the unified QA problem. Even if Full Fine tuning performs best, the very few shot results of prompt tuning with correct initialization performs similarly. For QA, that requires high annotation cost, this is a key finding for real life practitioners; Additionally, the experiments on pre-training added in the appendix are also really interesting, and would be better placed in the core article (maybe the aggregation only). \n\nThe protocol is clearly presented, and the appendix seems to provide a sufficient amount of detail for reproducibility. \n", "reasons_to_reject": "Only evaluated with T5.  ", "questions_to_authors": "None", "paper_topic_and_main_contributions": "In this article, authors evaluate prompt tuning against full fine tuning with T5 in a transfer learning setting for Question Answering (training on multiple source dataset and applying to in-domain and out-domain QA datasets). They compare several strategies that unify the knowledge from the source datasets, and summarize the results in several key findings, comparing impact of model size and number of few shot examples. ", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "Including the standard deviation in the core article instead of appendix would be better, as it is a key statistic when many dataset results are aggregated. This should also be discussed in the article."}}, {"review_id": "UpUt1pPD62", "reviewer": "Reviewer_Y5aV", "summary": "", "questions": "What is the meaning of PT(R) or PT(B) in Table 6 ? I cannot find any explanations. It would be helpful add the explanation in the caption of Table 6.", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for taking the time to review our work and providing your valuable feedback.\n\n>The investigation goal 1 “the ability to utilize a single model to address a range of different QA tasks” is the exact problem that LLM can solve.\n\nWhile LLMs have demonstrated a promising ability to adapt to various tasks using in-context examples, previous studies (https://arxiv.org/pdf/2302.06476.pdf, https://arxiv.org/pdf/2302.03494.pdf) have highlighted their limitations in handling complex spatial, temporal, and causal common sense reasoning tasks, which are covered in the scope of this study. It is evident that finetuned models significantly outperform zero-shot LLM models on tasks involving symbolic reasoning, logical reasoning, and question-answering. Since all of these complex tasks can be posed as question-answering tasks, the identified shortcomings underscore the necessity for a model capable of effectively addressing multiple complex QA datasets while maintaining competitive performance within a computational budget for real-world scenarios.\n\n>Although you may cannot tuning LLM such as chatGPT, but for me, applying In-context-learning with 16/32 demonstration example is still comparable with your explored model tuning or prompt tuning. \n\nPrior research, such as https://arxiv.org/pdf/2205.05638.pdf, indicates the superiority and cost-effectiveness of few-shot Parameter-Efficient Fine-Tuning over In-Context Learning, providing further motivation for this work. Additionally, insights derived from parameter-efficient learning for medium-sized language models should offer valuable guidance for approaches like LoRA-based finetuning (e.g., Llama), which is another popular and parameter-efficient method.\n\n> I will wonder that if the conclusion still hold when using larger models like T5-XXl, or models with 6B size or larger.\n\nOur observations from Table 6 demonstrate that prompt initialization achieves comparable performance to even more complex prompt-transfer approaches, even for the T5-Large model.\n\n>It would be interesting to investigate the robustness of these conclusions when using larger models.\n\nWhile this study only shows results on an encoder-decoder model like T5, it can be easily extended to other model architectures and sizes, such as 6B or larger. This work aims to evaluate the viability of PEFT-based techniques for a set of complex QA datasets. At the time of this study, there were no open-source LLMs available for training, thus we will leave these explorations for future work.\n\n>What is the meaning of PT(R) or PT(B) in Table 6 ? I cannot find any explanations. It would be helpful add the explanation in the caption of Table 6.\n\nLines 242-289 describe the models PT(R), PT(F), and PT(C). In summary, PT(R) refers to prompt-tuning with initialization from random tokens in the vocabulary, while PT(F) pertains to initialization from a set of existing source prompts belonging to the datasets sharing the same format. On the other hand, PT(C) corresponds to the initialization involving the entire set of source prompts. We will clarify the explanations and captions further in the final camera-ready draft.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The paper conducts a sufficient number of experiments and provides a systematic and comprehensive analysis of the results. The authors' findings are based on a rigorous evaluation of their proposed approaches and highlight the potential of prompt tuning and parameter-sharing for unified QA in low-resource settings.\n\n2. In addition to the quality of their experiments, the paper is well-organized and easy to follow, which enhances its readability and accessibility to a broad range of readers. \nWhile the paper lacks a comparison under larger models, the authors' goals are still meaningful and important, warranting continued research and development in this area. \n", "reasons_to_reject": "1. The paper did’t compared with any Large Language Model (LLM), such as chatGPT or Claude, which lessening the impact of the paper.  I understand the paper may be finished in advance, but there are three main concerns that the comparison is matters:\n    a. The investigation goal 1 “the ability to utilize a single model to address a range of different QA tasks” is the exact problem that LLM can solve.\n    b. The paper aim to explore QA under  low-resource setting — The investigation goal 4 “extensibility to new tasks without requiring expensive retraining”, which are also LLM is good at.\n \n    Although you may cannot tuning  LLM such as chatGPT, but for me, applying In-context-learning with 16/32 demonstration example is still comparable with your explored model tuning or prompt tuning. Besides, tuning a LLaMA 6B/17B are also affordable and convincing.\n\n\n2. Considering the paper is empirical study, only using T5-Base/large model may weak the generalizability of you findings.  For instance, the conclusion that \"parameter-sharing results in superior few-shot performance, but the trends are reversed in the full-shot setting\", and the observation that \"format-based prompt initialization achieves comparable performance to more complex prompt-transfer approaches\", I will wonder that if the conclusion  still hold when using larger models like T5-XXl, or models with 6B size or larger. It would be interesting to investigate the robustness of these conclusions when using larger models.", "questions_to_authors": "What is the meaning of PT(R) or PT(B) in Table 6 ? I cannot find any explanations. It would be helpful add the explanation in the caption of Table 6.", "paper_topic_and_main_contributions": "The paper argues that current QA models are typically designed for high-resource scenarios and require re-training to be useful in low-resource settings. To address this issue, the authors explore the potential of two paradigms: model tuning and prompt tuning, for unified QA in a low-resource setting.\n\nThe authors' results demonstrate that prompt tuning outperforms model tuning in a few-shot setting with a good initialization for out-of-distribution target tasks. Additionally, they show that parameter-sharing leads to superior few-shot performance, and that simple knowledge transfer techniques for prompt initialization can be effective in a low-resource regime.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "N/A"}, "extra_text": {"Missing_References": "1. He K, Huang Y, Mao R, et al. Virtual prompt pre-training for prototype-based few-shot relation extraction[J]. Expert Systems with Applications, 2023, 213: 118927.\n\nThe paper talk about prompt initialization and under few-shot setting. "}}]

Author rebuttal:

Meta-review: The paper investigates the question of building a unified QA model trained on a collection of QA datasets, primarily investigating if it is better to train it using full fine-tuning or prompt-tuning. 

Results on T5-base and T5-large show that prompt-tuning with proper initialization for the additional param can outperform full fine-tuning. 

The paper is set to explore a big question, but the execution is limited to a narrow setup that it is not clear if it generalizes. Specifically, 
- Previous work showed that prompt-tuning is not flexible enough to change the model behavior significantly. The positive results with prompt-tuning might just indicate that the full-finetuning baseline wasn't trained properly
- Results in Table 5 need to be contextualized in previous results from the literature and in existing out-of-the-box methods
- Giving the known limits of prompt-tuning, other methods like lora are more promising to explore.
- Experiments are limited to T5-base and T5-large, not including auto-regressive models like opt, pythia and llama.
Decision: Accept-Findings