MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension
Authors: Guoxin Chen
Yiming Qian
Bowen Wang
Liangzhi Li
Venue: EMNLP 2023 Findings (2023)
Keywords: Question Answering
Prompt Learning
Soft Prompt
Machine Reading Comprehension

Abstract: The large language models have achieved superior performance on various natural language tasks. One major drawback of such approaches is they are resource-intensive in fine-tuning new datasets. Soft-prompt tuning presents a resource-efficient solution to fine-tune the pre-trained language models (PLMs) while keeping their weight frozen. Existing soft prompt methods mainly focus on designing the input-independent prompts that steer the model to fit the domain of the new dataset. Those methods often ignore the fine-grained information about the task and context of the text. In this paper, we propose a multi-level prompt tuning (MPrompt) method for machine reading comprehension. It utilizes prompts at task-specific, domain-specific, and context-specific levels to enhance the comprehension of input semantics at different granularities. We also propose an independence constraint to steer each domain-specific prompt to focus on information within its domain to avoid redundancy. Moreover, we present a prompt generator that incorporates context-related knowledge in the prompt generation to enhance contextual relevancy. We conducted extensive experiments on 12 benchmarks of various QA formats and achieved an average improvement of 1.94\% over the state-of-the-art methods.

MPrompt: Exploring Multi-level Prompt Tuning for Machine Reading Comprehension
Guoxin Chen*** Yiming Qian Bowen Wang*. Liangzhi Li*
* *Mcetyou Al Lab Unnivrrsity of Chinese Academy of Sciences
Agency for Science, Technology and Research (A*STAR) Osaka University
Xiamen Key Laboratory of Women's Internet Health Management
gx. chen. chn@gmail. com qiany@ihpc. atmpp..a-sttee attesc..staa.ddde bowen. wang@is ids os. lillangzhieeiaayooziicco com
Abstract
the fine-tuning paradigm has become resource in- tensive.
The large language models have achieved supe- rior performance on various natural language tasks. One major drawback of such approaches is they are resource-intensiv in fine-tuning new datasets. Soft-prompt tuning presents resource-efficient solution to fine-tune the pre- trained language models (PLMs) while keep- ing their weight frozen. Existing soft prompt methods mainly focus on designing the input- independent prompts that steer the model to fit the domain of the new dataset. Those meth- ods often ignore the fine-grained information about the task and context of the text. this paper, propose multi-leve prompt tuning (MPrompt) method for machine reading com- prehension. It utilizes prompts at speciiie,,,,,, domain-specifie, and context-specific levels to enhance the comprehension of input semantics different granularities. We also propose an independence constraint to steer each domain- specific prompt to focus on information within its domain to avoid redundancy. Moreover, present a prompt generator that incorporates context-related knowledge in the prompt gen- cration to enhance contextual relevancy. We conducted extensive experiments on 12 bench- marks of various QA formats and achieved an average improvement of 1.94% over the state- of-the-art art !
Recently, soft-prompt tuning has been widely ex- plored as a parameter-effiieet approach to address- ing the aforementioned issues (Liu et al., 2023). For example, Li and Liang (2021) proposed Prefix- tuning, which prepends a sequence of optimizable prefixes each transformer layer while keeping the parameters of PLMs frozen. Prefix-tuming pro- vides lightweight alternative to fine-tuning and has achieved comparable performance with fewer trainable parameters. Lester al. (2021) proposed Prompt-tuning, which only prepends optimizable prompt vectors to the input sequence, which used fewer parameters compared to Prefix-tuning, Ma et al. (2022) discovered negative tokens in Prompt- tuning that have detrimental effect on downstream tasks and proposed XPrompt to mask these neg- ative tokens, resulting in improved performance. However, the aforementioned methods are input- independent, i.e., assigning uniform prompt to all inputs of a given task, which under-utilizes the in- put semantics for the answer generation in machine reading comprehension.
There is a growing trend towards designing input-dependent prompts (a.k.a dynamic prompts) for various tasks (Gu et al., 2021; Clive et al., 2022: Tang al., 2022). For example, Gu et al. (2021)
1 Introduction
proposed DialogPrompt for a dialog system, which dynamically generates prompt vectors according to the input dialogue context. Tang et al. (2022)
In recent years, pre-trained language models (PLMs) have been widely applied in question- answering tasks (Pandya and Bhatt, 2021), particularly in machine reading comprehen- sion (Baradaran et al., 2022), and achieved remark- able success through the seeraii-tten-ffnette paradigm (Roberts et al., 2020; Khashabi et al., 2020b). Despite the excellent performance, due to the explosive growth of parameter sizes in PLMs,
extracts input-related information from BERT (De- vlin et al., 2018) as contextualized prompts for natural language generation (Lewis et al., 2019; Raffel et al., 2020), which improves the relevance between the generated text and the input text. How- ever, to the best of knowledge, there has been little research exploring input-dependent prompt methods for aneesion-answering tasks, especially for machine reading comprehension. It is chal- lenging to apply input-independent methods to ma-
"Corresponding author.
'The code is available at httpps://gite thwp://ggttttu.... com/ Mhen-GXXMPrompt.

---

chine reading comprehension where the answer is context-sensitive
2022). It aims comprehend contexts and pro- vides answers to corresponding questions. In re- cent years, the focus of Machine Reading Com- prehension research has shifted from Extractive Question Answering (Seo et al., 2016; Wang et al., 2017; Tan et al., 2018) to Generative Question Answering (Izacard and Grave, 2020; Khashabi et al., 2020b, 2022; Jiang et al., 2022). For exam- ple, Lewis al. (2020) has explored retrieval- augmented generation scheme that combined pre- trained retrieval models to enhance the perfor- mance of the generative question answering models. Khashabi et al. (2020b, 2022) unified the input for- mat of different QA tasks into the same format and fine-tune the generative models (Raffel et al., 2020) for question answering. However, with the explo- sive growth in the parameter size of PLMs, the fine-tuning process becomes exponentially more resource intensive. One way to relax this computa- tional requirement through prompt learning (Li and Liang. 2021; Liu et al., 2023).
To address the above issues, we propose MPrropp, a novel Multi-level Prompt tuning ap- proach for machine reading comprehension. Our method utilizes the dataset and the context informa- tion to create three levels of prompts: task-specific, domain-specific, and context-specific The task- specific prompts are input-independent and gen- erate prompt based on the tasks. The domain- specific prompts utilize the domain knowledge generated from the dataset while context-specific prompts rely on the input context. These multi- level prompts endow PLMs with multiple fine- grained considerations of input semantics. To further enhance the domain-specific prompts and avoid information redundancy, we propose the in- dependence constraint steer each prompt to fo- cus on knowledge within the domain rather than cross-domain knowledge, Furthermore, we extract context-related knowledge from a scall-scale PLM, such as T5-small (Raffel et al., 2020), and integrate it into the prompt generation process to enrich the context sensitivity of prompts. With the help of achieve aver- age improvement of 1.94% over the state-of-the-arr methods on 12 benchmark datasets.
2.2 Prompt Learning
With the success of GPT-3 (Brown et al., 2020), prompt learning (Liu et al., 2023) has provided another efficient way utilize PLMs, which has attracted widespread attention. The format of prompts can be in camann-readabl natural language (discrete prompts) (Shin et al., 2020; Schick and Schütze, 2020), or embedding vectors (continuous prompts) (Lester et al., 2021; Li and Liang, 2021; Liu et al., 2021a,b; Ma et al., 2022). The con- tinuous prompts provide more flexible solution that encodes information into trainable embed- ding which presents the information pre-trained model more efficiently. For example, Lester et al. (2021) proposed Prompt-tuning, which achieves competitive performance by prepending trainable prompts to input sequences, and Ma et al. (2022) further improved the Prompt-tuning by pruning the negative prompt tokens.
these three levels of prompts,
Our main contributions are as follows:
We propose novel multi-level prompt tuning (MPrompt) for machine reading comprehen- sion which generates prompts at task-specific, domain-spectin and context-specific levels to improve answer generation.
We propose independence constraint to steer each domain-specific prompt focus on intra-domain information, avoiding infor- mation redundancy, at the same time enriching the Noomain-related semantics.
We propose a prompt generator based on a small-scale PLM to integrate context-related knowledge into prompt generation, which en- riches the context awareness and sensitivity of the generated prompts.
The aforementioned approaches did not suffi- ciently consider the full utilization of the input semantics and applied the same prompt for all ex- amples in the dataset, which potentially limits the delivery of the language models. Therefore, Tang et al. (2022) extracts contextualized prompts based on the input text from external PLMs, resulting in better performance in natural language genera- tion. Clive et al. (2022) proposes to combine task- specific prompts with dynamic prompts, enabling
2
Related Work
2.1
Machine Reading Comprehension
Machine Reading Comprehension (MRC) a chal- lenging task and hot topic in Question Answering Baradaran et al.,
(QA) (Pandya and Bhatt, 2021;

---

the model to have finer-grained control over the generated text.
crucial role as it contains the answer or the evidence in support of the answer. Mcanwhile, the context in QA datasets can often be divided into several do- mains. For example, in NewsQA (Trischler et al., 2016), the context can be grouped into different domains such as politics, economics, society, and so on. To improve the semantic understanding of context, the context from different domains should utilize different prompts, and each domain-specific prompt should imply a specific knowledge shared within the domain.
However, there has been little research exploring input-dependent prompt learning in question an- swering. In contrast to natural language generation, question-answering tasks emphasize understanding of the given question and context. Therefore, lack of input-dependent prompts may lead to under-leverage of the context information present in addition to the questions, particularly in machine reading comprehension tasks.
However, most QA datasets do not have explicit information about the domain of the context. To avoid additional annotation costs, we cluster the context C in an unsupervised manner to obtain different domains D E (D1...., Dn), where de- notes the number of domains, and each context can only belong to one domain. Each domain has its own shared prompt, therefore the domain- specific prompts D = (Di,... Dn), where Dj E Roxdp Vi E n.) D, denotes the prompt shared within the domain D;, P denotes the length of the domain-speciff prompts, dp denotes the di- mension of embedding from the prompt generator. Intuitively, domain-specinn prompts should en- capsulate information for each respective domain. Therefore, introduce the independence con- straint to steer D; to focus on the information within domain D;- Focusing on the knowledge specific to each domain can enhance contextual understanding, as confirmed by subsequent experi- ments. Specifically, for any pair of Da and DD C DD we introduce the Hilbert-Schmidt Independence Criterion (HSIC) (Gretton et al., 2005; Song al., 2007) to measure the independence between the prompts of two domains:
3 Methodology
Our proposed multi-level prompt tuning (MPrompt) framework is illustrated in Figure 1. The frame- work consists prompt generator and a genera- tive question answering model, whereas the former relies on a smaller-sized encoder-decoder archi- tecture. The prompt generator generates domain- specific and context-specifie prompts and elicits context-related knowledge from small-scale PLMs into the generation process.
3.1 Tpsk-specific Prompt
Many previous works (Li and Liang, 2021; Lester et al., 2021) have demonstrated that shareable prompt parameters learned from particular tasks effectively enhance the performance of pre- trained language models on downstream tasks. Therefore, following Li and Liang (2021), we con- struet task-specific prompts that share common prompt information within the task.
We prepend a prefix P E Rxd for the different types of attention class in the pre- pre-trained language models, where is the length of the task-specific prompt and d is the dimension of the embedding in generative QA model. For each attention class?, the prefix for key-value pairs T (Tı, ..., Tr) are learned through an ML.,, T = ML,((), where denotes the number of layers in the generative QA model, Ti = (TiKk, ) VI. E (1,...,AA L}. Ti,k and Ti, E pixd and T E IXx2dL The overall task-specific prompt is Trask = (Te, TDm, TDe].
1 (S, = 1p- tr(KHH)) (1)
where H is the centering matrix Hp 1p !1TT Kij ((Da; Day)) Lij (D,,, Do,). Da; E R1xd,, & and vb denote the kernel functions. HSIC 0 indicates independence, when o and vi are universal kernels. However, HSIC is not invari- ant to isotropic scaling, which can be addressed by normalizing HSIC which is known as Centered Kernal Alignment (CKA) (Nguyen et al., 2020; Raghu et al., 2021; Chen et al., 2023):
3.2 Domain-specific Prompt
In question answering scenarios, especially in ma- chine reading comprehension, the context plays a
'In encoder-decoder architecture models, there are typi- cally three types of attention: self-attention in the encoder. masked self-attention in the decoder, and cross-attention in the decoder. The corresponding task-specifi, prompts are denoted as Te. Tom, and TDc-
HSIC(D,, Db)
DD) HSIC(P,, D.) HSIC (D,, Dz)
(2)

---

Task-specific Prompt
Domain--peciifc
Context-speciffc
Tuned
Prompt Generator
Generative QA model
Answer
MLP
Past Key-values
Encoder
Decoder
Encoder
Decoder
Luse
MLP
MLP
Context
Question in Context
<p Answer
Figure 1: The overall framework of MPrompt.
where CKA [0, 1], and CKA implies inde- pendence.
small-scale PLM encode contexts and integrate them into the prompt generation process.
Computing the pair-wise independence requires m(n-1) iterations, which is slow for large To reduce computational costs, we randomly sample pairs of domains as θ to calculate the Lidp con- straints in each training iteration:
For a context c,, which belongs to the domain Dj. The encoder of the prompt generator takes the context G as its input, while the concatenation of domain-specinf prompt Dj and context-specinn prompt C serves as the input X' for the decoder,
Lidp Σ DJ).
X' x== [D;;C]]
(4)
(1,))0
where A' ((+x)xd,, should noted that we have removed the original decoder embed- ding layer. The output of the prompt generator is mapped to key-value pairs P = (PI..., PL} through the MLP,
3.3 Context-speeific Prompt
The domain-specific prompts provide shared intra- domain information, which provides iinegrrined knowledge compared to task-specific prompts. However, there are still diversities among contexts within the same domain, and utilizing such diverse information is critical for answering questions ac- curately.
P MLP MrP(PronnGGenenttooos... X)), (5)
where P ((tx))x2l1 P (Pi,K, PI,),, PLK and Pı,v E R(o+s)xd and L denotes the number of layers in the generative QA model. Intuitively, the knowledge related to the context q; is steered from the encoder of PLMs, and then integrated into the prompt generation process in the decoder. In this way, our approach allows for better learning of the semantics between prompt and context than previous work (Li and Liang, 2021; Lester et al., 2021; Ma et al., 2022), since both domain-specific prompt and context-specific prompt are closely re- lated to the context.
Therefore, we construct context-specific prompts to enhance the understanding of each context, which provides fine-grained knowledge compared to domain-specific prompts. Specifically, all contexts have shared sopeext-specific prompt C E Rxd,, where denotes the length of the context-specific prompt. Furthermore, we propose the prompt generator to ensure that generates different prompts for different contexts, especially for those contexts unseen the training data and discuss its other roles in the next section.
3.5 Applying Multi-level Prompts
3.4 Prompt Generator
Overall, P contains the information of domain- specific and context-specific prompts well knowledge from PLMs related to the context, while Trask contains the shared information within the task. order to exploit multi-level prompt infor- mation to enhance the performance on question answering, we integrate the above different levels
In general, task-specific prompts related to the task of specific datasets, while domain-specific and context-specific prompts both closely related to the context. To better leverage domain-specific and context- specific prompts to enhance PLMs' under- standing of the context semantics, we introduce a

---

of prompts into the encoder of the generative QA model. Specifically, for the self-attention computa- tion of layer/ in the encoder of the generative QA model, the original K2 and Vi are augmented as:
Datasets
Ques.
Cont.
Ans.
Type
len.
lcn.
Ion.
SQUAD2
118446
11873
11873
9.8
120
EX
NewsOA
72219
4341
4341
6.6
611
EX
NarQ4
63494
6922
21114
8.5
572
4.1
AB
DROP
9536
10.7
207
AB
MCTest
1480
320
840
26.2
213
3,9
MC (4)
ARCleusy
2250
569
2367
39.1
189
3.8
MC (4)
Ki- = (TELK: Plx: Kil, V [TR.V: Pu,v Vi]
ARC(chal.)
1119
46.2
185
MC
(6)
OBQA
4957
500
500
26.8
155
2.9
MC (4)
QASC
7208
926
5/26
30.1
253
1.6
(C)
RACE
25421
1436
1436
33.3
191
4.6
MC (4)
where Kj and V/ [ M denotes the length of the imput sequence. For the self- attention and cross-attention computation of layer in the decoder, K2 and Vi are augmented as:
BoolQ
6157
3270
3270
8.8
966
1.0
YN
BaelQ.NP NP
3798
98
Table 1: Dataset Statistics. NarQA and OBQA refer Narrative and OpenBookQA. "#Train" is ab- breviation for "the number of Training set". "Ques.". "Cont.", ", "Ans.", and "len." " are abbreviations for "Ques- tion", "Context", "Answer", and "Length", respectively. MC(4) indicates that the dataset contains candidates.
Ki = [TDM(DD) Kil, Vy [TDm(De)vvi Vil
where Ki and Vi E (
To train the multi-level prompts, the loss func- tion is a weighted sum of the two loss terms:
datasets in different formats were converted to a unified format to suit generative QA tasks. Due to space limitations, more details are available in Appendix A.I.
L - LNLL. + ALidp
(8)
where À is the hyperparameter used to control the independence constraint, LNLL is the text genera- tion loss, as follows:
Metrics. We evaluate each dataset using the metries most often used in previous work. For SQuAD2 and DROP, we used the F1 score with token overlap between the answer text and the gold answers. For NewsQA and NarrativeQA, we use ROUGE-L metric (Lin, 2004). For the multiple- choice and Yes/No QA, we use accuracy for evalu- ation (sometimes referred to as exact match), i.c., generated answer is considered correct only if it exactly matches the gold answers.
LILL = Σ log log
where y/2 denotes the t-th element of the target se- quence, and x represents the input sequence. is worth noting that, guided by Equation 8, we only update the MLP, task-specific, domain-specific, and context-specific prompts. while keeping all other parameters frozen.
Baselines. To comprehensively evaluate the performance of MPrompt, we compared it with a wide range of state-of-the-art soft-prompt methods, such as Fine-tuning (Khashabi et al., 2022), Prefix- tuning (Li and Liang, 2021), Prompt-tuning (Lester et al., 2021) and XPrompt (Ma et al., 2022).
4 Experiments
4.1 Datasets and Baselines
Datasets. To cover wide range of QA tasks evaluated approach on 12 benchmark datasets in the fields of Ex- tractive QA (EX): SQuAD2 (Rajpurkar et al., 2018), NewsQA (Trischler et al., 2016), Ab- stractive QA (AB): NarrativeQA (Kočiský et al., 2018), DROP (Dua et al., 2019), Multiple- choice QA (MC): MCTest (Richardson et al., 2013), ARC(easy, challenge) (Clark et al., 2016, 2018), OpenBookQA (Mihaylov et al., 2018),
4.2 Implementation
in experiments,
We convert cach dataset into unified text-to-text format suit generative question answering mod- els following (Khashabi et al.. 2020b, 2022). Our MPrompt is based on three scales of pre-trained UnifiedQA (Khashabi et al., 2020b) (which is a T5 model for question-answerie Large, XL with 220M, 770M and 3B parameters, respectively. For the prompt generator, we utilize UnifiedQA-Small with 60M parameters to ensure that there no excessive demand for GPU memory.
tasks): Base,
QASC (Khot al., 2020) RACE (Lai al., 2017).
and Yes/No QA (YN): BoolQ (Clark et al., 2019).
BoolQ-NP (Khashabi et al., 2020a). Table
In all experiments, we employ the AdamW op- timizer (Loshchilov and Hutter, 2017) and set B1 = 0.9, B2 = 0.999, and the weight decay is
presents the statistics of these datasets. Follow- ing Khashabi et al. (2020b), the above-mentioned

---

0.01. We train our method with a learning rate of 5e-5, 10% warmup ratio, X=le-4, 50 epochs and record the model with the best performance on the validation set. To ensure a fair comparison, we fix the length of task-specific prompts to 10 and adjust the lengths of domain- domain-specific and context- specific prompts to (5. 10, 13, 20, 30, 40, 50, 60), We use Kmeans (MacQueen, 1967) and Sentence- Transformers (all-mpnet-base-v2) (Reimers and Gurevych, 2019) to cluster the context and fix the number of clusters to 3 to obtain domain informa- tion D. The visualization of the clustering results by t-SNE (Van der Maaten and Hinton, 2008) is deferred to Appendix A.2. For all baselines, all hy- perparameter settings are based on the reported val- ues in the original paper to achieve optimal results. Our method is implemented with PyTorcl (Paszke al., 2019) and Transformers (Wolf al., 2020) library and experiments are conducted on Ubuntu 22.04 systems with NVIDIA RTX A100 or 4090 GPUs. Other implementation details and optimal hyperparameters are deferred to Appendix A.3.
t+d Cop
(-d+c
tedec
t+d
[+C
t-d+c web pg
0.69
0.64
0.59
0.54
0.19
0.64
0.39
0.24
SQuAD2
DROP
OBQA
BaoIQ-NP
Figure 2: Ablation Study. "t", "d", and "c" denote task- specific, domain-specithe and context- context-specific prompts. respectively. "w/o" is abbreviation for "without".
OBQA, QASC, and BoolQ-NP datasets. It is be- cause Prefix-tuning provides deeper prompts, while XPrompt removes negative prompts in Prompt- tuning. However, MPrompt achieves higher per- formance than Prefix-tuning and XPrompt at the same model sizes, demonstrating its effectiveness. (3) Due to the luxury of having high computational resources and full-weight update scheme in full fine-tuning, there is still a significant performance gap between soft-prompt tuning and full fine- tuning. However, As shown in Table 2, MPrompt matches the fine-tuning performance on tasks and even outperforms the fine-tuning performance of UnificdQA--Baa -ase and XL on most tasks. Specifi- cally for UnifiedQA-Baat MPrompt achieves the best performance on SQuAD2, NewsQA, NarQA, MCTest, ARC (easy), RACE, and BoolQ, result- ing in +0.69%, +0.62% +0.24% +1.31% %, +0.78% 0.21%, 16, and 0.25% improvements over fine-tuning, respectively. We incorporate context knowledge from other PLMs (such as UnifiedQA-smal in this paper) into prompt generation to enrich the seman- tics.
4.3 Performance Comparison
Table 2 displays the main experimental results of different methods 12 benchmark datasets. We conduct a comprehensive comparison between MPrompt and state-of-the-ait art methods, includ- ing Prompt-tuning (Lester et al., 2021), Prefix- tuming (Li and Liang, 2021), and XPrompt (Ma et al., 2022) for different parameter sizes of PLMs. The datasets cover wide range of question- answering scenarios, which is beneficial for the comprehensive evaluation of different methods.
We observe that: (1) Our method MPrompt out- performs other soft-prompt methods by large margin across all tasks and model scales. For ex- ample, MPrompt achieves absolute improvements of 2.17%. 1.85%, and 1.82% relative to Prefix- tuning on UnifiedQA -Base, Large, and XL respec- tively. It is due to the input-independent prompt learning methods applying a uniform prompt to all inputs for a given task, which evidently under- utilizing the input semantics in answer genera- tion. However, MPrompt significantly improves the performance in question cresstio-aasswringg tasks by enhancing the contextual comprehension of the PLMs with multiple levels of prompts. (2) Prefix- tuning and XPrompt have comparable performance at the same model size. Both algorithms out- perform Prompt-tuning on the NewsQA, DROP,
In summary, method achieved excellent per- formance compared to state-of-the-art soft prompt methods, closing and even surpassing the perfor- mance gap over fine-tuning. This demonstrates that MPrompt effectively enhances contextual compre- hension and enriches the semantics of the PLMs which significantly improves the quality of down- stream question-answeriig tasks.
4.4 Ablation Analysis
In this part, we perform an ablation study on the var- ious components of MPrompt, as shown in Figure

---

SQuAD2
NewsQA ROUGE-L
NarQA ROUGE-L
DROP
MCTes
ARC
ARCichall)
OBQA
QASC
RACE
BoolQ
BoolQ-NP
F1
ACC
ACC
ACC
ACC
ACC
ACC
ACC
ACC
Finc-tuning
46.06
72.45
78.52
Prompt-turing
68.07
$4.83
44.96
25.99
85.36
68.86
42.11
45.60
56.16
72.98
82.25
72.59
Prefix- tuming
42.49
55.00
76.31
220M
XPrompt
30.49
57.87
45.15
31.32
85.75
71.56
42.73
53,20
64,47
73.73
82.45
75.48
MPrompt
72.61
$9.99
46.30
87.74
73.23
44.97
69.98
75.70
77.25
Improvement
1.16
2.29
1.07
3.92
1.79
1 2.55
1 2.447
3.20
1.81
2.2.1
00:1
0.94
Fine-tuning
78.13
59.77
50.20
52.20
91.67
81.23
54,95
67.40
80.35
81.48
86.39
84.36
Prompt-tuning
72.40
56.45
48.76
90..4
76.47
$2.05
56.00
61.77
7x.43
79.27
Large
Prefix-tuning
75.20
59.24
48.21
43.79
9/. 201
79:71
52.67
64.60
78.08
79:50
85.45
81.83
XPrompt
48.56
MPrrmpt
76.52
60.35
49.37
50.08
93:45
80.93
$4.50
67.00
$0.15
81.19
86.17
82.94
Improveennt
1.322
1.116
1.6.9
1.122
1 1.222
1 11.83
2,40
2.007
1 1:1.4
1.7.2
1.10
Fine tuning
87.66
64.54
65.85
62.98
95.71
86.76
66.54
80.60
89.95
85.26
89.38
87.70
Prompt tuning
$2.91
60.24
53.69
45:36
93.33
83.75
63,40
71.00
85.54
85.28
88.39
84.65
XL
Prefix-tuming
84.86
61.83
57.34
58.83
95.27
85.77
66.98
78.20
86.01
85.44
88.93
86.26
3B
XPPrompt
84.73
62.24
56.66
51.11
94.39
85.98
65.69
76.30
86.87
85.91
89.47
86.49
MPrompt
39.41
86.95
81.80
Improvement
1.00
1.54
2.007
1.19
1 1.16
1.18
1.73
7 3,00
1: 2.2.7
1 1:227
1.34
f 1.13
Table 2: Comparison of state-of-art algorithm different datasets. The unit for all the metrics here is in percent- ag)(%). The numbers in blue indicate the performance gain (+) of our method compared to Prefix-tuning.
2. Firstly, observe decrease in performance when removing domain-specific or context-specific prompts. The domain-specific or context-specific prompts are constructed based on inputs of dif- ferent granularity, which enhances the semantic comprehension of the input. Secondly, when re- moving the independence constraint, there was a significant decrease in performance. The indepen- dence constraint steers domain-speciff prompts to focus intra-domain information rather than inter- domain information, which can effectively avoid in- formation redundancy. Furthermore, performance decreases when the prompt generator removed. The prompt generator ensures that context-specific prompts are generated differently for different con- texts, even those that never appear in the training data, which enhances the semantic understanding of the input context. Moreover, the prompt gener- ator elicits context-related knowledge from PLM and incorporates it into the prompt generation pro- cess, which helps improve the context awareness of the prompts.
SQUAD2
DROP
OBQA
BoolO NP
Figure 3: Evaluation of prompt length on the UnifedQA- base model. p and denote the length of the domain- specific and context -peeiffc prompts, respectively.
context-specific prompts impacts the final perfor- mance. We fixed the length of one prompt to 10 and varied the other in the range of (5, 10, 15, 20, 30, 40, 50, 60). As shown in Figure 3, in most cases, MPrompt shows stable performance for the length of domain-specific and context-speciff prompts. Moreover, since DROP and OBQA require reason- ing ability (Roberts et al., 2020), they are more sensitive to the prompt length compared to other datasets.
4.5 Sensitivity Analyses
4.5.2 The Weight of Loss Lidp
In this part, we conducted comprehensive sensitiv- ity analyses on our proposed method, including the length of prompts, the weight À of the loss Lidps different clustering results D, different scales of PLMs in the prompt generator, and the number of sampled domain pairs m.
We investigated the impact of loss weighing X on the results, as shown in Table 3. We found the change of weighting has minor impact on the SQuAD2 dataset and there is an optimal weight of 0.0001 for DROP, OBQA, and BoolQ-NP datasets. Lidp takes values between [0, 1]. too large À means that the model is not focusing on generating answers as its primary goal. An extremely small would make the sspeiii prompts lose focus on unique intra-domain information.
4.5.1 The Length of Prompts
MPrompt, the length of prompts is a key fac- tor that affects model performance. Here, we in- vestigate how the length of domain-specific and

---

À
SQuAD2
DROP
OBQA
BoolQ-NP
Small
Bosc
1
72.60
38.72
55.80
76.28
0.1
72.67
39.12
55.60
77.15
0.01
72.61
40.87
$7.80
74.93
0.75
0.001
72.63
41.21
58.20
76.38
0.0001
72.61
41.67
58.60
77.25
0.00001
72.63
39.20
56.00
76.25
0.60
Table 3: Evaluation of UnifiedQA- basse model.
4.5.3 Clustering Results
We investigated the impact of different numbers of clusters on performance, as shown in Table 4. Since the gold label of clustering results is not available in the question- -anteeiinnn datasets, it is difficult to determine the optimal number of clus- ters. Our evaluation shows, the performance of the model is not sensitive to the number of clusters. KMeans always outperforms randomly assigning cluster labels, which demonstrates that introduc- ing contextual cluster information to the model improves context comprehension.
0.35
SQuAD2
DROP
OBQA
BaaIQ-NP
Figure 4: Experiment of the prompt generator on differ- UnifiedQA models. "None" indicates that domain- specific and context-speciff, prompts were trained in the same way as task-specific prompts.
10, 15). Our evaluation Table shows that our algorithm is not sensitive to the number of sampled domain pairs m. Even with smaller m per batch, it still provides sufficient sampling frequency in training, which greatly reduces the computational
clusters SQuAD2
DROP
OBQA
BooIQ-NP
costs.
71.37
37.26
56,20
75.59
Random
71.63
37.25
55,40
75.91
m
SQuAD2
DROP
OBQA
BoolQ-NP
71.62
37.28
56.80
75.25
72.52
38.99
57.40
76.47
41.67
58.20
77.25
72.70
40.71
58.60
76.93
KMeans
72.70
40.71
76.93
72.71
40.93
57.90
76.01
5
72.75
40.01
58.80
76.88
10
72.65
40.11
59.00
77.17
Table Evaluation of the number cluster on UnifiedQA-base model. "Random" refers randomly assigning cluster labels.
15
72.76
41.30
58.60
76.96
Table 5: Evaluation of different the number of sampled domain pairs per batch
4.5.4 Different Scales of Prompt Generator In general, increasing the parameter number of PLMs brings abundant semantic knowledge. There- fore, we investigated the impact of PLMs with dif- ferent scales performance, shown in Figure 4. The prompt generator delivers significant per- formance improvements. Our evaluation shows, larger-scale PLMs tend to have better results, but require more computational resources. To balance the trade-off between cost and performance, the UnifiedQA-small already delivers satisfactory per- formance gains with small computational over- head (60M parameters).
Conclusion
In this paper, we propose a novel Multi-level Prompt (MPrompt) tuning method for machine reading comprehension. Our method strengthens PLMs' utilization of input semanties through three levels of prompts: task-specific prompts, domain- specific prompts, and context-specific prompts. The task-specific prompts are input-independent and generate prompts specific to a task. The domain-specific prompts utilize the domain knowl- edge generated from the dataset while context- specific prompts are relying on the input context. Our experiments show the combination of threc level prompts improves the answer generation per- formance on different sizes of PLMs and 12 bench- mark datasets. In future work, we will extend our method to more tasks such as summarization, trans- lation, and sentiment analysis.
4.5.5 Number of sampled domain pairs We investigated the impact of sampled domain pairs on the results. The number of clusters is set to 6, which requires 15 iterations per batch. We evaluate the number of sample pair m in {1, 3, 5,

---

Limitations
Jordan Clive, Kris Cao, and Marek Rei. 2022. Control prefixes for parameter-efficien text generation. In Proceedings of the 2nd Workshop Natural Lan- guage Generation, Evaluation, and Metrics (GEM), pages 363-382.
In our method, the length of prompts is the most critical parameter that affects performance. In our experiments, observe that MPrompt is sensitive prompt length for some challenging datasets. To obtain the optimal hyperparameter combination, it is inevitable to perform grid search on the length of prompts. Our model is designed for encoder- decoder structure, so the decoder-only structure like LLaMA, GPT, or Bloom is not applicable. Our model requires access to the parameter of the model which any black box model is not applicable to our algorithm.
Jacob Devlin, Ming-Wor Wei Chang, Kenton Lec, and Kristina Toutanova, 2018. Bert: Pre- Pre-training of deep bidirectional transformers for language understand- ing. arXiv preprint OrXiv:1810.0480
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner, 2019. Drop: A reading comprehension benchmark quiring discrete reasoning over paragraphs. arXiv preprint (rXxv:9903.0061.
Arthur Gretton, Olivier Bousquet, Alexander J. Smola, and Bernhard Schölkopf. 2005. Mcasuring statistical dependence with hilbert-schmidt norms. Algorith- mic Learning Theory, 16th International Conference, ALT 2005, volume 3734, pages 63-77.
Ethics Statement
Our work is developed with the highest ethical standards in mind. Our work should not be used for any entity that may violate human rights.
Xiaodong Gu, Kang Min Yoo, and Sang- Woo Lee. 2021. Response generation with context-aware prompt learning. arXiv preprint arXiv:2111.02645
References
Razieh Baradaran, Razich Ghiasi, and Hossein Amirkhani. 2022. A survey on machine reading com- prehension systems. Natural Language Engineering, 2866):68
Gautier Izacard and Edouard Grave. 2020. Leverag- ing passage retrieval with generative models for open domain question answering. arXiv preprint arXiv:2007.01282.
Tom Brown. Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. NeurIPS, 33: 1877-1991..
Zhengbao Jiang, Jun Araki, Haibo Ding, and Graham Neubig. 2022. Understanding and improving zero- shot multi-hop reasoning in generative question swering. arXiv preprint arXiv:2110004234
Daniel Khashabi, Tushar Khot, and Ashish Sabharwal. 2020a. More bang for your buck: Natural perturba- tion for robust question answering. arXiv preprint arXiv:2004. 04849..
Guoxin Chen, Yongqing Wang. Fangda Guo, Qinglang Guo, Jiangli Shao, Huawei Shen, and Xueqi Cheng. 2023. Causality and independence enhancement for biased node classification. In Proceedings of the 32nd ACM International Conference on Informa- tion and Knowledge Management, CIKM '23, page 203-212, New York, NY. USA. Association for Com- puting Machinery.
Daniel Khashabi, Yeganeh Kordi, and Hannaneh Ha- jishirzi. 2022. Unifiedqa-v2: Stronger generalization via broader cross- cross-format training. arXiv preprint a3Xiv:220..12359.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905. 10044.04.....
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Han- naneh Hajishirzi. 2020b. Unifiedqa: Crossing format boundaries with single qa system. arXiv preprint arXiv:2005.00700
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? Try ARC, the A12 reasoning challenge. arXiv preprint arXiv: 1803.05457.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composi- tion. AAAI, volume 34, pages 8082-8090.
Peter Clark, Oren Etzioni, Tushar Khot, Ashish Sab- harwal, Oyvind Tafjord, Peter Turney, and Daniel Khashabi. 2016. Combining retrieval, statisties, and inference to answer elementary science questions. In AAAI, volume 30.
Tomášs Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Ed- ward Grefenstette. 2018. The narrativeqa reading comprehension challenge. Transactions of the Asso- ciation for Computational Linguistics, 6:317-328.

---

Guokun Lai, Qizhe Xie. Hanxiao Liu, Yiming Yang. and Eduard Hovy. 2017. Race: Large-scale reading comprehension dataset from examinations. arXiv preprint arXiv: 170.0463....
Thao Nguyen, Maithra Raghu, and Simon Kornblith. 2020. Do wide and deep networks learn the same things? uncovering how neural network represen- tations vary with width and depth. arXiv preprint arXiv:2010.1532
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021.
The power of scale for parameter-effiieet prompt tuning. arXiv preprint (rXiv:210400899)
Hariom A Pandya and Brijesh S Bhatt. 2021. Ques- tion answering survey: Directions, challenges. datasets, evaluation matrices. arXiv preprint arXiv:2112.0377..
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdeirahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. 2019, Bart: De- noising sequence- toe--tt---oote to-sequence pre-training for natural language generation, translation, and comprehension. arXiv preprint arXiv:1910. 13461.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. NewriPS, 32.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Hein- rich Küttler, Mike Lewis, Wen-tau Yih, Tim Rock- täschel, et al. 2020. detrieval-augmenter genera- tion for Snowledge-intensiv nlp tasks. NewrlPS, 33:9459 9474.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with unified text-to-text trans- former. The Journal of Machine Learning Research. 21(1):5485-555
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint wwXiv:2101.001111111
Maithra Raghu, Thomas Unterthiner, Simon Kornblith, Chiyuan Zhang, and Alexey Dosovitskiy. 2021. Do vision transformers see like convolutional neural net- works? NeurlPS, 122111112222
Chin-Yew Lin. 2004. Rouge: A package for automatic
evaluation of summaries. In Text summarization
branches out, pages 81.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2023. Pre- train, prompt, and predict: systematic survey of prompting methods natural language processing. ACM Computing Surveys, 55(9):1-35.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint 1rXiv:1806.0338
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using siamese bert- networks. In EMNLP.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du Zhilin Yang, and Jie Tang. 20214 la. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint arXiv:2 1rXiv:2110.07602.
Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. EMNLP, pages 193-203.
Xiao Liu, Yanan Zheng. Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. 2021b. GPT understands, too. arXiv preprint 2rXiv:2103.10386
Adam Roberts, Colin Raffel, and Noam Shazeer. 2020. How much knowledge can you pack into the pa- rameters of language model? arXiv preprint arXiv:2002.08910.
Ilya Loshchilov and Frank Hutter. 2017. Decou- pled weight decay regularization. arXiv preprint
arXiv:1711.05101.
Timo Schick and Hinrich Schütze. 2020. Exploit- ing cloze questions for few shot text classification and natural language inference. arXiv preprint arXiv:2001.0767
Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. 2022. XPrompt: Exploring the extreme of prompt tuning. In EMNLP, pages 11033-11047.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannanch Hajishirzi. 2016. Bidirectional attention flow for machine comprehension. arXiv preprint 16Xiv:
MacQueen. 1967. Classification and analysis of mul- tivariate observations. In 5th Berkeley Symp. Math. Statist. Probabiliry, pages 281-297 University of California Los Angeles LA USA.
Taylor Shin, Yasaman Razeghi, Robert Logan IV. Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint 2rXiv:2010.15980
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can suit of armor conduct elec- tricity? A new dataset open book question an- swering. In EMNLP, pages 2381-2391.

---

Song, Alex Smola, Arthur Gretton, Karsten M Borg- wardt, and Justin Bedo. 2007. Supervised feature selection via dependence estimation. In ICMI, pages 823-830.
Chuanqi Tam, Furu Wei, Nan Yang, Bowen Du, Weifeng Lv, and Ming Zhou. 2018. S-Net: From answer extraction to answer synthesis for machine reading comprehension. In AAAI, volume 32.
Tianyi Tang, Junyi Li, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Context-tuning: Learning contextual- ized prompts for natural language generation. arXiv preprint arXiv:2201.0885 086670.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2016. Newsqa: machine comprehension dataset. arXiv preprint 1rXiv:1611.099
Laurens der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9111).
Wenhui Wang, Nan Yang. Furu Wei, Baobao Chang, and Ming Zhou. 2017. Gated self-matching networks for reading comprehension and question answering. In ACL, pages 189-198.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pier- ric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, al. 2020. Transformers: State-of-the-pn natural language processing. In EMNLP, pages 38-45.

---

A
Appendix
A.1 Datasets: Details
We evaluated our method on 12 datasets covering wide range of QA tasks. Due to some datasets (such ARC, OpenBookQA and QASC) lacking the context, following Khashabi al. (2020b, 2022), we used the datasets that contain retrieved contexts. Due to limited test access for some datasets, such as SQuAD2, NewsQA, DROP, QASC, BoolQ, and BooIQ-NP, we used the validation set as the test set and re-randomized equal number of sam- ples from the training set as the validation set. For MCTest, used the sum of mc160 and mc500. For RACE, used RACE-middle, which consists of English reading comprehension questions de- signed for Chinese middle school students. The datasets would be available our code.
A.2
Visualization of context clustering results with Kmeans
Tisk
Comext
leughh
SQuAD2
In the paper, we cluster the contexts by Kmeans and fix the number of clusters to 3, since we do not have access to the gold standard clustering results for each dataset. To observe the results of clustering, conducte visualization using t-SNE (Van der Maaten and Hinton, 2008), as shown in Figure Most of the datasets present better clustering results when the number of clusters 3, which will provide better domain information.
NewsUA
10
20
16
250
NarQA
10
50
80
16
100
DROP
ARClcaay!
ARCichol
10
40
80
OBQA
15
QASC
BoxQ)
10
30
0.1
10
BookQ NP
10
15
10
10
10
Table 6: Hyperparameter settings for our method. "Task len" indicates token length of task-specific prompts. "bsz" indicates batch size. "max_ ans length" indicates the maximum length of generated answers during infer- ence.
A.3 Implementation details
In Table 6, we report the hyperparameters used for training our models recorded in the experimen- tal section. For model inference (answer genera- num num__eamm beams 2, min Jength to 1, and early_ sitpllssttppinggggeee True. For MLP. we the hidden layer dimension to 512 and utilize the Tanh activation function. For domain-specific prompts and context-specific prompts, we initialize each prompt token as an embedded vector extracted from the prompt generator's vocabulary, as Lester et al. (2021) done.
tion),

---

SQuAD2
NewsQA
NarQA
(d) DROP
(c) MCTest
(f) ARC (casy)
g ARC (chal.)
OBQA
QASC
(i) RACE
(k) BoolQ
(I) BooIQ-NP
Figure 5: The visualization Kmeans clustering results for context by t-SNE.


Reviews:
[{"review_id": "RxkZfTB3yp", "reviewer": "Reviewer_Jx92", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nThank you for the time and effort you spent reviewing our paper. We appreciate your valuable feedback and constructive comments which we believe have helped us enrich this work. In the following, we address each of your concerns in detail.\n\nWe appreciate your suggestion to compare our method with other parameter efficient tuning techniques, such as Lora. From the inception of this work, we have been primarily focused on investigating the enhancements brought about by Soft Prompts in a specific domain (Machine Reading Comprehension). While both Lora and Soft Prompts belong to parameter efficient tuning techniques, they have slightly different theoretical foundations. Lora is a reparametrization-based method that adds an additional learnable linear layer, while Soft Prompts aim only at modifying the input text to optimize the model. To address your concern, we added comparative experiments with Lora and AdaLora based on the UnifiedQA-base model across 12 QA datasets, as shown in Table 1:\n\nTable 1: Comparison of MPrompt with Lora and AdaLora across 12 QA datasets. '$\\uparrow$' denotes the absolute improvement relative to Lora.\n_____\n\n| base       | SQuAD2 | NewsQA  | NarQA   | DROP   | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   | BoolQ  | BoolQ-NP |\n|:----------:|:------:|:-------:|:-------:|:------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|:------:|:--------:|\n|            | F1     | ROUGE-L | ROUGE-L | F1     | ACC    | ACC       | ACC         | ACC    | ACC    | ACC    | ACC    | ACC      |\n| Lora       | 75.10  | 47.15   | 45.27   | 19.13  | 86.43  | 68.06     | 43.77       | 36.40  | 47.30  | 73.54  | 82.02  | 65.46    |\n| AdaLora    | 75.11  | 47.14   | 45.26   | 19.14  | 86.42   | 68.06     | 43.77       | 36.40  | 47.30  | 73.54  | 82.02  | 65.53    |\n| MPrompt    | 72.61  | 59.99   | 46.30   | 41.64  | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  | 82.97  | 77.25    |\n| $\\uparrow$ | -2.49% | 12.84%  | 1.03%   | 22.51% | 1.31%  | 5.18%     | 1.19%       | 21.80% | 23.65% | 2.16%  | 0.95%  | 11.80%   |\n\n\nFrom Table 1, it can be observed that Lora does not outperform on most datasets requiring a high level of context understanding. Compared to Lora, MPrompt achieved an 8.49% improvement across 12 datasets. We posit that this might be due to two factors. First, Lora is more suitable for Natural Language Understanding (NLU) tasks as compared to Natural Language Generation (NLG). Second, Lora does not seem to effectively enhance the model's ability to comprehend the context of input. The findings presented in Table 1 further affirm the effectiveness of our proposed MPrompt method, demonstrating its capacity to enhance the model's contextual understanding while only necessitating minor parameter adjustments.\n\nHowever, we appreciate your insightful suggestion regarding the integration of context-specific and domain-specific information into Lora. We believe this could potentially address the issues we have observed with Lora's performance in Table 1. However, implementing it is not straightforward due to the theoretical differences between Lora and Soft Prompts. We intend to investigate this issue in detail in our future work. Nevertheless, in this paper, our main objective was to explore ways to enhance the performance of Soft Prompts in machine reading comprehension tasks, and our proposed method (MPrompt) showed significant improvement compared to the state-of-the-art methods across 12 QA datasets.\n\n\nIn addition, we compared Lora and MPrompt from the perspective of training costs, as detailed in Table 2. Although MPrompt incurs more significant training costs than Lora, considering the performance improvement, we believe this is acceptable.\n\nTable 2: Comparison of Training Resource Costs for Different Methods.\n___\n| base          | Trainable parameters | Training time cost per epoch | Test time cost per epoch |\n|:-------------:|:--------------------:|:----------------------------:|:------------------------:|\n| fine-tuning   | 222M                 | 117s                         | 26s                      |\n| prompt-tuning | 0.03M                | 82s                          | 33s                      |\n| prefix-tuning | 29M                  | 88s                          | 34s                      |\n| XPrompt       | 0.03M                | 306s (82s+137s+87s)          | 33s                      |\n| Lora          | 0.88M                | 84s                          | 31s                      |\n| AdaLora       | 4.86M                | 131s                         | 38s                      |\n| MPrompt       | 39M                  | 92s                          | 36s                      |\n\n\nTo wrap up, we wish to express our deep gratitude, for your detailed review and insightful feedback. Your guidance is instrumental in facilitating our further research. We hope that we have addressed all of your concerns effectively and look forward to any additional comments you might have.\n\nWith warm regards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- This is a well written paper reporting strong results over multiple QA datasets using prompt-based learning specific for reading comprehension. The technique outperforms previous state-of-the-art methods specific to soft prompt tuning.", "reasons_to_reject": "- Although the paper successfully discusses and compares against prompt-based learning techniques, it completely ignores other parameter efficient tuning techniques such as LoRA, DyLoRA, and AdaLora. LoRA is a very simple technique that has been shown to outperform prefix-tuning, prompt-tuning and in some cases outperform all fine-tuning. It is not clear how the context-sensitive or domain sensitive vectors could be extended to these recent techniques such as LoRA. The LoRA is simple enough to report its result on your QA datasets.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper has investigated the incorporation of 2 more prompt vectors designed for the context and domain of the input along with the task prompts for the task of reading comprehension. The technique introduces independence constraints for the domain-specific soft prompts. They conducted experiments on multiple QA datasets to show that the incorporation of  domain and context specific soft prompts is boosting the performance of naive prompt tuning.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "- Please at least discuss recent parameter efficient tuning techniques:\n[0]: https://github.com/huggingface/peft\n[1]: https://arxiv.org/abs/2106.09685\n[2]: https://arxiv.org/abs/2303.10512\n[3]: https://arxiv.org/abs/2210.07558"}}, {"review_id": "cE6y27JmLE", "reviewer": "Reviewer_2zHj", "summary": "", "questions": "1. Are there any comparisons about training resource cost between MPrompt and these baselines?\n2. If a new MRC dataset comes, how to adapt MPrompt on it? (in zero-shot, few-shot or fully supervised scenarios)", "limitations": "", "rating": 4, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe sincerely thank you for your thorough review and helpful remarks. They have been instrumental in identifying areas where we can enhance the clarity and depth of our paper. We have now revised the paper accordingly and below, we provide detailed responses to your concerns.\n\n### For your Question1\nFirstly, for the training resource cost, we tested the training cost of each method on the UnifiedQA-base model with the BoolQ dataset as an example, as shown in Table 1.\n\nTable 1: Comparison of Training Resource Costs for Different Methods.\n_____\n| base          | Trainable parameters | Training time cost per epoch | Test time cost per epoch |\n|:-------------:|:--------------------:|:----------------------------:|:------------------------:|\n| fine-tuning   | 222M                 | 117s                         | 26s                      |\n| prompt-tuning | 0.03M                | 82s                          | 33s                      |\n| prefix-tuning | 29M                  | 88s                          | 34s                      |\n| XPrompt       | 0.03M                | 306s (82s+137s+87s)          | 33s                      |\n| MPrompt       | 39M                  | 92s                          | 36s                      |\n\n\nFrom Table 1, it can be observed that even though XPrompt (an improvement on Prompt-tuning) maintains a consistent amount of trainable parameters, it is accompanied by a significant increase in training time. \nIn contrast, our method (MPrompt) achieved an absolute improvement of 2.25% across 12 QA datasets in various paradigms, despite introducing more trainable parameters compared to Prefix-tuning. However, in terms of the order of magnitude of the parameters and training time added, we believe this trade-off to be entirely acceptable.\n\n### For your Question2\nIn response to the second issue, we conducted zero-shot and few-shot experiments across 6 distinct multiple-choice datasets. For the domain labels, we assign the samples in the new datasets to the domain where the nearest cluster center is located.\n\n**For zero-shot scenarios**, we trained optimal MPrompts on each dataset separately and directly inferred on the other datasets.  The results are shown in Tables 2 and 3.\n\nTable 2: The zero-shot performance of **MPrompt** across 6 MC datasets. 'In-domain' refers to the best results achieved through training and testing on the same dataset.\n____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 66.67     | 42.49       | 32.80  | 48.06  | 73.40  |\n| ARC(easy)                                            | 77.26  | n/a       | 41.13       | 41.00  | 44.38  | 65.74  |\n| ARC(chall.)                                          | 66.31  | 57.37     | n/a         | 50.00  | 34.34  | 60.79  |\n| OBQA                                                 | 40.12  | 47.47     | 41.13       | n/a    | 20.30  | 40.53  |\n| QASC                                                 | 70.60  | 69.61     | 35.32       | 27.40  | n/a    | 60.52  |\n| RACE                                                 | 85.71  | 67.63     | 43.09       | 34.40  | 50.54  | n/a    |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |\n\nTable 3: The zero-shot performance of **Prefix-tuning** across 6 MC datasets.\n____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 61.85     | 39.75       | 32.20  | 48.81  | 73.05  |\n| ARC(easy)                                            | 75.00  | n/a       | 40.02       | 32.60  | 44.95  | 62.28  |\n| ARC(chall.)                                          | 72.14  | 53.62     | n/a         | 48.40  | 31.53  | 59.19  |\n| OBQA                                                 | 40.45  | 49.58     | 40.10       | n/a    | 22.79  | 43.31  |\n| QASC                                                 | 69.17  | 70.50     | 36.95       | 25.60  | n/a    | 66.57  |\n| RACE                                                 | 83.60  | 67.89     | 40.80       | 36.40  | 49.46  | n/a    |\n\nAs can be observed from Table 2, MPrompt retains good generalization performance even in zero-shot scenarios, which benefits from semantic prompts at different levels. For example, the model trained on MCTest achieves results comparable to 'In-domain' training on ARC(chall.) and RACE. In comparison with Prefix-tuning (as detailed in Table 3), MPrompt outperforms Prefix-tuning by 0.59% in the zero-shot scenario.\n\n**For few-shot scenarios**, we further fine-tuned the model by randomly drawing 1%, and 5% of the training data from new datasets. The results can be found in Tables 4, 5, 6, and 7, respectively.\n\nTable 4: The performance of **MPrompt** in few-shot (1%) scenarios across 6 MC datasets. '$\\uparrow$' denotes the average improvement compared to zero-shot scenarios.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   | $\\uparrow$ |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|:----------:|\n| MCTest                                               | n/a    | 66.50     | 42.66       | 41.00  | 48.81  | 73.75  | 1.86%      |\n| ARC(easy)                                            | 80.00  | n/a       | 41.55       | 48.40  | 48.70  | 72.14  | 4.26%      |\n| ARC(chall.)                                          | 80.60  | 57.20     | n/a         | 51.60  | 44.38  | 72.01  | 7.39%      |\n| OBQA                                                 | 42.62  | 49.96     | 41.55       | n/a    | 42.12  | 72.77  | 11.89%     |\n| QASC                                                 | 76.90  | 70.71     | 37.54       | 38.20  | n/a    | 69.43  | 5.87%      |\n| RACE                                                 | 85.60  | 67.63     | 43.17       | 38.80  | 49.89  | n/a    | 0.74%      |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |            |\n\nTable 5: The performance of **Prefix-tuning** in few-shot (1%) scenarios across 6 MC datasets.\n___\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 63.51     | 42.75       | 37.20  | 48.86  | 73.40  |\n| ARC(easy)                                            | 78.24  | n/a       | 40.02       | 34.20  | 44.84  | 70.98  |\n| ARC(chall.)                                          | 77.26  | 54.00     | n/a         | 48.60  | 43.20  | 71.05  |\n| OBQA                                                 | 41.31  | 50.08     | 40.53       | n/a    | 32.07  | 72.28  |\n| QASC                                                 | 74.48  | 70.12     | 37.71       | 36.60  | n/a    | 67.17  |\n| RACE                                                 | 85.60  | 66.85     | 41.80       | 38.80  | 48.08  | n/a    |\n\n\nTable 6: The performance of **MPrompt** in few-shot (5%) scenarios across 6 MC datasets.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   | $\\uparrow$ |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|:----------:|\n| MCTest                                               | n/a    | 69.36     | 42.66       | 51.60  | 57.99  | 73.54  | 6.35%      |\n| ARC(easy)                                            | 84.40  | n/a       | 41.04       | 50.60  | 59.07  | 73.12  | 7.74%      |\n| ARC(chall.)                                          | 83.10  | 65.15     | n/a         | 53.20  | 57.45  | 72.91  | 12.60%     |\n| OBQA                                                 | 84.05  | 51.47     | 42.06       | n/a    | 56.05  | 73.40  | 23.50%     |\n| QASC                                                 | 80.12  | 73.19     | 37.71       | 49.80  | n/a    | 71.66  | 9.81%      |\n| RACE                                                 | 85.00  | 71.68     | 43.52       | 45.80  | 57.67  | n/a    | 4.46%      |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |            |\n\nTable 7: The performance of **Prefix-tuning** in few-shot (5%) scenarios across 6 MC datasets.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 67.79     | 42.34       | 42.80  | 50.86  | 73.06  |\n| ARC(easy)                                            | 84.76  | n/a       | 40.36       | 46.40  | 48.70  | 72.26  |\n| ARC(chall.)                                          | 82.74  | 59.26     | n/a         | 49.20  | 45.68  | 73.15  |\n| OBQA                                                 | 76.67  | 51.56     | 40.78       | n/a    | 44.17  | 72.98  |\n| QASC                                                 | 81.90  | 73.91     | 38.91       | 49.20  | n/a    | 71.66  |\n| RACE                                                 | 85.60  | 70.54     | 42.62       | 44.20  | 52.59  | n/a    |\n\n\nAs can be observed from Tables 4 and 6, a small amount of data from the new dataset brings substantial benefits. For instance, the MPrompt trained on OBQA achieved an average improvement of 11.89% after receiving 1% of the training set.\nAdditionally, compared to the zero-shot scenario, there are improvements of 5.34% and 10.74% for 1% few-shot and 5% few-shot respectively. Most notably, in the scenario of 1% few-shot scenarios, we are already able to find results that are very close to the 'in-domain' training results. Furthermore, MPrompt outperformed Prefix-tuning by 2.15% and 2.72% in the few-shot scenarios with 1% and 5% data samples respectively. This further demonstrates the generalization capabilities of MPrompt.\n\n### For other concerns about more cases and missing reference papers\nWe will add more cases and visualization results in our paper and also include the missing reference papers.\n\nHere is a case:\n\nquestion: According to cell classification, prokaryotic cells are separated from eukaryotic cells. Which feature is often used to distinguish prokaryotic cells from eukaryotic cells? \\n (A) life processes (B) size differences (C) plasma membranes (D) energy molecules\n\nContext: The number and diversity of membranes in eukaryotic cells is a major feature that distinguishes them from prokaryotic cells. A plasma membrane surrounds both eukaryotic and prokaryotic cells. Prokaryotic cells are quite different from eukaryotes. Cell division in eukaryotes differs from that in prokaryotes. Describe two ways in which prokaryotic cells are different from eukaryotic cells. Eukaryotic cells a. differ from prokaryotic cells in that eukaryotes have a backbone. ____ DNA molecules that are used to carry foreign DNA into both prokaryotic and eukaryotic cells. organization of eukaryotic and prokaryotic cells and basic cell processes. Prokaryotic and eukaryotic cells have a cell or plasma membrane. How are prokaryotic cells different from eukaryotic cells?\n\nPrefix-tuning answer: plasma membranes (False answer)\n\nMPrompt answer: size differences (True answer)\n\nIn this case, the answer does not explicitly exist in the context, which requires a greater understanding of the context from the model. However, our method, by enhancing the understanding of the context through multi-level prompts, generates the correct answer.\n\n\nIn conclusion, we sincerely appreciate your thoughtful comments and advice. We believe that through addressing your concerns, our paper has been significantly improved. We look forward to any further suggestions you may have that can enhance our work.\n\nBest regards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The motivation of using multi-level information to generate prompt representation is intuitive.\n2. The experimental result is solid, with necessary fine-tuning and prompt tuning baselines provided and compared. 12 different MRC tasks in various paradigms are considered.\n3. The analysis about prompt tuning hyper-params for MPrompt is sufficient.", "reasons_to_reject": "1. More case studies are recommended to be provided, better along with the visualization result.", "questions_to_authors": "1. Are there any comparisons about training resource cost between MPrompt and these baselines?\n2. If a new MRC dataset comes, how to adapt MPrompt on it? (in zero-shot, few-shot or fully supervised scenarios)", "paper_topic_and_main_contributions": "The paper discusses about using multi-level prompt tuning to improve the fine-tuning results of PLMs on various MRC tasks. In detail, a soft prompt generator is used to leverage the task, domain and context information into prompt representations. In experimental results, the proposed MPrompt approach surpasses other fine-tuning and prompt tuning baselines. Comprehensive analysis about the detailed settings of prompt tuning are also provided.", "excitement": 4, "reproducibility": 5, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "1. IDPG: An instance-dependent prompt generation method\n2. Instance-wise prompt tuning for pretrained language models", "Typos_Grammar_Style_and_Presentation_Improvements": "Not found."}, "extra_text": {}}, {"review_id": "yC2E6anfbT", "reviewer": "Reviewer_Pw7P", "summary": "", "questions": "QuestionA: line235: I wonder how does the trained model generalize to\nnew domains that are not in the training data? How should we decide\nwhich group of domain prompts should we use for the new domains?\n\nQuestionB: line450: MPrompt outperforms the baselines. But I wonder\nhow efficient is MPrompt, when comparing with the baselines?  Does\nMPrompt use more trainable parameters? In general, I think a detailed\ncomparison on the efficiency of different methods would be very\nhelpful.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Dear Reviewer,\n\nWe would like to express our gratitude for your insightful comments and suggestions. Your feedback has allowed us to improve the quality of our study significantly. Please find our responses to your queries and concerns below.\n\n## For your concerns on the generalization ability of MPrompt (as well as QuestionA).\nGeneralization capability primarily refers to the performance when encountering new datasets, particularly in zero-shot and few-shot scenarios, as MPrompt can be retrained under fully supervised scenarios.\nThus, we evaluate the performance of our method (MPrompt) in zero-shot and few-shot scenarios across 6 MC datasets and compare it with Prefix-tuning, as shown in Table 1~6.\n\n### In the zero-shot scenarios.\nTable 1: The zero-shot performance of **MPrompt** across 6 MC datasets. 'In-domain' refers to the best results achieved through training and testing on the same dataset.\n____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 66.67     | 42.49       | 32.80  | 48.06  | 73.40  |\n| ARC(easy)                                            | 77.26  | n/a       | 41.13       | 41.00  | 44.38  | 65.74  |\n| ARC(chall.)                                          | 66.31  | 57.37     | n/a         | 50.00  | 34.34  | 60.79  |\n| OBQA                                                 | 40.12  | 47.47     | 41.13       | n/a    | 20.30  | 40.53  |\n| QASC                                                 | 70.60  | 69.61     | 35.32       | 27.40  | n/a    | 60.52  |\n| RACE                                                 | 85.71  | 67.63     | 43.09       | 34.40  | 50.54  | n/a    |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |\n\nTable 2: The zero-shot performance of **Prefix-tuning** across 6 MC datasets.\n____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 61.85     | 39.75       | 32.20  | 48.81  | 73.05  |\n| ARC(easy)                                            | 75.00  | n/a       | 40.02       | 32.60  | 44.95  | 62.28  |\n| ARC(chall.)                                          | 72.14  | 53.62     | n/a         | 48.40  | 31.53  | 59.19  |\n| OBQA                                                 | 40.45  | 49.58     | 40.10       | n/a    | 22.79  | 43.31  |\n| QASC                                                 | 69.17  | 70.50     | 36.95       | 25.60  | n/a    | 66.57  |\n| RACE                                                 | 83.60  | 67.89     | 40.80       | 36.40  | 49.46  | n/a    |\n\nAs evidenced in Tables 1 and 2, MPrompt delivers comparable performance to Prefix-tuning in zero-shot scenarios, even achieving a 0.59% improvement over Prefix-tuning. \nWe contend that the prompt generation process for domain-specific prompts, which incorporates contextual knowledge encoded from a small PLM, significantly reduces variance. This can also be supported by the ablation study in our paper (please refer to Figure 2 in our paper, the prompt generator has a key role to play in performance improvement).\n\n### In the few-shot scenarios.\nTable 3: The performance of **MPrompt** in few-shot (1%) scenarios across 6 MC datasets. '$\\uparrow$' denotes the average improvement compared to zero-shot scenarios.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   | $\\uparrow$ |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|:----------:|\n| MCTest                                               | n/a    | 66.50     | 42.66       | 41.00  | 48.81  | 73.75  | 1.86%      |\n| ARC(easy)                                            | 80.00  | n/a       | 41.55       | 48.40  | 48.70  | 72.14  | 4.26%      |\n| ARC(chall.)                                          | 80.60  | 57.20     | n/a         | 51.60  | 44.38  | 72.01  | 7.39%      |\n| OBQA                                                 | 42.62  | 49.96     | 41.55       | n/a    | 42.12  | 72.77  | 11.89%     |\n| QASC                                                 | 76.90  | 70.71     | 37.54       | 38.20  | n/a    | 69.43  | 5.87%      |\n| RACE                                                 | 85.60  | 67.63     | 43.17       | 38.80  | 49.89  | n/a    | 0.74%      |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |            |\n\nTable 4: The performance of **Prefix-tuning** in few-shot (1%) scenarios across 6 MC datasets.\n___\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 63.51     | 42.75       | 37.20  | 48.86  | 73.40  |\n| ARC(easy)                                            | 78.24  | n/a       | 40.02       | 34.20  | 44.84  | 70.98  |\n| ARC(chall.)                                          | 77.26  | 54.00     | n/a         | 48.60  | 43.20  | 71.05  |\n| OBQA                                                 | 41.31  | 50.08     | 40.53       | n/a    | 32.07  | 72.28  |\n| QASC                                                 | 74.48  | 70.12     | 37.71       | 36.60  | n/a    | 67.17  |\n| RACE                                                 | 85.60  | 66.85     | 41.80       | 38.80  | 48.08  | n/a    |\n\n\nTable 5: The performance of **MPrompt** in few-shot (5%) scenarios across 6 MC datasets.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   | $\\uparrow$ |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|:----------:|\n| MCTest                                               | n/a    | 69.36     | 42.66       | 51.60  | 57.99  | 73.54  | 6.35%      |\n| ARC(easy)                                            | 84.40  | n/a       | 41.04       | 50.60  | 59.07  | 73.12  | 7.74%      |\n| ARC(chall.)                                          | 83.10  | 65.15     | n/a         | 53.20  | 57.45  | 72.91  | 12.60%     |\n| OBQA                                                 | 84.05  | 51.47     | 42.06       | n/a    | 56.05  | 73.40  | 23.50%     |\n| QASC                                                 | 80.12  | 73.19     | 37.71       | 49.80  | n/a    | 71.66  | 9.81%      |\n| RACE                                                 | 85.00  | 71.68     | 43.52       | 45.80  | 57.67  | n/a    | 4.46%      |\n| In-domain                                            | 87.74  | 73.23     | 44.97       | 58.20  | 70.95  | 75.70  |            |\n\nTable 6: The performance of **Prefix-tuning** in few-shot (5%) scenarios across 6 MC datasets.\n_____\n| Trained on $\\downarrow$ - Evaluated on $\\rightarrow$ | MCTest | ARC(easy) | ARC(chall.) | OBQA   | QASC   | RACE   |\n|:----------------------------------------------------:|:------:|:---------:|:-----------:|:------:|:------:|:------:|\n| MCTest                                               | n/a    | 67.79     | 42.34       | 42.80  | 50.86  | 73.06  |\n| ARC(easy)                                            | 84.76  | n/a       | 40.36       | 46.40  | 48.70  | 72.26  |\n| ARC(chall.)                                          | 82.74  | 59.26     | n/a         | 49.20  | 45.68  | 73.15  |\n| OBQA                                                 | 76.67  | 51.56     | 40.78       | n/a    | 44.17  | 72.98  |\n| QASC                                                 | 81.90  | 73.91     | 38.91       | 49.20  | n/a    | 71.66  |\n| RACE                                                 | 85.60  | 70.54     | 42.62       | 44.20  | 52.59  | n/a    |\n\nIn few-shot scenarios, MPrompt's advantage over Prefix-tuning is more pronounced. Specifically, when dealing with 1% and 5% data samples, MPrompt showed improvements of 2.15% and 2.72%, respectively, compared to Prefix-tuning.\nTraining with a few samples allows for the adjustment of multi-level prompts, especially domain-specific prompts, to better adapt to the distribution of new datasets and consequently enhance performance.\n\n## For your concern about the efficiency of MPrompt (as well as QuestionB).\n\n\nTable 7: Comparison of Training Resource Costs for Different Methods.\n_____\n| base          | Trainable parameters | Training time cost per epoch | Test time cost per epoch |\n|:-------------:|:--------------------:|:----------------------------:|:------------------------:|\n| fine-tuning   | 222M                 | 117s                         | 26s                      |\n| prompt-tuning | 0.03M                | 82s                          | 33s                      |\n| prefix-tuning | 29M                  | 88s                          | 34s                      |\n| XPrompt       | 0.03M                | 306s (82s+137s+87s)          | 33s                      |\n| MPrompt       | 39M                  | 92s                          | 36s                      |\n\nWe have supplemented our study with a comparison of the training costs between MPrompt and the baseline, as illustrated in Table 7.\nThough XPrompt (an improvement over Prompt-tuning) maintains the same number of trainable parameters as Prompt-tuning, it increases the training time manifold, owing to the three-phase training process inherent in XPrompt.\nWhile our method (MPrompt) increases both the trainable parameters and training time relative to Prefix-tuning, it yields a 2.25% improvement on 12 datasets of machine reading comprehension. We believe that an additional cost of this magnitude is acceptable.\n\n## For your concerns about instance-level Prompt and task-specific Prompt.\nThank you for your valuable comments. Our algorithm utilizes three levels of information which are named task-specific, domain-specific, and context-specific. Each level of information carries different roles in our algorithm and contributes to the overall performance differently.\nOur Figure 2 and Table 8 below provides some supplementary information to illustrate the impact of having different level of prompts in the system.\n\nTable 8: Comparison of the effects of different level prompts (A supplementary to the ablation study in the paper)\n_____\n|       | SQuAD2 | DROP   | OBQA   | BoolQ-NP |\n|:-----:|:------:|:------:|:------:|:--------:|\n| t     | 71.45  | 35.72  | 55.00  | 76.31    |\n| d     | 71.22  | 37.85  | 53.30  | 76.62    |\n| c     | 71.67  | 37.23  | 55.00  | 76.12    |\n| d+c   | 72.06  | 40.79  | 56.60  | 76.23    |\n| t+d+c | 72.61  | 41.64  | 58.20  | 77.25    |\n\nAs gleaned from Table 8, we notice that the standalone use of task-specific, domain-specific, and context-specific prompts does not lead to sustained growth. This is partially attributable to the idiosyncrasies of the dataset.\nTask-specific, domain-specific, and context-specific prompts are essentially three fine-grained prompts that focus on different aspects of the data. We are motivated by the fact that there is no guarantee that a particular fine-grained prompt will be superior, and therefore provide information at different levels of granularity to the PLMs.\nTherefore, how to construct multi-level prompt and how to constrain the learning of prompt is what we want to focus on in our work.\n\n\nThank you once again, for your constructive criticism and valuable insights. Your inputs have greatly helped us refine our paper, and we've done our best to incorporate your suggestions in this revised version. We eagerly await your further recommendations.\n\nKind regards.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The proposed method MPrompt is intuitive and shows improvements over\n  several MRC/QA datasets;\n\n- The paper is well-structured and easy to follow.", "reasons_to_reject": "I have three major concerns:\n\n- I have some concerns on the *generalization ability* of MPrompt.\n  Firstly, MPrompt relies on clustering of the MRC dataset to determine\n  the domains. So I think it is difficult for MPrompt to solve new (or\n  OOD) examples. We can allocate the new example to the closest\n  cluster, but the performance is unknown and may have large\n  variances. Second, MPrompt is specifically tailored for the reading\n  comprehension task, and it seems not easy to be adapted to other\n  scenarios where efficient NLP methods are needed.\n\n- *Comparison with baselines.* MPrompt outperforms prefix/prompt tuning\n  and XPrompt, however, the extra cost is unknown. For example, I\n  think MPrompt introduces more trainable parameters than method like\n  prompt tuning; an extra LM is also leveraged to generate the\n  prompts. I think it is beneficial to have some detailed comparisons\n  on the cost or efficiency of these methods, which is crucial\n  for parameter-efficient training.\n\n- As described in this paper, generating instance-level prompts has\n  been shown to be more effective than using task-level prompts alone.\n  So MPrompt seems a direct adaptation of instance-level prompts to the machine reading\n  comprehension task, so I feel the novelty is limited.", "questions_to_authors": "QuestionA: line235: I wonder how does the trained model generalize to\nnew domains that are not in the training data? How should we decide\nwhich group of domain prompts should we use for the new domains?\n\nQuestionB: line450: MPrompt outperforms the baselines. But I wonder\nhow efficient is MPrompt, when comparing with the baselines?  Does\nMPrompt use more trainable parameters? In general, I think a detailed\ncomparison on the efficiency of different methods would be very\nhelpful.", "paper_topic_and_main_contributions": "This paper introduces Multi-level Prompt Tuning (MPrompt), for\nparameter efficient learning of the reading comprehension (MRC)\ntask. In addition to the task prompt vectors used in prefix/prompt\ntuning, MPrompt proposes to use extra prompt vectors that are dynamically\ncreated (generated by a small LM) for different domains and contexts\nin the reading comprehension task.\n\nTo decide the domains, an MRC data will be firstly clustered with\nKmeans to decide the number of domains; for each of the identified\ndomain, prompt vectors are generated from an LM. Furthermore, for each\nof the contexts in MRC, extra prompt vectors are also generated from\nan LM. Authors argue that this multi-level prompt tuning better learns\nthe MRC task, and show performance improvements on MRC datasets over\nprevious methods including prefix/prompt tuning and XPrompt.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces Multi-level Prompt Tuning (MPrompt), a method for enhancing the fine-tuning of pre-trained language models (PLMs) for reading comprehension tasks. MPrompt leverages dynamically generated prompts, including domain-specific and context-specific prompts, to improve PLM performance. It demonstrates superior results compared to baseline methods in various MRC datasets and investigates zero-shot and few-shot adaptation scenarios.

Reviewers raised concerns about MPrompt's generalization ability, efficiency, and novelty. They questioned how MPrompt handles new domains, its efficiency compared to baselines, and whether it offers a genuinely novel contribution.  The authors conducted new experiments to showcase MPrompt's generalization ability in zero-shot and few-shot scenarios, compared its efficiency with baselines, and provided evidence supporting the multi-level prompt approach as a valuable contribution. They also acknowledged and addressed the need for further case studies and visualization in the paper, indicating a willingness to improve their work based on reviewer feedback.

**Note:** The paper appears to be closely related to the works https://arxiv.org/pdf/2304.08467.pdf and https://arxiv.org/pdf/2212.10315.pdf, which focus on compressing prompts into soft-prompts. While the papers share similarities in utilizing soft prompts, it would be beneficial for the authors to reference these works and emphasize the distinctions in their approach, particularly in the context of reading comprehension tasks and the utilization of multi-level prompts.
Decision: Accept-Findings