APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models
Authors: Qifan Wang
Yuning Mao
Jingang Wang
Hanchao Yu
Shaoliang Nie
Sinong Wang
Fuli Feng
Lifu Huang
Xiaojun Quan
Zenglin Xu
Dongfang Liu
Venue: EMNLP 2023 Main (2023)
Keywords: Prompt Tuning
Parameter Efficient Learning
Attention Prompt

Abstract: With the continuous growth of large language models, the process of fine-tuning these models for new tasks has become increasingly parameter-intensive. Prompt tuning, a method that involves tuning a small set of soft prompts, has emerged as an effective and efficient approach for adapting large pre-trained language models. However, most existing prompt tuning approaches only introduce prompts at the input layer, limiting their performance and leaving large rooms for improvement. In this work, we propose a novel Attention Prompt tuning method, namely APrompt, for efficient adaptation of pre-trained language models. We first demonstrate that existing prompt tuning can be considered as a special case of attention prompt tuning. We then formally introduce APrompt, which incorporates query, key, and value prompts into the attention layer to guide the attention computation during fine-tuning. Experimental results on the SuperGLUE benchmark consistently demonstrate that our proposed approach outperforms state-of-the-art baselines and full fine-tuning method with pre-trained models at different scales. In addition, a comprehensive set of ablation studies validate the effectiveness of the prompt design, as well as the efficiency of our approach.

APrompt: Attention Prompt Tuning for Efficient Adaptation of Pre-trained Language Models
Hanchao Yu', Shaoliang Nie', Sinong Wang', Fuli Feng', Lifu Huang", Xiaojun Quan', Zenglin Xu and Dongfang Liu?
Qifan Wang', Yuning Mao', Jingang Wang";
MMeta AI MMeituan Lab UUnivesitty of Science and Technology of China Viirinna Tech SSun Yat-sen University 'Peng Chen Lab 'Rochester Institute of Technology wqfcrêfb. com, yuningmēfb. com
Abstract
With the continuous growth large language models, the process of fine-tuning these mod- els for new tasks has become increasingly parameter-intent intensive. Prompt tuning. a method that involves tuning a small set of soft prompts, has emerged as an effective and efficient ap- proach for adapting large pre-trained language models. However, most existing prompt tuning approaches only introduce prompts at the input layer, limiting their performance and leaving large rooms for improvement. In this work, we propose a novel Attention Prompt tuning method, namely APROMPT, for efficient adap- tation of pre-trained language models. We first demonstrate that existing prompt tuning can be considered as special case of attention prompt tuning. We then formally introduce APROMPT, which incorporates query, key, and value prompts into the attention layer to guide the attention computation during fine-tuning. Experimental results on the SuperGLUE bench- mark consistently demonstrate that our pro- posed approach outperforms state-of-thh-art -frt baselines and full fine-tuning method with pre- trained models at different scales. In addition, a comprehensive of ablation studies validate the effectiveness of the prompt design, as well as the efficiency of our approach.
$
0000
MOD0
0000
&
L
0000
1100
10 0000
Tuzing
P-Tunngg2
92-
87-
02-
77-
Prompt Tuntop
72-
P'Juning 22
X
Our
67-
TS-Rran (200M)
T5xt (SB)
Figure Muusstrtiion of APROMPT (ours) and previous works, including Fine-Tuning (Aribandi al., 2022), Prompt Tuning (Lester et al., 2021), and P-Tuning V2 (Liu et al., 2022) methods. Our method consistently improves over prompt tuning methods and also outper- forms fine-tuning method across tasks and model scales.
and PaLM 540B (Chowdhery et al., 2022) are be- coming increasingly prevalent. Despite their com- pelling performance, fine-tuning large-scale PLMs is highly parameter-inefficientt due to storing gradi- ents and updating for all model parameters. This inefficiency also arises from the requirement store and deploy a complete copy of the fine-tuned model for each individual task, resulting in compu- tational expenses that hinder fast model adaptation.
1 Introduction
To tackle the challenges associated with full fine-tumin fine-tuning, researchers have proposed parameter- efficient tuning approaches (Guo et al., 2021; He et al., 2022a; Hu et al., 2022) involving techniques such as partial tuning and extra module. Partial tuning methods (Yosinski et al., 2014) focus on fine-tuning the-tuning only portion of the backbone, such as the classifier head or the last few layers, while keeping the remaining layers frozen. On the other hand, extra module methods introduce learnable bias terms (Cai et al., 2020) or additional adapters (Houlsby et al., 2019) to the network for adapta- tion. These strategies operate within the pretrain-
Pre-trained Language Models (PLMs) have gained significant popularity in various natural language understanding tasks (Devlin et al., 2019; Lewis al., 2020; Raffel et al., 2020), exhibiting re- markable success under the tretrain-tteen-fnetuue paradigm. It has been consistently demonstrated in recent studies (Aribandi al., 2022; Zhang et al., 2022) that scaling up the size of these models leads to improved performance. Consequently, large language models such as LLaMA 65B (Touvron et al., 2023), GPT-3 175B (Brown et al., 2020),
"Corresponding authors.

---

then-finetnne paradigm and effectively reduce the number of learnable parameters. However, in gen- eral, these approaches tend to underperform the full fine-tuning models with large performance gaps. Recently prompt tuning approaches (Lester et al., 2021; Li and Liang, 2021; He et al., 2022b; Yang et al., 2023) have been proposed, which uti- lize set of learnable soft prompts prepended to the imput. These soft prompts consist of continuous embeddings that are updated during the tuning pro- cess while keeping the backbone frozen. Prompt tuning offers a conceptually simpler and more flex- ible method compared to other surameter-efficient tuning approaches. It has been demonstrated to per- form closer to full model tuning, especially with large-scale PLMs (Ma et al., 2022; Razdaibiedina al., 2023a). Prompt tuning provides promising parameter-efficiemt alternative to fine-tuning, as the soft prompts used in this approach are typically orders magnitude smaller, constituting less than 0.5% of the total model parameters. However, most existing prompt tuning approaches have mainly fo- cused on modifying the input layers and have not thoroughly explored the core architecture of the Transformer's self-attention mechanism. There- fore, they often underperform full fine-tuning and leave substantial room for improvement.
demonstrating that input prompts can be viewed as specialized form of attention prompts. This insight serves as valuable knowledge, enhancing our understanding of both the existing prompt tuning techniques and the novelty of our proposed approach.
We design novel attention prompt tuning by incorporating query, key, and value prompts into the self-attention computation along with the input prompts. By doing so, these atten- tion prompts play a crucial role in effectively guiding the model's fine-tuning process, en- abling faster and more accurate adjustments during the adaptation process.
We conduct comprehensive experiments on various tasks in the SuperGLUE benchmark, demonstrating the effectiveness of the pro- posed approach over several state-of-the-art prompt tuning and full fine-tum fine-tuning methods.
Related Work
Pre-trained Language Models Pre-trained Lan- guage Models (PLMs) (Yang et al., 2019; Ainslic et al.. 2020; Zaheer et al., 2020; Zhao et al., 2023) have demonstrated huge success across various nat- ural language processing tasks. Pioneering works such as BERT (Devlin al., 2019) and RoBERTa (Liu al., 2019) learn contextual representations with masked language model (MLM) and next sentence prediction tasks. Recently, a range of large-scale PLMs, including GPT-3 (Brown et al., 2020), T5 (Raffel et al., 2020), and PaLM (Chowd- hery et al., 2022), have emerged with diverse pre- training designs. However, the exponential increase in the number of parameters poses challenges in fine-tuning these models. It becomes computation- ally expensive to store and maintain all fine-tuned parameters for each tasks.
In this paper. we present a novel Attention Prompt tuning approach, APROMPT, for efficient and effective large language model adaptation. We begin by reexamining the prompt tuning approach and establish, both theoretically and empirically, that its input prompts can be considered as special- ized key-value prompts. We then formally intro- duce our APROMPT. Unlike previous prompt tun- ing methods, ARRMMP incorporates three sets of learnable prompts: query, key, and value prompts. These prompts are prepended to the respective ma- trices in the self-attention block within the Trans- former layer. During model tuning, these attention prompts are learned alongside the original input prompts, resulting in more effective guidance of attention computation for new tasks. Evaluation on the SuperGLUE benchmark showcases the superior performance of APROMPT compared to of the-art methods. The ablation study results provide strong evidence for the effectiveness and efficiency of the proposed attention prompts. We summarize the main contributions as follows:
Parameter-Efficient Tuning As the size of PLMs becomes larger, it increasingly unafford- able to update and save full model copies for each downstream application. Parameter-efficient tuning methods (Pfeiffer et al., 2020, 2021) arise in the era of LLM. Depending on whether new parameters are introduced, we divide parameter-efficien tuning methods into categories of partial tuning and extra module. Partial tuning methods simply update parts of the model such as the bias term (Zaken et al., 2022) or the last layers (Lee et al., 2019). Extra
We establish a connection between existing prompt tuning methods and our approach,

---

T5-Large
WSC
CB
Boolq
L
Prompt Tuning
78.31
88.41
83.65
Fixed Key-value Prompts
78.31
88.41
83.65
Key-value Prompts
78.88
88,93
84.06
Prompt Tuning Key-value Prompts
79.12
89.17
84,25
Figure 2: Input prompts P' are specialized key and value prompts with Pa- HHPP and P. = H.P.
Table 1: Performance (Accuracy) of Prompt Tuning and different key-value prompts variants on WSC, CB and Boolq tasks from SuperGLUB with T5-Large model.
module methods introduce task-specific parameters various locations of the model including Side- Tuning (Zhang et al., 2020) and Adaptors (Hu et al., 2022: Houlsby et al., 2019). There has also been effort unifying different parameter-effiient tuning methods for increased robustness and performance (Mao et al., 2022; He et al., 2022a).
ing results compared to other parameter-effiiiet tuning methods.
3.2 Connection with Key- Value Prompts
In this section, we investigate deeper on how the prompt tuning works and show that traditional input prompts are equavalent to constrained key- value prompts. Recall that in prompt tuning, the prompt tokens are first prepended to the input to- kens. The new sequence Xnew [P: X] is then fed into the Transformer encoder layer compute the contextual embeddings of the text tokens for the next layer. The self-attention defined as:
Prompt Tuning Prompt tuning (Han et al., 2023; Yan et al., 2023) inserts learnable parameters to the model as virtual tokens where the insertion can happen at the model input (Lester et al., 2021) or each layer (Li and Liang, 2021). Later variants im- prove prompt tuning methods for NLU (Liu al., 2022) and NLG (An et al., 2022) tasks respectively through series of optimization and adaptations. More recent studies add residual connections to improve the performance and stability of prompt tuning (Razdaibiedina et al., 2023b) and extend prompt tuning to the continual learning setting (Razdaibiedina al., 2023a). However, most of these methods only simply add prompts to input layers, which greatly limited their performances. Most recently, mixture prompt tuning has been proposed in MixPrompt (Yang et al., 2023) and E2VPT (Han et al., 2023), which combines the in- put prompts with key-value prompts. These meth- ods can be treated as special cases of our attention prompt tuning approach.
PP, = softmar O Knew Vnew Vd
Q = H Knew = Xnews Vnew H" X
where Q. Knew and Vnew are the new query, key and value embedding matrices, with H H² and H as the pre-trained model parameters that are frozen. is worth noting that for the query Q, there is no need to compute new one since only the original text tokens X are updated and used in the next layer. Then we have:
Knew H X PH [P..
3 Prompt Tuning Revisit
Similarly, we have View = [Pes VJ. Therefore, we can conclude that adding the prompt tokens P dur- ing prompt tuning is equivalent of prepending kcy prompts Ps and value prompts P, to the original key and value matrices respectively, as shown in Figure 2. Note that these key-value prompts constrained or coupled by the input prompts P.
3.1 Preliminary
Prompt tuning methods (Lester al., 2021; Liu al., 2022) are proposed as a group parame- ter efficient models for fast adaptation large- scale PLMS to downstream tasks. They intro- duce a set of task-specific prompts or prompt to- kens P E Rdxm and prepend them to the in- put sequence X E RdxR to form new input Xnew = [] Rdx(m+n), shown in left Fig- ure 2. Here is the length of prompt tokens, the input sequence length, and d is the dimension of the embedding vector. These prompts are learned on the downstream task during fine-tuning with the backbone frozen. Prompt tuning achieves promis-
3.3 Empirical Study
To further validate the findings, we conduct an ex- periment by comparing four methods, Prompt Tun- ing (Lester al., 2021), Fixed Key-value Prompt, Key-value Prompt, and Prompt Tuning + Key-value Prompt, on three tasks from SuperGL.UF with T5- Large backone. Fixed Key-value Prompt directly adds fixed key and value prompts computed from

---

Tuned *
Frozen
Head
MLP
MatMul
Transformer Encoder Layer L
Lv
Sortmax
LayerNorm
Scale
Transformer Encoder L Layer Le
M Malmul
MSA
Transforner Encoder Layer
LayerNom
(a) A ttention Prompt Tuning
(b) Transformer Encoder Layer
(c) Self-atiantion Layer
Figure 3: Overview our APROMPT model. introduce three sets trainable attention prompts, namely query, key, and value prompts. during the fine- tuning of the model, in addition to the input prompts. These prompts are incorporated into the query, key, and value matrices within the multi-head attention computation.
the optimal prompts P* learned in Prompt Tun- ing, i.e., Pk H* p* and P. = H without any tuning. Key-value Prompt learns the optimal key and value prompts during fine-tuning. Prompt Tuning Key-value Prompt learns both the input prompts and key-value prompts during fine-tuning.
computation. The overall model architecture of A PROMPT is depicted in Figure 3. Fundamen- tally, model enables the training of only threc components while keeping all other parameters frozen. These components are as follows: (1) Input prompts, denoted as P, which are inserted at the be- ginning of the input sequence each Transformer encoder layer. (2) Attention prompts, represented by Pq. PR. and P.. are incorporated into the query, key, and value matrices within the self-attention module, respectively. These prompts allow the model to learn attention patterns from the fine- tuning data. (3) A task-specific head, which is lightweight module dedicated to the specific task and can be trained efficiently.
The comparative results of these methods are presented in Table 1. Firstly, it is clear that the Fixed Key-value Prompts method achieves identi- cal results to Prompt Tuning, which aligns with our expectations and validates the equivalence between inpul prompis and constrained key-value prompts. Secondly, when allowing the key-value prompts to be Icarned during fine-tuning, we observe improved performance compared to fixed key-value prompts. The reason is that the fixed key-value prompts can be seen special case within the search space of unconstrained key-value prompts. Lastly, com- bining both input prompts and key-value prompts during fine-tuning leads to the highest performance. Our hypothesis that while key-value prompts theoretically have the potential to encompass the information contained in input prompts, they exist in different embedding spaces. In practice, input prompts provide additional value during the fine-tuning process. Further analysis and discus- sion are provided in the experimental section.
4.1 Input Prompts
In similar vein to traditional prompt tuning meth- ods (Lester al., 2021; Li and Liang, 2021), input prompts consist of set of d-dimensional embed- ding vectors, where the dimensionality matches that of the text tokens. These prompts are in- serted at the beginning of the input sequence in each Transformer encoder layer and interact with all the text tokens. They facilitate the learning of lask-specific embeddings, effectively guiding the model's performance on tasks.
Formally, these input prompts are defined P; = , P??...... P.N.. where P/¹ denotes the learn- able input prompts the juh Transformer encoder layer, and N is the total number of layers. Then the encoder layers are represented as:
4 Attention Prompt Tuning
Drawing inspiration from the insights presented in Section 3, we propose an attention prompt tun- ing approach by introducing the attention prompts in the Transformer layers to facilitate attention
- (p] E)

---

Z' Ly Z j j.
new attention computations are:
where Z' represents the contextual embeddings of the text tokens computed by the Jeh encoder layer. The different colors indicate trainable and frozen parameters, respectively. E is the text token embeddings initialized from the backbonc.
L(-) = MLP (LN (MSA (-))
MSA(.) foforrr Pieu Krae Vnew Va
where MLP and LN are the frozen multi-layer per- ceptron and layer norm, and MSA is the multi-head self-attention inside the Transformer encoder layer. In this way, the attention prompts can effectively guide the model adaptation to the new task.
4.2 Attention Prompts
While input prompts are effective in acquiring knowledge about the new task, they do not possess the capability guide the interaction of informa- tion within each encoder layer. During fine-tuning on new task with new data, the word distribu- tion may differ significantly from the examples seen during pre-training. Consequently, it becomes imperative to enhance the model's capacity for cap- turing new information from the fine-tuning data. This entails enabling better attention among the input tokens to effectively learn new patterns that emerge in the task tusk-specific context.
4.3 Task-specific Head
For each downstream task, we also fine-tune a task- specific head, which is a very small module dedi- cated to the specific task to generate the predictions.
Head (z*)
where ZN is the output contextual embedding from the top layer the encoder.
Experiments
5.1 Datasets
Q
K...
pore
Following previous works on prompt tuning (Lester et al., 2021: Liu et al., 2022), we use NLU tasks from the SuperGLUE benchmark to evaluate the performance of the language model (Raffel et al., 2020; Aribandi et al., 2022). Specifically, we use the following 8 datasets: BoolQ (Clark et al., 2019), CB (Jiang and de Marneffe, 2019), COPA (Roemmele al., 201 1). MRC (Khashabi et al., 2018), ReC (Zhang et al., 2018), RTE (Giampie- colo et al., 2007), WiC (Pilehvar and Camacho- Collados, 2019) and WSC (Levesque et al., 2012). More details are provided in the Appendix.
Figure 4: The new query, key and value matricies. In order to address this, we introduce a novel set of attention prompts that are integrated into the attention block within each encoder layer. These at- tention prompts can be categorized into two groups: query-key prompts and key-value prompts. The query-key prompts, denoted DQK and Ph consist of small matrices (comprising few rows) that are appended to the original query and key matrices within the attention module. By incor- porating these query-key prompts, we enhance the computation of attention maps among the to- kens, thereby improving the attention mechanism. The key-value prompts, represented by PKV and PKV, which are two supplementary matrices (a few columns) inserted to the key and value matri- ces, respectively. These key-value prompts provide additional information for the input tokens to at- tend to, thereby enhancing the representation of the learned embeddings. By incorporating both key prompts and key-value prompts, we aim to enable more effective information interaction and capture new patterns during the fine-tuning process. The new query, key and value matrices are augmented with these new attention prompts as shown in Figure 4. It is worth noting that the new key matrix is appended by both the key prompts from query-key and key-value prompts. Then the
5.2 Baselines
Our model is compared with five state-of-the-art prompt tuning and fine-tuning methods.
Fine-Tuning (Aribandi et al., 2022) is the stan- dard full fine-tuning approach of T5, where all the pre- trained parameters are fine-tuned.
Prompt-Tuning (Lester et al., 2021) is the vanilla prompt tuning approach which adds the input prompts in the first input layer.
P- P-Tuning v2 (Liu et al., 2022) builds on top of Prompt-Tuning by inserting set of individual prompts each Transformer layers.
XPrompt (Ma et al., 2022) designs more efficient prompts by pruning the least important token-level and piece-level prompts.

---

Boolq
CB F1/Acc
COPA
MRC FI/EM
ReC
RTE Acc
WIC Acc
WSC
Average
Method
Para
Acc
Acc
FI/EM
Acc
Score
T5-Base (220M)
Fing Fine-Tuning Tuning" (Aribandi el al., 2022)
100%
82.30
91.30
60.00
58.25
80.55
84.50
69.30
81.70
76.10
Prompt-Tuning (I (Lester ul., 2021)
0.06%
78.12
84,42
54.37
51.14
71.35
75.27
62.29
67.36
68.
P-Tuning v2 (1 et al., 2022)
0.53%
80.81
90.23
61.28
55.64
78.13
81.98
67.56
78.32
74.35
XPrompt (Ma et al., 2022)
0.04%
79.67
86.72
56.95
53.08
74.36
78.29
64.31
73.68
70.88
ResPrompt (Razdaibicdina et al., 2023b)
0.21% %
79.25
85,33
58.64
52.91
73.19
77.14
62.36
70.82
00.95
APROMPT (Ours)
0.45%
81.83
91.86
61.54
59.07
81.18
85.76
69.50
81.49
76.84
L5-Large (770M)
Fine- Teee-Tumnng (Aribandi et al., 2022)
100%
88.30
95.35
87.00
67.25
87.85
90.60
73,50
88.50
84.47
Prompt-Tuning (Lester al., 2021)
0.03%
83.65
88.41
82.67
63.28
82.46
85.19
71.05
78.31
79.25
P-Tuning v2 (Liu et al., 2022)
0.48%
87.92
95.56
86.20
70:47
89.03
89.14
71.81
86.59
84.59
XPrompt (Ma et al., 2022)
0.02%
85.54
91.39
67.32
85,47
87.30
73.22
80.28
ResPrompt (Razdaibicdina et 20233)
0.15%
83.51
90.64
82.79
65.16
84.72
86.97
71.13
80.36
80.66
APROMPT (Ours)
0.37%
90.35
95.83
88.32
71.98
90.64
90.47
74.67
90.13
86.55
TS-XL (3B)
Fine- Tuming" (Aribandi 2022)
100%
89.60
94.20
96.00
26.15
92.05
91.70
74.30
95.20
88.65
Prompt-Tuning (Lester 2021)
0.01%
87.58
91.25
73.49
90.14
89.35
74.21
87.16
85.59
P-Tuning (Liu 2022)
0.45%
90.11
94.08
95.33
75.21
92.39
92.13
75.46
94.25
88.62
XPrompt (Ma et al., 2022)
0.01%
14
92.73
95.18
75.01
91.18
92.16
74.85
89,43
87.46
ResPrompi (Ruzdaihiediin 202.3h
0.04%
46
92.54
93.12
75.17
91.20
91 64
75.32
89.15
87.08
APROMP (Ours)
0.32%
90.72
95,48
95.83
78.68
93.75
93.36
76.43
96.17
90.05
Table 2: Performance comparison result (%) on SuperGLUE development set. 'indicates the results reported in (Aribandi et al., 2022). 'Para' is the number of trainable parameters. The best results are in bold with underline representing the second best ones. For tasks with metrics, the average score is reported. All scores are averaged over 5 runs. Results are statistically significant with respect to all baselines on each PLM (all p-value < 0.005).
ResPrompt (Razdaibiedina et al., 2023b) adds
2018) optimizer with weight decay 1e-5
residual connections to improve the performance and stability of prompt tuning.
5.4 Main Results
The main performance comparison results are pre- sented in Table 2. There are several key observa- tions from these results. First, APROMPT sistently outperforms all prompt tuning baselines across different backbone models, showcasing the effective design of its attention prompts. For in- stance, when evaluating on Boolq with T5-Large, the Acc score of APROMPT demonstrates a signifi- cant improvement of 5.62% and 2.76% compared to two strong methods, XPrompt and P-Tuning v2, respectively. This highlights the limitation of existing prompt tuning approaches that primarily focus on designing input prompt tokens, failing to capture the intricate token interactions within new data. In contrast, the attention prompts em- ployed by our approach successfully bridge this gap, resulting in enhanced performance. Second, A PPRoMPT outperforms the full fine-tuning method in most cases, while other prompt tuning baselines still exhibit certain gaps, particularly when using smaller backbone models like Ba-Base. This obser- vation highlights the effectiveness of our approach across range of natural language understanding tasks. Moreover, our model achieves these results while training only around 0.4% the parame-
5.3 Implementation Details
APROMPT is implemented with the OpenPrompt library (Ding et al., 2022), which is unified and extensible toolkit for prompt tuning research. Our model is trained on 16 NVIDIA Tesla V100 GPUs. We translate cach SuperGLUE dataset into a text- to-text format following (Raffel et al., 2020). Three scales pre- treitrained models are used: T5-Base, T5- Large and T5-XL. with 200M, 770M and 3B pa- rameters, respectively. Following previous studies (Lester al., 2021; Ma et al., 2022), we train prompts for 100 epochs with constant learning rate of 0.3 and batch size of 16. There are three hyperparameters in our model: the lengthes of in- put, query-key and key-value prompts. For our method, we set the number of input prompts to 10 (as we found our model is less sensitive to it), and linearly search the best prompt length for both query-key and key-value prompts from {1, 5, 10, 20, 50). For all prompt tuning baselines, search the best input prompt length from (5, 10, 20, 50, 100). The best checkpoints are selected via early stopping on the development set. The models are trained using the Adafactor (Shazeer and Stern,

---

Model
Boolq
RTE
WSC
Fine-Tuning
71.31
75.72
70.35
67.52
Prompt-Tuning
69.48
75.16
71.73
67.47
TS-Buma
P-Tuning v2
73.36
77.94
73.48
70.50
XPrompt
71.53
76.57
72.68
69.75
ResPrompt
70.38
76.20
70.85
68.35
Boolq
RTE
WSC
APROMPT (Ours)
75.66
78.51
75.83
72.30
irout procppt
atth pronpt
pronpt
GK Prompt
AProrpe
96
Table 3: Performance comparision results with few- shot (32 samples) setting on Boolq, CB, RTE and WSC tasks for the TS-XL model. APROMPT consistently outperforms all baselines in low resource scenarios.
TS-Larg
28
Boola
ters in the backbone, making it significantly more parameter-efficient than the full fine-tuned model. It is worth noting that although APROMPT intro- duces additional attention prompts, the length of the input prompts are largely reduced (fixed to 10) and thus resulting in even less total trainable pa- rameters compared with P-Tuming v2. Third, is worth noting that the gap between fine-tuning and other prompt tuning methods diminishes as the size of the backbone models increases. This finding aligns with previous studies (Lester et al., 2021; Ma al., 2022) and underscores the trend of convergence between fine-tuning and alternative prompt tuning approaches.
wib leput procept
wio ao- pronpt
wib promptt
we OK Prompt
AProrppe
TS-J4
Boolq
GB
RTE
WBC
Figure 5: Ablation study on the impact of different prompt combinations on four tasks of SuperGLUE.
ically, we experimented with four additional mod- els: one without input prompts, one without query- key prompts, one without key-value prompts, and one without both query-key and key-value prompts. The Ace scores obtained on four SuperGLUE tasks with different backbone models are presented in Figure 5. The results reveal that the model's perfor- mance drops when any of the trainable prompts is removed, which aligns with our expectations. Fur- thermore, we observed that the performance drop of APROMPT without input prompts is relatively small compared to the models without attention prompts. This suggests the significance of both query-key and key-value prompts in comparison input prompts, thereby validating the analysis pre- sented in section 3. Once again, it is worth noting that combining all prompts in APROMPT leads to the best performance.
6 Analysis and Discussion
Results on Low-resource Scenario We con- ducted further evaluations to assess the perfor- mance of APROMPT and other baseline models in low-resource setting. Following (Schick and Schlitze, 2021), we randomly selected 32 exam- ples for each task as the new training set, using a fixed random seed. We fine-tuned the prompt model on this limited training set and reported the results on the full dev set using the best checkpoint in Table 3. It is evident that all methods experi- ence a significant drop in performance due to the limited data available for training. Nevertheless, APROMPT consistently outperforms the baseline models on tasks such as Boolq, CB, RTE, and WSC. Additionally, we observe that most prompt tuning approaches achieve better results compared to fine- tuning, indicating that despite the challenges of overfitting when training with limited data, prompt tuning methods exhibit superior generalization ca- pabilities compared to full fine-tuning.
Iey-nallee phompllangh
Japsaitle prompe lergh
10
gamnyoo beggh
50.0
12.07
10:35
02:11
02.39
02-42
21:31
20.25
22.1
1923
11:17
0252
9009
60.9
10:00
04:71
80.00
24:1
RTE
WSC
Figure 6: Ablation study of query-key and key-value prompt lengths. We vary the number of prompts for different combinations, and evaluate (Acc) on RTE and WSC tasks with T5-XL as the backbone model.
Impact of Different Prompts impact of different prompts in our model, we conducted an ablation study by exploring vari- ous prompt combinations in APROMPT. Specif-
To investigate the
Impact of Prompt Length In APROMPT, the lengths of query-key prompts and key-value

---

Position
Boolg
CB
RTE
WSC
Boolq
CB
RTE
WSC
First-luyer
78.56
85.48
90.28
90.88
96
First 12-layers
88.35
88.64
92.42
94.32
96
Last-layer
75.82
84.37
90.75
91.15
94
Last 12-layers
82.55
88.49
92.67
95.36
Alternatvv 12-layers
90.31
90.14
92.74
95.21
02
APROMPT (All)
90.72
95.48
93.36
96.17
90
88
Table 4: Performance comparison with different prompt positions on Boolq. CB, RTE and WSC for TS-XL.
0.00
0.25
0.50
0.75
1.00
vey-v prompts rañl
Figure 7: Performance comparison of different ratios of key-value prompts when fixing the total number trainable parameters using T5-XL.
prompts are the only hyperparameters that require tuning. To further analyze the impact of differ- ent prompt lengths on model performance, we conducted an ablation study by modifying both prompt lengths across {1, 5, 10, 20, 50}. We exper- iment over all possible length combinations, and detailed discussion on how balance these two prompts will be provided in later experiments. The model performance results of all prompt length combinations on RTE and WSC are shown in Fig- ure It can be seen that there is no universal op- timal prompt length that consistently achieves the best performance across both tasks. For instance, on RTE, the highest score is obtained with 10 key- value prompts and 5 query-key prompts, while on WSC, the best performance is achieved with 20 key-value prompts and 10 query-key prompts. We hypothesize that different tasks and datasets exhibit distinct data distributions, with 'hard' tasks poten- tially requiring longer prompts to effectively cap- ture the underlying patterns and knowledge within the data. However, this comes with the trade-off of an increased number of trainable parameters. Nonetheless, we observed that model's perfor- mance remains relatively stable within certain range of prompt lengths.
Effect of Attention Prompts Balancing The query-key and key-value prompts in APROMPT contribute differently to the model performance. To further investigate their correlation and ef- fectiveness, we conduct an experiment by fix- ing the total number of trainable parameters, and adjusting the ratio of key-value prompts from {0, 0.25, 0.5, 0.75, 1}. The model performances at different ratios on four SuperGLUE tasks are il- lustrated in Figure 7. We observe slightly different patterns different tasks. For example, WSC, key-value prompts with 0.75 ratio achieves the best score, while key-value prompts with 0.5 gives the best performance on CB. Nevertheless, APROMPT with ratio 0 (no key-value prompts) or 1 (no query- key prompts) underperforms other prompt combi- nations in most cases, indicating the effectiveness of both attention prompts.
Variants of APROMPT We compare APROMPT with its two variants analyze the performance- scale trade-off. Firstly, we remove the input prompts, retaining only the attention prompts, re- sulting in a variant named APROMPT- Addition- ally, we upply the pruming technique in XPrompt (Ma et al., 2022) to eliminate the least important prompts, creating a variant called APROMPT +. From the results in Table 5, we observe that the contribution of input prompts is not particularly significant, aligning with the findings in section 3. When applying prompt trimming, we note that while the number of trainable parameters decreases, the performance behavior varies across different tasks, leaving room for further exploration in terms of finding the optimal balance between the number of prompts and model performance.
Impact of Prompt Positions This study evalu- ates the impact of prompt positions the model performance. Concretely, we train five additional models with different prompt locations (applied to both encoder and decoder), including only first layer, last layer, first 12 layers, last 12 layers and alternative 12 layers. The performance compari- son results on MAVE are reported in Table 4. It is not surprising to see that inserting prompts to all encoder and decoder layers achieves the best performance. We also observe that only putting the prompts the first (input) or last (output) layer results in large performance drops, which is consis- tent with the observations in other prompt tuning works (Liu et al., 2022; Jia et al., 2022).
7 Conclusions
This work first connects existing prompt tuning with attention prompt tuning and show that input

---

Model
Para
Boolq
CB
RTE
WSC
Jai Prakash Gupta, Kai Hui, Sebastian Ruder, and Donald Metzler. 2022. Ext5: Towards extreme multi- task scaling for transfer learning. In The Tenth In- ternational Conference on Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. Open- Review.net.
APROMPT-
0.25%
90.41
95.28
93.12
95.82
APROMPT
0.32%
90.72
95.48
93.36
96.17
4PROMPT+ +
0.21%
91.05
95.34
93.51
96.02
Table 5: Different variants of our model with T5-XL.
prompts can be considered as a special case of key- value prompts in the attention layer. Inspired by the observation, we introduce three sets of new prompts, namely query, key, and value prompts, and incorporate them into the attention layer to guide the attention computation during fine-tuning, Experimental results on SuperGI.UE demonstrate the superior performance of our model over several ofte--of-the-ar art baselines.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel HerberttVVsss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemerr, Sam McCandlish, Alec Radford, Ily Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems 33: Annual Conference on Neural Information Process- ing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual.
Limitations
There are two limitations of our APROMPT model. First, while APROMPT outperforms other prompt tuning and fine-tuning approaches, identifying the optimal combination of attention prompts automat- ically poses challenge that remains unanswered. In our experiments, we conduct grid search to empirically determine the optimal prompt length. However, in the future, intend to explore sys- tematic solution that can identify either the optimal combination or a suboptimal one. Second, our cur- rent model learns task-specific prompts for each individual task. To address this, we plan to in- vestigate a parametric network that can guide the learning of task- aask-agnottic prompts, thereby enhanc- ing the model's flexibility and adaptability across multiple tasks.
Han Cai, Chuang Gan, Ligeng Zhu, and Song Han. 2020. Tinytl: Reduce memory, not parameters for efficient on on-device learning. In Advances Neural Information Processing Systems 33: Annual Confer- ence Neural Information Processing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Yi Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du, Ben Hutchinson, Reiner Pope, James Bradbury, Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito. David Luan, Hyeontaek Lim, Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai. Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lec, Zongwei Zhou, Xuezhi Wang, Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Meier-Hellstern, Douglas Fick, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- eling with pathways. CoRR, abs/2204.02311.
References
Joshua Ainslie, Santiago Ontañónn, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula. Sumit Sanghai, Qifan Wang, and Li Yang. 2020. ETC: encoding long and structured inputs in transformers. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing, EMNLP 2020, Online, November 16-20, 2020, pages 268-284, Association for Computational Lin- guistics.
Christopher Clark, Kenton Lee, Ming- Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACI-IIIT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 2924-2933 Associa- tion for Computational Linguistics.
Shengnan An, Yifei Li. Zeqi Lin, Qian Liu, Bei Chen, Qiang Fu, Weizhu Chen, Nanning Zheng, and Jian- Guang Lou. 2022. Input-tuning: Adapting unfa- miliar inputs to frozen pretrained models. CoRR, abs/2203.03131.
Vamsi Aribandi, Yi Tay, Tal Schuster, Jinfeng Rao, Huaixiu Steven Zheng, Sanket Vaibhav Mehta, Hon- glei Zhuang, Vinh Q. Tran, Dara Bahri, Jianmo Ni,

---

Jacob Devlin, Ming-We ding-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistiex; Human Language Tech- nologies, NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4171-4188 Association for Computational Linguistics.
Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMIR.
Edward J. Hu, Yclong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang. Lu Wang, and Weizhu Chen. 2022. Lora: Low- Low-rank adaptation of large language models. In The Tenth International Conference Learning Representations, ICLR 2022, Virtual Event, April 25-29, 2022. OpenReview.net,
Ning Ding, Shengding Hu, Weilin Zhao, Yulin Chen, Zhiyuan Liu, Haitao Zheng, and Maosong Sun. 2022. Openprompt: An open-source framework for prompt- learning. In Proceedings of the 60th Annual Meet- ing of the Association for Computational Linguistics, ACL 2022 System Demonstrations, Dublin, Ireland, May 22-27, 2022, pages 105-113, Association for Computational Linguistics.
Menglin Jia, Luming Tang, Bor-Chun Chen, Claire Cardie, Serge J. Belongie, Bharath Hariharan, and Ser-Nam Lim. 2022. Visual prompt tuming. In Com- puter Vision ECCV 2022 17th European Confer- ence, Tel Aviv, Israel, October 23-27, 2022, Proceed- ings, Part XXXIII, volume 13693 of Lecture Notes in Computer Science, pages 709-727. Springer.
Nanjjang Jiang and Marie-Catherina de Marneffe. 2019. Evaluating BERT for natural language inference: A case study on the commitmentbank. In Proceedings of the 2019 Conference on Empirical Methods in Nat- ural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, Novem- ber 3-7, 2019, pages 6085 Association for Computational Linguistics.
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of @ ACL 2007 Workshop on Textual Entailment and Paraphrasing, Prague, Czech Repub- lic, June 28-29, 2007, pages 1-9. Association for Computational Linguistics.
the
Demi Guo, Alexander M. Rush, and Yoon Kim. 2021. crrameter-efficienn transfer learning with diff prun- ing. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACI/JJNNP 2021, (Volume I: Long Papers), Virtual Event, August 1-6, 2021, pages 4884 4896. Association for Computational Linguis- tics.
Daniel Khashabi. Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chap- ter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLI 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Vol- ume I (Long Papers), pages 252-262.. Association for Computational Linguistics.
Cheng Han, Qifan Wang, Yiming Cui, Zhiwen Cao, Wenguan Wang. Siyuan Qi. and Dongfang Liu. 2023. E'2vpt: An effective and efficient approach for visual prompt tuning. CoRR, abs/2307.13 137700.
Jaejun Lee, Raphael Tang, and Jimmy Lin. 2019. What would elsa do? freezing layers during transformer fine-tuning. CoRR, abs/191 I... 103090..
Junxian He, Chunting Zhou, Xuezhe Ma, Taylor Berg- Kirkpatrick, and Graham Neubig. 2022a. Towards a unified view of parameter-efficien transfer learning. The Tenth International Conference on Learning Representations, ICLR 2022, Virtual Event, April 25- 29, 2022. OpenReview.net.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-effiiien prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing. EMNLP 2021, Virtual Event / Punta Cana, Domini- can Republic, 7-11 November, 2021, pages 3045- 3059, Association for Computational Linguistics.
Yun He, Huaixiu Steven Zheng, Yi Tay, Jai Prakash Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, Heng-Tze Cheng, and Ed H. Chi. 2022b. Hyperprompt: Prompt-based task-conditioning of transformers. In International Conference on Muchine Learning, ICML 2022, 17-23 July 2022, Baltimore, Maryland, USA, volume 162 of Proceedings of Machine Learning Research, pages 8678 8690. PMLR.
Hector J. Levesque, Ernest Davis, and Leora Morgen- stern. 2012. The winograd schema challenge. In Principles of Knowledge Representation and Rea- soning: Proceedings of the Thirteenth International Conference, KR 2012, Rome, Italy, June 10-14, 2012. AAAI Press
Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov. and Luke Zettlemoyer. 2020. BART: demoising sequence-toseqquncc pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computational Linguisties,
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Autariyan, and Sylvain Gelly. 2019. Parameter-effcient transfer learning for NLP. In Pro- ceedings of the 36th International Conference on Ma- chine Learning, ICML 2019, 9-15 June 2019, Long

---

ACL Online, July 5-10, 2020. pages 7871-7880. Association for Computational Linguistics.
Mohammad Taher Pilehvar and José Gamacho-Colladee 2019. Wic: the word-in-contess in-context dataset for evaluat- ing context-sensitive meaning representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computa- tional Linguisties: Humun Language Technologies. NAACL-HILT 2019, Minneapolis, MN, USA, June 2-7, 2019. Volume 1 (Long and Short Papers), pages 1267- 1273. Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Asso- ciation for Computational Linguistics and the 11th Internationaa Joint Conference Natural Language Processing, ACL/JCNLPP 2021, (Volume 1: Long Papers), Virtual Event, August 1-6, 2021, pages 4582- 4597. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine
Lee. Sharan Narang. Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. J. Mach. Learn. Res., 21:140:1-140:67.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengx- iao Du, Zhilin Yang. and Jie Tang. 2022. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), ACI. 2022, Dublin, Ireland, May 22-27, 2022, pages 61-68. As- sociation for Computational Linguistics.
Anastasia Razdaibiedina, Yuning Mao, Rui Hou, Ma- dian Khabsa, Mike Lewis, and Amjad Almahairi. 2023a. Progressive prompts: Continual learning for language models. In The Tenth International Confer- ence on Learning Representations, ICLR 2023, May 1-5, 2023. OpenReview.ne.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis. Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907..1192.
Anastasia Razdaibicdina, Yuning Mao, Rui Hou, Ma- dian Khabsa, Mike Lewis, Jimmy Ba, and Amjad Almahairi. 2023b. Residual prompt tuning: Improv- ing prompt tuning with residual reparameterization. In Proceedings of the 61th Annual Meeting of the As- sociation for Computational Linguisties. ACL 2023. Association for Computational Linguisties.
Fang Ma, Chen Zhang, Lei Ren, Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. 2022. Xprompt: Exploring the extreme of prompt tuning. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, EMNLP 2022, Abu Dhabi, United Arab Emirates, page 11033-11047. Association for Computational Linguistics.
Melissa Roemmele, Cosmin Adrian Bejan, and An- drew S. Gordon. 2011. Choice of plausible alter- natives: An evaluation of commonsense causal rea- soning, In Logical Formalizations of Commonsense Reasoning, Papers from the 2011 AAAI Spring Sym- posium, Technical Report SS-1 1-06, Stanford, Cali- fornia, USA, March 21-23, 2011 1, AAAI.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Alma- hairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. 2022. UniPELT: A unified framework for parameter-efficient language model tuning. In Pro- ceedings of the 60th Annual Meeting of the Associa- tion for Computational Linguistics (Volume I: Long Papers), pages 6253-6264, Dublin, Ireland. Associa- tion for Computational Linguistics.
Timo Schick and Hinrich Schütze. 2021. It's not just size that matters: Small language models are also few- shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. NAACL-HLT 2021, Online, June 6-11, 2021, pages 2339-2352. Association for Computa- tional Linguistics.
Jonas Pfeiffer, Aishwarya Kamath, Andreas Rücklé Kyunghyun Cho, and Tryna Gurevych. 2021. Adapterfusion: Non-destructive task composition for transfer learning. In Proceedings of the 16th Con- ference of the European Chapter of the Association for Computational Linguisties: Main Volume, EACL 2021, Online, April 19- - 23, 2021, pages 487-503. Association for Computational I. Linguistics.
Noam Shazeer and Mitchell Stem. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In Proceedings of the 35th International Conference on Machine Learning, 1CML 2018, Stockholmsmäs- san, Stockholm, Sweden, July 10-15, 2018, volume 80 of Proceedings of Machine Learning Research, pages 4603-4611, PMLR
Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurélien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. Llama: Open and efficient foundation language models. CoRR, Abs/2302.13971
Jonas Pfeiffer, Andreas Rückléé, Clifton Poth, Aishwarya Kamath, Ivan Vulic, Sebastian Ruder, Kyunghyun Cho, and Iryna Gurevych. 2020. Adapterhub: A framework for adapting transformers. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demon- strations, EMNLP 2020 Demos, Online, November 16-20, 2020, pages 46-54, 54, Association for Computa- tional Linguistics.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy,

---

and Samuel R. Bowman. 2019. Superglue: A stickier benchmark for veneral-purpost. language understand- ing systems. In Advances in Neural Information Processing Systems 32: Annual Conference on Neu- ral Information Processing Systems 2019, NeurlPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 3261-3275
23-28, 2020, Proceedings, Part III, volume 12348 of Lecture Notes in Computer Science, pages 698-714. Springer.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. CoRR, abs/1810.12885.
Liqi Yan. Cheng Han, Zenglin Xu, Dongfang Liu, and Qifan Wang. 2023. Prompt learns prompt: Exploring knowledge-aware generative prompt collaboration for video captioning- In Proceedings of the Thirty- Second International Joint Conference on Artificial August 2023,
Susan Zhang, Stephen Roller, Naman Goyal, Mikel Artetxe, Moya Chen, Shuohui Chen, Christopher Dewan, Mona T. Diab, Xian Li, Xi Victoria Lin, Todor Mihaylov, Myle Ott, Sam Shleifer, Kurt Shus- ter, Daniel Simig, Punit Singh Koura, Anjali Srid- har, Tianlu Wang, and Luke Zettlemoyer. 2022. OPT: open pre-trained transformer language mod- els. CoRR, abs/2205.01 01068..
Intelligence, IJAA 2023, Macao, SAR, China, pages 1622 1630. ijeai.org.
Yang, Qifan Wang, Jingang Wang, Xiaojun Quan, Fuli Feng. Yu Chen, Madian Khabsa, Sinong Wang. Zenglin Xu, and Dongfang Liu. 2023. Mixpave: Mix- prompt tuning for few-shor product attribute value extraction. In Proceedings of the 61th Annual Meet- ing of the Association for Computational Linguistics, ACL 2023, Association for Computational Linguis- tics.
Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang. Yupeng Hou, Yinggian Min. Be- ichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinhao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang. Zikang Liu, Peiyu Liu, Jian- Yun Nie, and Ji-Rong Wen. 2023. A survey of large language models. CoRR, abs/2303.18223.
Zhilin Yang. Zihang Dai, Yiming Yang. Jaime G. Car- bonell, Ruslan Salakhutdinov, and Quoe Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural In- formation Processing Systems 32: Annual Confer- ence on Neural Information Processing Systems 2019. NeurlPS 2019, December 8- 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Jason Yosinski, Jeff Clune, Yoshua Bengio, and Hod Lipson. 2014. How transferable are features in deep neural networks? In Advances in Neural Information Processing Systems 27: Annual Conference on Neu- ral Information Processing Systems 2014, December 8-13 2014, Montreal, Quebee, Canada, pages 3320- 3328.
Manzil Zaheer, Guru Guruganesh, Kumar Avinava Dubey, Joshua Ainslie, Chris Alberti, Santiago On- tañón, Philip Pham, Anirudh Ravula, Qifan Wang, Li Yang, and Amr Ahmed. 2020. Big bird: Trans- formers for longer sequences. In Advances in Neural Information Processing Systems 33: Annual Confer- ence on Neural Information Processing Systems 2020, NeurlPS 2020, December 62, 2020, virtual.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. Bitfit: Simple parameter-efficient fine-tuning for transformer based masked maguuag--modlls. In Proceedings of the 60th Annual Meeting of the As- sociation for Computational Linguistics (Volume 2: Shori Papers), ACL 2022, Dublin, Ireland, May 22- 27, 2022, pages 1-9. Association for Computational Linguistics.
Jeffrey O. Zhang, Alexander Sax, Amir Zamir,
Leonidas J. Guibas, and Jitendra Malik. 2020. Side- tuning: A baseline for network adaptation via addi- tive side networks. In Computer Vision ECCV 2020 16th European Conference, Glasgow, UK August

---

A Dataset Statisties
512 tokens, we truncate the input. We do not per- form any specific text preprocessing (c.g. remov- ing punctuation) but instead directly tokenize the raw text from SuperGLUE datasets using the cor- responding model tokenizer. For all experiments, we follow T5 (Raffel et al., 2020) formatting, We feed input examples along with their descriptors (e.g. 'sentencel" and 'sentence2"), and cast all clas- sification tasks into text-to-text format (e.g. 0 and classes in Boolq task are cast into 'True' and 'False') replicating guidelines from T5.
Table shows details of the eight datasets from SuperGL.UE benchmark (Wang al., 2019) that we used for our experiments, along with their training sizes and evaluation metrics. Following (Raffel et al., 2020) and (Lester et al., 2021), for tasks that the average of both scores as the final performance metric.
have evaluation metries
Dataset
Examples
Task
Domain
Metric
Boolq
9,427
QA
Wikipedia
Acc
CB
250
NLI
various
F1/Acc
B.3
Prompt initialization
COPA
400
QA
blogs, encyclop
Ace
MRC
5,100
QA
various
FI/EM
In our experiments, we initialize input prompts us- ing randomly sampled vocabulary embeddings sim- ilar to (Lester et al., 2021). We sample uniformly across the whole vocabulary, without limiting to top-k most common tokens. The attention prompts are randomly initialized.
ReC
101,000
QA
various
FI/EM
RTE 2,500
NLI
news, Wiki
Acc
WiC 6,000
WSD
lexical datahases
Acc
WSC 259
coref.
fiction books
Acc
Table 6: The details of 8 SuperGLUE tasks used in experiments. NLI denotes natural language inference, QA denotes questions and answers task. WSD denotes word sense disambiguation, EM denotes exact match scoring, Acc denotes accuracy.
Methods
Boolq
CB
RTE
WSC
TiniiT (Aribandi et al., 2022)
89.60
94,20
91.70
95.20
Partial-1 1 (Yosinski et al., 2014)
83.28
84.66
83.92
87.51
Adapter (Pfeiffer
2020)
85.75
89.17
B
Training Details
BitFit (Zaken al., 2022)
85,91
90.53
86.34
90.48
LoRA (Hu al., 2022)
89.48
94,66
91.91
95.82
APROMPT (Ours)
90.72
95.48
93.36
96.17
B.1
Settings
Our model is implemented with the OpenPrompt library (Ding et al., 2022), which is unified and extensible toolkit for prompt tuning research. Our model is trained on 16 NVIDIA Tesla V100 GPUs. We translate each SuperGI.UF dataset into a text- to-text format following (Raffel et al., 2020). We train our prompts for 100 epochs with constant learning rate of 0.3 and a batch size of 16. There are three hyperparameters in our model: the lengthes of input, query-key and key-value prompts. For our method, set the number of input prompts to 10 across all tasks, and linearly search the best prompt length for both query-key and key-value prompts from (1, 5, 10, 20, 50). For all prompt tuning baselines, we search the best input prompt length from (5, 10, 20, 50, 100). The best checkpoints are selected via early stopping on the development set. Adafactor (Shazeer and Stern, 2018) () optimizer is used in model training with weight decay le
Table 7:
Performance comparison with other non- prompt tuning based parameter efficient methods on Boolg, CB, RTE and WSC for T5-XL.
C Comparison with Other Parameter Efficient Methods
To conduct a full performance evaluation, we fur- ther conduct comparision of our approach with other non-prompt tuning based parameter efficient methods, including Parial tuning (Yosinski et al., 2014) (Partial-1 means only fine-tuning the first layer), Adapter (Pfeiffer et al., 2020), BitFit (Za- ken et al., 2022) and LoRA (Hu et al., 2022). The performance comparision results are reported in Table 7. It can be seen that APROMPT outperforms all the parameter efficient methods with large mar- gins. In fact, existing prompt tuning approaches also achieve better performances compared to these baselines.
B.2 Tokenization and Preprocessing
Mechods
Booly
Clil
OOPA
MRC
ReC
RTE
WSC
Prompe Tuning
2135m
3m
11m
4tm
180:55
57m
51m
3m
Following common practice (Lester et al., 2021), for all our experiments, we set the maximum input length (including the input prompt) to 512 tokens. We use padding to maximum length and mask out the padded tokens. In case of input exceeding
P-Tusing v2
3136a
10m
14m
I115m
216:2m
112800
1h14a
12t0
4347m
14m
1133m
28641m
2h26m
19m
Reshrmmp
St4Tm
10m
Ih21m
12m
PPROMPT
3821m
9m
15m
1k7
2th
1h34m
10m
Table 8: Training time of APROMPT on SuperGLUUE.

---

D Training Time
We further discuss and report the training time of different methods on all the tasks in SuperGLUE in Table 8. For Prompt-Tuning, the trainable parame- ters consist of the prompts designed for the input layer. In the case of P-Tuning V2, the trainable pa- rameters encompass the prompts associated with all layers. XPrompt focuses on trainable parameters related the pruned prompts, specifically for the input layer, following the pruning process. As for ResPrompt, the trainable parameters include both the input prompts and the residual network compo- nents. The total count of trainable parameters for each approach is detailed in Table 2.


Reviews:
[{"review_id": "7HnEmyXrb0", "reviewer": "Reviewer_tGWP", "summary": "", "questions": "Please refer to [reasons to reject].", "limitations": "", "rating": 3, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback! We discuss your raised points as follows:\n\n#### **Q1: Regarding Q is not calculated using X_new**\n\n**A1:** Thank you for your insightful question. We'd like to provide some clarification here. In the original Prompt Tuning work (Lester et al., 2021), the prompts are exclusively applied or appended to the input layer of the transformer, as illustrated in the second subfigure of Figure 1. It's important to note that while it's possible to compute updated features for these prompts after the first transformer layer, these prompts are not utilized in subsequent layers of the model. \n\nIn both P-Tuning V2 (Liu et al., 2022) and our own approach, the prompts are prepended to all layers of the transformer, as depicted in the two rightmost subfigures of Figure 1. However, it's crucial to emphasize that the prompts from different layers are learned independently and are not updated based on information from their previous layers. To illustrate this further, if you closely examine Figure 3(a), you'll observe that the prompts in the next layer are not updated or calculated from the previous layer (no connections to the previous layer). Meanwhile, the features of the text sequence are indeed updated from the previous layer.\n\nIn summary, in prompt tuning approaches, only the original text tokens are updated through the transformer layers. We thank you again for pointing this out. We will incorporate the above discussion during revision to make it more clear.\n\n#### **Q2: Regarding training time of other baselines**\n\n**A2:** Thank you for the great suggestion! We report the training time of all methods below (training time of Fine-Tuning is not reported as we directly import the results from the original paper). We’ll supplement these results into our revision.\n\n|  Method  | Boolq | CB | COPA | MRC | ReC | RTE | WiC | WSC |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Prompt-Tuning| 2h38m | 8m | 11m | 46m | 18h25m | 57m | 51m | 8m |\n| P-Tuning v2| 3h36m | 10m | 14m | 1h15m | 20h12m | 1h28m | 1h14m | 12m |\n| XPrompt| 4h47m | 14m | 29m | 1h53m | 28h41m | 2h26m | 2h19m | 19m |\n| ResPrompt| 3h47m | 10m | 23m | 1h21m | 22h32m | 1h51m | 1h34m | 12m |\n| APrompt | 3h21m | 9m | 15m | 1h7m | 21h | 1h24m | 1h16m | 10m |\n\n#### **Q3: Regarding the typo**\n\n**A3:** Thank you for pointing it out. We will revise accordingly.\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Views the existing prompt tuning method as a specialized form of attention prompts. \n\n2. Proposes a novel prompting method where the prompts are applied to the attention directly. \n\n3. Extensive experiments on three different model size and SuperGLUE benchmark shows the effectiveness of the proposed method. \n\n4. The paper is well-written and the figures help the understanding.", "reasons_to_reject": "1. I have a hard time understanding why in line 213, Q is not calculated using $X_{new}$. It seems like the authors argue this because only the original text tokens X are updated. However, to the best of my knowledge, the features of the prompts are also updated as it goes through the transformer layers. \n\n2. In Table 8, the authors report the training time of APrompt on the SuperGLUE dataset. But I feel its necessary to report the training time of other baselines as well.", "questions_to_authors": "Please refer to [reasons to reject].", "paper_topic_and_main_contributions": "The paper presents Attention Prompt tuning (APrompt), an innovative method for adapting large pre-trained language models. Unlike traditional tuning that focuses on input layers, APrompt incorporates prompts into the attention layer. Tested on the SuperGLUE benchmark, APrompt outperforms standard approaches across different model scales, with additional studies confirming its effectiveness and efficiency.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "In Figure 3 (c), Matmul -> MatMul"}, "extra_text": {}}, {"review_id": "J6qUJiyFen", "reviewer": "Reviewer_97Ph", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback!\n\n#### **Q1: Compare with non-prompt tuning approaches**\n\n**A1:** Thank you for your great suggestion! We completely agree with the reviewer that it is important to compare with other non-prompt tuning based parameter efficient approaches. In fact, we have compared with several widely used parameter efficient methods and reported the results in Appendix C (Table 7). The Adaptor (we should have named it Adaptor-H) method in Table 7 is a v2 version of the Adaptor work the reviewer mentioned. For completeness, we report the comparison results with the two important methods pointed out by the reviewer below (using T5-XL). We’ll include these additional results in our revised paper. Thank you again for the suggestion.\n\n|  Method  | Boolq | CB | RTE | WSC |\n| :-: | :-: | :-: | :-: | :-: |\n| Fine-Tuning | 89.60 | 94.20 | 91.70 | 95.20 |\n| **Adapter** | 84.46 | 85.89 | 85.53 | 88.75 |\n| Adapter-H | 85.37 | 86.54 | 85.75 | 89.17 |\n| **LoRA** | 89.48 | 94.66 | 91.91 | 95.82 |\n| APrompt | 90.72 | 95.48 | 93.36 | 96.17 |\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear writing.\n2. Thorough analysis and discussion.", "reasons_to_reject": "My main concern of the paper is on the baselines. As newly-proposed parameter efficient fine-tuning method, it should not only be compared with prompt tuning approaches, but also other methods such as Adaptor (https://arxiv.org/pdf/1902.00751.pdf) and LoRA (https://arxiv.org/abs/2106.09685), since they are more widely used.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes another parameter efficient fine-tuning method of large language models: attention prompt tuning. APROMPT incorporates three sets of learnable prompts: query, key, and value prompts. These prompts are prepended to the respective matrices in the self-attention block within the Transformer layer. During model tuning, these attention prompts are learned alongside the original input prompts. Comprehensive experiments on various tasks in the SuperGLUE benchmark show the effectiveness of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "vu8vU7YCdJ", "reviewer": "Reviewer_S6E8", "summary": "", "questions": "1. You should make a comparison among prompt tuning methods including their detailed trainable parameters and training time.\n2. LoRA [1] is an important parameter-efficient tuning method for large language models. You should compare with it.\n[1] LoRA: Low-Rank Adaptation of Large Language Models", "limitations": "", "rating": 4, "confidence": 5, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely thank you for your thoughtful feedback! We discuss your raised points as follows:\n\n#### **Q1: Detailed trainable parameters and training time of prompt tuning methods.**\n\n**A1:** Thank you for your excellent suggestion. In the context of these different techniques: \n\n* For Prompt-Tuning, the trainable parameters consist of the prompts designed for the input layer.\n* In the case of P-Tuning V2, the trainable parameters encompass the prompts associated with all layers.\n* XPrompt focuses on trainable parameters related to the pruned prompts, specifically for the input layer, following the pruning process.\n* As for ResPrompt, the trainable parameters include both the input prompts and the residual network components.\n\nThe total count of trainable parameters for each approach is detailed in Table 2. Furthermore, we report the training time of all methods below. We’ll supplement the above discussion and the training time results in our revision.\n\n|  Method  | Boolq | CB | COPA | MRC | ReC | RTE | WiC | WSC |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Prompt-Tuning | 2h38m | 8m | 11m | 46m | 18h25m | 57m | 51m | 8m |\n| P-Tuning v2 | 3h36m | 10m | 14m | 1h15m | 20h12m | 1h28m | 1h14m | 12m |\n| XPrompt | 4h47m | 14m | 29m | 1h53m | 28h41m | 2h26m | 2h19m | 19m |\n| ResPrompt | 3h47m | 10m | 23m | 1h21m | 22h32m | 1h51m | 1h34m | 12m |\n| APrompt | 3h21m | 9m | 15m | 1h7m | 21h | 1h24m | 1h16m | 10m |\n\n#### **Q2: Should compare with LoRA**\n\n**A2:** Thank you for the suggestion. We agree with the reviewer that LoRA is an important parameter-efficient tuning method. We conduct additional experiments and report the comparison results (using T5-XL) below. We will incorporate these results in Table 7 during revision.\n\n|  Method  | Boolq | CB | COPA | MRC | ReC | RTE | WiC | WSC |\n| :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: | :-: |\n| Fine-Tuning | 89.60 | 94.20 | 96.00 | 76.15 | 92.05 | 91.70 | 74.30 | 95.20 |\n| **LoRA** | 89.48 | 94.66 | 96.15 | 75.78 | 92.36 | 91.91 | 74.49 | 95.82 |\n| APrompt | 90.72 | 95.48 | 95.83 | 78.68 | 93.75 | 93.36 | 76.43 | 96.17 |\n\n#### **Q3: Missing references**\n\n**A3:** Thank you for pointing these related works to us! We will add references [2] and [3] in the revised paper ([1] is already cited).\n\nThanks again for your review! Please let us know if you have any further questions, and we are happy to discuss further.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel method. The proposed attention prompt tuning is novel compared to previous method and can be considered as a unified form.\n2. Insightful analysis. The empirical study is useful to build a connection between the proposed tuning and previous tuning.\n3.  Extensive experiments. The authors conduct many experiments to validate the proposed method.\n4. Good writing. ", "reasons_to_reject": "1. Missing some detailed comparison with previous methods.\n2. Missing some important baselines.", "questions_to_authors": "1. You should make a comparison among prompt tuning methods including their detailed trainable parameters and training time.\n2. LoRA [1] is an important parameter-efficient tuning method for large language models. You should compare with it.\n[1] LoRA: Low-Rank Adaptation of Large Language Models", "paper_topic_and_main_contributions": "This paper proposes an attention prompt tuning method for parameter-efficient tuning PLMs. This paper has proved that existing prompt tuning can be considered as a special case of attention prompt tuning. The main contributions lie in that extensive experiments has verified the effectiveness of this kind of prompt tuning.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "[1] LoRA: Low-Rank Adaptation of Large Language Models.\n[2] A Survey of Large Language Models.\n[3] Challenges and Applications of Large Language Models"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces APrompt as a parameter-efficient fine-tuning method for large PLMs. Rather than modifying input layers, APrompt integrates query, key, and value prompts into the attention layer of the transformer. These prompts are learned alongside the original input prompts during model tuning. APrompt's effectiveness is demonstrated through comprehensive experiments on various SuperGLUE benchmark tasks, outperforming standard approaches across different model scales. Additionally, the study highlights that existing prompt tuning can be seen as a special case of attention prompt tuning, emphasizing the method's practicality and efficiency.

All reviewers engaged in extensive discussions with the authors and all agree the contribution of this work is strong. All reviewers agree that the results are reproducible and the excitement of this work is more than average.
Decision: Accept-Main