Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts
Authors: Muling Wu
Wenhao Liu
Jianhan Xu
Changze Lv
Zixuan Ling
Tianlong Li
Longtao Huang
Xiaoqing Zheng
Xuanjing Huang
Venue: EMNLP 2023 Findings (2023)
Keywords: multi-task learning
token-wise
memory network
instance-dependent prompt

Abstract: Prompt tuning has been proven to be successful on various tasks by incorporating a small number of trainable parameters while freezing large pre-trained language models (PLMs). 
However, it is still unsettled how to generate more proper prompts for any individual examples and how to extend prompt tuning to multi-task learning scenarios by leveraging cross-task features. 
To address these challenges, we propose a token-wise prompt tuning (TPT), in which a bank of finer-grained soft prompt tokens is built for multi-task learning by memory network. 
The tokens are retrieved from the bank against an input example and assembled to an instance-dependent prompt. Extensive experimental results on $14$ datasets demonstrated that the models enhanced by our TPT performed far better than full parameter fine-tuned models and achieved state-of-the-art by tuning only $0.035\%$ parameters.

Parameter Efficient Multi-task Fine-tuning by Learning to Transfer Token-wise Prompts
Muling W Wenhao Liu Jianhan Xu!.2, Changze Lv ZiXuan Ling 1.2
Tianlong Li LongTao Huang". XiaoQing Zheng" Xuanjing Huang" 1.2
School of Computer Science, Fudan University, Shanghai, China
Shanghai Key Laboratory of Intelligent Information Processing
3 "Alibaba Group, Zhejiang, China
wwl.w22,whliu22,,czzzuu 2xxuu tllit2lleee@@ @m. fudan. edu. cn
{jianhanxu20, zhengxq, x xjwhang @ oftpa///eeeuoooooooooooou/////od edu. cn
Abstract
?
Prompt tuning has been proven to be successful various tasks by incorporating small num- ber of trainable parameters while freezing large pre-trained language models (PLMs). How- ever, it is still unsettled how to generate more proper prompts for any individual examples and how extend prompt tuning multi-task learning scenarios by leveraging cross-tar tass fea- tures. address these challenges, we propose token-wise wise prompt tuning (TPT), in which a bank of finer-grained soft prompt tokens is built for multi-task learning by memory net- work. The tokens are retrieved from the bank against an input example and assembled to an instance-dependent prompt. Extensive experi- mental results on 14 datasets demonstrated that the models enhanced by our TPT performed far better than full parameter fine-tuned mod- els and achieved state-of- the-hhr by tuning only 0.035% parameters.
Reiderst Modill
Powling Medule
Frowen
19
Tolkem-iorten Prromp Bonk
Retsioved Promppt
Irput
Token-wise prompting solution. An input text length I, {a1 f ,.,.,,., , is sent to a pooling module that produces feature vector á for the input. A retrieval module retrieves the most similar prompt to- kens, {1a, i.. ,' J. from a prompt bank based on the similarity scores estimated between è and soft prompt tokens (4, 12. ... ,.,, fn stored in the bank. The retrieved prompt tokens are assembled to an -nstance-dependent prompt and concatenated with the original input. The concatenated result is sent to a frozen LM for both in- ference and training, in which only the retrieved prompt tokens are tuned through error back pack-propagation.
Figure 1:
performance on various tasks by simply adding ad- ditional context to the input. Although promising. there are least two limitations: (a) It overlooks the inherent differences among instances. Even a well-learned prompt might not be suitable for all data instances within a large population, as high- lighted by Scao and Rush (2021). (b) It fails to leverage the rich cross-task features, as the learned prompts were exclusively designed for individual tasks, making it difficult for these prompts to be reused transferred across tasks (Vu et al., 2021).
1 Introduction
The architecture of Transformers (Vaswani al., 2017) has yielded impressive performances on var- ious natural language processing (NLP) tasks and has been widely established as the building block for PLMs. The dominant paradigm is to pre-train on large-scale unlabeled datasets and then fine-tune on task-related datasets (Devlin et al., 2019; Raf- fel et al., 2019; Radford and Narasimhan, 2018). However, performing full parameter fine-tuning for each task would be prohibitively expensive with the growing model scale. Thus, there has been growing interest in developing parameter-efficient fine-tuning (PEFT) methods (Houlsby et al., 2019; Hu et al., 2021; Ben-Zaken et al., 2021) that strive to achieve results comparable to full parameter fine- tuning with small number of trainable parameters. new
To overcome the first limitation, we propose a novel approach to automatically generate a more suitable prompt for each input example. Bari et al. (2022) proposed to retrieve non-trainable tokens. also referred to as hard prompt, from the embed- ding layer of the language model, which can en- hance both the training and inference processes of the model. Compared directly copying the fixed embedding layer as source of extra information for each example, training an additional embedding layer on the target task can provide more appropri-
Prompt learning (Lester et al., 2021), effective method of PEFT, can boost the model's
OOr code is available at https: thub. cow.ul.commmmm./// com/mlwu2//
TPT

---

ate information. Therefore, we go fürther on this line by decomposing the trainable soft prompt into finer-grained soft prompt tokens, these tokens con- stitute the token-wise prompt bank, which can be viewed as a trainable embedding layer. Memory network (Weston et al., 2014) is also used to store, tune and combine these tokens, illustrated in Figure 1. In contrast previous methodologies that consider the soft prompt single whole, our approach dissects it into fin-graiied prompt to- kens. This refined breakdown the prompt widens the scope of search space, facilitating a more ex- haustive combination of soft tokens, and ultimately leading to the generation of superior prompts.
multi-task learning scenarios, which demon- strates a remarkable boost in transfer learning on both seen and unseen tasks.
Empirical results on 14 different datasets demonstrate the effectiveness of TPT that out- performs existing prompt-based methods by a significant margin in accuracy and even out- performs the full parameter fine-tuning on both GLUE and SuperGI.UE datasets by tun- ing only 0.035% parameters.
Related Work
Task-dependent prompt. This line of research focuses on enhancing the generation of more effec- tive prompts for specific target tasks.
To address the second issue, we extend the pro- cess of constructing token-wise prompt bank (i.e., the memory network of token-wise soft prompts)
Specifically, Brown et al. (2020) introduced the utilization of a small set of manually crafted sen- tences as prompt, typically consisting of task de- scriptions and relevant examples. The prompt is fed to the frozen model as part of the input, offer- ing the potential to enable the pre-trained model to achieve comparable performance to fine-tuned models, particularly when well-designed.
multi-task learning scenarios. There are many features that can be shared between different tasks, and these features can be learned through multi- task learning (Mahabadi et al.. 2021). Additionally, prior work by Vu et al. (2020) demonstrated that performing prompt tuning on intermediate tasks be- fore doing it the target task can yield even better results. Following their recipe, first pre-train the token-wise prompt bank across multiple source tasks, then utilize this resulting bank as initializa- tion to train the token-wise prompt bank specifi- cally for the target task.
Auto-prompt (Shin et al., 2020), LM-BFF (Gao et al., 2021), and EFL (Wang al., 2021a) ex- tend this direction by automating the process of generating discrete prompts. However, optimizing prompts within discrete spaces presents challenges and is likely to be sub-optimal. Prompt tuning (Lester et al., 2021), Prefix tuning (Li and Liang, 2021), and P-tuning (Liu et al., 2021) adopt an alternative strategy by introducing continuous vec- tors, known soft prompt, in front of the input sequence. Only these continuous vectors need to be adjusted during training. so the optimization prob- lem discrete spaces is converted to a continuous optimization task, which can be handled through simple gradient descent.
Finally, we extend our approach, which is called token-wise prompt tuning (TPT), by combining token-wise prompt bank with task-specific prompt tuning, illustrated in Figure 2. In approach, all examples within given task share a task-level prompt, which is generated by way of task-specific prompt tuning. Additionally, for each individual ex- ample, an instance-level prompt is retrieved based on the similarity between the input example and tokens in the token-wiss wiiee prompt bank. These two prompts are concatenated together, incorporating both instance-level and task-level features, as part of the input facilitate model inference. More- over, extensive experimental results on 14 datasets demonstrated the effectiveness of our methods.
Moreover, later work has started to consider prompt tuning in the transfer learning scenario. Su et al. (2021) and SPoT (Vu et al., 2021) explore the transferability of prompts learned from different tasks, and address the sensitivity of prompt tuning to initialization through transfer learning. Wang et al. (2023) and PANDA (Zhong et al., 2022) pro- posed to learn a transfer prompt on the source tasks through knowledge distillation.
The contribution of this study can be summa- rized as follows:
This study is among the first ones to introduce token-wise prompt tuning by decomposing soft prompts into tokens and constructing a bank of trainable tokens by memory network.
Instance-dependent prompt. This line of research takes into account the individual characteristics of different examples and generates distinct prompts tailored to each specific example.
We extend the token-wise prompt bank to

---

In particular, Levine et al. (2022) and IDPG (Wu al.. 2022) generate instance-wise prompts via multi-layer perceptions (MLPs) based on the input encoded by the language model.
Prompt Transfer. Transfer learning methods at- tempt learn new target task given a collection of source tasks Tsource = {Ti, T2, ... .Tt) which have been a long-standing way to improve the ef- fectiveness and efficiency of NL.P (ystems(Ruder, 2017). Recent studies such as (Vu et al., 2021; ; Su et al., 2021) have demonstrated the applicability of transfer learning in the context of prompt tun- ing, also referred to prompt transfer. Instead of training the prompt from scratch over target task, these approaches employ the souree prompts P sourre = {PI, ... , which are trained ac- Pt),
Li et al. (2022) and Wang et al. (2021b) main- tain a prompt pool to store the prompts learned over source tasks, where each prompt is classified into specific categories and assigned key vectors. The encoded input serves as the query vector, and the target prompts are obtained by weighing the prompts in the prompt pool according to the re- sults of the query and key vector calculations. In addition, ATTEMPT (Asai et al., 2022) calculates weights simply based on the similarity between the input and the prompts learned from the source tasks, without the need for pre-computed clusters.
cording to equation (2) over source tasks. These source prompts P sourree can then serve as either ini- tialization vectors or weighted vectors for training the target prompt.
Unlike these approaches that weighted the soft prompt a whole, we instead utilize the finer- grained prompt tokens for combination, and only the tokens retrieved according to the input receive the gradient during training. Therefore, more appropriate prompt can be generated for each ex- ample. SPT (Bari et al., 2022) proposed to use retrieved non- trainiaie hard prompt prefix to guide the training of the prompt. In contrast, our method is to retrieve the trainable soft prompt and can be extended to scenarios of multi-task learning to incorporate cross-task features.
4 Method
Our proposed method TPT (As illustrated in Figure 2) consists of two stages: pre-training token-wise prompt bank (Section 4.1) and jointly prompt tun- ing (Section 4.2).
TPT pre-trains a token-wiss, prompt bank that integrates the cross-task features on various source tasks T Toorrce = {Ti, [Ti,T2.- ... Tr} and then utilizes this resulting bank as the initial token-wise prompt bank of the next stage to generate instance-level retrieved prompt for each example. In addition, all examples of the target task T Tlarget also share the same task-level soft prompt. These two kinds of prompts are concatenated with the input as the final input of the frozen LM and provide both additional instance-level and task-level features for that input, which enhance the model's training and inference processes, leading to improved performance.
3 Preliminaries
Prompt Tuning. Given a pre-trained LM with parameters 0, and target task Trarges with training data D (X,; 34 ) conventional full parameter fine-tuning (FT) seeks to maximize the likelihood of decoding the desired output Yi given input X; over training data D):
4.1 Pre-training Token-wise Prompt Bank
IIXX
ps X.)
(1)
We first pre-train token-wise prompt bank over t high-resource tasks Teource via the memory net- work. The examples from multiple datasets are mixed together. enabling the implicit integration of cross-task features within the learning process of the token-wise prompt bank, thereby endowing it with a powerful capacity for knowledge transfer. Formally, given the training data Dsource { D2, DD} of source tasks Tsources of the input sequence denoted
Unlike FT, prompt tuning freezes the pre-trained language model and only needs to train a very small number of parameters. Specifically, it prepends m randomly initialized vectors, also known as soft prompt P- = {P1, P..... )m). where p; E Rd be- fore the input X,, the optimization goal of prompt tuning as follows:
X; (2) (2, ..) E Rixd, where is the input length, d is the dimension of hidden state. The input X; and a randomly initialized token-wise prompt bank B are simultaneously sent to the re-
max P
Pa(s: X.))
where frozen, and only P is trainable.

---

Tolersaie Thompe Soot
Telomrarie Prongtfae Bank
Self Poongl
Prosen LM
Frun ?
Revireed Prompt
Ingut
Reteeved Pramgeed Seat
Ingst
Pre-training Token-wise Prompt Bank
Jointly Prompt Tuning
The overall process of TPT. The first step is to pre-train a token-wise prompt bank that absorbs cross-task features on multiple source tasks. The second step utilizes this prompt bank initialization, transfers the knowledge of the source tasks to the target task and generates retrieved prompt for each example, and jointly trains with the soft prompt on the target task. Instance-level retrieved prompt and task-level soft prompt provide richer contextual information for input help the model train and infer better.
Figure 2:
trieval module for calculation and an instance- retrieved prompt R; is generated for X, according to the similarity results. The retrieved prompt R; is prepended in front of the input sequence X; to form [R;:;;], which together serve the final in- put of the frozen LM. The training objective is to maximize the likelihood of conditional generating over D Doouree
prompt bank will attain the gradient for adjustment, and the other tokens that have not been retrieved remain untouched.
4.1.2 Similarity Score Estimation
The retrieval module controls which tokens to se- lect from the token-wise prompt bank each time instance-level retrieved prompt is generated by calculating the similarity between the example and prompt tokens in the token-win prompt bank.
(Di, D2, D which is:
max R
Σ X,EDs
poly; X:X
(3)
Specifically, the retrieval module will generate the similarity scores S, = {s{: s ... , be- tween input x, and the tokens in the bank, where 8) denotes the similarity score between X, and j-th token in the bank. The tokens located at the k in- dexes {ri, ri. ... r of that token-wise prompt bank, which possesses the highest similarity scores with the input are retrieved and concatenated into the retrieved prompt of that example, as follows:
4.1.1 Soft Prompt Decomposition
Unlike the previous prompt-based method, which weights or combines prompts as whole (Asai et al., 2022: Li et al., 2022: Wang et al., 2021b), TPT disassembles prompts into finer-grained prompt tokens to complete these operations, en- abling a more comprehensive amalgamation of soft tokens, and therehy expanding the range of possible combinations.
fri, rá.. r(] (Tpex(TopK(S)))
For this reason, what is stored in the bank is the soft prompts trained the source tasks, but smaller units of prompts, that are, soft prompt tokens, B = {f1, f2, ) where to E nx The n soft prompt tokens in the tosen-wise prompt bank will calculate the similarity scores with the input, and the & tokens with the highest scores will be retrieved and concatenated into the instance- level retrieved prompt R; {t.g, frg. " ,tyg }. where rf C .1, .n} indicates that the k-th token of the retrieved prompt R; generated for the i-th example X; corresponds to the index of the token stored in the token-wise prompt bank.
where Topk() function returns the largest k values of the given input, and Index() function returns the subscript of the given input value, which is the index in the bank.
To deal with inputs of various lengths, apply max-pooling over input X, generate the pooled input é E Rd with the same length of soft prompt tokens l in the token-wise prompt bank and the generated ft will be fed into the retrieval module to calculate similarity scores between &; and t, based on their inner product, which is:
i M MaxPoolingg
(5)
8) = (fr
(6)
During the training process, only the tokens (tyg bgg " , retrieved from the bisen-wise
where (.) denotes the operation of inner product.

---

4.2 Jointly Prompt Tuning
utilize 6 high-resource datasets source tasks and select 14 tasks from GLUE (Wang al., 2018), SuperGLUE (Wang et al., 2019) and Sci- (Khot et al., 2018) as target tasks for evaluation.
When jointly prompt tuning, the token-wise prompt bank trained in the first step is utilized as initializa- tion to generate the instance-level retrieved prompt according to the method described in section 4.1 Transfer learning is performed based on it, and the knowledge of the source tasks is transferred to the target task.
Source tasks. As in (Asai et al., 2022) and (Wang et al., 2023), we use 6 datasets with more than100k annotations source tasks: MNLI (Williams et al., 2017), QNLI (Demszky al., 2018), QQP (Wang et al., 2018), SST-2 (Socher et al., 2013). SQuAD (Rajpurkar et al., 2016), and ReCoRD (Zhang et al., 2018).
In addition, for all examples of the target task, we initialize a task-level soft prompt P {pi, P2. Pm.) where P E Rnxd which is shared by them. Instance-level retrieved prompt R; and task-level soft prompt P are concatenated in front of the input to form RR; P. X,). and fed into frozen LM together the contextual information. During the training process, retrieved prompt R; and soft prompt adjust simultancously, and the optimization objective transformed to maximize the likelihood of decoding the desired output y/ given input X over training data D = (X:; y1 M as follows:
Target tasks. We selected a total of 14 tasks as tar- get tasks for evaluation, among which, 8 tasks are selected from GLUE: MNLI, QQP, QNLI, SST-2, STS-B (Cer et al., 2017), MRPC (Dolan and Brock- ett, 2005), RTE (Bar-Haim et al., 2006), and COLA (Warstadt et al., 2018), tasks are selected from Su- perGLUE: MultiRC (Khashabi et al., 2018), BoolQ (Clark et al., 2019), CB (de Marneffe et al., 2019), WiC (Pilehvar and Camacho-Collados, 2018), and WSC (Levesque et al., 20,1), and another dataset SciTail (Khot et al., 201 (). SeiTail is a scientific entailment dataset, which is chosen for few-shot evaluation.
mAx
Po [M
(7)
In contrast to vanilla prompt tuning, which only provides a shared task-level soft prompt P for all examples of the target task, TPT provides an addi- tional instance-level retrieved prompt R; specific to each example X; of the target task com- plementary. This instance-level prompt R; cap- tures the particular information related to the input X,, while the task-level prompt encompasses the overall information from the training data D to which X; belongs. By incorporating these distinct levels of features, a more comprehensive contex- tual framework is established for X., which can facilitate enhanced model training and inference capabilities.
5.2 Models
Following the standard approach in previous prompt-based method (Lester et al., 2021; Asai et al., 2022; Wang et al., 2023), we mainly ex- periment using the publicly available pre-trained T5-base model with 220M parameters. In our abla- tion study, we also consider T5-Small (60M) and T5- (770M) models.
5.3 Baselines
We compare TPT with the following baselines: (1) full parameter fine-tining (FT), where all the model parameters are tuned during adaptation on each downstream task, while other methods solely focus on adjusting the specific components mentioned below. Specifically, (2) prompt tuning (PT) (Lester et al., 2021), where target prompt vectors are ini- tialized by randomly sampled top vocabularies; (3) SPoT (Vu et al., 2021) and ATTEMPT (Asai et al., 2022) initialize target prompts by retrieving or ag- gregating prompts trained over source tasks; (4) Adapter (Houlsby et al., 2019) and AdapterDrop (Rücklé et al., 2020) insert trainable modules in the middle of the model; (5) BitFit (Ben-Zaken Zaken et al., 2021) only needs to adjust the bias term. (6) MPT
5 Experiments
Following the previous prompt-based methods (Lester et al., 2021; Asai et al., 2022), we perform our experiments on 14 different datasets with full- dataset and few-shot settings, and the experimental results show the effectiveness of our TPT in various scenarios.
5.1 Datasets and Tasks
The TPT method is divided into the first stage of pre-training on source tasks, and the second stage of task adaptation on the target task. Specifically,

---

SuperGIAUR
Method
Param
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoL.A
Multi
BaolQ
WiC
WSC
CB
Avg.
FT
220M
86.8
91.6
93.0
91.6
89.7
90.2
71.9
61.8
$1.9
72.8
51.1
70.2
59.6
85.7
73.9
Adapters
19M
$6.5
90.2
53.2
93.8
90.7
85.3
71.9
64.0
84.5
75.9
82.5
67.1
67.3
85.7
75.7
AdapterDrop
91.4
62.7
BaFF
280K
55.3
90.1
63.0
94.2
90.9
86.8
67.6
58.2
83.3
74.5
79.6
70.0
59.6
78.6
72.5
ATTEMPT
232K
$4.3
90.3
93.0
93.2
89.7
85.7
T3.4
57.4
83.4
74.4
78.8
06.88
13.8
78.6
70.5
FT
77K
$1.3
59.7
122.8
90.9
89.5
68. 1
54.7
10.6
72.2
58.7
61.7
48.9
51.9
67.9
57.8
77K
93.0
93.4
90.0
57.1
MPT
77K
55.9
90.3
93,1
83.8
90.4
89.1
62.4
74.8
79.6
69.0
67.3
79.8
71.1
TPT
639K
85.5
90.1
93.2
91.77
89.8
89.7
82.3
59.5
85.6
74.1
80.1
69.8
67.3
94.6
77.2
Table Results on GLUE and SuperGLUE.
the results are based b5-base models. The middle of the table shows the results of the prompt-basct method, the top of the table shows the results of other PEFT methods, and the bottom of the table the result of our proposed TPT. For these experiments, we exclude SQuAD and ReCoRD from source prompts inventories for comparison with prior work. We use Pearson Correlation for STS-B. F1 for MultiRC (Multi), and accuracy for other tasks as metrics. "param/task" denotes the number of parameters trained for each task GLUE.
(Wang et al., 2023) learns target prompt through knowledge distillation.
85
Aur. SuperGLUE
Adupter
Avg. GLUE
BeRi
8
PI SPat ATTEMPT
5.4 Implementation Details
To pre-train the token-wise prompt bank, we con- duct a 5-epoch training phase on a mixture of high-resource source tasks. For jointly prompt tun- ing, we reuse the trained token-wise prompt bank to generate the instance-level retrieved prompt and the prompt trained the target task inter- mediate task initialize the soft prompt. Unless specified, we use T5-base as our base LMs for TPT and more details in Appendix A.2.
TPT-
le5
1e6 1e7
leB
leb
le6
le7 Narameters
1eB
Paremetters
Figure Parameter efficiency on GLUE (left) and Su- perGLUE (right). All results are based TS-base (Raf- fel et al., 2019).
baselines. Specifically, we achieved SOTA high- resource datasets: MNLI (9).2%), SST-2 (94.7%), and on low-resource datasets: RTE (82.3%), WSC (67.3%), CB (94.6%), which demonstrate the ef- fectiveness of TPT across various data resource sce- narios. Compared to vanilla prompt tuning, TPT obtains relative improvement of 13.4% on GLUE and 19.4% on SuperGLUE, surpassing the perfor- mance of vanilla prompt tuning across all datasets by large margin. This result further shows that the instance-level retrieved prompt composed of prompt tokens is complementary to the task-level soft prompt generated by prompt tuning, and the supplementary effect universal. Moreover, TPT outperforms full parameter fine-tuning (FT) on GLUE by 0.7% and SuperGLUF by 4.7%, despite tuning 0.245% many task-specific parameters.
a dataset does not have public test split with an- notations, we use development set as our test set split the development set into our development and test sets, following (Davison, 2021).
In few-shot experiments, for each number of shots following (Mahabadi al., 2021; Asai al., 2022), we randomly sample 3 times from the training set with different random sceds and report the mean performances. In addition, the task-level soft-prompt is initialized randomly or following (Vu et al., 2021) to use the prompt trained on the MNLI dataset as initialization.
5.5 Results
We present main results, which are the full-data adaption in Section 1, few-shot adaption in Sec- tion 5.5.2, and parameter efficiency in 5.5.3.
5.5.2 Few-shot adaption
5.5.1 Full-data adaption
Following (Mahabadi al., 2021: Asai et al., 2022; Wang al., 2023), we conduct few-shot experi- ments on BoolQ, CB and SciTail, further ver- ify the effectiveness of TPT under the resource- constrained setup. Table 2 shows the results our approach and other baselines, which includes
Table 1 presents the per-task performance of differ- ent methods on all datasets.
As shown Table 1, TPT has established state- of-the-art (SOTA) performances on the datasets of GLUE and SuperGLUE compared with these

---

b-shot
FT (220M)
AD (L.M)
PT (77K)
ST (77K)
HF (638K)
АTP (232K)
MPT (77K)
TPT(535K)
BoolQ
50.5
53.1
61.6
50.5
48.0
61.8
62.2
62.2
16
56.5
51.4
61.9
50.6
50.2
60.0
63.3
63.5
32
58.4
54.5
61.7
61.2
58.3
65.3
68.9
07.4
CB
57.7
51.1
53.5
71.4
60.7
82.1
73.6
78.6
16
77.0
74.8
63.5
64.3
76.3
78.5
78.6
80.4
32
80.0
74.8
67.8
64.3
81.4
85.7
82.1
86.3
SciTail
79.6
79.5
57.7
69.6
82.0
80.2
80.2
81.0
83.2
60.8
71.9
86.5
79.5
87.3
85.5
32
81.9
85.0
60.2
71.9
85.8
80.2
86.3
85.2
Table 2: Few-shot results (A: = 16, 32). FT, AD, PT, ST, HF, and ATP denote Fine-tuning, Adapter, Prompt tuning. SPoT, HyperFormer (Mahabadi al., 2021), and ATTEMPT.
full parameter fine-tuning, Adapter, prompt tuning, SPoT, HyperFormer, ATTEMPT, and MPT. To be specific, TPT outperforms other methods in certain cases, achieving both SOTA BoolQ (4, 16-shot) and CB (16, 32-shot). These results clearly indi- cate that TPT can effectively use cross-task features in source tasks to target tasks few-shot domain adaptation.
range of model scales, spanning from 60M to 770M parameters. As the model size increases, TPT ex- hibits increasingly pronounced advantages. Partic- ularly, when the model size is large, TPT surpasses other baselines on all three datasets.
75
5.5.3 Parameter efficiency
Accuracy Oka
80
(%g
70
88
60
Figure 3 compares the performance of different models versus their number of updated parameters on GLUE and SuperGLUE. In addition, TPT-f is variant of TPT, that is, the parameters of the bank are frozen during the joint training process, and only the parameters corresponding to the task-level soft prompt need be adjusted.
00
Figure 4: Performance with different backbone LMs.
In addition, we also conducted experiments on a larger scale model (T5-3B), and table shows that our TPT method still has strong competitiveness the era of large models.
Specifically, TPT outperforms all other base- lines on both GLUE and SuperGLUE with only a small number of parameter adjustments, espe- cially over full-parameter fine-tuning. TPT-f still maintains very high accuracy (y-axis) when adjust- ing smaller number of parameters per task (x- axis). TPT-f adjusts many parameters vanilla prompt tuning, but the performance GLUE and SuperGLUE is more than prompt tuning by large margin, which proves that the TPT and TPT-f have a high degree of parameter effectiveness.
Method
GLUE
PT
79.7
TPT
88.1
Table 3: Experimental results on large language model (T5-3B).
Effectiveness of token-wise prompt bank. We also conduct experiments to assess the effectiveness of solely utilizing the retrieved prompt, abbreviated as RP, generated from the token-wise prompt bank. Instead of concatenating the instance-level re- trieved prompt with the task-level soft prompt and jointly performing prompt tuning, we exclusively prepend the instance-level retrieved prompt, which is composed of tokens retrieved from the token- wise prompt bank, to the input during task adap- tation. To be specifie, RP-S signifies training the
5.6 Ablation Study
Model Scaling, We empirically analyze how in- creasing the backbone LM size affects TPT perfor- mance. Figure 4 shows the performance of TPT well as full parameter fine-tuning, Adapter, AT- TEMPT, prompt tuning and MPT with three dif- ferent T5 models (T5-small, T5-base, T5-large). These results show that TPT largely benefits from backbone LM size increase, which is aligned with the finding of (Lester et al., 2021). Furthermore, TPT demonstrates effectiveness across a wide
"The large language model (T5-3B) can bring better per- formance, but the training process is relatively unstable.

---

prompt bank for the target task from scratch, while RP-M involves pre-training a token-wise prompt bank on multiple source tasks and subsequently em- ploying it as initialization to train the token-wise prompt bank for the target task. In addition, the training method of PR-W is similar to that of RP-S. But unlike RP-S, which decomposes soft prompts into finer grained prompt tokens and then retrieves and adjusts these tokens in the token-wise prompt bank, RP-W W treats soft prompts whole and then retrieves and adjusts these prompt in the prompt bank.
directly adding them to an agreement vector, leads to better outcomes.
Method
GLUE
SuperGLUE
Addition
83.4
75.2
Concatenation
85.6
77.2
Table 5: The impact of different combinations of instance-dependent prompts and task-specific prompts.
Prompt Initialization. We explore the impact of soft prompt initialization in the context of the joint prompt tuning process. Our investigation focuses on three distinct initialization strategies: (1) Ran- dom Initialization: This approach involves replicat- ing embeddings from the most frequent tokens in the vocabulary. (2) SPoT Initialization: Following the methodology of SPoT, we employ the prompt trained on the MultiNLI dataset as the initialization for the sentence-level classification target task. (3) Target Task Initialization: We utilize the prompt trained specifically for the target task the initial- ization. By examining these different strategies, we aim to understand the effects of soft prompt ini- tialization on the overall performance of the joint prompt tuning process.
The results presented in Table 4 reveal that even when using only the retrieved prompt, RP-S out- performs vanilla PT and RP-W by large margin and RP-M surpasses ATTEMPT by a large mar- gin, which validates that the method of dismantling soft prompts into finer-gras mner-grained prompt tokens and then combining them can generate a more suitable prompt for each example and also demonstrates the effectiveness of our loken-wise prompt bank. Moreover, RP-M performs better than RP-S, which indicates that multi-task learning source tasks facilitate beneficial transfer effect both seen and unseen target tasks.
The results presented in table 6 demonstrate that initializing the task-level soft prompt in three dif- ferent ways for joint prompt tuning is much better than vanilla prompt tuning, which shows that the soft prompts initialized by these different meth- ods have significantly improved performance on all datasets after being prefixed with the instance-level prompt we proposed for jointly prompt tuning, thus verifying that our proposed instance-level retrieved prompt is complementary to all these different task- level soft prompts.
Method
GLUE
SuperGLLLE
72.2
57.8
RP-W
78.6
62.5
RP-S
83.5
67.2
ATTEMPT
83.4
70.5
RP-M
84.2
74.7
Table 4: The effectiveness of roken-wiss prompt bank. "PR" indicates that only the instance-level prompt re- trieved from the token-wise prompt bank is used. "-S" means to train the bank from scratch on the target task, "-M" means perform multi-task learning on multiple source tasks, and then perform transfer learning on the target task to train the bank, and "-W" means to treat the soft prompt as whole.
Furthermore, employing the prompt trained on the target task as the initialization yields the most favorable outcomes, while the randomly initialized prompt exhibits relatively poor results. This ob- servation also indicates the more task-related soft prompt can play greater role during the joint prompt tuning process.
Combination Methods. We also explore the im- pact of the two different methods of combining instance-level prompts and lask-level prompts on performance: (1) Following the approach of AT- TEMPT (Asai et al., 2022), the values of corre- sponding positions in the two prompts are directly added. (2) Prepending the instance-level prompt in front of the lask- prompt as (Bari et al., 2022).
Conclusions
In this study, we have introduced TPT, novel parameter-efficient fine-tuning method designed to address the challenges of generating more suit- able prompts for individual examples and extend- ing prompt tuning to multi-task learning scenarios for capturing cross-task features. TPT harnesses the power of a memory network to construct a finer-
The results of table 5 show that the second method yields superior performance. This find- ing suggests that processing the task-level features and instance-leve features separately, rather than

---

Method
GLUE
SuperGLUE
References
Random
84.4
70.7
Intermediate task
81.7
72.9
Akari Asai, Mohammadreza Salehi, Matthew E. Peters, and Hannanch Hajishirzi. 2022. Attempt: Parameter- efficient multi-task tuning via attentional mixtures of soft prompts. In Conference on Empirical Methods in Natural Language Processing.
Target task
85.5
77.2
Table 6: The impact of different initialization methods. The results are indicated by "Intermediate task", where the prompts are initialized with those trained on the in- termediate task, and by "Target taks" where the prompts are initialized with those tuned on the target task.
Roy Bar-Haim, Ido Dagan, Bill Dolan, Lisa Ferro, Danilo Giampiccolo, Bernardo Magnini, and Idan Szpektor. 2006. The second pascal recognising tex- tual entailment challenge.
grained token-wiss prompt bank comprising soft prompt tokens in multi-task learning settings. Our extensive experimental results have demonstrated the effectiveness of TPT. It has demonstrated supe- rior performance over full parameter fine-tuning in specific cases while requiring significantly fewer adjusted parameters. Moreover, these experiments have also verified the compatibility of our method with existing prompt-based approaches, thereby contributing new insights to the field of parameter- efficient fine-tuning.
M Saiful Bari, Aston Zhang, Shuai Zheng. Xingjian Shi, Yi Zhu, Shafiq R. Joty, and Mu Li. 2022. Spt: Semi- parametric prompt tuning for multitask prompted learning. ArXiv, abs/2212.10929.
Elad Ben-Zaken, Shauli Ravfogel, and Yoav Gold- berg. 2021. Bitfit: Simple parameter-efficient fine-tuning for transformer-bass masked language- models. ArXiv, abs/2106. 10199.
Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert- Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Bemer, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners.
Limitations
We have demonstrated the potential of integrating instance-dependent prompts, derived from token- wise prompts, with task-specific prompts to en- hance performance. It would be intriguing to ex- amine the feasibility of generating task-specific prompts on-the-fly, leveraging the assembly and retrieval of their token-wise, fine- fine-graines grained prompts. Additionally, our future research will focus on the creation of a generalized token-wise wise soft prompts model, which can be applicable across wide spec- trum of NLP tasks, rather than being restricted to a select few.
Daniel Matthew Cer, Mona T. Diab, Eneko Agirre, lñigo Lopez- Gazzio, and Lucia Specia. 2017. Semeval- 2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In International Workshop on Semantic Evaluation.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. Boolq: Exploring the surpris- ing difficulty of natural yes/no questions. ArXiv. abs/1905..1000 10044.
Ethics Statement
Joe Davison. 2021. Compacter: Efficient low-rank hy- percomplex adapter layers.
This work fully comply with the ACL Ethics Policy, All the authors declare that there is no ethical issues in this paper submitted to ACL 2023 for review.
Marie-Catherie de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019. The commitmentbank: Investi- gating projection in naturally occurring discourse.
Dorottya Demszky, Kelvin Guu, and Percy Liang. 2018. Transforming question answering datasets into natural language inference datasets. ArXiv, abs/1809.02922.
Acknowledgements
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Natural Science Foun- dation of China (No. 62076068), and Shanghai Municipal Science and Technology Project (No. 21511 2021100
Jacob Devlin, Ming-Was Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understand- ing, ArXiv, abs//aal.... 100.0480..
William Dolan and Chris Brockett. 2005. Automati- cally constructing a corpus of sentential paraphrases.

---

International Joint Conference on Natural Lan- guage Processing.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. 2021. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetworks. In Anmual Meeting of the Association for Computational Linguistics.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In Association for Computational Linguis-
lics (ACL).
Mohammad Taher Pilehvar and José Hamacho-Colladoo 2018. Wic: the word-in lel--i--ooon context dataset for evaluat- ing context-sensitive meaning representations. In North American Chapter of the Association for Com- putational Linguistics.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin de Laroussilhe, Andrea Ges- mundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for nlp. In Inter- national Conference Machine Learning.
Alee Radford and Karthik Narasimhan. 2018. Im- proving language understanding by generative pre- training.
Edward J. Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. ArXiv, abs/2106.09968.
Colin Raffel, Noam M. Shazeer, Adam Roberts, Kather- ine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2019. Exploring the limits of transfer learning with a unified text-to-text transformer. ArXiv, abs/ 191991100.......
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading com- prehension over multiple sentences. In North Amer- ican Chapter of the Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev. and Percy Liang. 2016. Squad: 100,000+ ques- tions for machine comprehension of text. ArXiv, abs/1606.05250.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. In AAA/ Conference on Artificial Intelligence.
Andreas Rücklé, Gregor Geigle, Max Glockner, Tilman Beck, Jonas Pfeiffer Nils Reimers, and Iryna Gurevych. 2020. Adapterdrop: On the efficiency of adapters in transformers. In Conference on Empir- ical Methods in Natural Language Processing.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: method for stochastic optimization. CoRR, abs/1412.6980.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficiet prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Sebastian Ruder. 2017. An overview of multi-task learn- ing in deep neural networks. ArXiv, abs/l 706.05098.
Teven Le Scao and Alexander M. Rush. 2021. How many data points a prompt worth?
Taylor Shin, Yasaman Razeghi, Robert Logan IV au2, Eric Wallace, and Sameer Singh. 2020. Auto- prompt: Eliciting knowledge from language models with automatically generated prompts.
Hector J. Levesque, Ernest Davis, and L. Morgenstern. 2011. The winograd schema challenge. In Interna- tional Conference on Principles of Knowledge Rep- resentation and Reasoning.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, A. Ng. and Christopher Potts. 2013, Recursive deep models for semantic compositionality over sentiment treebank. In Conference on Empirical Methods in Natural Lan- guage Processing.
Yoav Levine, Itay Dalmedigos, Ori Ram, Yoel Zeldes, Daniel Jannai, Dor Muhlgay. Yoni Osin, Opher Lieber, Barak Lenz, Shai Shalev-Shwartz, Amnon Shashua, Kevin Leyton- Brooon, and Yoav Shoham. 2022. Standing the shoulders of giant frozen lan- guage models. ArXiv, abs/2204. 10019.
Yusheng Su, Xiaozhi Wang. Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang. Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, Lei Hou, Maosong Sun, and Jie Zhou. 2021. On transfcrability of prompt tuning for natural language processing. In North Ameri- can Chapter of the Association for Computational Linguistics.
Junyi Li, Tianyi Tang, Jianyun Nie, Ji rong Wen, and Wayne Xin Zhao. 2022. Learning to transfer prompts for text generation. In North American Chapter of the Association for Computational Linguistics.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning! Optimizing continuous prompts for generation.
Ashish Vaswani, Noam M. Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017, Attention is all you need. In NIPS
Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang. and Jie Tang. 2021. Gpt understands, too.

---

Tu Vu, Brian Lester, Noah Constant, Rami AI-Rfou, and Daniel Matthew Cer. 2021. Spot: Better frozen model adaptation through soft prompt transfer. ArXiv, abs/2110.n. 10.00904.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. ArXiv, abs/2208.
Tu Vu, Tong Wang. Tsendsuren Munkhdalai, Alessan- dro Sordoni, Adam Trischler, Andrew Mattarella- Micke, Subhransu Maji, and Mohit lyyer. 2020. Ex- ploring and predicting transferability across nlp tasks. ArXiv, abs/2005, .00770.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. Superglue: A stick- ier benchmark for general-purpott language under- standing systems. In Neural Information Processing Systems.
Alex Wang. Amanpreet Singh. Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2018. Glue: A multi-task benchmark and analysis plat- form for natural language understanding. ArXiv, abs/1800 Abs//804.07461.
Sinong Wang. Han Fang, Madian Khabsa. Hanzi Mao, and Hao Ma. 2021a. Entailment as few-shot learner.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Rogério Schmidt Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter- efficient transfer learning, ArXiv, abs/2303.02861.
Zifeng Wang, Zizhao Zhang, Chen- Yu Lee. Han Zhang, Ruoxi Sun, Xiaoqi Ren, Guolong Su, Vincent Perot, Jennifer G. Dy, and Tomas Pfister. 2021b. Learning to prompt for continual learning. 2022 IEEE/CVF Conference Computer Vision and Pattern Recog- nition (CVPR), pages 139-149.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- 2018. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 641.
Jason Weston, Sumit Chopra, and Antoine Bordes. 2014. Memory networks. CoRR, abs/1410.391)
Adina Williams, Nikita Nangia, and Samuel R. Bow- man. 2017. A broad-coverage challenge corpus for sentence understanding through inference. In North American Chapter of the Association for Computa- tional Linguistics.
Zhuofeng Wu, Sinong Wang, Jiatao Gu, Rui Hou, Yux- iao Dong, V. G. Vinod Vydiswaran, and Hao Ma. 2022. Idpg: An instance-dee instanee-dependent prompt gener- ation method. In North American Chapter of the Association for Computational Linguistics.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. ArXiv, abs/181 12885.

---

GLUE
SuperGLUE
Method
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
Avg.
Multi
BoolQ
WIC
WSC
Avg.
Random
94.2
80.9
54.0
72.0
78.7
69.6
Intermediate task
84.8
90.0
93.2
93.8
89.3
89.7
80.1
56.5
84.7
73.3
79.5
67.7
63.5
80.4
72.9
Target task
84.6
90.1
93.2
94.7
89.8
89.7
82.3
39.8
85.5
74.4
80.1
69.8
67.3
94.6
77.2
Table 7: Experimental results of different initialization methods of task-level soft prompt to perform TPT on GLUE and SuperGLUE.
GLUE
SuperGLUE
Method
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
Avg.
Multi
BoelQ
WIC
WSC
CB
Avg.
PT
81.3
89.7
92.8
90.9
89.5
68.1
54.7
10.6
72.2
58.7
61.7
48.9
51.9
67.9
37.5
RP-W
81.7
59.7
92.8
91.7
88.9
68.9
58.5
57.0
78.6
65.7
62.2
52.2
61.4
67.9
62.5
RP-S
82.9
89.7
92.8
92.9
89.1
88.5
72.9
58.8
53.5
74.3
62.4
52.7
64.4
82.1
67.2
ATTEMPT
84.3
90.3
93
93.2
89.7
85.7
73.4
57.4
83.4
74.4
73.8
06.8
53.8
78.6
70.5
RP-M
83.9
00.0
93.4
93.2
89.3
88.2
78.7
56.7
84.2
73.8
78.5
67.6
6144
89.3
74.7
Table Experimental results of single instance-level retrieved prompt on GLUE and SuperGLUEE
GLUE
SuperGLUE
Method
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
Avg.
Multi
BoolQ
WFC
WSC
CB
Avg.
Add
82.6
89.3
92.9
93.2
89.5
86.3
78.4
54.8
83.4
72.4
78.4
67.1
65.4
92.9
75.2
Concatenate
84.6
90.1
93.2
94.7
89.8
89.7
82.3
59.8
85.5
74.4
80.1
69.8
67.3
94.6
77.2
Table 9: Experimental results of TPT on GLUE and SuperGLUE through different prompt combination methods.
GLUE
SuperGLUE
Method
Param
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RIE
CoL.A
Avg.
Multi
BoolQ
WIC
WSC
CB
Avg.
IPT-
77K
84
90
93.1
93.8
89.5
88.7
80.9
58.8
84.9
74.3
79.2
66.3
64.4
94.6
75.8
00.1
93.2
89.7
82.3
67.3
Table 10: Experimental results of TPT variant TPT-f on GLUE and SuperGLUF.
Appendix
uations, specifically focusing on two distinet combi- nation methods employed to combine the instance- dependent prompt and target-specific prompt. The table highlights the outcomes achieved by utilizing these combination approaches.
A
More details
A.1 Detailed Results
The following provides more detailed information for the experimental section of this paper.
Finally, Table 10 presents a comprehensive anal- ysis of TPT-f, a variant of TPT, in the context of GLUE and SuperGLUE evaluations. TPT-f effee- tively reduces the number adjustable parameters in comparison to TPT, while demonstrating com- parable performance in terms of achieved results.
Table 7 provides a comprehensive breakdown the outcomes obtained from GLUE and Super- GLUE evaluations, specifically focusing on the task-level soft prompt initialization. This approach involves combining the task-specific soft prompt retrieved prompt in or-
with the cestance-dependee der optimize prompt tuning. The table compares the results for three distinct methods employed in this initialization process.
A.2 Training details
Hyperparameters. As used in TPT, we use the prompt length m = 100 for each prompt and use the learning rate of 0.3 for prompt tuning to train the task-specific prompt and set weight decay to be x 10-5 In addition, we also utilize learning rate of 0.3 for pre-training token-wise prompt bank and jointly prompt tuning and optimize the objective function using Adam (Kingma and Ba, 2014), In particular, use the learning rate of 0.1 for SuperGI.UF, and Yelp, WinoGrande, SciTail and PAWS multi-task experiments, and 0.3 for the other experiments. At the same time, we also
Table 8 presents a comprehensive analysis of the outcomes obtained from GLUE and Super- GLUE assessments when exclusively relying on the instance-level retrieved prompt the supple- mentary context. The aim of this investigation is evaluate the efficacy of the token-wise prompt bank in generating appropriate prompts for each input example.
Table provides a detailed presentation of the results obtained from GLUE and SuperGLUE eval-

---

try different schedulers and when train the task-level soft prompt, we choose the constant learning rate of 0.3 and for the other experiments, we also try the linear scheduler.
Few-shot Adaptation Experiments Details. Fol- lowing (Mahabadi et al., 2021), we run few-shot adaptation experiments three times and take the mean of the performance. We cite the performance of the full parameter fine-tuning (FT), Adapter (AD), HyperFormer (HF) from (Mahahadi al., 2021), prompt tuning (PT), SPoT (ST), ATTEMPT (Asai et al., 2022) (ATP), and MPT (Wang et al., 2023) and random initialize the task-level soft prompt or utilize the prompt trained on MNLI.
Per-device batch size for TPT and prompt tun- ing, For T5 small and base, set Per-GPU batch size be 100 and 32, while for T5-large, we use the batch size of 16.


Reviews:
[{"review_id": "RWXSlQ2qbG", "reviewer": "Reviewer_ZdZA", "summary": "", "questions": "See \"Reasons to reject\".", "limitations": "", "rating": 2, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n\n**Q1: There are some missing baselines such as SPT, and HyperPrompt that might be strong contenders in multi-task prompt-tuning.**\n\nR1:Our work involves comparing with several strong baseline methods in prompt tuning developed previously, such as ATTEMPT (EMNLP2022) and MPT (ICLR 2023).These approaches utilize the 'T5-base' as backbone, and both the methods proposed by ATTEMPT and MPT are initially trained on six high-resource tasks before adapting to the target task, which are different from SPT. \n\nThe method proposed by SPT is first trained on GLUE and then adpted on SuperGLUE, thus only experimental results on SuperGLUE are available, and its performance is inferior to ours => 74.66 (SPT) vs. 77.2 (Ours)\n\nHyperPrompt performs better than us after introducing additional parameters and joint training with the backbone.However, our backbone parameters are frozen, and their training parameters are significantly larger than ours (approximately 3000 times), as shown in the table below:\n\n|             |  Param   | GLUE | SuperGLUE |\n| :---------: | :------: | :--: | :-------: |\n|    Ours     | 0.00035x | 85.6 |   77.2    |\n| HyperPrompt | 1.04000x | 86.8 |   78.9    |\n\nFurthermore, HyperPrompt is more similar to Prefix tuning, requiring the introduction of parameters in front of every layer's K and V, unlike methods like prompt tuning that only need to modify the embedding layer.In situations where the model needs to switch frequently between different tasks, methods like HyperPrompt, which require parameter loading within the model, can result in longer latency.\n\nFinally, in the revised version, we will replicate HyperPrompt's experiment of freezing the backbone and only training additional parameters at the T5-base scale, allowing for a fairer comparison with our TPT method.\n\n\n\n**Q2: Even though they utilize more parameters than other baselines, their TPT represents poor performance than previous works on various tasks in most of their experiments.**\n\nR2:Our method has achieved favorable performance on numerous datasets, and when compared to these baselines, especially full-parameter fine-tuning, TPT has achieved better overall results in both GLUE (84.9 (FT) vs.  85.6 (Ours)) and SuperGLUE (73.9 (FT) vs. 77.2 (Ours)).\n\n\n\n**Q3: The authors claim that “training additional embedding layer on the target task can provide more appropriate information than a fixed embedding layer, such as SPT”. However, there is no analysis to verify this or prove that their method could actually solve/alleviate the problems.**\n\nR3: SPT's performance on SuperGLUE is inferior to ours (74.66 (SPT) vs. 77.2 (Ours)), providing experimental evidence of the superiority of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "They introduce token-wise prompt tuning by decomposing soft prompts by constructing a bank of trainable tokens by memory network on multi-task learning scenarios. The results indicate that their proposed model achieves higher performance than baselines on several benchmarks.", "reasons_to_reject": "1) Lack of baselines\n\nIn this paper, they propose prompt-tuning for multi-task learning scenarios. However, there are some missing baselines such as SPT[1], and HyperPrompt[2] that might be strong contenders in multi-task prompt-tuning.\n\n2) Poor Performance\n\nThe experimental results show that most improvements from their proposed method are brought by several tasks (e.g. QNLI, SST-2 ...). However, even though they utilize more parameters than other baselines, their TPT represents poor performance than previous works on various tasks(e.g. MNLI, QQP, STS-B ...) in most of their experiments. \n\n3) Lack of Analysis\n\nIn this paper, the authors claim that “training additional embedding layer on the target task can provide more appropriate information than a fixed embedding layer, such as SPT”. However, there is no analysis to verify this or prove that their method could actually solve/alleviate the problems. Moreover, the authors did not provide any interpretation for performance degradation on several tasks than other baselines, which occurs in most experiment results.\n\n[1] Bari et al., SPT: Semi-Parametric Prompt Tuning for Multitask Prompted Learning, arXiv 2022\n\n[2] He et al., HyperPrompt: Prompt-based Task-Conditioning of Transformers, ICLR 2022\n\n", "questions_to_authors": "See \"Reasons to reject\".", "paper_topic_and_main_contributions": "This paper proposes two main limitations for prompt tuning: 1) prompt tuning overlooks the inherent differences among instances that may not be suitable for all data instances 2) prompt tuning approaches are hard to be reused or transferred across tasks. In this paper, they suggest token-wise prompt tuning (TPT) by combining token-wise prompt bank with task-specific prompt tuning on multi-task learning scenarios. ", "excitement": 2, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "He et al., HyperPrompt: Prompt-based Task-Conditioning of Transformers, ICLR 2022"}, "extra_text": {}}, {"review_id": "8FKkmrWNqF", "reviewer": "Reviewer_GwUS", "summary": "", "questions": "Question A: Do you think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?\n\nQuestion B: How long does TPT need for training and inference and how is this time consumption compared to full-parameter tuning?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n\n**Q1: The experiment only covers a model size of 220M. I think a more comprehensive evaluation of a larger range of models with a variance of scales should be included (e.g. T5-large on full GLUE dataset and even larger models), especially for those larger models given the continuous growth of the scales of pre-trained language models.**\n\nR1: Our work involves comparing with several strong baseline methods in prompt tuning developed previously, such as ATTEMPT (EMNLP2022) and MPT (ICLR 2023). Their method utilizes t5-base as a backbone, and to facilitate a more accurate comparison with their approach, we have also employed t5-base as our backbone.\n\nFurthermore, in the ablation study section, we investigated the impact of model scale. On the three datasets we compared, our approach achieved the best performance when scaled up to T5-large. This demonstrates the potential for our method to achieve even greater performance gains when increasing the model scale. \n\nIn the revised version, we will also include experiments with larger-scale backbones, such as T5-3B or llama.\n\n\n\n**Q2: The first comparison in Table 3 (PT vs RP-S) seems unfair. To justify the claim \"dismantling soft prompts into prompt tokens can generate a more suitable prompt\", I think the baseline should be pools of soft prompts with similarity-based retrieval methods rather than vanilla soft-prompt (since the major contribution of the work is dismantling rather than dismantling + retrieval). Moreover, I wonder if ATTEMPT is a multi-task pre-training-based method. If not, then I don't think the second comparison is a fair one.**\n\nR2 : Well, It's a real fantastic question! Previously, retrieved-based methods were designed to retrieve soft prompts trained over various source tasks.  There hasn't been a baseline that solely retrieves soft prompts trained over the target task. Therefore, we have designed a baseline that retrieves soft prompts in a manner similar to TPT, but exclusively from the prompt pool corresponding to the target task. The experimental results are as follows:\n\n|          | MNLI | QQP  | QNLI | SST2 | STSB | MRPC | RTE  | COLA | AVG  |\n| :------: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: | :--: |\n|    PT    | 81.3 | 89.7 | 92.8 | 90.9 | 89.5 | 68.1 | 54.7 | 10.6 | 72.2 |\n| Retrieve | 81.7 | 89.7 | 92.8 | 91.7 | 88.9 | 68.9 | 58.5 | 57.0 | 78.6 |\n|   RP-S   | 82.9 | 89.7 | 92.8 | 92.9 | 89.1 | 88.5 | 72.9 | 58.8 | 83.5 |\n\nThe experimental results demonstrate that retrieval-based methods lead to performance improvements, but they still fall significantly behind our TPT method. Thus, this also validates our viewpoint: dismantling soft prompts into prompt tokens can generate a more suitable prompt.\n\nAdditionally, our approach can be viewed as a variant of ATTEMPT, as both involve 'pre-training' on the same source tasks and then adapting to the target task.\n\n\n\n**Q3: Given the strong ability of currently emerged language models (e.g. ChatGPT) and the accompanying simpler adaption methods (e.g. in-context learning), the proposed method seems outdated**\n\nR3 : In-context learning has proven to be a highly effective approach. However, large-scale models are sensitive to such hard prompts, making it labor-intensive to design adapted prompts for different tasks.  Additionally, for certain specialized domains or tasks, executing in-context learning may require experts to design highly professional prompts. And prompt tuning automates this process by searching for soft prompts in a continuous space.\n\nFinally, in the revised version, we will use larger-scale backbones for relevant experiments and compare their capabilities with llm's in-context learning.\"\n\n\n\n**Q4: Do you think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?**\n\nR4 : In the ablation study section, we investigated the impact of model scale. On the three datasets we compared, our approach achieved the best performance when scaled up to T5-large. This demonstrates the potential for our method to achieve even greater performance gains when increasing the model scale. \n\nTherefore, I believe that our method is equally applicable to llm. In the revised version, we will also include experiments with larger-scale backbones, such as T5-3B or llama.\n\n\n\n**Q5: How long does TPT need for training and inference and how is this time consumption compared to full-parameter tuning?**\n\nR5 : During training, we may spend slightly more time, but the inference time remains the same. In the PEFT domain, the focus is on the number of parameter adjustments, and since we have fewer adjustable parameters, the computational and storage resources required are significantly reduced.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. This paper presents token-wise prompt tuning, which is a novel and interesting prompt-tuning approach.\n2. The performance shown in the experiment is good.\n3. The writing is clear and the structure is well-organized.", "reasons_to_reject": "1. Experiment: the experiment only covers a model size of 220M. I think a more comprehensive evaluation of a larger range of models with a variance of scales should be included (e.g. T5-large on full GLUE dataset and even larger models), especially for those larger models given the continuous growth of the scales of pre-trained language models. \n2. Ablation: the first comparison in Table 3 (PT vs RP-S) seems unfair. To justify the claim \"dismantling soft prompts into prompt tokens can generate a more suitable prompt\", I think the baseline should be pools of soft prompts with similarity-based retrieval methods rather than vanilla soft-prompt (since the major contribution of the work is dismantling rather than dismantling + retrieval). Moreover, I wonder if ATTEMPT is a multi-task pre-training-based method. If not, then I don't think the second comparison is a fair one.\n3. Given the strong ability of currently emerged language models (e.g. ChatGPT) and the accompanying simpler adaption methods (e.g. in-context learning), the proposed method seems outdated.", "questions_to_authors": "Question A: Do you think the method or idea proposed in this paper can directly facilitate or have any implications on the usage and adaption of cutting-edge large language models (e.g. ChatGPT)?\n\nQuestion B: How long does TPT need for training and inference and how is this time consumption compared to full-parameter tuning?", "paper_topic_and_main_contributions": "This paper focuses on parameter-efficient fine-tuning for pre-trained language models, specifically, prompt-tuning. The paper proposes a token-wise prompt selection and training method. These prompts are firstly trained on multi-task corpus and then specified to target tasks. The experiment results in this paper verify the effectiveness and efficiency of the proposed method.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "Not applicable", "Typos_Grammar_Style_and_Presentation_Improvements": "line 64: \", these\" -> \". These\"\n\nline 302: $\\mathbf{x}_i$ -> $\\mathbf{X}_i$"}, "extra_text": {}}, {"review_id": "rfkPha9fMI", "reviewer": "Reviewer_viJ7", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 4, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n\n **Q1: This paper is similar to previous prompt tuning paper, with the added element of retrieval for prompt initialization. While the approach is straightforward, it doesn't offer significant innovation compared to earlier studies**\n\nR1 : Our work is built upon prompt tuning, with a focus on addressing the limitations of vanilla prompt tuning. Additionally, we introduce a novel approach of disentangling the soft prompt into prompt tokens, which significantly improves performance in a token-wise prompt tuning manner, demonstrating the effectiveness of our method.\n\n\n\n**Q2: In this paper, the authors primarily utilized the T5-small and T5-large models, which seem somewhat outdated. It might be beneficial for the authors to explore more recent decoder-only models, like the LLaMa model, for a more current analysis**\n\nR2: Our work involves comparing with several strong baseline methods in prompt tuning developed previously, such as ATTEMPT (EMNLP2022) and MPT (ICLR 2023). Their method utilizes t5-base as a backbone, and to facilitate a more accurate comparison with their approach, we have also employed t5-base as our backbone.\n\nFurthermore, in the ablation study section, we investigated the impact of model scale. On the three datasets we compared, our approach achieved the best performance when scaled up to T5-large. This demonstrates the potential for our method to achieve even greater performance gains when increasing the model scale. \n\nIn the revised version, we will also include experiments with larger-scale backbones, such as llama.\n\n\n\n**Q3: This paper concentrates solely on classification tasks, including GLUE and SuperGLUE, which can be considered somewhat dated. Generation tasks, by contrast, present more utility and challenges than classification tasks**\n\nR2 : Similar to Q2, to facilitate comparisons with the previous baselines, our method was also tested exclusively on classification tasks. Finally, in the revised version, we will evaluate the performance of our TPT method on more complex generation tasks.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "this paper is well written and easy to understand. \n\n", "reasons_to_reject": "1. This paper is similar to previous prompt tuning paper, with the added element of retrieval for prompt initialization. While the approach is straightforward, it doesn't offer significant innovation compared to earlier studies.\n\n2. In this paper, the authors primarily utilized the T5-small and T5-large models, which seem somewhat outdated. It might be beneficial for the authors to explore more recent decoder-only models, like the LLaMa model, for a more current analysis.\n\n3. This paper concentrates solely on classification tasks, including GLUE and SuperGLUE, which can be considered somewhat dated. Generation tasks, by contrast, present more utility and challenges than classification tasks.\n\nIn general, this article feels like it's from a few years ago and isn't particularly captivating.\n\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method that could retrieve tokens from prompt bank and utilizes them for prompt tuning.", "excitement": 1, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "T9hRCu19Y0", "reviewer": "Reviewer_11wC", "summary": "", "questions": "", "limitations": "", "rating": 2, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your valuable comments!\n\n **Q1：The method involves several complex components, such as memory networks and a token-wise prompt bank, which might increase the complexity of implementation and tuning.**\n\nR1: Although our method introduces multiple components, it still only requires adjustments to a very small percentage of parameters (0.035%) compared to full-parameter fine-tuning. Therefore, it should not pose significant implementation challenges.\n\n\n\n**Q2：While the method attempts to integrate cross-task features through multi-task learning, ensuring effective transfer of these features across different tasks might be challenging. **\n\nR2 : In the ablation experiments section, we demonstrated that through multi-task learning, we can achieve excellent transfer learning results on both seen and unseen tasks, as shown in the table below:\n\n|      | GLUE | SuperGLUE |\n| :--: | :--: | :-------: |\n| RP-S | 83.5 |   67.2    |\n| RP-M | 84.2 |   74.7    |\n\n\n\n**Q3：The method emphasizes achieving performance comparable to full parameter fine-tuning by tuning only 0.035% of parameters. However, this might limit the model's flexibility and ability to adapt to different tasks**\n\nR3 : Compared to full-parameter fine-tuning, our method achieves better performance while adjusting fewer parameters. Additionally, when we increase the model size or trainable parameters, we can further enhance our performance.\n\n\n\n**Q4：While the paper compares the proposed method with existing approaches, the comparison is limited to a few methods. A more comprehensive comparison with a wider range of state-of-the-art methods would provide a better understanding of the relative performance.**\n\nR4 : Our work involves comparing with several strong baseline methods in prompt tuning developed previously, such as ATTEMPT (EMNLP2022) and MPT (ICLR 2023). \n\nFurthermore, in the revised version, we will include the following: \n\n1.  A comparison with SPT, with experimental results as shown below:\n\n   |      | SuperGLUE |\n   | :--: | :-------: |\n   | SPT  |   74.7    |\n   | TPT  |   77.2    |\n\n   \n\n2. Replication of HyperPrompt's experiment, where backbone parameters are frozen and only additional parameters are adjusted, for a fairer comparison with our TPT method", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Novel approach: The concept of token-wise prompt banks is a novel contribution to prompt-based methods in NLP. It provides a more comprehensive and contextually relevant framework for training and inference.\n2. Experimental results: The paper presents experimental results on a range of NLP tasks, demonstrating the effectiveness of the proposed method. The results show significant improvements over existing methods, validating the approach.\n3. Clear presentation: The paper is well-structured and clearly presents the methodology, experiments, and results. The authors provide sufficient details to understand the proposed approach and its implementation.", "reasons_to_reject": "1.Implementation Complexity: The method involves several complex components, such as memory networks and a token-wise prompt bank, which might increase the complexity of implementation and tuning.\n2.Utilization of Cross-Task Features: While the method attempts to integrate cross-task features through multi-task learning, ensuring effective transfer of these features across different tasks might be challenging.\n3.Parameter Efficiency: The method emphasizes achieving performance comparable to full parameter fine-tuning by tuning only 0.035% of parameters. However, this might limit the model's flexibility and ability to adapt to different tasks.\n4.Limited comparison: While the paper compares the proposed method with existing approaches, the comparison is limited to a few methods. A more comprehensive comparison with a wider range of state-of-the-art methods would provide a better understanding of the relative performance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper proposes a method to enhance prompt-based methods for natural language processing tasks. The authors introduce the concept of token-wise prompt banks, which consist of fine-grained prompts that can be combined to generate suitable prompts for specific examples. They demonstrate the effectiveness of their approach through experiments on various NLP tasks, showing improved performance compared to existing methods. \n\nThe main focus of the article is to propose a method called token-wise prompt tuning (TPT) to enhance prompt-based methods for natural language processing tasks. The authors aim to address the challenges of generating proper prompts for individual examples and extending prompt tuning to multi-task learning scenarios. They introduce the concept of token-wise prompt banks, which consist of fine-grained prompts that can be retrieved and assembled to create instance-dependent prompts. The article primarily focuses on demonstrating the effectiveness of TPT through experimental evaluations of various NLP tasks.\n\nThe token-wise prompt tuning (TPT) approach addresses the limitations of previous methods in two key ways:\n\n1. Individualized prompts: Previous methods often use a single prompt for all examples within a task, overlooking the inherent differences among instances. TPT tackles this limitation by generating instance-dependent prompts. It decomposes the trainable soft prompt into finer-grained soft prompt tokens and retrieves tokens from a token-wise prompt bank based on the similarity between the input example and the tokens in the bank. This allows for the generation of more suitable prompts tailored to each example, enhancing the model's performance.\n\n2. Cross-task feature leveraging: Previous methods design prompts exclusively for individual tasks, making it challenging to reuse or transfer prompts across tasks. TPT extends the construction of token-wise prompt banks to multi-task learning scenarios. It pre-trains the token-wise prompt bank across multiple source tasks, capturing cross-task features. This resulting bank is then utilized as initialization to train the token-wise prompt bank specifically for the target task. By incorporating cross-task features, TPT demonstrates a remarkable boost in transfer learning on both seen and unseen tasks.", "excitement": 2, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This work proposes a new approach for multi-task efficient fine-tuning. Proposed methods creates a prompt bank using a set of "meta-training" tasks and then combines individual items in the prompt bank at evaluation to achieve superior transfer. Reviewers agree on the clarity of writing and presentation and find the proposed method novel. There are some concerns about comparison to relevant work and using of larger models, authors provide comparison to other relevant work and commit to adding larger models by the camera-ready.
Decision: Accept-Findings