Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learning
Authors: Zhen Wang
Rameswar Panda
Leonid Karlinsky
Rogerio Feris
Huan Sun
Yoon Kim
Venue: ICLR 2023 poster (2023)
Keywords: Prompt Tuning
Multitask Learning
Transfer Learning

Abstract: Prompt tuning, in which a base pretrained model is adapted to each task via conditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. However, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vectors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from multiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it to each downstream target task. Extensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-art methods, including the full finetuning baseline in some cases, despite only tuning $0.035\%$ as many task-specific parameters.

Published as a conference paper at ICLR 2023
MULTITASK PROMPT TUNING ENABLES PARAMETER-BFFICIEN TRANSFER LEARNING
Zhen Wang Rameswar Panda² Leonid Karlinsky² Rogerio Feris²
Huan Sun'
Yoon Kimè
'The Ohio State University, 2MIT-IBM Watson Al Lab, Massachusetts Institute of Technology {wang. 9,15, sun. 397. edu, (rpanda, leonidka, rsferis @eieris)ebbm.... com, yoonkim@mit..cm edu
ABSTRACT
Prompt tuning, in which a base pretrained model is adapted to each task via con- ditioning on learned prompt vectors, has emerged as a promising approach for efficiently adapting large language models to multiple downstream tasks. How- ever, existing methods typically learn soft prompt vectors from scratch, and it has not been clear how to exploit the rich cross-task knowledge with prompt vec- tors in a multitask learning setting. We propose multitask prompt tuning (MPT), which first learns a single transferable prompt by distilling knowledge from mul- tiple task-specific source prompts. We then learn multiplicative low rank updates to this shared prompt to efficiently adapt it each downstream target task. Ex- tensive experiments on 23 NLP datasets demonstrate that our proposed approach outperforms the state-of-the-ar methods, including the full finetuning baseline in some cases, despite only tuning 0.035% as many task-specific parameters]
INTRODUCTION
Finetuning pretrained language models (PI.Ms) has led to significant improvements across various downstream NLP tasks (Devlin et al., [219 Howard & Ruder 2018 Raffel et al. 2020). However, the conventional paradigm of full task-ipecific finetuning (FT) is difficult to scale to multiple tasks, given that modern PLMs have hundreds of millions (or even billions) of parameters. There thus has been a growing interest in developing perameter-efficien
methods for model tuning (Houlsby et al. 2019 Lester al.] 2021 Ding al. 2022), where the goal is to learn only small number of additional parameters per task while achieving performance comparable to full finetuning.
Prompt tuning (PT), which prepends able continuous prompt vectors to the in- put, has emerged as a promising approach for parameter-efffcieet transfer learning with PLMs (Liu et al., 2021a Li Liang. 2021] Lester et al. [021 Liu et al. [2022b] 2021b).
Source Training
Target Adaptation
Soft Promett Per Tesk
Retrieve or Aggregate Prompts
Source Tasks
T
Target
T5
PT freezes the PLM parameters and only learns small set of task-specific prompt vectors. However, despite their impressive performance, there is still a large gap between prompt tuning and full finetuning (Lester et al., 202I). Addi- tionally, this approach is sensitive initializa- tion and often requires more training time than finetuning (Su al. 2022 Zhong et al. 20022).
Existing Approaches
Source Promens
Sean
Distilation
Tranatrredd
Tj
Source
Target
Prompt Decomposition
(b) Our Approach
Recent work has proposed to address these is- sues transferring prompt vectors from vari- ous tasks (Su et al. 2022 Zhong et al.) (002). These methods first train soft prompts on mul- tiple source tasks and then use these pretrained prompts to initialize the prompt for further fine- tuning on a target task based on (potentially learned) similarity measure (uu et al. 2022 Asai et al.
A conceptual overview of our approach. Tn- stead of retrieving aggregating source prompts (top), multitask prompt tuning (MPT. bottom) learns a single transferable prompt. The transferable prompt is learned via prompt decomposition and distillation.
Figure 1:
Work done during an internship at MIT-IBM Watson Al Lab. Project page: https://zhhnwanggg// 02. thtb.b.... io/mpt. html

---

Published as a conference paper at ICLR 2023
85.0
75
Avg. SuperGLUE
82.5
Avg. GLUE
70
FT
Adapter
80.0
BitFiL
PT
77.5
65
SPoT
ATTEMPT
75.0
ATTEMPT>
60
MPT
72.5
MPT*
104
10a
106
107
10
104
10a
10
107
10a
# of Parameters
# of Parameters
Figure 2: Parameter efficiency GLUE (left) and SuperGLUE (right). Our multitask prompt tuning (MPT) approach, which transfers single shared prompt learned from multiple source tasks using prompt decompossi- tion and distillation, maintains high accuracy (g-axis) while finetuning only a small number parameters per task (r-axis). All results are based on T5-Base (Raffel et al. 2020). Baselines include: Adapters (Houlsby)
et al.
2019). BitFit (Zaken et al. 22022). PT (Lester et al. [202]]. J. SPoT (Vu et al. 2022), and ATTEMPT (Asai
2022
"Indicates multitask training on target tasks. Best viewed in color.
2022) (see Figure 1 top). In this paper, we extend this line of work and introduce multitask prompt tuning (MPT), which uses multitask data to learn a single prompt that can be efficiently trans- ferred to target tasks. While conceptually simple, learning a shared prompt space can be practically challenging as requires learning commonalities across different source tasks while minimizing in- terference. Therefore, decompose the soft prompt of each source task (which can be represented as a prompt matrix) into a multiplication of a shared matrix and a low-rank task-specific matrix, and find that this decomposition is more effective than simply sharing the prompt matrix across all tasks. This decomposition is learned through knowledge distillation from soft prompts obtained from regular prompt tuning. To transfer new tasks, we perform low-rank multiplicative updates to the shared prompt matrix. Figure || (bottom) illustrates our approach.
Extensive experiments on 23 NLP datasets across diverse tasks demonstrate the effectiveness of our proposed approach over state-of- state-of-the-art prompt transfer methods. On the SuperGLUE bench- mark (Wang et al. 2019), MPT with T5-Base (Raffel al., 2020) yields 16.3% improvement over the vanilla prompt tuning baseline (PT, et al. 2021). and also outperforms the most compet- itive multitask prompt transfer baseline (ATTEMPT, Asai et al. 2022) despite tuning much fewer task-specific prompt parameters (77.6K vs 232K). On some benchmarks, MPT exceeds the perfor- mance of full finetuning while only requiring 0.035% tunable parameters per task (see Figure 2 We also find that MPT is very effective for few-shot learning with 4-32 labels for each target task.
RELATED WORK
Parameter-efficiet transfer learning. Parameter-efficien transfer learning for pretrained lan- guage models is an active research area (Ding et al. (022). Adapters (Houlsby et al. 2019: Ma- habadi et al. 2021) and its variants (Hu et al. 2021 Karimi Mahabadi et al. (021) insert trainable layers, while BitFit (aken al. 2022) only updates the bias parameters without changing any other model parameters. Diff pruning (Guo et al. 2021) and FISH (Sung et al. 2021) learn sparse updates to the original PLM. Another popular choice is prompt tuning (Lester al. (22T) which only up- dates soft prompt vectors prepended to the input. Prefix-tuning of optimizing continuous prompts for natural language generation tasks is presented in Li & Liang (2021). UNIPELT learn combine different tuning methods via gating mechanism (Mao et al.) (022). HyperPrompt (He et al. 2022) introduces mask-condiiione hyperprompts that condition the model on task-specific information for constructing prompts. LST (Sung et al.) aims to reduce the training memory of parameter-efficicnt tuning by ladder side network. Discrete (i.e., hard) prompts have also been shown to be effective in many cases (Schick & Schütze, 202 la Gao et al. 2021 Malkin et al.) 2022). However, our ap- proach is most related to the transferability of prompts (Wang et al., 2021 et al. 2022 Su et al.) 2022), which focuses boosting the performance of prompt tuning across many tasks. SPoT (Vu et al. 2022) selects one prompt using a similarity measure, and ATTEMPT (Asai et al.) 2022 adopts an attention mechanism
the source prompts initialize the prompt for target task. Unlike ex- isting works, our approach learns a single shared prompt by decomposing and distilling knowledge from source prompts for efficient adaptation diverse of target tasks.
2

---

Published as a conference paper at ICLR 2023
Multitask learning. Multitask learning, which focuses simultancously solving multiple related tasks with a single model, has been studied from multiple perspectives Zhang & Yang 2021 Ruder
2017). A common approach is to transfer model that has been fine-tuned multiple source tasks
to another target task (Vu et al. 2020; Raffel et al. 2020 Aghajanyan et al., 2021a Zhong et al.)
2021 Clark al., 2019b] Singh al.
2022). A few recent works show zero-shot and few-shot
transfer capabilities of language models through massive multitask learning over a large number of
tasks (Sanh al. 2022; Wang et al. 2022; Liu et al. 2022a, Wei et al. 2021). Designing specific parameter shariig strategies is also another recent trend in multitask learning (Ruder et al. 2019]
Sun al.) 2020; Misra et al. 2016). While our proposed approach is inspired by these methods, this paper focuses on multitask prompt transfer for sarfiiter-efficienn adaptation of language models, which still remains challenging and largely understudied problem.
Knowledge distillation. Knowledge distillation has been used to improve performance and effi ciency across many tasks (Gou et al., 202T), including model compression (Hinton al. 2015 Jiao ct al. 2020 (Sanh et al.) (2019). transfer learning (Furlancllo et al. 2018 Xu et al. 2020). machine
translation (Zhou et al. 2019)) question answering (Hu et a011) and document retrieval (Shakeri et al. 2019). Concurrently with our work, PANDA (Zhong al. 2022) uses knowledge distillation
with a new metric to better predict prompt transferability across different combinations of source- target tasks. PANDA focuses on transferring from one source task to another target task using similarity measure (similar to SPoT (Vu et al. 2022)), while our MPT approach leverages multitask learning to better exploit cross-task knowledge for prompt transfer.
APPROACH
Given a set of source tasks S = {S1, S,.,, and target tasks T (T1, }r}. our goal is to learn single soft prompt over S that can be adapted each task Ti in a parameter-efficient way. Simply training a single soft prompt on S and then finetuning on each T; is sub-optimal as it can fail to leverage commonalities across source tasks while minimizing interference the same time. To this end, multitask prompt tuning (MPT) aims to compress task-shared knowledge in S into a single prompt matrix és via knowledge distillation improve performance on while filtering out task-specific information that is less useful for transfer learning.
Prompt tuning. Given a pre-trrined language model with parameters O and one target task with training data (X, YY) = [ yı W. the standard approach is to directly finetune all the parameters by maximizing the conditional probability P +), which can be parameter-ineffccien when considering a group of target tasks T. An alternative that is more parameter-efficient is prompt tun- ing (PT), which randomly initializes a small number of learnable prompt vectors (i.c., soft prompts) to be prepended to the input embeddings of the PLM while freezing model parameters é (Lester et al. 2021 Liu et al. 2022b). Formally, for a sequence of input tokens with token embeddings as T tt, f2,..... tpl Raxd PT prepends a learnable prompt matrix P R with the same dimension as the token embedding d, where is hyperparameter. PT then optimizes the following loss function with respect P.
CPLM = - Σ log Ply: a. O,;
where the input to the language model is given by the concatenated matrix [P:T] E RR++) xd While this approach has been successful for some tasks and models, researchers have observed that vanilla PT can sometimes lead to lower performance (especially on smaller PLMs), slow con- vergence, and high sensitivity to parameter initialization (Lester et al., [2021] Su et al. 2022 Zhong et al. (022). Recent works address these issues by first training prompts on multiple source tasks and then using these prompts to initialize the prompts for target task via some similarity measure (Asai et al. 2022) Vu et al. 2022). We extend this line of work and propose framework for transferring multitask knowledge into single soft prompt enable more performant and parameter-efficiee transfer learning to downstream target tasks T.
3.1 MULTITASK PROMPT TUNING
Our proposed framework, dubbed MPT, consists of two stages: source training and target adapta- tion. MPT first focuses on source training to generate single soft prompt matrix to be reused in the
3

---

Published as a conference paper at ICLR 2023
second stage for target task adaptation. Specifically, prompt matrices for the source tasks are decom- posed into a task-shared matrix and a low-rank task-specific matrix (prompt decomposition), where the former is shared across all tasks. This decomposition into shared and task- components is learned through knowledge distillation. Once learned, the shared prompt matrix is adapted to a downstream target task via low-rank multiplicative updates.
Prompt decomposition. The goal of prompt decom- position is enable efficient knowledge sharing across source tasks S, while still allowing each task to main- tain its own parameters encode sisk-specific knowl- edge. We decompose the soft prompt Ps for the k-th task into two parts, as shown Figure 3 Let P E T denote the shared prompt across all tasks, and further let ug E IK' 2% E IRd be the task-specific vectors for each task k. The task-specific vectors form rank-one matrix WR = Ut & v which has the same dimensions as the shared prompt P The task prompt P for k-th source task then parameterized as:
Task-specific vectors
41
1/2
Shared prompt P
Wi
W2
P
A:
P = P o Wx = (us @ w
(2)
where o denotes the Hadamard product between two ma- trices. Our parameterization of prompt decomposition is inspired by prior low-rank methods (Li al. 2018 Agha- janyan et al., 20216; Wen et al. (2020). such that general information across the set source tasks can be cap- tured by "slow" weights p shared across tasks, while the "fast" weights We could then encode task-specific knowledge for S& in low-rank subspace.
MPT
Figure 3: An illustration prompt decom- position two tasks. The shared matrix P is combined with task tpec-fp vectors to obtain the task-specific prompt matrices P. for t E (1,
Prompt distillation. Learning the prompt decomposition directly from the multitask datasets S tended make the shared component P' overfit larger tasks. We found knowledge distillation from separately-trained source prompts to be an effective strategy for learning good decompos- able prompts. Specifically, we first obtain teacher prompt P pleeacher) for the k-th source task by conventional prompt tuning. We then randomly initialize a corresponding student prompt as PA = p o (#+ 6 vf)). where all student prompts share p and have their own task-specific vectors as described above. We then use distillation transfer cross-task knowledge into the shared prompt matrix (Sanh et al., 2019). The first loss is match the output probability distributions of students and teachers by minimizing their KL-Divergence with respect to the shared prompt matrix P and the task-specific parameters Uy and Uk.
L Lagits Σ
Σ
KL [
[P(Py: yi B1; θ, p(teacherr)
P yi Ri; O P.)
KE|S| (ESS
We use a temperature I' to control the smoothness of the output distribution for both teacher and stu- dent models as Pj ž exp(z;/T") /T"), where is the logit score for class j and Z is the normalization factor. We also have an additional mean squared loss on teacher model hidden states,
LHiddeen Σ Σ
H
H k.
(teacher)
KE|Si (ES.
where H (teacher) and HA. denote the hidden states of teacher and student networks respectively, k. which consist of sequence of hidden vectors for i-th input. Such additional distillation loss from
intermediate states has been shown to improve results in distilling PLMs (Jiao al. 2020 Shheefer
& Rush, 2020). The total loss function for training student source prompts for obtaining a single shared prompt to be transferred to the target side is then,
L LPLM + X(CLogit CHidden),
(5)
where LPLM Eke(s) CLM represents the aggregated task losses for all source tasks, and À is weight to balance the impact of distillation loss terms.
4

---

Published as a conference paper at ICLR 2023
3.2 SOURCE TRAINING AND TARGET ADAPTATION
Training the single source prompt to be transferred to target tasks requires two steps. First, the teacher prompts for all source tasks are pretrained individually through vanilla prompt tuning. Then, we perform multitask training on S = {S,,... S to jointly learn the single shared prompt via the knowledge distillation loss function Equation 5 We also adopt a simple stochastic task sampling strategy, which dynamically changes the number of tasks per batch. For each batch of multitask samples, we randomly select a number K from [2, x| first, then randomly choose K tasks from S and their corresponding samples to constitute mini-batches Such dynamic task sampling strategies are common in the PLM multitask learning literature (Raffel et al., 2020).
For target adaptation, initialize the target prompt for target task To to be the Hadamard product of the shared prompt matrix and the task low-rank prompt matrix, i.e., P, = P (u, @ v. and optimize with the regular task loss in Equation [ with respect to P U2, Vt, where we use separate learning rates for P vs. Uc 8G (see Appedix A). We remark that MPT can also be used for multitask learning on group of target tasks T = (71, T2...., Tr where P is shared across T.
Parameter-efficiency Each task contains the shared prompt a vanilla soft prompt and smaller number of task-specific vectors (1 + d). Thus, the total number of tunable parameters for a single target task is x d) + (1 + d). After training, this can further be compressed into single matrix of size 1 x d For a group of target tasks, the total number of tunable parameters is (1 x d)= + (1 + d)r, where T is the number of target tasks. We list and compare different methods in terms of the number of trainable parameters in Table
that has the same dimensions as
EXPERIMENTS
We conduct experiments across a comprehensive range of NLP datasets to show that MPT outper- forms strong baselines in both full-dataset (Tables 2 and few-shot (Tables 3 adaptations, while being more parameter-efficien compared to existing methods (Figure 2}.
4.1 EXPERIMENTAL SETUP
Datasets and tasks. As in Asai et al. (002) we evaluate MPT using 6 datasets with more than 100k annotations as source tasks: MNLI (Williams et al., 2017), QNLI (Demszky al. 2018).
QQP (Wang et al. (018)) SST-2 (Socher et al. 2013). SQuAD (Rajpurkar et al.] 2016). and
ReCoRD (Zhang et al. 2018). We use 23 datasets from four benchmarks target tasks: Mul-
tiC (Khahhabi et al. 2018). BoolQ (Clark et al. 2019)). WiC (Pilchvar & Camacho-Collados
2018 WSC (Levesque et al. 2012), and CB (De Marnefte et al.,
2019) from SuperGL.UE (Wang
et al.
2019); RTE (Gaamicccolo et al. (007). CoLA (Warstadt et al.
2019).STS-B (Cer et al.
2017)
MRPC (Dolan & Brockett ).05) MNLI, QQP, QNLI and SST-2 from GLUE (Wang et al.
2018);
Natural Questions (Kwiatkowski et al. 2019), HotpotQA (Yang et al.]
2018). NewsQA (Trischler
ct al. (017) and SearchQA (Dunn et al. 2017 from MRQA (Fisch et al. 2019); WinoGrande (Sak-
aguchi et al. 20221), Yelp-2 (Zhang al. 2015). SciTail (Khot et al.) 2018) and PAWS- (Zhang
ctt al. 2019) from the "Others" benchmark in (Asai et al. (022);: and E2E (Novikova et al., (017)) and
WebNL.G (Gardent et al. 2017) for experiments on adapting natural language generation tasks.
Models. Following the standard approach in prompt tuning (Lester et al. 2021 Asai et al. 2022), mainly experiment using the publicly available pretrained T5-Base model with 220M parame- ters (Raffel et al. 2020 We use 100 prompt vectors for all benchmarks (hence Ps E R 1000xd). In our ablation study, we also consider T5-Small (60M) and L5-Large (770M) models.
Baselines. We compare MPT with the following baselines: (1) Full finetuning (FT), where all the model parameters are tuned during adaptation on each downstream task. (2) Vanilla prompt tuning (PT) (Lester et al. (02T), where target prompt vectors are initialized by randomly sam- pled top vocabularies. (3) Existing prompt transfer methods, including SPoT (Vu et al., (022) and ATTEMPT (Asai et al., 2022), which initialize target prompts by retrieving or aggregating source
prompts. (4) Popular parfiiter-efficient methods including Adapters (Houlsby et al., 2019) and
BitFit (aaen et al. 2022). On GLUE, we also compare with several of methods that
However for comparison against prior work we show the number of tunable parameters, i.c.,
5

---

Published as a conference paper at ICLR 2023
Table Results on GLUE and SuperGLUE. The metrics are Pearson correlation for STS-B, (Multi), and accuracy for other tasks as evaluation metrics. MPT results are averaged over three runs, and subscripts denote standard deviation. The column "param/task" represents the number of trainable parameters for each task in GLUE. (Top) Model adaptation to each target task with no parameter sharing on the target side (so params/task for MPT is just (1 + d)). (Bottom) Model adaptation to group of tasks (marked by d) + (1 d). See Section 3.2 for more details.
MultRCC
"), where param/task for MPT
purami
SuperGLA
Fiscuning
199
290K
85.3
50.9
85.5
67.6
70.0
59.6
PT
16.8
$1.3
89.3
62.8
98.9
89.5
68.1
54.7
10.6
75.3
56.7
61.7
43.9
51.9
67.9
$7.8
SPOT
76.8K
85.4
90.1
93.0
93.4
90.0
79.7
09.3
57.3
52.3
1400
77.2
07.8
50.0
45.4
02.9
ATTEMPT
2238
$4.3
00.3
03.0
93.2
$9.7
85.7
73.4
57.4
R3.4
74.4
78.8
66.8
33.8
78.6
70.5
MPT
776K
85,9um
90.3vm
40.3amm
93.4pps
90.66
79.4
62.4co
$5.6
74.
67.7em
79.8
74.6
Tinenning
2531
85.7
32.0
17.5
M022
15.
31.9
Adketess
902
HoperDecoder*
1.83
90.5
83.7
71.7
55.9
ATTEMPT
968
45.8
00.0
63.1
91.3
90.8
86.1
70.9
64.3
65.3
744
76.
66.5
69.2
83.1
74.1
MPT
10.5K
S4.3gz
90.6p3
43.0g24
90.3
90.4g
89.2
62.761
63.5us
55.%
74.Kun
79 79.2ux
70
07.3gm0
76.101
Table 2: Results MRQA and Others. We F1 for MRQA tasks and accuracy for others the evaluation metrics. MPT results are averaged over three runs subscripts indicate standard deviation.
MRQA SQA
Others Sciaall
Method
param/task
NQ
News
Avg.
Yelp
PAWS
Avg.
Fanetuning
220M
81.1
74.7
96.7
93.8
94.1
87.1
Adapters
1.9M
74.2
77.6
81.4
65.6
74.7
59.2
96.9
94.5
94.3
86.2
BiFit
280K
70.7
75.5
71.7
64.1
72.0
57.2
94.7
94.7
92.0
84.7
PT
76.8K
67.9
72.9
75.7
61.1
69.4
49.6
95.1
87.9
55.8
72.1
SPoT
76,8K
68.2
74.8
75.3
58.2
69.1
50.4
95.4
91.2
91.1
82.0
ATTEMPT
232K
MPT
77.6K
72.0g.1
75.80.14
77.2g0
63.7gu
72.2gmm
56.5037
96.4pm
95.5u26
93.5u11
85.5u32
adapt a pretrained model to all the target tasks using multitask learning, such as HyperFomer (aa-) habadi et al., (021). HyperDecoder (Ivison & Peters 2022). multitask variants of FT and Adapters. We directly quote numbers reported published papers when possible or use publicly available
source code (Karimi Mahabadi et al. 2021] Mahabadi et al. 2021 Asai et al. 2022) under the same
backbone and experimental settings for fair comparison.
Implementation details. For source training, train MPT on the mixture of source tasks for epochs with the examples-proportion sampling described in Section 3.2 For prompt distillation, calculate the hidden state loss for hidden states from both the encoder and decoder of T5. For target adaptation, we reuse the shared prompt from MPT and take averaged source task-specific vectors to initialize the target task-specific vector. We run all the experiments three times with different random seeds and report the mean and standard deviations. In few-shot experiments, for each number of shots k randomly sample 10 times from the training set with different random seeds and report the mean performances. Note that for few-shot learning, the source prompt learning still uses the full set of the source tasks, See Appendix A for the full experimental setup including hyperparameters.
mixing strategy (Raffel et al. 2020) and stochastic task
4.2 RESULTS AND ANALYSIS
Full-dataset adaptation. Tables I and 2 show the per-task performance of different methods on all four benchmarks. As seen from Table (top), MPT establishes new state-of-the-art re- sults for parameter-efffcciet finetuning PT [Lester et al. 2021). MPT obtains relative improvement of 13% on GLUE and 16% Super- GLUE with the same number of task-specific parameters, highlighting the benefits of transferring knowledge from multiple source tasks. MPT also consistently outperforms other parameter-efficient methods such as SPoT (Vu al.) 2022), ATTEMPT (Asai et al. 2022), and BitFit (Zaken et al.) 2022). despite updating far fewer parameters. Adapters is the most competitive in terms of aver- age accuracy on both benchmarks, but MPT is far more parameter efficient and requires 4x fewer task-specific parameters. More surprisingly, MPT outperforms the full finetuning baseline on both benchmarks, despite tuning 0.035% as many task-specific parameters. See Figure for the compar- ison against different methods in terms of accuracy and parameter-efficinng
both GLUE and SuperGLUE. When compared vanilla
Table (bottom) shows the results when finetuning against a group target tasks. ATTEMPT and MPT particularly performant in this setting, even when compared against state-of-the

---

Published as conference paper at ICLR 2023
Table Few-shot learning results with & {4, 16. 32) on BoolQ. CB, and SciTail. FT: Finetuning. AD:
Adapters, PT: Prompt tuning, ST: SPoT. HF: HyperFormer, ATP: ATTEMPT. Numbers in brackets denote the number of parameters tuned for each task. MPT is very competitive or even better than existing methods in the majority of the cases while tuning much fewer task task-specific parameters.
k-shot
FT (220M)
AD (1.9M)
PT (76.88)
ST (76,8K)
HP (638K)
ATP (232K)
MPPT (77.6K)
4
50.5
53.4
61.6
50.5
48.0
61.8
62.2
BoolQ
16
56.5
51.4
61.9
50.6
50.2
60.0
63.3
32
58.4
54.5
61.7
61.2
58.3
63.3
68.9
57.7
51.1
53.5
71.4
60.7
82.1
73.6
CB
16
77.0
74.8
63.5
64.3
76.3
78.5
78.6
32
80.0
74.8
67.8
64.3
81.4
85.7
82.1
79.6
79.5
57.7
69.6
82.0
80.2
80.2
SciTail
80.0
83.2
60.8
71.9
86.5
79.5
87.3
81.9
85.0
60.2
71.9
85.8
80.2
86.3
Table 4: Few-shot learning results on GLUE and SuperGLUE for vanilla prompt tuning (PT) and MPT with 4, 16, and 32 training examples. MPT consistently outperforms PT, demonstrating the generalizability of MPT
prompts tasks with only few training examples.
ksshot
Method
MNLI
QOP
QNLI
8ST-2
GLUE
STS-B
MRPC
RTE
CoLA
Avg.
Mulii
BoolQ
SuperGLLL
WiC
WSC
CB
Avg.
40.1
40.4
68.1
27.4
54.7
61.8
61.6
51.2
60.4
57.7
MPT
59.4
82.0
86.2
56.5
89.1
68.1
62.6
31.8
67.3
62.2
62.2
52.9
67.3
73.6
63.6
PT
41.5
59.9
68.1
28.5
60.3
48.9
44.2
49.8
32
PT
37.0
623
56.7
509
875
68.1
54.7
23.2
55.1
59.2
61.7
52.0
67.3
67.8
61.7
multitask baselines such as HyperFormer (Mahabadi et al.] 2021) and HyperDecoder (Ivison & Peters 2022). which train single model on different target tasks. This reveals the potential of our MPT to further leverage multitask knowledge on the target side, enabling even more parameter- efficient adaptation of pretrained language models.
Table shows the performance of different methods on the MRQA and Others benchmark. Our approach significantly improves the average performance of PT by +2.8% on MRQA and +13.5% on the Others benchmark, while adding only 0.01% % more task-specific parameters. Similarly, MPT obtains 85.5% average accuracy on WinoGrande, Yelp, SciTail, and PAWS, outperforming BitFit (84.7%), which updates 10x more task-specific parameters. When we increase the prompt length from 100 300, we also found an average improvement of 0.8% on MRQA and 0.6% Others, closing the gap between MPT and Adapters. While our improvements being highly parameter- efficient are encouraging, the accuracy gap between MPT and the full finetuning still significant in MRQA, which indicates opportunities for future work in multitask prompt tuning.
Few-shot adaptation. Following prior works (Mahabadi et al., 202T Asai et al. (022). we first conduct few-shot experiments on BoolQ, CB, and SciTail tasks to measure how the pretrained MPT prompts can generalized to new tasks with only a few training examples available (k = 4, 16, 32). Table shows the results of Adapters, HyperFormer, PT, and SPoT. As can be seen from Table vanilla PT struggles for few- shot adaptation (esp., CB and SciTaiil,, suggesting that randomly initialized prompts generalize to new tasks with only few labeled examples. SPoT improves the performance of PT on CB and SciTail tasks, and MPT outperforms both PT and SPoT. We also observe that other methods in Table (Finetuning, Adapters, HyperFormer, and ATTEMPT) have trouble in the few-shot set- ting. Moreover, Table the few-shot learning performance comparison between PT and MPT on all the GLUE and SuperGLUE tasks. As shown in Table 4 we can observe that not only MPT outperforms the vanilla PT by a large margin most the datasets, but also MPT can perform very well on many datasets to reach their full-dataset performance with 16 or 32 shots, such as QQP, QNLI, STS-B, and WSC. These results clearly indicate that MPT can effectively use cross-task knowledge in source tasks target tasks where there only few labeled examples.
approach and other baselines, which includes full finetuning,
hard to
Natural language generation tasks. We next conduct experiments to test whether prompt de- composition learned from source NLU tasks can generalize to target NLG tasks, We transfer the T5-Large prompt trained using diverse source tasks to two NLG tasks: E2E (Novikova et al, 2017) and WebNLG (Gardent et al. 2017). Table shows that our proposed MPT significantly outperforms standard PT (Lester et al. 202 on both NLG tasks across all the metrics. Our BLEU improvements PT are 3.03% and 5.25% on E2E and WebNLG G tasks respectively, showing the

---

Published as a conference paper at ICLR 2023
Table Results on NLG tasks. The source prompt decomposition learned against NLU tasks adapted to target NI.G tasks. MPT consistently outperforms PT on both tasks.
E2E
WebNLG
BLEU
NIST
METEOR
Rouge-L
CIDEr
BLEU
METEOR
TER
PT
29.11
5.00
0.343
51.50
1.72
46.02
0.37
46.89
MPT
32.14
5.35
0.363
52.88
1.86
52.27
0.40
41.36
FT
Adaster
PT
ATTEMPT
MPT
90
90
NuMMC
MLRRC
Accuracy
Real
Accuracy
70-
70
20
W5C
WSC
50
Molii.
Bool
vec.
wat.
Mullc
40
TS-Small
TS-aaae
Large
15 15.smal
Stoe
13 Large
15 15-small
75-0a5e
13 L3-Lagg
SPaT
MPP
MultiRc
Boolo
Figure (Left) Performance various baselines as function model size (from T5-Small to T5-Large).
(Right) Correlation of prompt matrices on SuperGLUE tasks. Best viewed color.
effectiveness of our approach on both NLU (e.g.. classification, NLL, QA tasks) and NLG tasks. This an impressive result, particularly since the source tasks are all NLU tasks, i.e., MPT can transfer knowledge from NLU tasks to NLG tasks.
Model scaling. We conduct scaling experiments to analyze how MPT performs with increasing pretrained model sizes three SuperGLUE tasks as in Asai et al. (2022). Figure 4 (left) shows the performance of MPT as well as full finetuning (FT), Adapter, PT, and ATTEMPT with three different T5 models (T5-Small, T5-Base, T5-Large). These results show that MPT not only able to achieve the best parameter efficiency, but also is effective across different model scales ranging from 60M 770M parameters.
Analyzing prompt matrices. We conduct qualitative analyses on prompts learned using MPT to investigate whether cross-task knowledge is indeed encoded in the task-shared prompt, making easier for target tasks to effectively adapt and encode their own knowledge. Following Vu et al. (2022), we use the prompt matrices to compute cosine similarities between pairs target tasks after adaptation, where each task is represented by the composition of task-shared and task-specific prompts (averaged to obtain a single vector). Figure (right) shows the visualization of cosine similarity matrices for SPoT and MPT SuperGLUE tasks. We find that task embeddings can effectively cluster similar tasks together (e.g.. MultiRC is similar to BoolQ).
4.3 ABLATION STUDIES
Prompt decomposition and distillation. Table 6 presents the results SuperGLUE. where fix all the hyper-parameters across all settings and rerun MPT source training to get various ablated versions of the transferred prompt.
To measure the effect of prompt decomposition, we re-
Table 6: Ablation results prompt decom- position and distillation.
place the vanilla source prompt with our decomposable prompt of task-shared and task-specific components and train it without prompt distillation (third row Table (). which gives 3.5% average performance improvement on SuperGL.UF. over the baseline (first row in Table (6. This ablation clearly demonstrates the importance of the prompt decomposition strategy in MPT and shows that the shared component can effectively capture the rich cross-task knowledge that is beneficial for target downstream tasks.
Decomposiion
Disullation
SuperGLUE Avg.
69.5
70.6
73.0
74.1
To test the effect of prompt distillation, we train a vanilla prompt shared by all the source tasks with the same training loss of MPT in Equation The teacher models are kept the same for this ablation and MPT. Compared with the simple baseline (first row Table (6. adding prompt distillation (second row) produces 1. 1.1% average performance improvement. Furthermore, observe that
8

---

Published as a conference paper at ICLR 2023
Table 7: MPT performance MRQA and Others with more source tasks.
MRQA
Others
NQ
HP
SQA
News
Avg.
WG
Yelp
SciTail
PAWS
Avg.
MPT (w/6 6 Source Tasks)
72.0
75.8
77.2
63.7
72.2
56.5
96.4
95.5
93.5
85.5
MPT (w/ 12 Source Tasks)
72.1
76.4
77.9
64.0
72.6
56.6
96.8
95.9
92.9
85.6
prompt distillation combined with prompt decomposition yields the best average performance of 74.1% on the SuperGI.UE benchmark. This confirms that distilling knowledge from separately- trained source prompts is an effective strategy for learning good decomposable prompts.
Distillation objective. We further investigate the individual components of prompt distillation to measure their influences on the final performance. We remove the loss of hidden states from Equa- tion 5 and find that it produces average performance of 73.7% on SuperGLUE, verifying the effectiveness of regularizing hidden states in conjunction with logits to reach its full performance. which is consistent with findings in [anhh al. (2019). Finally, consider variant of distillation loss to match the teacher and student prompts directly by adding an MSE loss to minimize the dis- tance between the two prompts. Replacing our proposed distillation losses with this prompt distance loss and jointly training it with prompt decomposition yield an average SuperGLUE performance of 73.6%, which performs worse than the distillation losses based on logits and hidden states.
Prompt length. While our experiments use / = 100 prompt vec- tors, we show in Figure 5 that using longer prompts obtains im- provements up to / = 300, reaching 76.8% on SuperGLUE. How- ever, further increasing the prompt length from 300 to 400 leads to an absolute 1.8% drop in accuracy, possibly due to overfitting.
77.5
75.0
Avg. SuperGLUE
72.5
70.0
Target adaptation strategy. When transferring the shared prompt from source to target tasks, we find that only updat- ing task-shared component (i.e., removing task-specific vectors) or only updating task-specific vectors (i.e., freezing task-shared component) produces suboptimal results (62.5% and 71.3% on SuperGLUE). This shows the importance of updating both com- ponents (which have different learning rates) for target adaptation.
67.5
PT
MPT
65.0
62.5
60.0
57.5
100
200
300
Prompt Length
Stochastic task sampling. MPT uses multitask training strat- egy in Section 3.2, which stochastically samples a number of
Figure 5: Performance on Super- GLUE as a function of prompt length for PT and MPT.
tasks within each mini-batch. Ablating the stochastic task sam- pling results in 73.7% on SuperGLUE (lower than the full perfor- mance of 74.1%), which demonstrates the slight benefit of this simple multitask training strategy.
Number of source tasks for pretraining. For our main experiments, we selected 6 NLP tasks fol- lowing[Asai et al. (0022). To investigate the effect of more source tasks, we incorporated 6 additional diverse source tasks on top of the original 6 tasks, including topic classification (AGNews (Zhang al. 2013)), multi-choice QA (CommmonsenseCA) (Talmor et al.) 2019), Open (Mihaylov et al. 2018), ARC (Clark et al. 2018)), adversarial (ANLI (Nie et al. 2020)) and commonsense reasoning (Winogradde (Sakaguchi et al. 202I)). Table shows the results on MRQA and Others benchmarks. MPT with 12 tasks is still quite effective for target adaptation on both benchmarks, slightly outperforming MPT trained using 6 tasks. While is unclear how much MPT would ben- efit from even more source tasks, it would be interesting to see whether MPT trained on large-scale benchmarks such as CrossFit (Ye et al., 2021 -which consist of 160 NLP tasks-can enable even more parameter-efficient (and accurate) transfer learning.
5 CONCLUSION
We introduced and studied multitask prompt tuning (MPT), which learns a single transferable prompt by decomposing and distilling knowledge from multiple source tasks and their task-specific source prompts. MPT decomposes the task prompt as the Hadamard product of a shared prompt matrix and a rank-one task-specific matrix. The shared component is then transferred and adapted to target tasks for further tuning. Empirically transfer learning to target downstream tasks across diverse NLP benchmarks, even outperforming the full finetuning baseline in some cases, despite tuning much fewer task-specific parameters.
found this approach enables parameter-efffiient

---

Published as a conference paper at ICLR 2023
ACKNOWLEDGEMENTS
We are grateful to the anonymous reviewers for their constructive comments and suggestions. ZW sincerely thanks Peihao Wang for the insightful discussion during the internship. YK partially supported by an MIT-IBM Watson Al grant. We also acknowledge support from the IBM Research Al Hardware Center and the Center for Computational Innovation Rensselaer Polytechnic Institute for the computational resources on the AiMOS Supercomputer.
REFERENCES
Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 5799-5811, 2021
Armen Aghajanyan, Sonal Gupta, and Luke Zettlemyer. Intrinsic dimensionality explains the ef- fectiveness of language model fine-tuning. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 1 11th International Joint Conference on Natu- ral Language Processing \Volume Long Papers), pp. 7319-7328, 2021b.
Akari Asai, Mohammadreza Salchi, Matthew E Peters, and Hannaneh Hajishirzi. Attentional mix- tures of soft prompt tuning for parameter-efficient multi-task knowledge sharing. arXiv preprint arXiv:2205. 1961, 2022.
Daniel Cer, Mona Diab, Eneko Agirre, Inigo Lopez-Gazpio, and Lucia Specia. Semeval-2017 task 1: Semantic textual similarity-uutilineee and cross-lingual focused evaluation. arXiv preprint arXiv: /708. 00055, 2017.
Christopher Clark, Kenton Lee, Ming-Wer Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv: 1905. 100.: 2019a.
Kevin Clark, Minh-Thang Luong, Urvashi Khandelwal, Christopher D Manning, and Quoc Le. Bam! born-again multi-task networks for natural language understanding. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pp. 5931-5937, 2019b.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge. arXiv preprint 11Xiv:1803.0000
2018.
Marie-Catherine De Marneffe, Mandy Simons, and Judith Tonhauser. vestigating projection in naturally occurring discourse. In proceedings of Sinn und Bedeutung, volume 23, pp. 107-124, 2019.
The commitmentbank: In-
Dorottya Demszky, Kelvin Guu, and Percy Liang. Transforming question answering datasets into natural language inference datasets. arXiv preprint arXiv:1809.02990 2018.
Jacob Devlin, Ming-Wes Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL, 2019.
Ning Ding, Yujia Qin, Guang Yang, Fuchao Wei, Zonghan Yang, Yusheng Su, Shengding Hu, Yulin Chen, Chi-Min Chan, Weize Chen, et al. Delta tuning: A comprehensive study of parameter efficient methods for pre-trained language models. arXiv preprint arXiv:2203.06904, 2022.
Bill Dolan and Chris Brockett. Automatically constructing a corpus of sentential paraphrases. In Third International Workshop Paraphrasing (IWP2005), 2005.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint 17055:1704.05179 2017.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of 2nd Machine Reading for Reading Comprehension (MRQA) Workshop at EMNLP, 2019.
10

---

Published as a conference paper at ICLR 2023
Tommaso Furlanello, Zachary Lipton, Michael Tschannen, Laurent Itti, and Anima Anandkumar. Born again neural networks. In International Conference on Machine Learning, pp. 1607-1616, 2018.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners In Proceedings of the 59th Anmual Meeting of the Association for Computational Lin- guistics and the Ith International Joint Conference on Natural Language Processing ( (Volume I: Long Papers), pp. 13816-3830, 2021.
Claire Gardent, Anastasia Shimorina, Shashi Narayan, and Laura challenge: Generating text from rdf data. In Proceedings the 10th International Conference on Natural Language Generation, 2017.
The webnlg
Danilo Giampiccolo, Bernardo Magnini, Ido Dagan, and William B Dolan. The third pascal rec- ognizing textual entailment challenge. In Proceedings of the ACL-PASCAL, workshop on textual entailment and paraphrasing, pp. 1-9, 2007.
Jianping Gou, Baosheng Yu, Stephen Maybank, and Dacheng Tao. Knowledge distillation: A survey. International Journal of Computer Vision 1796))1789-1819 2021.
Demi Guo, Alexander M. Rush, and Yoon Kim. Parameter-efficient transfer learning with diff pruning. In Proceedings of ACL, 2021.
Yun Hc, Steven Zheng, Tay, Jai Gupta, Yu Du, Vamsi Aribandi, Zhe Zhao, YaGuang Li, Zhao Chen, Donald Metzler, et al. Hyperprompt: Prompt-baset cask-oonditooning of transformers. In International Conference on Machine Learning, pp. 8678-8690, 2022.
Geoffrey Hinton, Oriol Vinyals, Jeff Dean, et al. Distilling the knowledge in neural network. arXiv preprint arXiv: 2(7), 2015.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, An- drea Gesmundo, Mona Attariyan, and Sylvain Gelly. Parameter-efficient transfer learning for nlp. In International Conference on Machine Learning, pp. 2790-27999 2019.
Jeremy Howard and Sebastian Ruder. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 928-339, 2018.
Edward J Hu, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, et al. Lora: Low-rank adaptation large language models. International Conference on Learning Representations, 2021.
Minghao Hu, Yuxing Peng, Furu Wei, Zhen Huang, Dongsheng Li, Nan Yang, and Ming Zhou.
Attention -uuided answer distillation for machine reading comprehension. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2077-2086, 2018.
Hamish Ivison and Matthew E Peters. Hyperdecoders: Instance-specific decoders for multi-task nlp. arXiv preprint (xXix:2203.08300 2022.
Xiaoqi Jiao, Yichun Yin, Lifeng Shang, Xin Jiang, Xiao Chen, Linlin Li, Fang Wang, and Qun Liu. Tinybert: Distilling bert for natural language understanding. Findings of the Association for Computational Linguistics: EMNLP 2020, pp- 4163 4174, 2020.
Rabeeh Karimi Mahabadi, James Henderson, and Sebastian Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022- 1035, 2021.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. Look- ing beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Com- putational Linguistics: Human Language Technologies. Volume I (Long Papers), pp. 252-262 2018.
11

---

Published as a conference paper at ICLR 2023
Tushar Khot, Ashish Sabharwal, and Peter Clark. Scitail: A textual entailment dataset from science question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32, 2018.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. Natural questions: benchmark for question answering research. Transactions of the Association for Computational Lingnistics, 7:453 466, 2019.
Brian Lester, Rami A1-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 3045-3059 2021.
Hector Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge. In Thir- leenth international conference on the principles of knowledge representation and reasoning, 2012.
Chunyuan Li, Heerad Farkhoor, Rosanne Liu, and Jason Yosinski. Measuring the intrinsic dimension of objective landscapes. In International Conference on Learning Representations, 2018.
Xiang Lisa Li and Perey Liang. Prefix-tuning: Optimizing continuous prompts for generation. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers),
pp. 4582-4597, 2021.
Haokun Liu, Derek Tam, Mohammed Muqeeth, Jay Mohta, Tenghao Huang, Mohit Bansal, and Colin Raffel. Few-shot parameter-efficient fine-tuning is better and cheaper than in-context learn- ing. arXiv preprint arXiv:2205.0566
2022a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre- train, prompt, and predict: A systematic survey of prompting methods in natural language pro- cessing. arXiv preprint arXiv:2107.13586, 2021a.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning can be comparable to fine-tuning universally across scales and tasks. arXiv preprint 10X002210..0762. 2021b.
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning: Prompt tuning can be comparable to fine-tuning across scales and tasks. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 61-68, 2022b.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghani, and James Henderson. Parameter- efficient multi-task fine-tuning for transformers via shared hypernetwors.. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing ( (Volume 1: Long Papers), pp. 565-576, 2021.
Nikolay Malkin, Zhen Wang, and Nebojsa Jojic. Coherence boosting: When your pretrained lan- guage model not paying enough attention. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 8214 8236, 2022.
Yuning Mao, Lambert Mathias, Rui Hou, Amjad Almahairi, Hao Ma, Jiawei Han, Scott Yih, and Madian Khabsa. Unipelt: A unified framework for parameter-efficient language model tuning. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 6253-6264, 2022.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct elec- tricity? new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 2381-2391, 2018
Ishan Misra, Abhinav Shrivastava, Abhinav Gupta, and Martial Hebert. Cross-stitch networks for multi-task learning. In Proceedings the IEEE conference on computer vision and pattern recog- nition, pp. 39949994494 4003, 2016.
12

---

Published as a conference paper at ICLR 2023
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. Adversar- ial nli: A new benchmark for natural language understanding. In Proceedings of the 58th Anmual Meeting of the Association for Computational Linguistics, pp. 4885 4985- 2020.
Jekaterina Novikova, Ondiej Dušck, and Verena Rieser. The e2c dataset: New challenges for end- to-end generation. arXiv preprint arXiv:1706.0922: 09254. 2017.
Mohammad Taher Pilchvar and Jose Camacho-Collados Wic: the word-in-context dataset for eval- uating sontext-sensitiee meaning representations. arXiv preprint wrXiv:1808.09122 2018.
Edoardo M Ponti, Alessandro Sordoni, and Siva Reddy. Combining modular skills in multitask
learning, arXiv preprint arXiv:2202.13914 2022.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the limits of transfer learning with a unified text-to-text transformer. Mach. Learn. Res., 21(140):1-67, 2020.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing. pp. 2383 22922 2016.
Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv preprint arXiv:1706.05098, 2017.
Sebastian Ruder, Joachim Bingel, Isabelle Augenstein, and Anders Søgaard, Latent multi-task chitecture learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 4822-4829, 2019.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. Winogrande: An adver- sarial winograd schema challenge at scale. Communications of the ACM, 64(9)999-106, 2021.
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108, 2019.
Victor Sanh, Albert Webson, Colin Raffel, Stephen Bach, Lintang Sutawika, Zaid Alyafeai, Antoine Chaffin, Arnaud Stiegler, Teven Le Scao, Arun Raja, al. Multitask prompted training enables zero-shot task generalization. In The Tenth International Conference on Learning Representa- tions, 2022.
Timo Schick and Hinrich Schütze. Exploiting cloze-questions for few-shot text classification and natural language inference. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pp. 255-269, 2021a.
Timo Schick and Hinrich Schütze. It's not just size that matters: Small language models are also few-shot learners. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp. 2339-2352, 2021b.
Siamak Shakeri, Abhinav Sethy, and Cheng Cheng. Knowledge distillation in document retrieval. arXiv preprint 4rXiv:1911.11665 2019.
Sam Shleifer and Alexander M Rush. Pre-trained summarization distillation. arXiv preprint arXiv:2010.13002. 2020.
Janvijay Singh, Fan Bai, and Zhen Wang. Frustratingly simple entity tracking with effective use of multi-task learning models. arXiv preprint TrXiv:2210.06444 2022.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language pro- cessing, pp. 1631-1642, 2013.
13

---

Published as a conference paper at ICLR 2023
Yusheng Su, Xiaozhi Wang, Yujia Qin, Chi-Min Chan, Yankai Lin, Huadong Wang, Kaiyue Wen, Zhiyuan Liu, Peng Li, Juanzi Li, et al. On transferability of prompt tuning for natural language processing. In Proceedings of the 2022 Conference of the North American Chapter of the Associ- ationfor Computational Linguisties: Human Language Technologies, pp. 3949-3969, 2022.
Ximeng Sun, Rameswar Panda, Rogerio Feris, and Kate Saenko. Adashare: Learning what to share for efficient deep multi-task learning. Advances in Neural Information Processing Systems, 33: 8728 8740, 2020.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. In Advances in Neural Information Processing Systems.
Yi-Lin Sung, Varun Nair, and Colin A Raffel. Training neural networks with fixed sparse masks. In Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and Wortman Vaughan (eds.), Advances in Neural Information Processing Systems, volume 34, pp. 24193-24205. Curran Associates, Inc., 2021. URL https::/ppocceediiiiiiii neurins. 567 3017333cc55111 cb2653f548f8709558888 15r.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqaa A question answering challenge targeting commonsense knowledge. Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguisties: Human Lan- guage Technologies, Volume (Long and Short Papers), pp. 4149-4158, 2019.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pp. 191-200, 2017.
Tu Vu, Tong Wang, Tsendsuren Munkhdalai, Alessandro Sordoni, Adam Trischler, Andrew Mattarella-Micke, Subhransu Maji, and Mohit lyyer. Exploring and predicting transferabilityy across nlp tasks. In Pruceedings of the 2020 Conference on Empirical Methods in Natural Lan- guage Processing (EMNLP), pp. 7882-7926, 2020.
Tu Vu, Brian Lester, Noah Constant, Rami Al-Rfou, and Daniel Cer. Spot: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Asso- ciation for Computational Linguistics (Volume 1: Long Papers), pp. 5039-5059, 2022.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding. Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Newral Networks for NLP, pp. 353-355, 2018.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy. and Samuel Bowman. Superglue: A stickier benchmark for general-purpose language understanding systems. Advances neural information processing systems, 32, 2019.
Chengyu Wang, Jianing Wang, Minghui Qiu, Jun Huang, and Ming Gao. Transprompt: Towards an automatic transferable prompting framework for few-shot text classification. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 2792-2802, 2021.
Yizhong Wang, Swaroop Mishra, Pegah Alipoormolabashi, Yeganeh Kordi, Amirreza Mirzaei, An- jana Arunkumar, Arjun Ashok, Arut Selvan Dhanasekaran, Atharva Naik, David Stap, et al. Benchmarking generalization via in-context instructions on 1,600+ language tasks. arXiv preprint arXiv:2204.07775, 2022.
Alex Warstadt, Amanpreet Singh, and Samuel R Bowman. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625 0411,611,77 2019.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du An- drew M Dai, and Quoc V Le. Finetuned language models are zero-shot learners. In International Conference on Learning Representations, 2021.
Yeming Wen, Dustin Tran, and Jimmy Ba. Batchensemble: an alternative approach to efficient ensemble and lifelong learning. In International Conference on Learning Representations, 2020.
14

---

Published as a conference paper at ICLR 2023
Adina Williams, Nikita Nangia, and Samuel R Bowman. A broad-coverage challenge corpus for sentence understanding through inference. arXiv preprint arXiv:1704.05426, 2017.
Guodong Xu, Ziwei Liu, Xiaoxiao Li, and Chen Change Loy. Knowledge distillation meets self- supervision. In European Conference on Computer Vision, pp. 588-604, 2020.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. Proceedings of the 2018 Conference on Empirical Methods Natural Language Processing, pp. 2369, 2018.
Qinyuan Ye, Bill Yuchen Lin, and Xiang Ren. Crossfit: A few-shot learning challenge for cross-task generalization in nlp. arXiv preprint arXiv:2104.0883. 2021.
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. Bitfit: Simple parameter-efficien fine-tuning for transformer-baset masked language-models, Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp. 1-9, 2022.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint 11f0.:1811.288888 2018.
Xiang Zhang. Junbo Zhao, and Yann LeCun. Character-level convolutional networks for text clas- sification. Advances in neural information processing systems, 28, 2015.
Yu Zhang and Qiang Yang. A survey on multi-task learning. IEEE Transactions on Knowledge and Data Engineering, 2021.
Yuan Zhang, Jason Baldridge, and Luheng He. Paws: Paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers),
pp. 1298-1308. 2019.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. Panda: Prompt transfer meets knowledge distillation for efficient model adaptation. arXiv preprint arXiv:2208. 2022.
Ruiqi Zhong, Kristy Lee, Zheng Zhang, and Dan Klein. Adapting language models for zero-shot learning by meta-tuning on dataset and prompt collections. In Findings of the Association for Computational Linguistics: EMNLP 2021, pp. 2876-2878, 2021.
Chunting Zhou, Jiatao Gu, and Graham Neubig. Understanding knowledge distillation in non- autoregressive machine translation. In International Conference on Learning Representations, 2019.
15

---

Published as a conference paper at ICLR 2023
A EXPERIMENTAL SETUP
For initial training of source prompts, we train MPT on the mixture of source tasks for epochs with the propppespprooottionall mixing strategy (Raffel al. 2020) and stochastic task sampling. For prompt distillation, we calculate the hidden state loss for hidden states from both the encoder and decoder of T5. For target adaptation, we reuse the shared prompt from MPT and take averaged source task- saecifpecific vectors to initialize the target task-specific vector. We train 20 epochs on small datasets, 10 epochs on large (more than 10k examples) datasets, and 5 epochs on the MRQA datasets. We run all the experiments three times with different random seeds and report the mean and standard deviations. few-shot experiments, for each number of shots k, we randomly sample 10 times from the training set with different random seeds and report the mean performances. For the few-shot setting, the source prompt learning still uses the full set of the source tasks.
During source training, we set the default learning rate as 0.3 for both task-shared and task-specific components. However, during target adaptation, we use a strategy two-speed learning rates for those two components, in Ponti al. (2022). Specifically, for the task-shared and task-specific components, respectively, during target task adaptation. Fol- Dowing[Lested et al. (202T), initialize the teacher and student prompts by randomly sampling tokens from T5's vocabulary (Raf-) fel et al. 2020). We set the default batch size for T5-Base as 32 and for model scaling experiments, the batch sizes for T5-Small and T5-Large are 100, and 12 respectively. The default input length for most tasks are set to 256, except MultiRC and MRQA benchmarks have input length of 348 and 512. We set the distillation loss coefficient À in Equation 5 to 0.9 and keep it fixed for all our experiments.
set the learning rate to 0.3 and 0.4
the default number of tunable tokens per each prompt to 100 and
For all datasets, we use the development set as the testing set if the original testing set is not publicly available. If the training set is small, we split the original development set into the development and testing set; otherwise, we separate a development set from the training set and use the original development set for testing. We limit the number of training data for Yelp to 100k.
16


Reviews:
[{"review_id": "t9U64P2p6U", "reviewer": "Reviewer_Tz5t", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on the prompt tuning of Transformer language models in multi-task settings. They proposed a method named multitask prompt tuning (MPT), which aims to enhance the transferability of source prompts (i.e., the learned soft prompts for source tasks). Specifically, they learn a single transferable prompt by knowledge distillation from multiple source tasks and then learn the rank-one matrix for adapting the transferable shared prompt to a given target task -- prompt decomposition, distillation, and adaptation. On the benchmark GLUE and SuperGLUE, they compare the proposed MPT method with many other prompt tuning baselines. The MPT outperforms the baseline methods and has a smaller number of the parameters.  ", "summary_of_the_review": "Overall, I enjoy reading the paper. The idea is pretty novel and its performance is very good, especially when we consider the parameter efficiency. It has a few limitations which are not stated and covered, though. Also, I think the evaluation can be further improved according to my above suggestions. ", "strength_and_weaknesses": "## Strength\n\n- A novel method for multi-task prompt tuning. The design of the prompt decomposition and distillation is intuitive and reasonable. \n- Great empirical results on GLUE and SuperGLUE. The proposed method MPT outperforms many recent baseline methods for prompt tuning. \n- Comprehensive analysis with ablation studies and qualitative analysis with heat maps. \n\n## Weakness\n- The evaluation does not consider the NLG tasks, such as those in the GEM benchmark. This can be a big limitation. \n- The evaluation is based on the setup where only 6 source tasks are used. This is a pretty small size and it seems that many source tasks are closely related to each other. I would suggest authors use benchmarks such as CrossFit (Ye et al. 2022) to do a more large-scale analysis, where the transferring is more challenging as some source tasks can be relatively less related to the target tasks. \n- The current method design only considers a single shared prompt for transfer. I think when you have a large number of source tasks, this can be a weak point. As it is less likely that a very diverse set of tasks can use a single prompt to share all knowledge.  ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "sxn0Kvr7Pl", "reviewer": "Reviewer_FfyU", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper presents a soft (continuous) prompt tuning method called MPT. In traditional soft prompt tuning, prompts are often sensitive to initialization when trained from scratch and performance may still lag behind full model fine tuning. In this work, the manuscript presents a method for multitask prompt tuning where a single soft prompt is learned that can be transferred to target tasks. The authors find a low rank decomposition based on a source task matrix and a task-specific low rank matrix is more performative than sharing the prompts directly across tasks. This decomposition is is learned via a knowledge distillation style approach.\n\nThe authors evaluate performance on 21 NLP datasets reflecting a variety of tasks and report significant improvements on SuperGLUE vs vanilla prompt tuning, along with all the smaller training parameter benefits of parameter efficient transfer learning. They further find that MPT performs well in few-shot learning for models in the 60M to 770M parameter space. The paper presents comprehensive ablation experiments", "summary_of_the_review": "I think this paper makes several nice empirical contributions, is very clearly written with comprehensive evaluations and ablation experiments. This covers some reasonable questions in the space of soft prompt tuning and MTL so it merits acceptance. ", "strength_and_weaknesses": "Strengths\n- Parameter-efficient transfer learning is an important research area \n- The prompt decomposition method is quite straightforward (decomposition + distillation)\n- Comprehensive evaluation (21 datasets) and baseline methods\n- Nice breadth of additional experiments (few-shot performance, LM param scaling, prompt lenght)\n- Ablation studies highlight benefits of combining decomposition + distillation\n\nWeaknesses\n- Ideally the manuscript would explore some class of larger language models (3 - 11B param range), though this presupposes some level of compute that is not available to all researchers, so it is not a strong criticism.\n- Experiments would benefit from replicates to characterize variance.\n- The core methods aren't super novel (decomposition + distillation), but the combination seems to provide empirical benefits.  \n- Code is not immediately available. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "jpo_q3ZnKI", "reviewer": "Reviewer_6YdE", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposed a new method for multi-task prompt tuning, which uses source tasks to learn a single shared prompt and then adapts to target tasks with decomposition and distillation. The design of the decomposition makes the resulting prompting learning more performant yet more parameter-efficient. ", "summary_of_the_review": "See above. ", "strength_and_weaknesses": "Summary Of Strengths\n- the paper is clearly written and presented;\n- the idea of leveraging decomposition is new and insightful, which not only makes the prompt learning more performant but results in fewer parameters. \n- extensive ablations are provided (decomposition, distillation, adaptation strategy, and training strategy) to illustrate the design choices, which may pave the road for future researchers and practitioners in the prompt learning area. \n- the efficacy of the decomposition and distillation for multi-task prompt tuning is verified across benchmarks (GLUE, SuperGLUE, MRQA) and scales (up to 700M); \n\nSummary Of Weaknesses\n- the additional training compute is unclear compared to fine-tuning, is the fine-tuning also conducted using the same schedule as MPT; Also, in the SPoT paper, they found the best results were achieved with multi-task fine-tuning 79.2 (T5-base), is it also a valid baseline to compare with?\n- the poor baseline performance \n\n   (i) It seems the baseline of BitFit / LoRA / Adapter performs worse than the one reported in [1], could the author elaborate on the reasons?\n\n   (ii) Also, is there any intuition  why SPoT yields such worse performance on SuperGLUE, which should be comparable to Model Tuning in the original paper (though they use more source tasks compared to the one used here)\n\n- the adaptation in Table 1 and 2 is still per-task adaption but not in a multi-task manner (the design choice of this is unclearly presented), and how to select the group is unclearly presented; \n\n- the generalization of the proposed method is uncertain, is it limited to T5-variants, or is it also applicable to GPT (casual mask) models?\n\n- will the variance of different runs also be given in Table 1 and 2, which can help show the significance of the results? \n\n- for the few-shot setting, is the source prompt learning still using the full set of the source tasks, or is that also few-shot?\n\n[1] Sung, Yi-Lin, Jaemin Cho, and Mohit Bansal. \"Lst: Ladder side-tuning for parameter and memory efficient transfer learning.\" NeurIPS (2022).\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "O53URwU9Sx", "reviewer": "Reviewer_iVfY", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies multitask prompt tuning to learn better soft prompt representations for tasks. Their proposed approach has two stages: 1) They first train a single source prompt representation for each individual task using the conventional prompt training approach; 2) then they learn a shared prompt representation and task-specific prompt representations on all tasks by applying the prompt distillation from the teacher prompt obtained from step #1. They evaluated their approach on a few well-established benchmarks, including GLUE and SuperGLUE, and demonstrated their prompt tuning approach is better than the prior prompt tuning approaches with less parameters. ", "summary_of_the_review": "The paper proposes a novel multitask prompt tuning approach. Their approach shows better effectiveness and efficiency compared to prior prompt tuning baselines. The results & findings can be more convincing if the weaknesses can be resolved. ", "strength_and_weaknesses": "Strengths:\n- A novel multitask prompt tuning approach by the separation of shared and task-specific prompt representations and knowledge distillation. The idea is simple and technically sound.\n- Evaluations show their approach achieves better results than prior prompt tuning approaches with fewer parameters on widely-adopted benchmarks. Their approach is also on-par or slightly better than the finetuning baseline. \n- Their approach demonstrates significantly better few-shot capability than finetune baseline and other prompt tuning approaches.\n- Code & data will be released.\n\nWeaknesses:\n- Compared to Adapter or finetuning baseline, the proposed approach is still worse on certain datasets (table 1 & 2). It would be better to show if the performance gap w.r.t. Adapter can be closed by adding the same number of prompt parameters as the Adapter.\n- It would be more convincing if the few-shot experiments can be performed on GLUE and SuperGLUE, instead of 3 separate datasets. Similar to the scaling experiment (i.e., Figure 4)\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster