Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning
Authors: Haeju Lee
Minchan Jeong
Se-Young Yun
Kee-Eung Kim
Venue: EMNLP 2023 Findings (2023)
Keywords: prompt tuning
parameter-efficient fine-tuning
transfer learning
bayesian method
TL;DR: Our method constructs a global posterior distribution over soft prompts from the source tasks, utilizing it as a prior for the target task to leverage the source posterior for guidance.

Abstract: Prompt tuning, in which prompts are optimized to adapt large-scale pre-trained language models to downstream tasks instead of fine-tuning the full model parameters, has been shown to be particularly effective when the prompts are trained in the multi-task transfer learning setting. These methods generally involve individually training prompts for each source task and then aggregating them to provide the initialization of the prompt for the target task. However, this approach critically 
ignores the fact that some of the source tasks could be negatively or positively interfering with each other. We argue that when we extract knowledge from source tasks via training source prompts, we need to consider this correlation among source tasks for better transfer to target tasks. To this end, we propose a Bayesian approach where we work with the posterior distribution of prompts across source tasks. We obtain representative source prompts corresponding to the samples from 
the posterior utilizing Stein Variational Gradient Descent, which are then aggregated to constitute the initial target prompt. We show extensive experimental results on the standard benchmark NLP tasks, where our Bayesian multi-task transfer learning approach outperforms the state-of-the-art methods in many settings. Furthermore, our approach requires no auxiliary models other than the prompt itself, achieving high degree of parameter-efficiency.

Bayesian Multi-Tans Transfer Learning for Soft Prompt Tuning
Haeju Lee
Minchan Jeong!
Se-Young Yun!
Kee-Eung Kim
Kim Jaechul Graduate School of AI, KAIST
(1hg912, mcjeong, yunseyoung, kekim)ekaist.ac.kk
Abstract
Prompt tuning, in which prompts are optimized adapt large-scale pre-traine pre-trained language mod- els to downstream tasks instead of fine-tuning the full model parameters, has been shown to be particularly effective when the prompts are trained in the multi-task transfer learning set- ting. These methods generally involve individ- ually training prompts for each source task and then aggregating them to provide the initializa- of the prompt the target task. However, this approach critically ignores the fact that some of the source tasks could be negatively positively interfering with each other. We argue that when we extract knowledge from source tasks via training source prompts, we need to consider this correlation among source tasks for better transfer to target tasks. To this end, we propose Bayesian approach where we work with the posterior distribution prompts across source tasks. We obtain representative source prompts corresponding the samples from the posterior utilizing Stein Variational Gradient Descent, which are then aggregated constitute the initial target prompt. We show extensive experimental results on the standard benchmark NLP tasks, where our Bayesian multi-task transfer learning approach outper- forms the state-of-the-art methods in many set- tings. Furthermore, our approach requires no auxiliary models other than the prompt itself, achieving high degree of parameter-efficiency:
$2
S3
81
S4
SVGD Particles
for Task
Adaptution
Transfer Leaming from Average Procupt
BMTPT
Two key steps for Bayesian Multi-Task Prompt Tuming (BMTPT) are illustrated. First, we merge the posterior distributions of each source task to form global posterior distribution. This distribu- tion is approximated using Stein Variational Gradient Descent (SVGD), particle-based variational inference method. Finally, we adapt to the target task by using the derived posterior from the source tasks prior.
Figure 1:
One of the promising approaches is prompt tun- ing (PT, Lester et al. 2021), where few adaptable vectors added as prompts to the input of the downstream task (Lester al., 2021: Li and Liang, 2021). PT freezes the PLM model parameters and limits the learning to prompts, yet achieves im- pressive performance. However, it still challeng- ing to achieve the same level of performance as the full fine-tuning, as well as to mitigate sensitivity to initialization (Zhong et al., 2022).
1 Introduction
Large-scale pre-trained language models (PLMs) have been recently fine-tuned for various NLP tasks (Devlin et al., 2019; Raffel et al., 2020a). Due the computational challenges of training the extensive parameters in PLMs, there is growing focus on methods that efficiently tune fewer pa- rameters (Houlsby et al., 2019; Ben Zaken et al., 2022).
To address these challenges, recent works (Wang al., 2023; Asai al., 2022; Vu al., 2022) proposed to adopt multi-task transfer learning ap- proach, where the prompt trained from multiple source tasks to be applied to the target task. Specif- ically, they train valued vectors for prompts
"These authors contributed equally this work !oode: https:// /tt::/gitttit Sonn/heyzude/BMP

---

However, these strategies may not fully address the inherent heterogeneity within source task distri- butions. They can falter, especially when attempt- ing aggregate prompts that have been trained across various tasks. Particularly, these issues per- sist even when each source task's posterior distribu- tion follows Gaussian distribution. Further discus- sion on this subject can be found in Appendix A. (i.e. soft prompts) on source tasks and use them the initialization of prompt for the target task. However, it is unclear whether aggregating such addividually-trainee prompts provides a reliable ini- tialization point and fully harnesses the benefits of multi-task transfer learning.
vectors that are prepended the input. We can formally describe PT as follows: consider an in- put sequence æ, and a soft prompt Rixd with length and dimension d, which matches the lan- guage model's (LM) embedding dimension. The soft prompt is prepended to the sequence æ and then processed by the LM, resulting in the predic- tion of the target sequence y.
In this paper, we propose Bayesian Multi- Task Prompt Tuning (BMTPT) as a practical yet effec- tive solution to this challenge. Unlike traditional methods of prompt tuning grounded in transfer learning, our approach engages with the posterior distribution of prompts across a multitude of source tasks. For the transference of knowledge gleaned from source tasks, we utilize the source prompts' posterior distribution the prior for the designated target task. This Bayesian method of transfer learn- ing augments the conventional transfer learning framework, which primarily learns the initialization point of the target prompt from the source tasks. Specifically, BMIPT employs Stein Variational Gradient Descent (SVGD, Liu and Wang 2016), a particle-based Variational Inference (V1) method, to approximate the source prompts' posterior dis- tribution. Further elaboration on this method is provided in Section 2.2.
Our work aligns closely with recent efforts transfer soft prompts from source tasks in order initialize prompts for target tasks. For instance, SPoT (Vu et al., 2022) retrieves a source task prompt based on similarity to initialize the target task prompt, while ATTEMPT (Asai et al., 2022) employs an attention mechanism to initialize the prompt for the target task using information from the source prompts" The most recent method for prompt tuning transfer, MPT (Wang al., 2023), decomposes source prompts into shared and task- specific parts to reduce interference between tasks during aggregation.
We validate our approach through experiments on 21 datasets across diverse NLP tasks and output formats. The experimental results demonstrate that BMTPT achieves comparable or superior perfor- mance to strong state-of-the-arr parameter-efficcon fine-tuning methods (Asai et al., 2022; Wang et al.. 2023) as well full fine-tuning, while utilizing very small number of parameters and requiring no auxiliary models other than the prompt itself.
Hence, an integrated approach regarding source task distributions may prove advantageous if rep- resentative knowledge set can be constituted for transfer to the target task. This paper takes Bayesian approach to transferring prompts from source tasks to target tasks. Instead of learning prompts individually then aggregating them, we use the full posterior distribution of prompts across the source tasks. Since this is intractable, we ap- proximate the posterior via sampling, and leverage these samples for training the prompt for the target task, which corresponds to setting the posterior the prior of the target prompt.
2 Background
2.1 Transfer Learning for Prompt Tuning
Fine-tuning entire models for downstream NLP tasks, particularly with Large Language Model (LLM), can be expensive in terms of training costs. Therefore, efffieter-efffcient tuning focuses on limiting the updates to a small set of parame- ters. Various approaches have been proposed, such as Adapter (Houlsby et al., 2019) and its vari- ants (Karimi Mahabadi et al., 2021a; Hu et al., 2022) that involve inserting trainable layers, and BitFit (Ben Zaken et al., 2022) that only trains bias weights while keeping other weights intact.
2.2 Particle Based VI and SVGD
Variational Inference (VI) is widely used ap- proach in machine learning for distribution ap- proximation, notable in Bayesian Neural Networks (BNNs) (Blundell et al., 2015; Graves, 2011). De- spite its computational simplicity, it often restricts
Recently, there has been growing interest in prompt tuning (PT). This approach involves up- dating only the 'soft prompt', a set of continuous
AAlhhough ATTEMPT includes a randomly initialized tar- get prompt in the attentional mixture, the argument in this section still applics.

---

the family of distributions, a limitation that less present in methods like MCMC (Gilks al., 1995; Doucet et al., 2001; Robert and Casella, 2004). Particle-based VI methods provide an alterna- tive approach by drawing upon the strengths of both VI and MCMC. Unlike traditional VI methods, particle-based VI does not restrict itself to spe- cific family of distributions. This flexibility allows it approximate wider range of complex and diverse distributions (Liu and Wang, 2016; Zhang et al., 2020; Naesseth et al., 2018). However, its theoretical guarantees are not yet fully understood. Assumptions often made, such as the presence of in- finite particles or adherence to simple distributions like Gaussian, may not hold in practical scenar- ios (Nacsseth et al., 2018; Salim et al., 2022; Sun al., 2022: Liu et al., 2023).
soft prompt 0 is prepended the sequence æ. prior to its processing by the LM. The underlying objec- tive is to predict the target sequence y. We denote the dataset for the k-th source as D and define DS - - The target task is represented DJ during task adaptation. Thus, the i-th instance in the D dataset will be represented as (af, v²) ). Note that the log-likelihood log p(D) (05) for the k-th source task can be represented as follows:
(D
log p( = Σ
log PLM [0S; af1)
In this formulation, PLM denotes the likelihood de- termined by the LM and the corresponding crite- rion.
Next we state our Bayesian objective, aiming to optimize the target task prompt using the posterior of source prompts for a transfer learning scheme.
Stein Variational Gradient Descent (SVGD, Liu and Wang 2016) is a signifieant advancement in particle-bused VI. SVGD applies a transformation to particles to make them more representative of the target distribution through an iterative process. Specifically, for let particles tend to position them- selves as though they were samples drawn from the distribution p, The update rule of SVGD is described as follows:
Problem Statement. The objective is to maximize the posterior probability of the target prompt 07, as expressed by the following equation:
argmax P(D")10") "p0""''') 0T
(1)
where p(DT10") is the likelihood and p(0"
0, + O: + (0i) where (7 is
is the prior that is learned from the source tasks in prior the target task adaptation:
M
kOO 0.Vo + Vo, ( 0.),
p(0" (D$) PL p(0 (0S) p(0
M
(2)
where k(---) is the positive definite kernel function like RBF
In this context, the prior distribution p(O (DS) serves as a guide for the target task adaptation. We model p( as the multivariate Gaussian with mean 0$, since without any information on the target task, it is natural to have AT = 0S.
Despite its merits, SVGD can face mode col- lapse (Chen and Ghattas, 2020; Liu et al., 2022).
One workaround. Damped SVGD (Ba et al., 2022),
mitigates this by adjusting the deterministic bias in the update rule, which is used in our work. For a more thorough mathematical explanation, kernel details, and information about damped SVGD, we direct readers to Appendix B.
This problem formulation provides a general framework subsuming conventional transfer leamn ing method for prompt tuning. For example, we could approximate the above integral in Eq. (2) defining the prior on 0J using a prompt trained from source tasks g i.e. ((B$(DS)=00)) which would be roughly equivalent to the conven- tional transfer learning setting where the source prompt serves as the initialization of the target prompt.
(0S).
3 Problem Setting
In this section, we formally introduce core ele- ments, symbols, and problem statements that form the basis of our approach. We denote the trainable parameter of the soft prompt 0 R charac- terized by its length and the dimension d of the Language Model (LM). For clarity, we use g and 8" to denote the soft prompts for source tasks and target tss, respectively. This implies that the
Assuming an uninformative prior for the source prompt 0 (e.g. uniform distribution) as well independent selection of source tasks, the posterior distribution p(031 DD) for source tasks is formu- lated as the product of the posteriors of each task.

---

Remark. Assuming the uniform prior for 0 and independent selection of source tasks, the global posterior p(OS (D$) is proportional to the product of posteriors:
Algorithm 1 Bayesian Multi-Task Prompt Tuning
Input:
DS DJ source tasks and target task
K
O (00.) M
initialized particle set
p(03 D. = UKE DJ)
II r(0' D.). A=1
Source Posterior Learning: for t f- to T do
4 Approach
@+1 f- O of p(-(Ds) (et) (SVGD iteration; Section 2.2)
Instead of optimizing individual prompts for each source task in isolation, our method primarily re- volves around learning the posterior distribution of source prompts across all source tasks. This ap- proach assigns larger probability mass to those prompts capable of addressing greater number of source tasks, thereby potentially becoming more suitable candidate prompts for the target task as well. We implement this concept by using particles to approximate the posterior distribution. The fol- lowing subsections provide a detailed explanation of this methodology.
end for
Store 0 + Or, for all i E [M]
Target Task Adaptation:
J(07) = - log poDpp(D'00) log 1
M Σ p(0 (0,))
T < argming ((T)
Output:
T trained weight for the target task
4.1 Main Strategy
where a and are constants arising from the multi- variate isotropic Gaussian assumption of p(0 (OS). Combining Eq. (4) and Eq. (5), the final loss for target adaptation is therefore:
The optimization of the target task prompt is mod- eled as MAP inference in Eq. (1) using p(0" (DS) the prior. We approximate this with M parti- cles {0$) Ji=1 (each particle corresponds to soft prompt) drawn from p(- (DS) using SVGD:
argmin Ø
log ((DT((1)))))
0 0$1
p(0T = E [p(oT10s) 055~~0 p(. [D.)
202
Monte Carlo
Sampling
M
p(0 (0%)
where 0
This objective suggests that, during target adaptation, we can initialize 0J with the average value of the optimized particles T Øs.
1 EM M
0.
M
For task adaptation, i.e. obtaining the prompt for the target task, the objective Eq. (1) is achieved based on the approximation provided by Eq. (3):
argmin log p(D) 0T)) -log Dogp(07 (DS)
4.2
Additional Strategies
8T
M
4.2.1 Source Task Sampling
log (D - log M Σ ((0 i=1
argmin
As transfer learning prepares for unknown arbi- trary target tasks, usually is considered preferable that various source tasks are learned. However, if the number of source tasks increases, we have to calculate the training losses of all source tasks. Therefore it is necessary to alleviate the bottleneck coming from a large number of source tasks. To this end, we use an approximate posterior distribu- tion instead of the true global posterior distribution. Specifically, during each source posterior learning iteration, we uniformly sample ε tasks from the K source tasks (x < K) without replacement and constitute a batch with the data entries from that k tasks.
=(T(
(4)
The pseudo-code of BMTPT algorithm is shown in Algorithm 1.
For practical purposes, we can minimize the second term of the objective J (OT) by applying Jensen's 's incquality, as demonstrated below:
-log
T
log P 1
M
M
1=1
1
TT
S
+
2
M4
(5)

---

4.2.2
Composition of o and Multi-target Task Adaptation
Eistimation (MLE) loss with respect D7 and mi- nus of the average of log priors.
At the start of the target adaptation, we compose 0² with element-wise multiplication of full-rank matrix which is initialized with 0 and low-rank matrix whose elements are all where both matri- ces are learnable and have the shape of (1, d). The low-rank matrix is made by ab" where a = 1' and b = 1d and both a, b are trainable components. Importantly during target adaptation, we adopt two-speed learning rate scheme for the full-rank and low-rank matrices by setting higher learning rate for the low-rank matrix (Ponti et al., 2022; Asai al., 2022; Wang et al., 2023). This facil- itates multi-target task adaptation, by employing multiple low-rank matrices to assign each low-rank matrix to each target task, while sharing the full- rank matrix among all target tasks. In doing so, the full-rank matrix captures the shared knowledge across tasks, while the respective low-rank matrices capture the task-specific knowledge (Wang al., 2023). We also apply this scheme single-target adaptation, as we empirically observed that the use of two-speed learning rate promotes faster perfor- mance convergence.
4.4 Efficiency of BMTPT
Recent prompt tuning transfer methods primarily focus on measuring the efficiency during target adaptation, overlooking the need to evaluate the efficiency of source task training phase, which is helpful for identifying potential bottlenecks. We highlight the efficiency of BMIPT in comparison to the most recent prompt tuning transfer methods, ATTEMPT (Asai et al., 2022) and MPT (Wang et al., 2023), in both source and target stage. It is noteworthy that both methods require additional neural networks beyond soft prompts during either source task training target adaptation: MPT in- volves a teacher network that is of the same size as the LM backbone it uses distillation during source task training, and ATTEMPT involves the training of an attention module during target adap- tation.
BMTPT, on the other hand, proves be efficient in both the source posterior learning and target adaptation stages, when evaluated under criteria of computational and space complexity. The ad- ditional intricacies that BMTPT introduces, com- pared to vanilla prompt tuning, are the use of SVGD during source posterior learning and the computation of regularization terms derived from the prior during target adaptation (Eq. (6)). In terms of computational complexity, given that the SVGD step used in BMTPT primarily involves computing RBF kernel values among a limited number of par- ticles, the computational cost is minimal. Likewise, the regularization calculation during target adap- tation is also negligible. On the aspect of space complexity, BMTPT continues exhibit efficiency. During source posterior learning, BMTPT companies SVGD particles only, the memory space that BMTPT requires is occupied by the backbone LM parameters and the SVGD particles which are comprised of M d trainable parameters. Since we employ a small number of particles, the memory consumption by SVGD particles is almost negligi- ble. During target adaptation, as we compose one target task prompt with shared matrix (full-rank) and task-specific matrix (low-rank), BMTPT quires (1 d)/N + (1+d) trainable parameters per one target task, when we adapt on N target tasks. This makes BMTPT train only 0.035% parame- ters compared to full fine-tuning. For a detailed
4.3
Training Process
4.3.1
Source Task Posterior Approximation
Unlike previous transfer learning methods in prompt tuning that individually train prompts for each source task, we approximate the global pos- terior distribution of source tasks, by employing M particles. Here, a particle corresponds to one instance of soft prompt. Each particle is initialized with randomly sampled tokens, following Lester et al. (2021). We pack batch as the following: each particle 0 (which is an instantiation of soft prompt, and S i S M) is prepended to input texts from K' source tasks, forming a batch of size M K. It is worth noting that the p(-) in the SVGD update rule can be interpreted as the minus of the cross-entropy loss of the language model when given the input with the particle (prompt) prepended. Also, we employ limited number of SVGD particles, usually M < 10. We perform 100K SVGD updates to sample @s.
4.3.2 Target Task Adaptation
With the initialized AT, we start target task adapta- tion. The loss for the adaptation process is Eq. (6), which is the combination of Maximum Likelihood

---

analysis, direct the reader to Appendix C.
motivation is demonstrated by the experimental results in Section 6.
4.5
Contrasts and Contributions
4.5.1
Constrast with Conventional Multi-Tash Learning
5 Experiment
5.1 Datasets and Tasks
Both BMTPT and traditional multi-task learning algorithms have common point in that they utilize multi-source data. However, BMTPT uses multi- source data to find posterior distribution across the multi-source data and transfer the posterior to target domain, under the Bayesian perspective. Tra- ditional multi-task learning methods, on the other hand, optimize network parameters with respect to MLE objectives in general.
As in previous works (Asai et al., 2022; Wang et al., 2023), We use set of 6 extensive datasets as souree tasks and assess the performance of our algorithm on a range encompassing entailment, paraphrase detection, sentiment analysis, question answering (QA), and commonsense reasoning.
distinet target tasks,
Source Tasks During source posterior learning, the following datasets from GLUE (Wang et al., 2019b), SuperGLUE (Wang et al., 2019a), and MRQA 2019 shared task (MRQA; Fisch et al. 2019), comprising over 100,000 annotations in total. Specifically, we utilize source tasks, MNLI (Williams et al., 2018), QNLI (Dem- szky et al., 2018), QQP (Wang et al., 2019b) and SST-2 (Socher et al., 2013) from GLUE, SQuAD (Rajpurkar et al., 2016) from MRQA, and ReCoRD (Zhang et al., 2018) from SuperGLUE.
4.5.2 Distinction from MPT
BMTPT constructs soft prompt using full-rank and low-rank matrices during the target adaptation stage, similar to MPT (Wang et al., 2023). How- ever, the methodologies diverge in their application and intent. MPT separates shared (full-rank) and task-specific (low-rank) components during source training with the underlying intuition that discover- ing common patterns among various source tasks can promote efficient transfer. After source train- ing, MPT re-uses the shared component and aver- ages task-specific components to initialize full-rank and low-rank matrices, then applies element-wise multiplication of the full and low-rank matrices to form a target prompt. BMTPT, on the other hand, does not use the decomposition at source poste- rior learning. We employ SVGD particles which are instantiations of full-rank prompts, to learn the source posterior. Then, the beginning of target adaptation, we prepare a full-rank matrix and low- rank matrix to form a target prompt. The full-rank matrix initialized with the average of SVGD par- ticles and the low-rank matrix is initialized with 1 ixd (see Section 4.2.2). Notably, the intention be- hind this prompt decomposition is different from that of MPT; our aim is only to facilitate multi target task adaptation.
Target Tasks For target adaptation, we test our algorithm with 21 datasets from four benchmarks: MNLI, QQP, QNLI, SST-2, RTE (Gi- ampiccolo et al., 2007), CoLA (Warstadt et al., 2019), STS-B (Cer et al., 2017) and MRPC (Dolan and Brockett, 2005) from GLUE; BoolQ (Clark et al., 2019), CB (de Marneffe et al., 2019), Mul- üRC (Khashabi et al., 2018), WiC (Pilehvar and Camacho-Collados. 2019) and WSC (Levesque et al., 2012) from SuperGLUE; Natural Questions (NQ; Kwiatkowski et al. 2019), HotpotQA (HQ; Yang et al. 2018), NewsQA (News; Trischler et al. 2017) and SearchQA (SQA; Dunn et al. 2017) from MRQA; WinoGrande (Sakaguchi et al.. 2020), Yelp-2 (Zhang et al., 2015), SciTail (Khot et al., 2018) and PAWS-Wiki (Zhang et al., 2019) from the "Others" benchmark Asai et al. (2022). We direct readers to Appendix for the performance and analysis on MRQA and "Others" benchmarks.
4.5.3 Unique Motivation behind BMTPT
At its core, BMTPT aims to focus on the essence of transfer learning by enhancing it via transferring prior for target adapta- tion. On the other hand, existing prompt transfer methods such as SPoT, ATTEMPT, and MPT tend to rely on the transferability between individual NLP tasks (c.g., SQuAD is more helpful for solv- ing MRPC than SST-2). The efficacy of this unique
useful distribution
5.2 Implementation Details and Baselines
Implementation Details Throughout the experi- ments, we use TS-base as the base LM for BMTPT and all of the baselines, and we use prompt length 100. Unless specified differently, we

---

GLUI
SupeGiLIE
Mehood
# Punmm
MNLI
QNEI
S51-2
STS-B
MRIK
REB
COLA
ArE
Mulkire
BoolQ
WKC'
WSC
CB6
Ang.
Vine tuning
LMI
$6.8
91.6
93.0
94.6
$9.7
90.2
71.9
61.8
$4.9
72.8
81.1
70.2
59.6
55.7
73.9
Atlapiers
1.M
N6.5
90.2
93.2
90.7
85.3
71.9
64.0
75.9
H2.5
67.1
67.3
76.7
BidFi
280K
65.3
90.1
93.0
94.2
90.9
56.8
67.66
58.2
63.3
74.5
79.6
70.0
59.6
78.5
72.5
PT
76.8K
$1.3
89.7
92.8
90.9
$9.5
68.1
54.7
10.6
72.2
58.7
61.7
18.9
51.9
67.9
57.8
PT
76.8K
$5.8
86.9
93.2
92.9
90.5
$7.1
77
83.2
72.2
77.
67.3
72.3
SPOT
76.8K
65.4
90.1
93.0
93.4
$0.0
79.7
69.8
57.1
62.3
74.0
77.2
67.0
50.0
46.4
62.9
ATTEMPT
232K
$4.3
90.3
93.0
93.2
$9.7
$5.7
73.4
57.4
K3.1
74.1
78.8
66.3
53.6
78.6
70.5
MPT
27.6K
$5.9
90.3
93.8
90.4
89.1
79.4
74.8
79.6
69.0
79,8
74.1
BMIPT (Ours)
77,6K
86.2as5
90.3,
93.4, 11
94.Am
90.9
87.2g1
N1.3
M6.6.
$8.7
72.4g
80.5
67.460
67.5um
35.7
74.6
Fine-suning"
91.1
75.4
Aibplers*
1.9M
86.3
90.5
93.2
93.0
$9.9
90.2
20.3
61.5
64.4
Hyppatoommer
90.0
93.0
25.4
HypetDecoder*
76.5K
56.0
90.5
93.4
94.0
90.5
$7.7
71.7
55.9
53.7
ATTEMPT
232K
K3.8
90.0
93.1
03.7
90.8
86.1
79.9
64.3
85.2
74.4
78.3
66.5
69.2
$2.1
74.1
MPT
77.6K
93.0
82.7
79.2
70
BMTPT* (Ours)
77.6K
85.90m
90.2
83:2
95.Aus
91.2uz
66.9,s
80.9:44 85.6c.9
$8.7
72.3531
80.1032
67.704
67.bun
$9.5um
75.3
Table 1: Experiment results for GLUE and SuperGLUE using T5- along with the number of trained parameters. BMTPT results are averaged across three runs, with subscripts indicating the standard deviation. The evaluation metrics Pearson correlation for STS-B, F1 for MultiRC, and accuracy for the other tasks. Top rows single- task adaptation with no parameter sharing during the target task adaptation, while bottom rows employ multi-task adaptation. The best performance among parameter-efficient fine-tuning methods is bolded. BMTPT consistently outperforms most baselines in GLUE and is comparable in SuperGLUE, affirming its robustness across language tasks.
GLUE
SuperGLUB
R-shot
Method
MNLI
QOP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
Avg.
Multire
BooiQ
WiC
WSC
CB
Avg.
PT
40.1
63.2
40.4
53.0
38.8
68.1
56.3
27.4
54.7
61.8
61.6
51.2
60.4
53.5
57.7
MPT
82.0
86.2
56.5
89.1
62.6
34.8
62.6
73.6
63.6
BMTPT (Ous)
43.0
82.4
89.2
60.3
90.0
76.7
55.8
67.8
70.7
60.6
62.7
56.1
67.3
78.6
65,1
PT
41.5
62.3
59.9
50.9
87.8
68.1
54.7
28.5
56.7
60.3
61.9
48.9
44.2
63.5
55.8
16
MPT
61.6
84.7
90.6
63.2
89.1
70.1
64.8
32.1
69.5
64.5
63.3
49.8
67.3
78.6
64.7
BMTPT (Ours)
65.2
85.5
91.3
70.9
89.7
77.0
63.5
68.4
76.4
60.4
63.7
62.4
67.3
75.0
65.8
PT
37.0
62.3
56.7
50.9
87.5
68.1
54.7
23.2
55.1
59.2
61.7
$2.6
67.3
67.8
61.7
322
MPT
63.6
88.5
91.0
75.9
89.7
74.5
59.7
30.8
71.7
63.3
68.9
53.9
67.3
82.1
67.1
BMIPT (Ours)
88.9
91.6
89.1
90.4
78.2
59.4
67.4
64.2
82.1
Table 2: Few-shot experiment results for GLUE and SuperGLUE using T5-base, using 4, 16, and 32 training instances. BMTPT results are averaged across three runs. In tasks with limited training data, BMTPT consistently surpasses MPT and prompt tuning.
GLUE
SuperGLUE
BMTPP Variations
MNLI
QQP
QNLI
SST-2
STS-B
MRPC
RTE
CoLA
Avg.
Multre
BoolQ
WIC
WSC
CB
Avg.
Standard BMTPT
86.2
90.3
93.4
94.4
90.9
87.2
81.3
86.6
88.7
72.4
80.3
67.4
67.3
85.7
74.6
RMTPP w/ large
89.1
90.9
94,1
95.5
92.3
89.3
85.6
87.7
90.6
76.5
84.4
72.4
67.3
85.7
76.8
BMTPT w/ TS. 3B
92.3
91.4
94.3
95.2
93.3
89.4
85.7
89.4
91.4
79.4
88.3
73.7
67.3
89.3
79.6
Source task sampling (1)
86.2
90.1
92.9
94.3
91.2
88.2
81.3
83.6
88.5
71.8
80,9
66.8
67.3
85.7
74.5
Source task sampling (2)
85,8
90.3
93.1
94.8
90.9
88.3
80.4
86.9
83.8
72.0
80.8
69.1
67,3
83.7
75.0
w/ 10 particles
85.7
90.4
93.3
93.8
90.8
90.8
77.0
85.2
88.4
72.7
78.9
68.6
67.3
83.1
74.1
w/o price
85.3
87.1
93.0
94.3
90.9
88.4
79.7
83.9
87.8
71.9
78.2
66.8
67.3
82.1
73.3
Table 3: Table corresponding Section 6.2. We examined BMTPT in larger models and evaluated three components of BMTPT: source task sampling, performance based on the number of particles, and the prior.
employ 5 particles for SVGD and use 6 source tasks as mentioned in Subsection 5.1, therefore forming a batch of size 30 (5 x 6). Also we use a =105 for target adaptation loss, denoted at Eq. (6). For two-speed learning rate, we set 0.3 as the full-rank matrix learning rate and 0.4 as low-rank matrix learning rate. We use a batch of size 32 during target adaptation. For multi-target task adaptation, first form a batch of input texts from target tasks, using example-proportional
mixing strategy (Raffel et al., 2020b), then prepend corresponding target prompt to each input text in the batch. We ran all the experiments three times using different random seeds and provide the mean and standard deviations of the results. In cases where a dataset lacks a publicly available test split with annotations, we adopt either the original development set our test set or perform split within the original development set to create separate development and test sets, following

---

Mahabadi al. (2021).
The results presented in the bottom part of Ta- ble 1 demonstrate the ability of BMTPT to ef- fectively utilize multi-task knowledge during fine- tuning on a group of target tasks. This highlights that BMTPT can benefit from multi-target adap- tation setting, by further reducing the number of trainable parameters.
Baselines We conduct a comprehensive compar- ison of BMTPT with various baseline methods, including full finetuning (FT), vanilla prompt tuming (PT) (Lester et al., 2021), existing prompt transfer methods such SPoT (Vu et al.. 2022), ATTEMPT (Asai et al., 2022) and MPT (Wang al., 2023), well as popular parameter-effiieet approaches like Adapters (Houlsby et al., 2019) and BitFit (Ben Zaken al., 2022). On GLUE, we additionally compare with several statc-of- the-art multi-task learning methods including HyperFormer (Karimi Mahabadi et al., 2021b) and HyperDecoder (Ivison and Peters, 2022), along with multi-task variants of FT and Adapters. Also, to compare our algorithm with conventional multi-task transfer learning, we implement and evaluate a vanilla multi-task transfer method that learns a single prompt upon the combined loss of source tasks and transfers it to the target task. We either directly quote reported numbers or utilize publicly available source code under the same backbone for a fair comparison, as outlined in the respective papers (Mahabadi et al., 2021; Karimi Mahabadi et al., 2021b; Asai et al., 2022: Wang al., 2023).
We also compare the performance of BMTPT and vanilla multi-task transfer that introduced Scction 5.2, in Table 1. Surprisingly, vamilla multi- task transfer shows strong performance in GLUE and SuperGLUE tasks, outperforming competitive baselines. This result supports Section 2.1 which claims that previous methods (Vu et al., 2022; Asai et al., 2022; Wang et al., 2023) not the op- timal transfer technique. It is worth noting that BMTPT outperforms vanilla multi-task transfer. To understand this advantage, we may delve into the Bayesian perspective of BMTPT, which includes conventional transfer learning. While vanilla multi- task transfer only learns an initialization point that contains relatively limited source task informa- tion (Shwartz-Ziv al., 2022), BMTPT learns pos- terior from the source tasks and adopts it as prior during target adaptation, enabling richer and more insightful adaptation process.
6.1.2 Few-Shot Experiments
6 Results
We also present the results of the few-shot experi- ments on the GLUE and SuperGLUE datasets. For the 4-shot experiments, the learning rates were re- duced to one-third of their original values to accom- modate the decreased batch size relative to standard experiments. The performance figures for BMTPT are averaged over three runs, each initialized with different random seed. These outcomes suggest that the prior used in target adaptation effectively positions the prompts an optimal initial point for task adaptation low-resource conditions.
In Section 6.1, we provide the main findings on GLUE and SuperGLUE benchmarks. For findings on MRQA and "Others" benchmarks, please refer to Appendix D. In Section 6.2, we further provide set of analyses.
6.1 Main Results
6.1.1 GLUE and SuperGLUE
As shown in the top part of Table I, BMTPT achieves new state-of-the-art results in parameter- efficient fine-tuning for both GLUE and Super- GLUE, outperforming other prompt tuning transfer methods (Vu et al., 2022; Asai et al., 2022; Wang et al., 2023). Compared to vanilla PT (Lester et al., 2021), BMTPT demonstrates a relative improve- ment of 16.5% on GLUE and 16.8% on Super- GLUE. This highlights the advantages of trans- ferring knowledge using Bayesian approach. It is worth mentioning that BMTPT outperforms the full fine-tuning baseline on both benchmarks, despite only tuning 0.035% of the parameters compared to full fine-tuning.
6.2 Analyses
6.2.1 Model Scaling
We perform scaling experiments to analyze the performance of BMTPT as the size of the pre- trained model increases. The result demonstrates that BMTPT can largely benefit from scaling LM to larger models. This aligns with the finding by (Lester et al., 2021), which suggests that prompt tuning is effective especially when applied larger backbone LMs. Note that BMTPT achieves compa- rable performance to fully fine-tuned models even

---

with T5-base, meaning that BMTPT is effective across various model scales.
performance, implying the efficacy of learning an informative source posterior and leveraging it dur- ing target adaptation to facilitate effective transfer learning.
6.2.2 Effectiveness of Source Task Sampling
To evaluate the effectiveness of Source Task Sam- pling discussed in Section 4.2, we conducted exper- iments under two settings: (1) subsampling 3 tasks from a pool of source tasks (refer to Section 5.1), to examine if Source Task Sampling can mitigate performance degradation limited computation resource scenario, and (2) diversifying the source task set include 12 tasks and subsampling tasks from this expanded set, to investigate the potential benefits of Source Task Sampling with an expanded source task set. For the second setting, we expand the source task set with AGNews (Zhang et al.. 2015), CommonsenseQA (Talmor et al., 2019), OpenBookQA (Mihaylov et al., 2018), 8), ARC (Clark et al., 2018), adversarial NLI (Nie et al., 2020), and Winogrande (Sakaguchi et al., 2020).
7 Conclusion
We present Bayesian Multi-Task Prompt Tuning (BMTPT), a Bayesian approach for transferring soft prompt. Our method defines a posterior distribution over prompt on source tasks, and approximates the posterior using SVGD, then initializes the target prompt with aggregation of source prompts while regularizing the training of the target prompt using transferred posterior, Empirically we found this approach achieves comparable or superior performance over strong parameter-efficient fine-tuning baselines.
Limitations While showing compelling ex- perimental results with only the use of soft prompt, BMTPT has its limitations. Primarily, appending the soft prompt to the input text leads to an extension in the overall input length, consequently increasing the memory footprint. This is a well-known issue PT (Karimi Ma- habadi et al., 2021a), and BMTPT is not immune to it. Furthermore, in BMTPT, since multiple particles are used and source task sentences are appended to each, the batch size grows to be a multiple of the number of particles. This could potentially increase memory demands during the source posterior learning. However, this can be mitigated by implementing Source Task Sampling or reducing the number of particles. Experiments determining the optimal number of particles have not been performed in our study, and future research could potentially explore this aspect to ascertain the most appropriate number of particles.
From Table 3, we can see that setting (1) shows minimal performance degradation compared to the case with 6 source tasks. This finding indicates the successful application of the Source Task Sam- pling technique in low computation resource sce- narios. Also, setting (2) demonstrates slight perfor- mance enhancements, suggesting that Source Task Sampling can derive benefits from diversifying the source task set.
6.2.3 BMTPT Performance on Different Numbers Particles
Since SVGD is a particle-based VI method, the number of particles employed may affect the perfor- mance of our method. Therefore we investigate the effect of the number of particles on target adapta- tion performance by comparing 5-particle BMTPT and 10 10-particle BMTPT (Table 3). We found that the 10-particle case does not yield better results than the 5-particle case. Because of the instabil- ity reported in the original SVGD paper (Liu and Wang, 2016) and similar empirical finding from Yoon et al. (2018) we speculate that this absence of enhancement might be attributed to the inherent characteristics of SVGD, including its sensitivity kernel function parameters.
Acknowledgements This work was supported by the "Development of Efficient Fine-Tuning and Zero-Shot Generalization Methods" project funded by KT (KT award B220002586), ITT grant funded by MSIT (No.2019-0-00075, AI
Graduate School Program (KAIST); No.2020- 0-00940, Foundations of Safe Reinforcement Learning and Its Applications Natural Language
6.2.4 Effect of Prior
To assess the impact of the prior term in Eqq (6). we conducted an ablation experiment by remov-
Processing; No.2022-0-00311, Development of Goal-Oriented Reinforcement Learning Tech- niques for Contact-Rich Robotic Manipulation of Everyday Objects), Artificial intelligence
ing the prior term from the target adaptation loss.
The ablated version of BMTPT exhibited poorer

---

industrial convergence cluster development project funded by the Ministry of Science and ICT(MSIT, Korea) & Gwangju Metropolitan City, NRF of Korea (N2F1119 Field-oriented Technology Development Project for Customs Administration through NRF of Korea funded by the MSIT and Korea Customs Service (NRF002MS31AA10995353 ETRI grant (227S1 100, Core Technology Research for Self- Improving Integrated AI System), KAIST-NAVER Hypercreative Al Center.

---

References
Dorottya Demszky, Kelvin Guu, and Percy Liang, 2018. Transforming question answering datasets into nat- ural language inference datasets. arXiv preprint
Akari Asai, Mohammadreza Salchi, Matthew Pe- ters, and Hannaneh Hajishirzi. 2022. ATTEMPT: Parameter-efficient multi-task tuning via attentional mixtures of soft prompts. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 6655-6672, Abu Dhabi, United Arab Emirates. Association for Computa- tional Linguistics.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
Jimmy Ba, Murat A Erdogdu, Marzyeh Ghassemi, Shengyang Sun, Taiji Suzuki, Denny Wu, and Tian- zong Zhang. 2022. Understanding the variance col- lapse of SVGD in high dimensions. In International Conference on Learning Representations.
William B. Dolan and Chris Brockett. 2005. Automati- cally constructing corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing (IWP2005).
Elad Ben Zaken, Yoav Goldberg, and Shauli Ravfogel. 2022. BitFit: Simple parameter-effiiient fine-tuning for transformer-basel masked language-models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 1-9, Dublin, Ireland. Associ- ation for Computational Linguistics.
Arnaud Doucet, Nando Freitas, and Neil Gordon, ed- itors. 2001. Sequential Monte Carlo Methods in Practice. Springer New York.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05177
Charles Blundell, Julien Cornebise, Koray Kavukcuoglu, and Daan Wierstra. 2015. Weight uncertainty in neural network. In International conference on machine learning, pages 1613-1622.
PMLR.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eun- sol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading compre- hension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1- 13, Hong Kong, China. Association for Computa- tional Linguistics.
Daniel Cer, Mona Diab. Eneko Agirre, lñigo Lopez- Gazpio, and Lucia Specia. 2017. SemEval-2017 task I: Semantic textual similarity multilingual and crosslingual focused evaluation. In Proceedings of the 11th International Workshop on Semantic Evaluation (SemEval-2017), pages 1-14, Vancouver, Canada. Association for Computational Linguisties.
Danilo Giampiccolo, Bernardo Magnini. Ido Dagan, and Bill Dolan. 2007. The third PASCAL recogniz- ing textual entailment challenge. In Proceedings of the ACL-PASCAL Workshop on Textual Entailment and Paraphrasing, pages 1-9, Prague. Association for Computational Linguistics.
Peng Chen and Omar Ghattas. 2020. Projected stein variational gradient descent. In Advances in Neural Information Processing Systems, volume 33, pages 1947-1958, Curran Associates, Inc.
Christopher Clark, Kenton Lee, Ming- Ming-Wei Chang, Tom Kwiatkowski. Michael Collins, and Kristina Toutanova. 2019. BoolQ: Exploring the sur- prising difficulty of natural yes/no questions. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume I (Long and Short Papers), pages 2924-2936, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
W.R. Gilks, S. Richardson, and David Spiegelhalter, ed- itors. 1995. Markov Chain Monte Carlo in Practice. Chapman and Hal//CRC.
Alex Graves. 2011. Practical variational inference for neural networks. In Advances in Neural Information Processing Systems, volume 24, Curran Associates, Inc.
Neil Houlsby, Andrei Giurgiu, Stanislaw Jastrzebski, Bruna Morrone, Quentin De Laroussilhe, Andrea Gesmundo, Mona Attariyan, and Sylvain Gelly. 2019. Parameter-efficient transfer learning for NLP. In Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 2790-2799. PMLR.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question an- swering? try arc, the ai2 reasoning challenge. arXiv preprint Xiv: 18011803000 05457.
Marie-Catherine de Marneffe, Mandy Simons, and Ju- dith Tonhauser. 2019 The commitmentbank: Inves- tigating projection in naturally occurring discourse. Proceedings of Sinn und Bedeutung, 1372):
Edward J Hu, yelong shen, Phillip Wallis, Zeyuan Allen- Zhu, Yuanzhi Li, Shean Wang. Lu Wang. and Weizhu Chen. 2022.
LoRA: Low-rank adaptation of large

---

language models. In International Conference on Learning Representations.
on Knowledge Representation and Reasoning, pages 552-561. Institute of Electrical and Electronics Engi- neers Inc. 13th International Conference on the Prin- ciples of Knowledge Representation and Reasoning, KR 2012; Conference date: 10-06-2012 Through 14-06-2012.
Hamish Ivison and Matthew Peters. 2022. Hy- perdecoders: Instance-specific decoders for multi- task NLP. In Findings of the Association for Computational Linguisties: EMNLP 2022, pages 1715-1730, Abu Dhabi, United Arab Emirates. Asso- ciation for Computational Linguisties.
Xiang Lisa Li and Percy Liang. 2021. Prefix-
tuning: Optimizing continuous prompts for gener- ation. In Proceedings of the 59th Anmual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4582-4597, Online, Association for Computa- tional Linguisties.
Rabeeh Karimi Mahabadi, James Henderson, and Se- bastian Ruder. 20 Compacter: Efficient low-rank hypercomplex adapter layers. Advances Neural Information Processing Systems, volume 34, pages 1022-1035. Curran Associates, Inc.
Rabeeh Karimi Mahabadi, Sebastian Ruder, Mostafa Dehghami, and James Henderson. 2021b. Parameter- efficient taski-task fine-tuning for transformers via shared hypernetworks. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference Natural Language Processing (Volume I: Long Papers), pages 0155576, Online. Association for Computational Linguistics.
Qiang Liu and Dilin Wang. 2016. Stein variational gradient descent: A general purpose bayesian infer- ence algorithm. Advances Neural Information Processing Systems, volume 29, Curran Associates, Inc.
Tianle Liu, Promit Ghosal, Krishnakumar Balasubra- manian, and Natesh S. Pillai. 2023. Towards under- standing the dynamics of gaussian-stein variational gradient descent.
Daniel Khashabi, Snigdha Chaturvedi. Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking be- yond the surface: A challenge set for reading compre- hension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguisties: Human Language Technologies, Volume I (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguisties.
Xing Liu, Harrison Zhu, Jean-Francois Ton, George Wynne, and Andrew Duncan. 2022. Grass- mann stein variational gradient descent. In Proceedings of The 25th International Conference on Artificial Intelligence and Statistics, volume 151 of Proceedings of Machine Learning Research, pages 2002-2021. PMLR.
Rabech Karimi Mahahadi, James Henderson, and Se- bastian Ruder. 2021. Compacter: Efficient low-rank hypercomplex adapter layers. In Advances in Neural Information Processing Systems.
Tushar Khot, Ashish Sabharwal, and Peter Clark. 2018. Scitail: A textual entailment dataset from science question answering. Proceedings of the AAAI Conference Artificial Intelligence, 32(().
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct elec- tricity? a new dataset for open book question an- swering. In Proceedings of the 2018 Conference Empirical Methods in Natural Language Processing. pages 2381-2391, Brussels, Belgium. Association for Computational Linguisties.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Red- field, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Ken- ton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wes Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Nat- ural questions: A benchmark for question answer- ing research. Transactions of the Association for Computational Linguistics, 7:452-466
Christian Nacsseth, Scott Linderman, Rajesh Ran- ganath, and David Blei. 2018 Variational sequen- tial monte carlo. In Proceedings of the Twenty-Firs International Conference on Artificial Intelligence and Statistics, volume 84 of Proceedings of Machine Learning Research, pages 968-977. PMLR.
Brian Lester, Rami A1-Rfou, and Noah Constant. 1021. The power of scale for parameter-effict efficient prompt tuning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3045-3059 Online and Punta Cana, Domini- can Republic. Association for Computational Lin- guistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natu- ral language understanding. In Proceedings of the 58th Annual Meeting the Association for Computational Linguistics, pages 4885-4901, On- line. Association for Computational Linguistics.
Hector J. Levesque, Ernest Davis, and Leora Mor- genstern. 2012. The winograd schema challenge. In 13th International Conference on the Principles of Knowledge Representation and Reasoning. KR 2012, Proceedings of the International Conference
Mohammad Taher Pilehvar and Jose Camacho-Collados. 2019. WIC: the word-in-context dataset for

---

evaluating sontext-sensitve meaning representa- tions. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1267-1273, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
Lukang Sun, Avetik Karagulyan, and Peter Richtarik. 2022. Convergence of stein variational gradient de- scent under weaker smoothness condition.
Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. CommonsenseQQ:: A ques- tion answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies. Volume (Long and Short Papers), pages 4149-4158, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
Edoardo M Ponti, Alessandro Sordoni, Yoshua Ben- gio, and Siva Reddy. 2022. Combining modu- lar skills in multitask learning. arXiv preprint arXiv:2202.13914,
Colin Raffel, Noam Shazeer, Adam Roberts, Kather- ine Lee. Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020a. Exploring the limits transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Adam Trischler, Tong Wang. Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehen- sion dataset. In Proceedings of the 2nd Workshop Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguisties.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020b. Exploring the limits of transfer learning with a unified text-to-text trans- former. The Journal of Machine Learning Research, 21(1):5485- 5551.
Tu Vu, Brian Lester, Noah Constant, Rami Al- Rfou', and Daniel Cer. 2022. SPoT: Better frozen model adaptation through soft prompt transfer. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100.000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Lin- guistics.
1: LLong Papers), pages 5039-5059, Dublin, Ireland. Association for Computational Linguisties.
Alex Wang. Yada Pruksachatkun, Nikita Nangia, Aman- preet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman, 2019: Superglue: A stickier benchmark for general-purpose language understand- ing systems. In Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc.
Christian P. Robert and George Casella. 2004. Monte Carlo Statistical Methods. Springer New York.
Keisuke Sakaguchi, Ronan Le Bras, Chandra Bhaga- vatula, and Yejin Choi. 2020. Winogrande: An ad- versarial winograd schema challenge at scale. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8732-8740.
Alex Wang, Amanpreet Singh, Julian Michael, Fe- lix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understand- ing. In International Conference on Learning Representations.
Adil Salim, Lukang Sun, and Peter Richtarik. 2022, A convergence theory for SVGD in the population limit under talagrand's 's inequality tl. In Proceedings of the 39th International Conference on Machine Learning, volume 162 of Proceedings of Machine Learning Research, pages 19152 PMLR.
Zhen Wang, Rameswar Panda, Leonid Karlinsky, Roge- rio Feris, Huan Sun, and Yoon Kim. 2023. Multitask prompt tuning enables parameter-efficient transfer learning. The Eleventh International Conference on Learning Representations.
Bernhard Schölkopf and Alexander J. Smola. 2018. Learning with Kernels. The MIT Press.
Alex Warstadt, Amanpreet Singh, and Samuel R. Bow- man. 2019. Neural network acceptability judgments. Transactions of the Association for Computational Linguistics, 7:625-64)
Ravid Shwartz-Ziv, Micah Goldblum, Hossein Souri, Sanyam Kapoor, Chen Zhu, Yann LeCun, and An- drew Gordon Wilson. 2022. Pre-train your loss: Easy bayesian transfer learning with informative priors.
Adina Williams, Nikita Nangia, and Samuel Bow- man. 2018. A broad-coverage challenge cor- pus for sentence understanding through infer- ence. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages 1112- 1122, New Orleans, Louisiana. Association for Com- putational Linguistics.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over sentiment tree- bank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. pages 1631-1642. Seattle, Washington, USA. Asso- ciation for Computational Linguistics.

---

Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio. William Cohen, Ruslan Salakhutdinov. and Christo- pher D. Manning. 2018. HotpoiQA: A dataset for diverse, explainable multi-hop question answer- ing. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing. pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sung- woong Kim, Yoshua Bengio, and Sungjin Ahn. 2018. Bayesian model-agnostic meta-learning In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc.
Jianyi Zhang, Yang Zhao, and Changyou Chen. 2020. Variance reduction stochastic particle-ptimmzzaioo sampling. In Proceedings of the 37th International Conference on Machine Learning. volume 119 of Proceedings Machine Learning Research, pages 1.307-11316. PMLR.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and ma- chine commonsense reading comprehension. arXiv preprint 1xX.:18110.1288.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text clas- sification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019. PAWS: Paraphrase adversaries from word scram- bling. In Proceedings of the 2019 Conference the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. Volume I (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Asso- ciation for Computational Linguistics.
Qihuang Zhong, Liang Ding, Juhua Liu, Bo Du, and Dacheng Tao. 2022. Panda: Prompt transfer meets knowledge distillation efficient model adaptation. arXiv preprint arXiv:2208.1014 10160.

---

A Analogy with Gaussians for the Aggregation of Prompts Trained on Diverse Tasks
Consider a scenario with K source tasks, each characterized by a posterior p(0)Dx)) N A where Dx represents the dataset of k-th task, and is a soft prompt. Under the uniform prior, maximizing the likelihood (MLE) is equivalent to MAP estimation, and would lead each source prompt trained on task k to the mode f4g- By combining individual posteriors with assuming independent selection of tasks, we can construct the global posterior p = U P&) x IT p(()Dz)) The goal of transfer learning is to maximize this posterior, anticipating that the overall knowledge captured from source tasks will lead to good starting point for a target task. Note that the posterior, which is a product of Gaussian distributions, (>))"((( Axpu). Since the mean is the
is a Gaussian distribution whose mean Mglobal- -E mode of a Gaussian, Pglobal would be good candidate for the initialization point of the target prompt. However, unless the covariances differ only by a scaling factor, weighted sum of the individual modes { k= is unlikely to equal gglohal-
B Details for SVGD
B.1 Choice of SVGD
Stein Variational Gradient Descent (SVGD) is nonparametric variational inference technique that amalgamates the benefits of Markov Chain Monte Carlo (MCMC) and variational inference (Liu and Wang, 2016). Our utilization of SVGD over conventional variational inference (VI) methods is driven by multiple factors, each rooted in the limitations and attributes of standard VI approaches.
The target posterior distribution we aim to approximate is complex, potentially even multi-modal. Standard VI methods, constrained by specific family of distributions, often fail to capture such intricate structures. Therefore, they can show an inherent bias toward particular tasks. In contrast, SVGD employs particle-based approach to dynamically generate more expansive class of approximating distributions. This capability allows SVGD to represent complex and multi-modal distributions with greater accuracy. Furthermore, traditional VI methods like Variational Autoencoders (VAE) necessitate sampling. In contrast, SVGD requires the log derivatives of the prior at each point, commonly referred to as the score function. Additionally, while most VI methods aim to minimize surrogates of KL divergence through optimization, SVGD employs a first-order update method with a competing mechanism between particle repulsion and gradient descent.
generator-based and
B.2 Mathematical Explanation
Whereas gradient descent guides particles towards the optimal direction of fastest objective decrease, SVGD identifies the optimal transformation to minimize the KL divergence between the current and target distributions.
To find the optimal direction in the unit ball B of the Reproducing Kernel Hilbert Space H. which is the closed linear span of (k(0,-) -) 0 RD). that minimizes the KL-divergence towards the target distribution p. SVGD uses the point transformation Ti (0) (I (0). We will use the same notation for probability density with probability measure μu, there is no confusion. Specifically, it finds o that satisfies:
{- (T
P.P
argmax SEB
(a #p 1p)
0 u.
do
aa=0
where T#p(A) (T ((T-((()) (A))** The closed-form solution of the above is given by:
$1.
(o)ok((0)((0.)) + ()
Here, log p(0) is the log- likeliiod of p. The SVGD algorithm updates the distribution as follows:
M4l (I + )
UUner the uniform prior assumption, this relation can derived from Pw) x p(D 0) for each Please refer the Remark the Section 3. IIf random variable X follow distribution ya, the distribution Tif can he seen as the distribution of T(X).

---

where a is the step size. Discretized version of the above update rule for finite of particles {0:), SVGD iterativlyy transports the particles using the following update rule for m [M]]
M
0, <- 0, == where (0,) is M Σ Vo, log logp(0))k(0),0) + Vo,
The behavior inherent to SVGD is orchestrated by the two terms in the update, which define the key control mechanisms. Firstly, the first term entails the sharing of gradient information among particles, guiding their update trajectory. Additionally, the influence of neighboring particles is modulated by kernel distance weighting. The second term, Vo, k(,, preventing them from converging to a single mode.
; 0z), introduces a repelling force between the particles,
B.3 Detailed Explanation for RBF Kernel
In the execution of the Stein Variational Gradient Descent (SVGD) for our set of particles denoted {0: J. adopted the Radial Basis Function (RBF) kernel, which is defined follows:
k(O1, 02) = exp (-10²- 011 h
where h- (Median {10, - - i,j E [M]}}
log +1)
In this formulation, h is parameter frequently adjusted according to the distances between particles. As part of our methodology, adhere to the median heuristic, strategy supported by previous stud- ies (Schölkopf and Smola, 2018; Ba et al., 2022). This entails designating the bandwidth as the median of the set of mutual distances between particles.
B.4 Damped SVGD
The variant of Stein Variational Gradient Descent (SVGD) we employ in this work is Damped SVGD, delineated in the work by Ba et al. (2022). SVGD, in its typical implementation, is prone variance collapse when applied in a finite regime with particles, rather than updating distributions directly. This necessitates adaptation of the SVGD's update rule ensure proper approximation of the distribution with particles. The Damped SVGD specifically addresses this issue by moderating the influence of its own gradient descent term. This adjustment can seen clearly the update rule for 0; in a configuration M As compared to the standard update rule, the modification reads:
MM
1
damped (0:)
Ve, log p0 ((0)) 0.) V., M X Ve, (og) 0,)
M
j
(0) (1 X) M Vo.
(0) M Vg. lo.)
the Damped SVGD paper, the parameter À can be chosen using one of two strategies: taking as Amin min (1, + M)) for "fully damped," or taking À as a value between Amin and for "intermediate." In experiments, we use "intermediate" by consistently choosing the value min (1, (1,e-1(5, 155 M)) taking both selections into account. In our standard setting, this yields À 0.368. This variant of SVGD improves upon the original by mitigating the issue of variance collapse in some degree.
Computational Complexity Analysis of BMTPT
Our computational analysis verifies that BMTPT is computationally efficient for both source task posterior learning and task adaptation stages. Note that the additional computation necessary for BMTPT occurs after the prompt receives the pack-propagated gradient information from the LLM.

---

C.1 Definitions of Notations
M: Number particles.
I: Length of the prompt.
d: Hidden dimension of LLM (Large Language Model).
dprompt d l: Dimension of the prompt.
ggad Number of operations for the gradient backpropagation through the backbonc LLM.
K: RBF Kernel matrix with dimensions M x M (Kij k(0;,0))
Matrix of prompt parameters with dimensions M x dprompt-
V log p: Gradient of log-probability for each particle.
a: Stepsize.
C.2 Source Task Training
In the source task training phase, have multi-partice formulation governed by SVGD with RBF Kernel. The formulation involves various matrix and vector products, which we denote
A
KV log p (diag( (diaa((K)
-
h
The computational complexity for BMTFT during this phase can be summarized as (Tgraa) + M2 O (dppompt). This indicates that BMTPT requires additional M (d(pumpp) calculations over the vanilla prompt tuning. However, since Tgaad is the dominating factor and M 25 in our experiments, this increase is computationally acceptable. The average wall-clock time recorded during the training of the source task, based on 5 updates, is as follows: 0.42 seconds for the backward pass through the language model (LM) and 0.0035 seconds for Damped SVGD. We used single GeForce RTX 3090 GPU for these computations.
C.3 Task Adaptation Stage
During the task adaptation stage, the additional computational complexity of BMTPT is mainly due the upper bound of log prior term, which takes O (dpromppt computations. Therefore, the computational complexity for a single update is (Tgraa)) + (d Here well, the dominating factor is T Tgma- Similarly, we report the wall-clock time observed on our device during the target adaptation phase, specifically for the SuperGLUE- CB task with a batch size 32. The forward pass through the LM takes an average of 0.16 seconds, while the forward pass for the prior term requires 0.00011 seconds. We used single GeForce RTX 3090 GPU.
D Experiment on MRQA and "Others" Benchmark
MRQA
Others
Method
# Params
NQ
Aug.
Yelp
Avg.
Fine-luning
75.1
74.7
87.1
Adapter
1.9M
74.2
77.6
81.4
65.6
74.7
59.2
96.9
94.3
86.2
70.7
75.5
PT
76.8K
67.9
72.9
75.7
61.1
69.4
49.6
95. 1
87.9
55.8
72.1
SPoT
76.8K
68.2
74.8
75.3
58.2
69.1
50.4
93.4
91.2
91.1
82.0
ATTEMPT
232K
70.4
$7.6
84.9
MPT
77.6K
72.0
75.8
77.2
63.7
72.2
56.5
96.4
95.5
93.5
85.5
BMIPT (Ours)
77.6K
69.6g.1
82.9k
76.2um
62.4km
72.8
55.6
97.6ue
95/4u4
93.7cx2
85.6
Table 4: Experiment results MRQA and Others. We evaluate MRQA tasks using Fl score and Others using accuracy. BMTPT results are averaged over three runs with standard deviation indicated by subscripts.
For MRQA and several "Others" datasets, BMTPT remains competent among parameter-efficien baselines, showing the versatility of our approach outside GLUE and SuperGLUE.


Reviews:
[{"review_id": "Udf60TD9Ey", "reviewer": "Reviewer_qA2o", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your insightful review and acknowledgment of our work's contributions. We are pleased that our approach has been noted for its comparative advantages in various environments. The effectiveness of our method, enabled by optimizing prompt vectors through a Bayesian approach, aims to offer a more effective approach to transfer learning.\n\n### **Contrast with Traditional Multi-Task Learning**\nWe agree that our methodology seems similar with typical multi-task learning, in terms of the use of multi-source data in utilizing SVGD with prompt particles. However, this is largely a byproduct of our Bayesian formulation. Our aim is to create the prompt with more robust representation of general knowledge by leveraging posterior information via Bayesian approach. In doing so, we utilize SVGD to update the particles (each particle is an instantiation of a prompt) to approximate the joint posterior distribution from source domain datasets. This unique formulation and its implementation differentiate our work from existing multi-task learning paradigms in the context of prompt tuning.\n\n### **Comparison with MPT**\nOur work bears resemblance to MPT in how we construct a soft prompt by composing full-rank and low-rank matrices during the target adaptation stage. However, it is important to understand that both our motivation and our detailed approach to composing full-rank and low-rank matrices differ significantly from the strategies employed by MPT.\n\nMPT uses prompt decomposition to separate shared and task-specific information during source task training. In contrast, BMTPT employs a similar decomposition but applies it to the target task(s) during the task adaptation stage. We create a full-rank matrix by aggregating SVGD particles that have learned the source posterior and combine it with a low-rank matrix through element-wise multiplication. The low-rank matrix is introduced per target task, and it is designed to handle multi-target task adaptation scenarios.\n\n### **Empirical Performance**\nAs the reviewer noted, our empirical results may not explicitly surpass on all of the existing NLP tasks. However, we would like to emphasize that our method demonstrates superior performance across a significant portion of the GLUE/SuperGLUE datasets. Moreover, it achieves this without requiring any components beyond the soft prompts and the backbone model, unlike other baselines, which do require additional elements (e.g., ATTEMPT incorporates an attention module, MPT utilizes a separate backbone model for distillation).\n\nAdditionally, we re-present the tables for the experiments include the number of tunable parameters for comparison. Our BMTPT demonstrates superior performance compared to baselines that utilize a larger number of parameters or additional networks. \n\nIn the tables, it is evident that other baselines such as Adapters (1.9M params), BitFit (280K params) and ATTEMPT (232K params) utilize considerably more parameters compared to our method, BMTPT (77.6K params). Despite this, we demonstrate performance that is largely superior, or at the very least, comparable, across both single and multiple tasks within the GLUE/SuperGLUE datasets.\n\n### Result on GLUE Datasets (Single Target Task)\n| Method        | #params  | MNLI  | QQP  | QNLI | SST-2 | STS-B | MRPC | RTE  | CoLA | Avg. |\n|---------------:|:---------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| Adapters      | 1.9M    | **86.5**  | 90.2 | 93.2 | 93.8  | 90.7  | 85.3 | 71.9 | 64.0 | 84.5 |\n| BitFit        | 280K    | 85.3  | 90.1 | 93.0 | 94.2  | **90.9**  | 86.8 | 67.6 | 58.2 | 83.3 |\n| PT            | 76.8K   | 81.3  | 89.7 | 92.8 | 90.9  | 89.5  | 68.1 | 54.7 | 10.6 | 72.2 |\n| Vanilla transfer PT | 76.8K  | 85.8  | 86.9 | 93.2 | 92.9  | 90.5  | 87.1 | 77.0 | 83.2 | 87.1 |\n| SPoT          | 76.8K   | 85.4  | 90.1 | 93.0 | 93.4  | 90.0  | 79.7 | 69.8 | 57.1 | 82.3 |\n| ATTEMPT       | 232K    | 84.3  | **90.3** | 93.0 | 93.2  | 89.7  | 85.7 | 73.4 | 57.4 | 83.4 |\n| MPT          | 77.6K   | 85.9  | **90.3** | 93.1 | 93.8  | 90.4  | **89.1** | 79.4 | 62.4 | 85.6 |\n| BMTPT (Ours)  | 77.6K | 86.2  | **90.3** | **93.4** | **94.4**  | **90.9** | 87.2 | **81.3** | **86.6** | **88.7** |\n\n### Result on GLUE Datasets (Multiple Target Tasks)\n| Method        | #params  | MNLI  | QQP  | QNLI | SST-2 | STS-B | MRPC | RTE  | CoLA | Avg. |\n|---------------:|:---------:|:-------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|:------:|\n| Adapters     | 1.9M    | **86.3**  | **90.5** | 93.2 | 93.0  | 89.9  | **90.2** | 70.3 | 61.5 | 84.4 |\n| HyperFormer  | 280K    | 85.7  | 90.0 | 93.0 | 94.0  | 89.7  | 87.2 | 75.4 | 63.7 | 84.8 |\n| HyperDecoder | 76.8K   | 86.0  | **90.5** | **93.4** | 94.0  | 90.5  | 87.7 | 71.7 | 55.9 | 83.7 |\n| ATTEMPT      | 232K    | 83.8  | 90.0 | 93.1 | 93.7  | 90.8  | 86.1 | 79.9 | 64.3 | 85.2 |\n| MPT          | 77.6K   | 84.3  | 90.0 | 93.0 | 93.3  | 90.4  | 89.2 | **82.7** | 63.5 | 85.8 |\n| BMTPT (Ours) | 77.6K | 85.9  | 90.2 | 93.2 | **95.3**  | **91.2**  | 86.9 | 80.9 | **85.6** | **88.7** |\n\n### Result on SuperGLUE Datasets (Single Target Task)\n| Method              | #params  | Multirc  | BoolQ  | WiC | WSC | CB | Avg. |\n|---------------------|:-------:|:--------:|:------:|:---:|:---:|:--:|:-----:|\n| Adapters            | 1.9M    | **75.9**     | **82.5** | 67.1| **67.3** | **85.7**| **75.7**  |\n| BitFit              | 280K    | 74.5     | 79.6   | **70.0**| 59.6 | 78.6| 72.5  |\n| PT                  | 76.8K   | 58.7     | 61.7   | 48.9| 51.9 | 67.9| 57.8  |\n| Vanilla transfer PT | 76.8K   | 72.2     | 77.9   | 65.5| **67.3** | 78.6| 72.3  |\n| SPoT                | 76.8K   | 74.0     | 77.2   | 67.0| 50.0 | 46.4| 62.9  |\n| ATTEMPT             | 232K    | 74.4     | 78.8   | 66.8| 53.8 | 78.6| 70.5  |\n| MPT                 | 77.6K   | 74.8     | 79.6   | 69.0| **67.3** | 79.8| 74.1  |\n| BMTPT (Ours)        | 77.6K| 72.4 | 80.3| 67.4| **67.3**| **85.7**| 74.6 |\n\n### Result on SuperGLUE Datasets (Multiple Target Tasks)\n| Method              | #params  | Multirc  | BoolQ  | WiC | WSC | CB | Avg.  |\n|---------------------|:-------:|:--------:|:------:|:---:|:---:|:--:|:-----:|\n| ATTEMPT            | 232K    | 74.4     | 78.3   | 66.5| **69.2** | 82.1| 74.1  |\n| MPT                | 77.6K   | **74.8**     | 79.2   | **70.2**| 67.3 | **89.3**| **76.1**  |\n| BMTPT (Ours)       | 77.6K | 72.3 | **80.1** | 67.7| 67.3 | **89.3**| 75.3 |\n\nFurthermore, we would like to direct the reviewer to the empirical performance in the few-shot experiment, as displayed in the initial response to reviewer *g8bk*. The result demonstrates that our prior guides the prompts toward an efficient initial point for task adaptation in low-resource scenario.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. A Bayesian multi-task transfer learning approach to prompt tuning that outperforms state-of-the-art methods in many settings.\n2. A soft prompt tuning method that allows for more flexible and efficient transfer learning by using a continuous relaxation of the prompt vectors.\n3. An extensive experimental evaluation on standard benchmark NLP tasks that demonstrate the effectiveness of the proposed approach and its superiority over existing methods.", "reasons_to_reject": "1. Technically, the applied approaches are not novel techniques and the main novelty is applying them to solve a multitask learning problem. Multi-task prompt tuning is only one specific instance of multi-task learning. \n2. Regardless of learning the posterior distribution of source tasks, the practical implementation seems to be kind of similar to the mentioned reference, MPT, esp., in terms of prompt decomposition, parameter efficiency, and the source training/target adaptation process. The authors might need to explain or clarify clearly the fundamental differences with MPT in the paper.\n3. The empirical results are marginal or nearly the same as the baselines. The only vast improvement comes from the CoLA task in GLUE, improving it from 60+ in baseline to 86. The rest of all tasks are nearly the same or very marginal improvement over ATTEMPT or MPT. Therefore, the empirical improvement isn't impressive.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper is about Bayesian Multi-Task Transfer Learning for Soft Prompt Tuning. The main problem addressed in this paper is how to improve the performance of pre-trained language models on target tasks by fine-tuning them with task-specific prompts. The paper proposes a novel approach to prompt tuning that takes into account the correlation among source tasks for better transfer to target tasks. The technical approaches, Partical-based Vartiinal Inference and Stein Variational Gradient Descent, are applied to solve the multi-task prompt tuning problem. Specifically, the proposed approach aims to learn a global posterior approximation of source tasks such that it can be transferred to downstream target tasks easily and efficiently.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "jKdocaThrw", "reviewer": "Reviewer_AbeE", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 2, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your detailed review and for acknowledging the innovation and efficiency of our work. We are gratified that you recognize the value of our approach, which uses Stein Variational Gradient Descent (SVGD) to obtain representative source prompts. Our method aims for both effectiveness and parameter efficiency, without requiring any additional mechanisms.\n\n### **Computational Complexity Analyses** \nThe additional computational requirements for BMTPT are primarily due to the calculation of Damped SVGD updates during source posterior learning and the computation of regularization terms originating from the prior during target adaptation. Given that Damped SVGD primarily involves computing RBF kernel values among a limited number of particles (in our case, five particles), the computational cost is minimal. Likewise, the regularization calculation from the log-likelihood of the prior during target adaptation is also negligible. As a result, BMTPT can learn the prompt posterior from source tasks without imposing a significant computational burden.\n\nThis justifies the use of BMTPT for soft prompt tuning, as it delivers high performance with minimal computational demands. As evidenced by the tables in our initial response to reviewer qA2O, BMTPT exhibits superior performance across a large portion of the GLUE/SuperGLUE datasets. Furthermore, we have also discovered that our method shows advantages in the few-shot setting, indicating greater efficiency in real-world scenarios. Please refer to the table in the initial response to reviewer *g8bk*.\n\n\n### **Theoretical Framework of the Proposed Method**  \nWe appreciate the reviewer's inquiry about the theoretical substantiation for our methodology. Our approach is fundamentally rooted in Bayesian inference, a paradigm which naturally allows us to update our understanding of uncertainty based on the data from the source task. Within this Bayesian framework, conventional variational inference techniques have been shown to fall short in capturing complex, multi-modal distributions.\n\nIn this regard, we employ Stein Variational Gradient Descent (SVGD). Unlike traditional methods, SVGD is a non-parametric approach whose particles can represent flexible class of approximating distributions, which enables it to model intricate distributions with higher fidelity.\n\n### **Choice of Backbone Model** \nWe chose to use T5-{base, large, 3B} as backbone for fair comparison with notable baselines such as PT, SPoT, ATTEMPT and MPT, because these baselines adopted T5 as backbone in their experiments.\n\nHowever, to address the reviewer's concern, we are currently running experiment using BERT-large as backbone, to showcase that BMTPT is effective on LMs other than T5. We will report the performance of BERT-large based BMTPT and compare it with regular prompt tuning and **P-tuning V2** (used BERT-large as backbone) in the discussion period.\n\n### **Need for Visual or Interpretable Analysis Incorporating Concrete Natural Language Statements**  \nWe will subsequently include empirical results for natural language generation tasks in the discussion period. Should you have specific experiments in mind, we would welcome your recommendations.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The proposed method demonstrates commendable innovation, standing apart from mere amalgamation of existing models. The novel approach showcases tangible efficacy without introducing additional parameters.", "reasons_to_reject": "1.The absence of a computational complexity analysis, coupled with marginal and non-significant experimental performance improvements, raises concerns regarding the practical significance. If the proposed model introduces high computational complexity and yields only marginal gains, its real-world utility may be limited.\n\n2.The paper lacks comparative analysis with some important baselines, such as P-tuning v2. \n\n3. Additionally, the theoretical substantiation for the proposed method is insufficiently detailed.\n\n4. The exclusive use of a single language model, T5-base, as the backbone prompts doubts about the general applicability of the proposed approach across a broader spectrum of models.\n\n5. Furthermore, the paper lacks visual or interpretable analyses that incorporate concrete natural language statements. \n\nConsidering these points, I respectfully recommend that the authors thoroughly address these shortcomings to enhance the paper's overall quality and potential for contribution before reconsidering it for acceptance.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper solves the problem of  multi-task transfer learning for soft prompt tuning, considers the correlation among source tasks for better transfer to target tasks, and propose a Bayesian approach where they obtain representative source prompts corresponding to the samples from the posterior utilizing Stein Variational Gradient Descent (SVGD). Extensive experimental results on the standard benchmark NLP tasks show the advantage of the proposed Bayesian multi-task transfer learning approach. Besides, it has a high degree of parameter-efficient.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "P-Tuning: Prompt Tuning Can Be Comparable to Fine-tuning Across Scales and Tasks"}, "extra_text": {}}, {"review_id": "tPuryy82fE", "reviewer": "Reviewer_g8bk", "summary": "", "questions": "1. I am curious if the authors could include precise params/task numbers like ATTEMPT and MPT do in their papers. While section 4.4 computes this parametrically, having the exact values (especially in the table) would enhance comparisons.\n\n2. I am wondering about the choice of SGVD over regular variational inference, given its complexity. I am curious to know whether the authors explored using VI instead of SVGD. If they did, how did it perform? Additionally, I am curious to know how the optimization process might differ with VI. I assume that the only thing that would change is learning the posterior of source task prompts and drawing samples from $p(. | \\mathcal{D}^\\mathcal{S})$ in learning target prompts? \n\n3. I was wondering if the authors have done experiments in the few-shot scenario?\n\n4. I'm wondering if the authors could include comparisons with other methods when scaling up to larger models like T5-large and T5-3B.", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate your comprehensive evaluation and the acknowledgment of our study's contributions. It's gratifying to see that BMTPT geared toward understanding a unified distribution for source prompts, was identified as valuable for its efficacy in correlating tasks.\n\n### **Regarding the Number of Tunable Parameters**\nWe emphasize that our method is parameter-efficient, using nearly the same number of parameters as MPT. We also highlight the overall GLUE performance and the number of parameters as follows:\n\n### **Table for Average Sigle-Target Performance on GLUE Datasets**\n| Method        | FT | Adapters   | BitFit  | PT  | Vanila Transfer PT | SPoT  | ATTEMPT | MPT | BMTPT  |\n|---------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| #params   | 220M | 1.9M  | 280K | 76.8K | 76.8K | 76.8K | 232K  | 77.6K | 77.6K |\n| GLUE Avg.  | 84.9 | 84.5  | 83.3  | 72.2  | 87.1  | 82.3 | 83.4 | 85.6 | **88.7**  |\n\nWhile our multi-task performance on SuperGLUE is marginally lower than that of MPT, the overall trend in empirical results is notable, particularly in the context of parameter efficiency.\n\n### **Reason for using SVGD**\nWe appreciate your inquiry regarding our decision to use Stein Variational Gradient Descent (SVGD) over traditional variational inference methods. Our choice is motivated by several considerations that emerge from the limitations and features of standard variational inference methodologies.\n\nFirstly, the source task posterior we aim to approximate possesses a complex, perhaps multi-modal distribution. Conventional variational inference, restricted by distribution families, often fails to capture such complexity adequately, and can be overly biased towards specific task. On the other hand, SVGD adaptively generates a richer class of approximating distributions using particles. This enables SVGD to approximate complex and multi-modal target distributions with higher fidelity.\n\nFurthermore, the theoretical foundations of SVGD offer an additional layer of reliability. SVGD framework is originated from the first-order optimization on the functional space, which provides solid backgrounds for the optimization strategy.\n\nIn response to the reviewer's question, we clarify the distinctions between traditional VI and SVGD. Traditional VI methods like Variational Autoencoders (VAE) are generator-based and necessitate sampling. In contrast, SVGD requires the log derivatives of the prior at each point, commonly referred to as the score function. Additionally, while most VI methods aim to minimize surrogates of KL divergence through optimization, SVGD employs a first-order update method with a trade-off mechanism between particle repulsion and gradient descent.\n\n### **Few-Shot Learning Experiments on GLUE and SuperGLUE**\nIn this section, we present the requested few-shot experiments conducted on the GLUE and SuperGLUE datasets. For the 4-shot experiments, we adjusted the learning rates to one-third of their original values, to account for the reduced batch size compared to our standard experiments. The reported performance for BMTPT is averaged over three runs, each operated with a different random seed.\n\nThese results demonstrate that our prior in target adaptation guides the prompts toward an efficient initial point for task adaptation in low-resource scenario.\n\n### 4-shot (GLUE)\nMethod | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Avg. |\n|--------|------|-----|------|-------|-------|------|-----|------|------|\nPT     | 40.1 | 63.2| 40.4 | 53.0  | 88.8  | 68.1 | 56.3| 27.4 | 54.7 |\nMPT    | **59.4** | 82.0| 86.2 | 56.5  | 89.1  | 68.1 | **62.6**| 34.8 | 67.3 |\nBMTPT(Ours)  | 43.0 | **82.4** | **89.2** | **60.3** | **90.0** | **76.7** | 55.8 | **67.8** | **70.65** |\n\n### 16-shot (GLUE)\n Method | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Avg. |\n|--------|------|-----|------|-------|-------|------|-----|------|------|\n PT     |  41.5 | 62.3| 59.9 | 50.9  | 87.8  | 68.1 | 54.7| 28.5 | 56.7 |\nMPT    |61.6 | 84.7| 90.6 | 63.2  | 89.1  | 70.1 | **64.8**| 32.1 | 69.5 |\nBMTPT(Ours)  | **65.2** | **85.5** | **91.3** |\t**70.9** | **89.7** | **77.0** | 63.5 |\t**68.4** | **76.4** |\n\n### 32-shot (GLUE)\n Method | MNLI | QQP | QNLI | SST-2 | STS-B | MRPC | RTE | CoLA | Avg. |\n|--------|------|-----|------|-------|-------|------|-----|------|------|\n PT     |37.0|62.3|56.7|50.9|87.5|68.1|54.7|23.2|55.1|\nMPT    |63.6|88.5|91.0|75.9|89.7|74.5|**59.7**|30.8|71.7|\nBMTPT(Ours)  |**66.3**|**88.9**|**91.6**|**89.1**|**90.4**|**78.2**|59.4|**67.4**|**79.0**|\n\n### 4-shot (SuperGLUE)\nMethod | Multi | BoolQ | WiC | WSC | CB | Avg. |\n|--------|------|-----|------|-------|------|------|\nPT     | 61.8 | 61.6 | 51.2 | 60.4  | 53.5  | 57.7 |\nMPT    | **62.6** | 62.6 | 52.9 | **67.3**  | 73.6  | 63.6 |\nBMTPT(Ours)  | 60.6|**62.7**|**56.1**|**67.3**|**78.6**|**65.1**|\n\n### 16-shot (SuperGLUE)\nMethod | Multi | BoolQ | WiC | WSC | CB | Avg. |\n|--------|------|-----|------|-------|------|------|\nPT     | 60.3 | 61.9 | 48.9 | 44.2 | 63.5 | 55.8 |\nMPT    | **64.5** | 63.3 | 49.8 | **67.3** | **78.6** | 64.7 |\nBMTPT(Ours)  |60.4\t| **63.7**| **62.4**| **67.3** |75.0 |**65.8**|\n\n### 32-shot (SuperGLUE)\nMethod | Multi | BoolQ | WiC | WSC | CB | Avg. |\n|--------|------|-----|------|-------|------|------|\nPT     | 59.2 | 61.7 | 52.6 | **67.3** | 67.8 | 61.7 |\nMPT    | **63.3** | **68.9** | 53.9 | **67.3** | **82.1** | **67.1** |\nBMTPT(Ours)  |63.2|64.2|**55.5**|**67.3**|**82.1**|66.5|\n\n### **Comparison on the Larger Models with Baselines**\nOur experimental results on GLUE and SuperGLUE, using T5-large and T5-3B, are detailed in Table 3. A direct performance comparison with ATTEMPT and MPT, cutting-edge prompt tuning methods, could not be facilitated due to the absence of specific performance metrics in their respective papers (see Figure 4 in both ATTEMPT and MPT). Instead, only a degree of score augmentation corresponding to increases in backbone sizes is provided. However, it can be seen that BMTPT achieves on-par-with performance compared to ATTEMPT and MPT, on BoolQ, MultiRC and WiC. In the other tasks within the GLUE/SuperGLUE benchmark, BMTPT generally surpasses both ATTEMPT and MPT when using T5-base. We believe this superiority pattern would be maintained when scaled up to T5-large or T5-3B.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Acceptable results showing that the method outperforms the existing PEFT methods on GLUE.\n2. This work can encourage more work on Bayesian multi-task prompt training in the future.\n3. The paper is clearn and well-written.", "reasons_to_reject": "1. The multi-task results on Super-GLUE are worse than MPT.\n2. The method does not compare to existing work in the literature when scaling the model in section 6.2.", "questions_to_authors": "1. I am curious if the authors could include precise params/task numbers like ATTEMPT and MPT do in their papers. While section 4.4 computes this parametrically, having the exact values (especially in the table) would enhance comparisons.\n\n2. I am wondering about the choice of SGVD over regular variational inference, given its complexity. I am curious to know whether the authors explored using VI instead of SVGD. If they did, how did it perform? Additionally, I am curious to know how the optimization process might differ with VI. I assume that the only thing that would change is learning the posterior of source task prompts and drawing samples from $p(. | \\mathcal{D}^\\mathcal{S})$ in learning target prompts? \n\n3. I was wondering if the authors have done experiments in the few-shot scenario?\n\n4. I'm wondering if the authors could include comparisons with other methods when scaling up to larger models like T5-large and T5-3B.", "paper_topic_and_main_contributions": "The paper introduces BMTPT as a a novel Bayesian approach to multi-task prompt tuning. In contrast to previous approaches in the literature that learn source prompts individually, BMTPT learns a global distribution over the source prompts which enables better capturing the correlations between these tasks. The approach then initializes and learns the target task prompts from the source prompts. Results on transferring from 6 source tasks to 21 target tasks show that the method outperform existing parameter efficient fine tuning methods in the literature.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "1. Boldening the highest numbers in tables can enhance readability."}, "extra_text": {}}, {"review_id": "LHvfH5WdhT", "reviewer": "Reviewer_f8vf", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### **Clarification on Soft Prompts and Multi-Source Data**  \nThank you for your comprehensive review and for acknowledging the clarity and thoroughness of our paper. We appreciate your recognition of our Bayesian approach, which utilizes SVGD for representative source prompts. However, we would like to point out that each soft prompt learns the joint posterior distribution from multi-source data, rather than assigning each source task to a specific prompt. These prompts are then aggregated to serve as the initial target prompt, offering a systematic method for transfer learning from multiple source tasks to target tasks.\n\n### **Performance Relative to Baselines**  \nWe acknowledge the reviewer's observation regarding the relative performance of our proposed BMTPT, as highlighted in Table 1. It is imperative to note that while the model may not have shown a pronounced advantage over the baselines in certain tasks, its robust performance across multiple benchmarks is noteworthy. Specifically, the BMTPT either outperforms or matches existing baselines on a significant number of tasks within the GLUE/SuperGLUE framework. This generalized performance reflects the model's reliability across various linguistic tasks, serving as an important metric beyond mere task-specific performance. Please also refer to the **Empirical Performance** section in the initial response to reviewer *qA2o*.\n\nWe wish to point out that our method performs well in few-shot scenarios, as shown in the initial response to reviewer *g8bk*. The result indicates both effective initial task adaptation and increased efficiency in real-world conditions.\n\nIn the previous version of the manuscript, we inadvertently presented the performance metrics for multi-task ATTEMPT rather than MPT in Table 2. Below, we provide the corrected Table 2 with the performance metrics for MRQA and \"Others\" datasets for the MPT model:\n\n### Result on MRQA Datasets\n| Method        | params | NQ    | HP    | SQA   | News  | Avg.  |\n|---------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| Adapter       | 1.9M | **74.2**  | 77.6  | **81.4**  | **65.6**  | **74.7**  |\n| BitFit        | 280K | 70.7  | 75.5  | 77.7  | 64.1  | 72.0  |\n| Prompt tuning | 76.8K | 67.9  | 72.9  | 75.7  | 61.1  | 69.4  |\n| SPoT          | 76.8K | 68.2  | 74.8  | 75.3  | 58.2  | 69.1  |\n| ATTEMPT       | 232K | 70.4  | 75.2  | 77.3  | 62.8  | 71.4  |\n| MPT           | 77.6K | 72.0  | 75.8  | 77.2  | 63.7  | 72.2  |\n| BMTPT (Ours)  | 77.6K | 69.6  | **82.9**  | 76.2  | 62.4  | 72.8  |\n\n### Result on  ''Others'' Datasets\n| Method        | params | WG    | Yelp  | SciTail | PAWS  | Avg.  |\n|---------------:|:-------:|:-------:|:-------:|:-------:|:-------:|:-------:|\n| Adapter       | 1.9M | **59.2**  | 96.9  | 94.5    | **94.3**  | **86.2**  |\n| BitFit        | 280K | 57.2  | 94.7  | 94.7    | 92.0  | 84.7  |\n| Prompt tuning | 76.8K | 49.6  | 95.1  | 87.9    | 55.8  | 72.1  |\n| SPoT          | 76.8K | 50.4  | 95.4  | 91.2    | 91.1  | 82.0  |\n| ATTEMPT       | 232K | 57.6  | 96.7  | 93.1    | 92.1  | 84.9  |\n| MPT           | 77.6K | 56.5  | 96.4  | **95.5**    | 93.5  | 85.5  |\n| BMTPT (Ours)  | 77.6K | 55.6  | **97.6**  | 95.4 | 93.7  | 85.6  |\n\nNote that BMTPT significantly improves the average performance of PT by +3.4% on MRQA and +13.5% on the \"Others\" benchmark.\n\n### **Fundamental Motivation Behind BMTPT**  \nWe appreciate the query about the fundamental motivation behind our method. **At its core, BMTPT aims to focus on the essence of transfer learning by enhancing it via transferring a useful distribution as a prior for target adaptation. On the other hand, existing prompt transfer methods such as SPoT, ATTEMPT, and MPT tend to rely on the transferability between individual NLP tasks (e.g., SQuAD is more helpful for solving MRPC than SST-2)**. By employing a Bayesian framework, we leverage SVGD updates and prior injecting objectives, which are computationally efficient and straightforward to implement.\n\n### **Contribution and Applicability**  \nThis Bayesian approach allows for the optimization for various NLP tasks without requiring additional neural network structures or procedures like distillation (ATTEMPT uses an attention module and MPT utilizes a separate backbone model for distillation). Consequently, our model provides a unified and more efficient method that performs competitively or surpasses existing approaches including other PEFT methods and full fine-tuning. We believe this represents a significant contribution to the field, substantiating both the theoretical significance and practical applicability of our method.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. Clear writing of the paper.\n2. Thorough experimental analysis that tried multiple model sizes and various source/target tasks.", "reasons_to_reject": "1. My main concern of the paper is that the proposed method doesn’t show a significant improvement compared to other existing methods in Table 1 lower part.\n2. I don’t quite understand the motivation of the method. Intuitively, what are the expected benefits of the proposed method compared to existing methods?", "questions_to_authors": "", "paper_topic_and_main_contributions": "As the paper’s name suggests, this paper proposes a Bayesian approach to conduct transfer learning from multiple separately trained soft prompts from other source tasks to target tasks as the target soft prompt initialization. In this method, representative source prompts are calculated corresponding to the samples from the posterior utilizing Stein Variational Gradient Descent, which are then aggregated to constitute the initial target prompt.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces BMTPT to address the challenge of improving the performance of PTLMs on target tasks through task-specific prompt fine-tuning. BMTPT takes into consideration the correlation among source tasks to enhance the transfer to target tasks. Particle-based Variational Inference and Stein Variational Gradient Descent (SVGD) are utilized to solve the multi-task prompt tuning problem. Specifically, BMTPT aims to learn a global posterior approximation for source tasks, facilitating efficient transfer to downstream target tasks. Experimental results on standard benchmark NLP tasks demonstrate the effectiveness and parameter efficiency of the proposed Bayesian multi-task transfer learning approach, outperforming existing methods by capturing task correlations and leveraging this knowledge to improve target task performance.


The majority of reviewers accept that the soundness of this work is moderate and the excitement level is decent. Reviewer AbeE has some concerns but the reviewers also provided substantive responses. However, the reviewer did not engage in discussions and I found responses by the authors at least relevant.
Decision: Accept-Findings