BayesPrompt: Prompting Large-Scale Pre-Trained Language Models on Few-shot Inference via Debiased Domain Abstraction
Authors: Jiangmeng Li
Fei Song
Yifan Jin
Wenwen Qiang
Changwen Zheng
Fuchun Sun
Hui Xiong
Venue: ICLR 2024 poster (2024)
Keywords: Prompt;Pre-Trained;Few-shot;Debiased;Domain Abstraction
Primary area: learning theory
TL;DR: Abstracting training domains in a debiased manner to generate discriminative prompts, which provide unambiguous guidance for PLMs.

Abstract: As a novel and effective fine-tuning paradigm based on large-scale pre-trained language models (PLMs), prompt-tuning aims to reduce the gap between downstream tasks and pre-training objectives. While prompt-tuning has yielded continuous advancements in various tasks, such an approach still remains a persistent defect: prompt-tuning methods fail to generalize to specific few-shot patterns. From the perspective of distribution analyses, we disclose that the intrinsic issues behind the phenomenon are the over-multitudinous conceptual knowledge contained in PLMs and the abridged knowledge for target downstream domains, which jointly result in that PLMs mis-locate the knowledge distributions corresponding to the target domains in the universal knowledge embedding space. To this end, we intuitively explore to approximate the unabridged target domains of downstream tasks in a debiased manner, and then abstract such domains to generate discriminative prompts, thereby providing the de-ambiguous guidance for PLMs. Guided by such an intuition, we propose a simple yet effective approach, namely BayesPrompt, to learn prompts that contain the domain discriminative information against the interference from domain-irrelevant knowledge. BayesPrompt primitively leverages known distributions to approximate the debiased factual distributions of target domains and further uniformly samples certain representative features from the approximated distributions to generate the ultimate prompts for PLMs. We provide theoretical insights with the connection to domain adaptation. Empirically, our method achieves state-of-the-art performance on benchmarks.

Published as a conference paper at ICLR 2024
BAYESPROMPT: PROMPTING LARGE-SCALE PRE- TRAINED LANGUAGE MODELS ON FEW-SHOT INFER- ENCE VIA DEBIASED DOMAIN ABSTRACTION
Jiangmeng Li 2x Fei Song 13r Yifan Jin 13, Wenwen Qiang 't, Changwen Zheng
Fuchun Sun Hui Xiong
National Key Laboratory of Space Integrated Information System, Institute of Software Chinese Academy of Sciences, Beijing, China
State Key Laboratory Intelligent Game, Beijing, China
UUniversity of Chinese Academy of Sciences. Beijing, China "Tsinghua University, Beijing, China
'The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, China {j iaingenng220 9, songfe nongfei2022, 2022,, yi fan2020, qi angwenwenn iiscass... acc. ac. changwen& scaa ac ccsun@maiin ts tignggua...nnno.. en, xi xionghui @ust .h
on
ABSTRACT
As a novel and effective fine-tuning paradigm based on large-scale pre-trained lan- guage models (PLMs), prompt-tuning aims to reduce the gap between downstream tasks and pre-training objectives. While prompt-tuning tonpt-ttuing has yielded continuous advancements in various tasks, such an approach still remains persistent defect: prompt-tuning methods fail to generalize to specific few-shot patterns. From the perspective of distribution analyses, we disclose that the intrinsic issues behind the phenomenon are the over-muliitdiioous conceptual knowledge contained PLMs and the incomplete knowledge for target downstream domains, which jointly result in that PLMS mis-locate the knowledge distributions corresponding to the target domains in the universal knowledge embedding space. To this end, intu- itively explore to approximate the complete target domains of downstream tasks in debiased manner, and then abstract such domains generate discriminative prompts, thereby providing the de-ambiguous guidance for PLMMs Guided by such an intuition, we propose a simple yet effective approach, namely BayesPrompt, to learn prompts that contain the domain discriminative information against the interference from
irrelevant knowledge. BayesPrompt primitively lever- ages known distributions to approximate the debiased factual distributions of target domains and further uniformly samples certain representative features from the approximated distributions to generate the ultimate prompts for PLMs. We provide theoretical insights with the connection to domain adaptation. Empirically, our method achieves state-of- the-art performance on benchmarks
INTRODUCTION
Benefiting from key ingredients of the massive candidate datasets, vast trainable model parameters, and choreographed training architecture, PLMS (Dai & Le, 2015), as artificial general intelligence approaches, achieve impressive successes in general natural language processing fields. However, for specialized downstream tasks, PLMs hit developmental bottleneck, which especially falls short of the expectations of researchers in few-shot scenarios (Wang et al., 2020). The intrinsic reason behind such an issue is that PLMs contain pver-multituiious conceptual knowledge", resulting in that domain-inreless irrelevant knowledge may interfere with the inference on downstream tasks, especially for
Equaa contribution.
Corresppnding author.
'The code implementation of our method is available at this link.
"The knowledge contained by PI MLSs exhihits inherent polysemy.

---

Published as a conference paper at ICLR 2024
egg. Apple products were sold very well in the last quarter, and experts attribute the phenomenon to the waning of the COVID-19 pandemie.
Domain Information is Unknown
PLMs
Fruit
?
Company
All Knowledge of PLMs
prplepr were sold very well in the last quarter, and experts attribute the phenomenon to the waning the COVID-19 pandemic. "
e.g.
Domain Information is Known
PLMs
Company
Prompt
Domain Knowledge
Google, Microsoft, A mazon, Facebook, NVIDIA...
iPhone, MacBook, iOS, macOS, iPod, Arrdsd...
Steve Tim Cook, Larry Page, Sam A Itman...
All Knowledge of PLMs
Figure 1: Examples of leveraging the domain information prompting PLMSs The points within the blue dashed circle represent the factual knowledge distribution of task-related data in PLMs.
few-shot datasets (Li et al., 2023). We demonstrate an illustrative example Figure for ease of understanding.
To remedy this deficiency, recent works propose well-designed prompts to guide PLMs thereby avoiding the inference outliers on downstream tasks (Wei et al., 2021 Liu et al., 2023b). However, manually constructing such prompts requires expertise and costly workloads (Schick 2020; Li et al., 2022b). To this end, the data-driven trainable prompts emerge and yield significant performance boosts on the downstream inference PLMs (Gao et al., 2021; ;; Liu et al., 2021; Gu et al., 2021), but such learning paradigm of prompt still suffers from long-standing the limited and discrete semantic information contained in the training samples from downstream domains can barely support the conventional trainable prompts to acquire sufficient supervision, such that the guidance of the generated prompts is trivial to PLMs. Especially, such a challenge further exacerbates the performance of PLMs in few-shot scenarios.
Schütze,
challenge:
To further understand the implicit and essential reason behind the defect of PLMs in the few-shot scenarios, we revisit the operational principle introduced in downstream inferences of PLMs from the distribution perspective. For the traditional inference paradigm without prompts demonstrated in Figure 2 (a), certain samples, e.g., sentences, may contain information that directly confounds the inference of PLMs. We ascribe this phenomenon to the fact that the confounding samples concurrently belong nultiple domain distributions in the knowledge embedding space of PLMs, and the model cannot determine the desired domain without prompts that contain domain discriminative information.. Consequently, the wvrrmuulitudinous conceptual knowledge can not only empower PLMs to understand universal concepts but also interfere with the inference specific tasks. For the inference paradigms with trainable prompts demonstrated in Figure 2 (b) and (c), the information contained in the limited training samples of candidate downstream domains may lead to the knowledge ambiguity of PLMs, while the information contained in the corresponding complete domains can effectively cope with such an issue. We conjecture that the limited training samples lead the trainable prompts to learn the biased empirical distribution of the target domain, which only contains partial information and is inconsistent with the factual distribution of the target domain, resulting in the covariate shift problem (Heckman, 1979; Shimodaira, 2000), thereby still providing certain ambiguous guidance for PLMs.
"The information that effectively characterizes features of the actual downstream domain.
2

---

Published as a conference paper at ICLR 2024
PLMs
PLMs
PLMs
Prompt
Prompi
Knowledge of Single Sample
Incomplete
Complete
Knowledge
Knowledge
There bat the grass.
e.g. There a on the grass.
e.g. He put the suitcase the bank.
e.g. He put suitcase in the bank."
c.g. dealer showed the rules.
e.g. There is a bat on grass.
Single Sample
Incomplete Domain
Complete Domain
(a)
(b)
Figure 2: The intrinsic reasons why patchy target domain knowledge may incur negative impacts on the inference of PLMs. The points in the red solid circles adhere the factual distributions corresponding to the task-related data in PLMs, while the points in the blue dashed circles represent the approximated domain distributions obtained through specific learning paradigm.
To this end, we intuitively explore to approximate the complete training domains on downstream tasks in debiased manner, and then abstract such domains to generate discriminative prompts, thereby providing the de-ambiguous guidance' 4 for PLMs. Specifically, we propose novel approach, dubbed BayesPrompt, which primitively leverages known distributions, e.g., Gaussian distribution, to approximate the debiased factual distributions of downstream domains and further uniformly samples certain representative features from the approximated distributions to generate the ultimate prompts for PLMs. On top of this, the behavior of BayesPrompt can be treated as the debiased domain abstraction S We elaborate on the procedures of the proposed approach in a nutshell. The distribution approximation is achieved by using Stein Variational Gradient Descent (SVGD) (Liu & Wang, 2016), which is a general-purpont Bayesian inference algorithm. In practice, we observe that selecting the conventional Gaussian distribution as the known distribution degenerates the approximation of downstream domain distributions, and thus the Gaussian Mixture Model (GMM) (Reynolds et al., 2009) is constructed fit the sample distribution. The resulting distribution and sample representations are then used to initialize the target distribution and particles for the SVGD algorithm. Through iterative updates of SVGD, a new set of particles is generated that approximates the target distribution. As for the sampling strategy, we adopt uniform sampling based on empirical evidence. By sampling from the target distribution, we obtain prompts that contain domain discriminative information, which can mitigate interference from domain-irrelevan knowledge. The sufficient experimental analyses prove that BayesPrompt achieves state-of-the-an performance, Contributions:
We disclose a standinn issue challenging the downstream inference of PLMs for current methods, especially in few-shot scenarios, which is further described by providing intuitive illustrations for ease of understanding.
We propose BayesPrompt, orthogonal existing methods, to approximate the factual distributions of downstream domains a debiased manner, and further abstract such domains thereby generating discriminative prompts for PLMs.
We conduct extensive evaluations on various experimental settings, including the few-shot ex- periments and standard experiments, empirically prove the effectiveness of BayesPrompt.
*Using feature that well describes the actual downstream domain as a prompt to guide the PLMs to remember the knowledge of the target domain.
distribution fits actual downstrcam domain distribution well.
"Abstracting feature that can well describe the actual downstream domain based on the debiased factual distribution.
3

---

Published as a conference paper at ICLR 2024
2
RELATED WORKS
PLMs have demonstrated their ability to capture rich knowledge from massive corpora (Li et al., 2022c). However, there exists a significant gap between the objectives of pre-training and fine- tuning. Prompt-tuning methods are fueled by the birth of GPT-3 (Brown et al., 2020) and have achieved impressive successes in a wide range of tasks. By leveraging language prompts as contexts, downstream tasks can be expressed as certain objectives similar to pre-training objectives. With appropriate manual prompts, a series of studies (Ben-David et al., 2021; Lester et al., 2021; Lu et al., 2021; Jiang et al., 2022; Ma et al., 2022) have been proposed, demonstrating the advancement of prompt-tuning. (Han et al., 2022) propose model, called PTR, which creatively applies logic rules to construct prompts with several sub-prompts. (Ding et al., 2021) apply prompt-tuning to entity typing by constructing an entity-oriented verbalizer and templates. (Hu et al., 2021) incorporate external knowledge into the verbalizer with calibration. (Li et al., 2022a) use external knowledge bases to construct snowledge-injected prompts. (Chen et al., 2022b) inject latent knowledge contained in labels into prompt construction and synergistically optimize their representation with structured constraints. (Wang et al., 2023) exploit knowledge to improve the efficiency of text classification. (Liu et al., 2023a) leverage prompt-tuning to incorporate background information and relational information for causal reasoning. These previous studies have established that the effectiivness of prompt-based learning is largely attributed to the implicit knowledge embedded in PLMs. However, they did not focus the factual distributions of target domains in PLMs. Recently, advancements in large language models have enabled the emergence of various forms of prompt-tuning (Brown al., 2020; Wei al., 2021b; 2022). However, the performance improvements emerge only with sufficient model scale. In this paper, we propose BayesPrompt to attenuate the impact of domain- irrelevant knowledge while enhancing the adaptability to different model scales. As compared to the abooe--mntioons prompt-tuning methods, BayesPrompt achieves a good balance among model effectiveness, model generalization, model scale, and human workloads.
3 PRELIMINARY
Before introducing BayesPrompt, we take Relation Extraction (RE) as example to recap the preliminaries of prompt-tuning. Formally, we suppose the RE dataset as = (X, Y), where X denotes the set of examples, and Y denotes the set of relation labels. To be concise, to represent all entities briefly, as a single entity may be composed of multiple tokens. Therefore, each example e X consists of several tokens, x: {w01 W2, Wsy-.. Way-.., Wn) The RE task takes a query sentence x with the corresponding entity pair (w,, W,) as the input, aiming to learn a distribution P(yl(w,, w.)) over all possible pre-defined relations y E Y.
w, and w.
A typical prompt consists of a template T(-) and a set of label words V, where the template T()) defines the location and number of the added auxiliary words, and V refers to a of label words in the vocabulary of Language Model (LM). In addition to retaining the original tokens in x, one or more [M ASA' is placed into Zprompt for the LM fill the label words. Given each example æ for RE, the template T(-) is leveraged to insert pieces of texts with entity pair (w.,, wo) into x to map as apprompt = Tar), where the Tprompt is the corresponding input of LM with a [ [MASK] token in it. Inspired by (Shin et al., 2020; Li & Liang, 2021; Liu et al., 2021; Zhou et al., 2022), we suppose the bijective mapping from the relation label space to the label word space as M Y + V, and M (z) presents the label words corresponding to label y. Thus, the probability distribution over V at the masked position can be implemented by
( x) P (LMASK) = M (T)).
By filling [MASK] tokens in the input, the RE task can be transformed into a masked language modeling problem, where the goal to infer the appropriate label words at the masked positions.
4 METHODOLOGY
Our objective is to learn prompts that contain the domain discriminative information against the interference from the domain-irrelevant knowledge by approximating the debiased factual distribu- tions of downstream domains. To this end, propose BayesPrompt to determine the discriminative
4

---

Published as a conference paper at ICLR 2024
[CLS] The [4, farm [/E,] comprises [xy] turbines [/Eg]. [SEP]
[sub]
farm
[sub]
Mask
[oby]
turhitas
[obj]
[SEP]
comaable Continoous Tokens
Component Who WhlleleZee11
Dffacticl Product Producereel.at)
Enirty Tokess
MLM Head
[relation]
Mask Tokens
Relation Probablittes
[ M
X1
Bgg
Uniform Sampling
Knowledge Injection
(X.V)
X2
Encuder
GMM
SVGD
(8 Jrnce
[sub]
4p
[obi]
R
[relation]
4
K Knwwledge Injectioo
Figure 3: The overall framework of BayesPrompt. "[E]" and "[/E]" denote the start and end tags indicate entities for inferring relations, respectively.
prompts. BayesPrompt provides unambiguous guidance for PLMs by debiasing training domains on downstream tasks and abstracting such domains in a uniform sampling manner.
The overall framework of BayesPrompt is illustrated Figure 3. Specifically, the example o; first fed into the encoder obtain its representation h4 As practical observation discussed in Section we notice that using the conventional Gaussian distribution as the known distribution degenerates the approximation of downstream domain distributions. Therefore, build GMM model the representation distribution as follows:
Pu.Po,PP GMM R,),
(2)
where denotes the representations of all examples, and R2 is the number of Gaussian components that are determined by relation categories. The parameters each Gaussian component in the fitted GMM are represented by the outputs P, P. and P.. They denote the mean vectors, covariance matrices, and weights of each Gaussian component, respectively.
We adopt SVGD (Liu & Wang, 2016) to approximate the de- biased factual distributions of downstream domains. SVGD general-purpose variational inference algorithm that aims to approximate target distribu- tion. converges more rapidly than MCMC (Hastings, 1970) because it utilizes the gradient of the target distribution and fol- lows deterministic update prin- ciple. our work, the Gaussian mixture distribution determined by Pa. Po. and Ps fed into SVGD as the target distribution, and the representations of training examples are considered as the set of
Table 1: Disassembled relations prepared for label prompt words.
Relation Labels
Component-Whole(el..)))
("Component", "Whole")
Messago-Toppc((l..e)
("Message", "Topic")
Cause-Effect(el.ee2)
("Cause", "Effect")
Instrument-Agencc((ll.ee)
("Instrument", "Agency""
Content-Contaaiee((l...
("Content", "Container") ("Product", "Producer")
Product-Prodducer(e/
Membeen-Collectionnll..c
("Member". "Collection")
initial particles {6%) ) 'm=1" where is equal to the number of examples. By applying a form of functional gradient descent that minimizes the KI. divergence (Kullback & Leibler, 1951), SVGD iteratively transports the set of initial particles to match the target distribution. At the iteration É, the particle 6m O is updated by
1 M & (0 Vog logp (0g) Vo (0f.00)
rc+1 - 0t ego (0t). where ó (0)
M j-1
where ét is the step-size at the ε-th iteration. denotes the RBF kernel, and pe represents the target distribution determined by GMM. A particle obtains information from other particles by requesting their gradients, and further determines its update direction. The relevance of other
5

---

Published as a conference paper at ICLR 2024
particles is evaluated based the kernel distance, thereby assigning larger weights to particles that are closer (Yoon et al., 2018). The last term Var k (0j, employs the repulsive force between particles to prevent them from collapsing into a trivial point. Through iterative updates, the resulting particles e' = {6mm m=1
approximate the debiased factual distributions of downstream domains.
By uniform sampling from O', we can obtain the latent knowledge w, which represents debiased domain abstraction that provides de-ambiguous guidance for PLMs. Subsequently, the prompt with knowledge injection can be constructed for the RE task.
To fully leverage the substantial semantic knowledge embedded in relation labels, we define label prompt word lp to represent the implicit semantics of the relation. Inspired by (Chen et al., 2022b), we obtain a set of semantic words S, by disassembling the relation label and set SR S,2,......... S. where m the number of relation labels. The specific disassembly process is shown in Table 1. We suppose the probability distribution over the semantic words sets Sa as 6R = [0r1. dr2..... d.m], where the probability distribution is estimated through frequency statistics. Furthermore, impose the weighted average function for o, each word among S, to initialize the label prompt word Ip. which can inject the semantic knowledge of relations as follows:
é (lp) or (S,).
where é (l,) is the embedding of the label prompt word lns and e represents the word-embedding layer of LM. By injecting the semantic knowledge about the label into the label prompt word ly. we can facilitate the process of inferring relations.
Note that the Type Marker (Zhou & Chen, 2021) approach can enhance performance by incor- porating the type information of entities, but it requires additional annotations for entity types that are commonly unavailable. Inspired by the label prompt word lp. we further construct the type prompt word ip to inject entity type in- formation into prompts. For the relation label "perıdate of birth", is evident that the sub- ject entity matching such a relation belongs to "person". and the object entity matching such relation belongs to "date". With the prior knowl- edge contained in a specific relation, we can intu- itively acquire the scope of potential entity types, instead of relying on annotations. However, the applicability of this method is limited. For in- stance, given the relation "Cause-Effect(el..e cannot estimate the potential entity types, since this relation does not provide any infor- mation on the entities involved. To remedy this deficiency, we initialize the type prompt word tp with the latent knowledge w obtained by uni- form sampling:
Algorithm 1 BayesPrompt
Input: A set of training samples {x,, V:)
Generating features by using fixed PLM encoder F. F.).
h4 = E(x;), and H = [hh)
# Initializing a GMM to model the representa- tion distribution R2 lol. and Pp. Pr. (H, R,)
Initialized particles: (0%) } H.
# Adopting SVGD to approximate the debiased factual distribution for iteration é do @ 6 + EgΦ (0%.). where (0) 1 Σ K (0%, Vog logo (09) Vogk (05, 0]]
end for
Updated particles: O' ( M
m=
for t-th training iteration do
# Sampling to initialize prompts w O', and é (t,p) =
(Ip) = o, -. e(S))
# Minimize the loss to train the RE task min --=-Xx Exex ylogP (y z)
(tpp
(S)
where è (lp) ) is the embedding of type prompt word tp. The improved performance shows the
end for
effectiveness of the discriminative information contained in latent knowledge w.
To fully associate the initialized label prompt word 4p and type prompt word f, with the surrounding
context, we perform further optimization of their representation by the loss function computed as the cross-entropy between y and P (y) 2) as follows:
J
Σ
ylogP ([MASK] = M (T))
(6)
X xex
where |X| represents the number of samples in the training dataset. The procedure of BayesPrompt is summarized in Algorithm 1.

---

Published as a conference paper at ICLR 2024
Table 2: Fl scores (%) of prompt-tuning models with different settings. The best results are in bold. (BB-K) denotes the comparison between BayesPrompt and KnowPrompt, and (BB-R) denotes the comparison between BayesPrompt and RetrievalRE.
Fow-Sher Setting
Datasets
Split
FINE TUNING
GDPNET
PTR
KnowPromph
RetrievalRIE
BayesPrompt
((B-K)
((B-R)
K=1
18.5(+1.4)
10.3(+2.5)
14.7(=111)
28.6(=6.2)
33.3(=16)
35.1(+2.9)
SemBval
K=5
41.5(+2.3)
53.9(=1.9) 42.7(=2..)
(#+1())))
69.7(+1.7)
71.6(3.3)
+4.3
+1.23
K=16
66.1(40.4)
67.5(=0.8)
80.6(=1.2)
80.9(=1.6)
81.8(=10)
818((1.22
K=1
7.6(=3.0)
4.2(+3.8)
8.6(+2.5)
(7.6(=188)
19.5(=155)
22.5(+2.5)
TACRED
K=5
16.6(+2.1)
(5.5(+2.3)
24.9(+3.1)
28.8(+2.0)
30.7(+1.7)
31.4(+0.6)
+3
+1.27
K-16
26.8(+1.8)
28(+1.8)
30.7(+2.0)
34.7(+1.8)
(+.1()))2)
36.2((0.8)
K=1
7.2(=1.4)
5,1(+2.4)
9.((0)77)
17.8(=2.2)
18.7(=1.8)
21.9(22.0)
TACREV
K=5
16.3(+2.1)
17.8(+2.4)
(6.9(=1.5)
30.4(+0.5)
30.6(+0.2)
312((+0.)
+2.43
+1.37
K=16
25.8(+1.2)
26.4(=1.2)
31.4(+0.3)
33.2(+1.4)
35.3(+0.3)
35.6(+0.7)
5 THEORETICAL INSIGHTS WITH CONNECTION TO DOMAIN ADAPTATION
The gap between prompting problem and domain adaptation. The "domain adaptation (Qiang et al., 2021)" is learning from a source data distribution a well- well-performing model on a different (but related) target data distribution, while, such purpose has gap with our purpose in BayesPrompt Our method aims to fit the distribution of a few-shot domain, but we are not going to align the distributions of the target few-shot domain and the domain of PLMs. The intuition behind such behavior is that the distribution of the PLM domain is subject to the Gaussian distribution but the distribution of the few-shot domain is not the Gaussian distribution, such that arbitrarily aligning the distributions to fine-tune the PLM degenerates its ability to capture discriminative information, which also empirically proved by (Kumar al., 2022).
The reason for such a statement is that the innate assumption behind the Gaussian distribution is the sufficient samples, i.e., when the statistical samples are sufficient, the Gaussian distribution well fits the factual distribution of such discrete samples. However, for the few-shot scenario, the data is limited, such that the distribution can not be well fit by the Gaussian distribution. Therefore, if we directly fine-tune the PLM by aligning the distribution of the PLM domain (s.t. Gaussian distribution) with the distribution of the few-shot domain (s.t. another distribution), the knowledge of PLMS can be perturbed, and further, the discriminative information cannot be learned by PLMs.
Do the theoretical assumptions on a shared label space from domain adaptation hold in prompt- tuning? The behavior of BayesPrompt can be treated as implicitly adapting the target downstream domain to a subset of a specific PLM domain by leveraging a well-learned prompt. We will discuss the feasibility of transforming the theoretical objective of BayesPrompt to that of domain adaptation and why the domain adaptation cannot work in this scenario.
For "the feasibility of the transformation", in the prompt-tuning scenario, the downstream domain can be treated as the target domain, and the specific subset of the PLM domain can be treated as the source domain, i.c., the domain distribution alignment is performed between the specific subset of the PLM domain and the downstream domain, which have the shared labels. Furthermore, due to the mechanism behind prompt-tuning, i.e., the PLM network is frozen, we determine that both the gre-training data and the data from the downstream dataset are fed into the shared network to be projected into the shared latent space, and there is no any extra non-linear projection during the prediction. The only difference in features from the pre-training domain and the downstream domain the data, such that we derive conclusion that the specific subset of the PL.M domain and the downstream domain have shared label space, and the analyses based on the assumption of domain adaptation are theoretically feasible.
For "the reason why domain adaptation cannot work in the proposed scenario", following the aforementioned analyses, we disclose that the downstream domain can be bounded by the discrete data, but the specific subset of the PLM domain, which has the shared labels with the downstream domain, cannot be certainly determined, such that conventional domain adaptation methods cannot be directly leveraged to achieve the objective.
7

---

Published as a conference paper at ICLR 2024
Table 3: Standard RE performance of F1 scores (%) benchmarks. "w/o" indicates that the model does not use additional data, yet "w/" indicates the model requires extra data for downstream tasks.
Standard Setting
Methods
Extra Data
SomEval
TACRED
TACREV
RE-TACRED
Average
Fine tuning pre models
FINE TUNING
w/o
87.6
68.7
76.0
84,9
79.3
SPANBERT
w/
70.8
78.0
85.3
78.0
KNOWBERT
89.1
71.5
79.3
89.1
82.3
LUKE
80.6
76.7
MTB
89.5
70.1
79.8
GDPNET
w/o
71.5
79.3
75.4
Prompt-tuning pre-trained madels
72.4
83.7
KnowPrompt
w/o
90.2
72.4
82.4
91.3
84.1
RetrievaIRE
w/o
90.4
72.7
82.7
91.5
84.3
BayesPrompt
90.6
72.9
83,0
91.4
84.5
6
EXPERIMENTS
Benchmarking BayesPromiy on various datasets, We evaluate BayesPrompt on four RE datasets, namely SemEval 2010 Task 8 (SemEval) (Hendrickx et al., 2019). TACRED (Zhang al., 2017), TACREV (Alt et al., 2020), and ReTACRED (Stoica et al., 2021). We adopt the Fl score as the primary evaluation metric for the experiments on all the aforementioned datasets. In experiments, we select KnowPrompt (Chen et al., 2022b) our primary baseline, which is representative model that injects the latent knowledge contained labels into the prompt construction, thereby empowering the inference of relations. We further compare BayesPrompt with RetrievalRE (Chen et al., 2022a), which is the follow-up work of KnowPrompt. In the few-shot learning sctting, we perform 1-, 5-, and 16-shot experiments to assess the effectiveness of our method in low- resource scenarios. Following the benchmark experimental settings (Chen et al., 2022a), we report the F1 score and the standard deviation different benchmark datasets in Table 2. The results demonstrate that on average, BayesPrompp beats KnowPromp by 3.24% among benchmark datasets. For RetrievalRE, BayesPrompt achieves an average improvement of 1.29% among benchmark datasets. The statistical results further demonstrate the effectiveness of BayesPrompt. Besides, we also perform the significance test, i.e., t-test, with the primary baseline, and observe that the P values are consistently lower than 0.05, e.g., 0.045 on the SemEval dataset, indicating that the improvement of BayesPrompt is significant.
Validation of the robustness of BayesPrompt against knowledge ambiguity. We perform further experiments to prove that BayesPrompt is able to mitigate the negative impact of knowledge ambiguity, and the results are shown in Table 3. Considering the importance of entity knowledge in facilitating the understanding of relational semantics for models, we select SpanBERT (Joshi et al., 2020), KnowBERT (Peters et al., 2019), LUKE (Yamada al., 2020), and MTB (Soares et al., 2019) as basclines, which are representative models leveraging the external knowledge to enhance learning objectives, input features, model architectures, pre-training strategies. As shown in Table BayesPrompt outperforms the compared knowledge-enhancee models, indicating that despite the task-specific knowledge already contained in knowledge-enhanced PLMs. remains challenging for fine-tuning to sufficiently leverage such knowledge for downstream tasks. We compare BayesPrompt with the principal baseline, i.e., KnowPrompt, in the standard setting, and the results demonstrate that BayesPrompp can yield 0.4% performance rise on average. When compared to the RetrievalRE. BayesPrompt still achieves an average performance improvement of 0.2%, further highlighting its superiority. The intriguing outcome demonstrates the merit of the proposed method: even when the amount of data changes, BayesPrompt remains robust in alleviating the negative impact of the knowledge ambiguity, i.c., the model maintains consistent performance.
Training Complexity of BayesPrompt. We compare the training complexity of BayesPrompt and KnowPrompt on the SemEval and TACREV datasets through experiments. The results are presented in Table 4, which indicates that BayesPrompt has slightly higher time complexity than the benchmark method due to the necessity of adding prompts containing domain discriminative information for downstream tasks during each prediction to provide de-ambiguous guidance for PLMs. Nevertheless, the results in Table 2 and Table 3 demonstrate that although BayesPrompt has
8

---

Published as a conference paper at ICLR 2024
Table 4: The complexity comparisons between BayesPrompt and KnowPromp on SemEval and TACREV. Note that for fair comparisons, this experiment is based on I GPU of NVIDIA 3090.
Methods
Parameters
Datasets
Training time cost for an epoch
1-shot
5-shot
16-shot
full dataset
SemEval
16s
19s
26s
198s
KnowPrompt
355M
TACREV
217s
228s
239s
2096s
SemEval
27s
31s
43s
242s
BayesPrompt
355M
TACREV
332s
344s
376s
2422s
SemEval
50.6
1-shot
full dataset
95
20.4
71.6
61.3
35-3
60
75
313
21
65
40
55
SemPual
TACRID
TACRFV
RATACAFD
SemPaal
TAGRED
TACRIV
RATACHEO
-c-Siusian
Figure 4: (a) GMM vs. Gaussian results the SemEval dataset regarding different shots. (b) and (c) the ablation study on different datasets, where del TPW refers to the removal of type prompt words.
a slightly higher time complexity, its performance improvement is consistent and substantial. For instance, in the 1-shot setting of the SemEval dataset, the training time cost of each epoch the main baseline is 16 seconds, while the training time cost of each epoch at BayesPrompt is 27 seconds. However, the main baseline achieves an F1 score of 28.6%, whereas BayesPrompt achieves F1 score of 35.1%.
Ablation study. For the approximation of debiased factual distributions. we comprehensively consider the Gaussian distribution and GMM as the candidate known distributions. The empirical results are demonstrated in Figure 4 (a), and we observe that our method using GMM achieves relatively suitable and effective performance. Figure 4 (b) and (c) demonstrate the effects of the discriminative prompts. Specifically, the 1-shot setting on TACRED, the performance drops from 22.5% to 20.2% when removing type prompt words, indicating that the discriminative prompts are effective for few-shot inference. We conduct further empirical analyses in Appendix A, including the analysis of case study, extended ablation study, etc.
7 CONCLUSIONS
We first clarify that the over-multitudious conceptual knowledge contained in PLMs and the in- complete knowledge for the target downstream domain are the essences incurring the defects of of-e--of-the-ar prompt-tuning approaches. To remedy such defects, we propose BayesPrompt to approximate the debiased factual distribution of downstream domain, and further uniformly sample certain representative features from the approximated distribution to generate the prompts contain- ing the domain discriminative information. Experiments demonstrate the consistent performance superiority BayesPrompt over baselines.
Limitations and broader impacts. The training complexity of BayesPrompt is slightly higher than the baselines, and the distribution approximation possesses the potential for further optimization. Prompt-tuning requires that the target domains and the pre- pre-training domain violate the identically dis- tributed assumption. The scientific question explored by this work only has positive and inspirational impacts the prompt-tuning community.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGEMENTS
The authors would like to thank the editors and reviewers for their valuable comments. This work supported by the Fundamental Research Program, China, Grant No. JCKY2022130C020, the National Funding Program for Postdoctoral Researchers, Grant No. GZC20232812, the CAS Project for Young Scientists Basic Research, Grant No. YSBR-040, the Youth Innovation Promotion Association CAS, No. 106, 2022 Special Research Assistant Grant Project, No. E3YD5901, and the China Postdoctoral Science Foundation, No. 2023M743639.
REFERENCES
Christoph Alt, Aleksandra Gabryszak, and Leonhard Hennig. Tacred revisited: A thorough evaluation of the tacred relation extraction task. arXiv preprint arXiv:2004. /4855, 2020.
Eyal Ben- Davi-D Nadav Oved, and Roi Reichart. Pada: A prompt-based autoregressive approach for adaptation to unseen domains. arXiv preprint arXiv:2 /2206, 2021.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et Language models are few-shot learners. Advances in neural information processing systems, 33:1877 2020.
Xiang Chen, Lei Li, Ningyu Zhang, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Relation extraction as book examination: Retrieval-enhaneed prompt tuning. In Proceedings of the 45th International ACM SIGIR Conference on Research and Development in Information Retrieval, pp. 2443-2448, 2022a.
Xiang Chen, Ningyu Zhang, Xin Xie, Shumin Deng, Yunzhi Yao, Chuanqi Tan, Fei Huang, Luo Si, and Huajun Chen. Knowprompt: Knowledge-aware prompt-tuning with synergistic optimization for relation extraction. In Proceedings of the ACM Web Conference 2022, pp. 2778-2788, 2022b.
Andrew M Dai and Quoc V Le. Semi-supervised sequence learning. Advances in neural information processing systems, 28, 2015.
Ning Ding, Yulin Chen, Xu Han, Guangwei Xu, Pengjun Xie, Hai-Tao Zheng, Zhiyuan Liu, Juanzi Li, and Hong-Gee Kim. Prompt-learning for fine-grained entity typing. arXiv preprint a10iv:2108. 10604, 2021.
Tianyu Gao, Adam Fisch, and Danqi Chen. Making pre-trained language models better few-shot learners. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pp. 3816-3830, Online, August 2021. Association for Computational Linguisties.
Yuxian Gu, Xu Han, Zhiyuan Liu, and Minlie Huang. Ppt: Pre-trained prompt tuning for few-shot learning. arXiv preprint X1Xx:2109.00333 2021.
Xu Han, Weilin Zhao, Ning Ding, Zhiyuan Liu, and Maosong Sun. Ptr: Prompt tuning with rules for text classification. Al Open, 3:182-192 2022.
W Keith Hastings. Monte carlo sampling methods using markov chains and their applications. 1970.
James J Heckman. Sample selection bias as specification error. Econometrica: Journal of the econometric society, pp. 153-161, 1979.
Iris Hendrickx, Su Nam Kim, Zomitsa Kozareva, Preslav Nakov, Diarmuid O Séaghdha, Sebas- tian Padó, Marco Pennacchiotti, Lorenza Romano, and Stan Szpakowicz. Semeval-2010 task 8: Multi-way classification of semantic relations between pairs of nominals. arXiv preprint 191 10422, 2019.
Shengding Hu, Ning Ding, Huadong Wang, Zhiyuan Liu, Jingang Wang, Juanzi Li, Wei Wu, and Incorporating knowledge into prompt verbalizer
Maosong Sun. Knowledgeable for text classification. arXiv preprint arXiv: 100.00035. 2021.
10

---

Published as a conference paper at ICLR 2024
Yuezihan Jiang, Hao Yang, Junyang Lin, Hanyu Zhao, An Yang, Chang Zhou, Hongxia Yang. Zhi Yang. and Bin Cui. Instance-wist wise prompt tuning for pretrained language models. arXiv preprint MxXiv:2206.01958 2022.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel Weld, Luke Zettlemoyer, and Omer Levy. Spanbert: Improving pre-training by representing and predicting spans. Trunsactions of the Association for Computational Linguistics, 8:64-77, 2020.
Solomon Kullback and Richard A Leibler. On information and sufficiency. The annals of mathemati- cal statistics, 22(1):79-86, 1951.
Ananya Kumar, Aditi Raghunathan, Robbie Jones, Tengyu Ma, and Percy Liang. Fine-tuning can distort pretrained features and underperform out-of-distribution. arXiv preprint arXiv:2202.10054, 2022.
Brian Lester, Rami Al-Rfou, and Noah Constant. The power of scale for parameter-efficient prompt tuning. arXiv preprint arXiv:2104. 08491, 2021.
Haochen Li, Tong Mo, Hongcheng Fan, Jingkun Wang, Jiaxi Wang, Fuhao Zhang, and Weiping Li. Kipt: Knowledge-injecte prompt tuning for event detection. In Proceedings of the 29th International Conference on Computational Linguistics, pp. 1943-1952, 2022a.
Jiangmeng Li, Wenyi Mo, Wenwen Qiang, Bing Su, and Changwen Zheng. Supporting vision-language model inference with eausality-pruning knowledge prompt. arXiv preprint 1rXiv:2205.11100 2022b.
Jiangmeng Li, Yanan Zhang, Wenwen Qiang, Lingyu Si, Chengbo Jiao, Xiaohui Hu, Changwen Zheng, and Fuchun Sun. Disentangle and remerge: interventional knowledge distillation for few-shot object detection from conditional causal perspective. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pp. 1323-1333, 2023.
Shaobo Li, Xiaoguang Li, Lifeng Shang, Zhenhua Dong, Chengjie Sun, Bingquan Liu, Zhenzhou Ji, Xin Jiang, and Qun Liu. How pre-trained language models capture factual knowledge? a causal-inspired analysis. arXiv preprint 2203::220331674 2022c.
Xiang Lisa Li and Percy Liang. Prefix-tuning: Optimizing continuous prompts for generation. arXiv preprint 1xXi::2111.00190 2021.
Jintao Liu, Zequn Zhang, Zhi Guo, Li Jin, Xiaoyu Li, Kaiwen Wei, and Xian Sun. Kept: Knowledge enhanced prompt tuning for event causality identification. Showledge-Basee Systems, 259: 10064, 2023a.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, prompt, and predict: A systematic survey of prompting methods in natural language processing. ACM Computing Surveys, 55(9):1-35, 2023b.
Qiang Liu and Dilin Wang. Stein variational gradient descent: A general purpose bayesian inference algorithm. Advances neural information processing systems, 29, 2016
Xiao Liu, Kaixuan Ji, Yicheng Fu, Weng Lam Tam, Zhengxiao Du, Zhilin Yang, and Jie Tang. P-tuning v2: Prompt tuning be comparable fine-tuning universally across scales and tasks. arXiv preprint arXiv:2110.0760) 2021.
Yao Lu, Max Bartolo, Alastair Moore, Sebastian Riedel, and Pontus Stenetorp. Fantastically ordered prompts and where to find them: Overcoming few-shot prompt order sensitivity. arXiv preprint N.Xi:2200408786.6 2021.
Fang Ma. Chen Zhang, Lei Ren. Jingang Wang, Qifan Wang, Wei Wu, Xiaojun Quan, and Dawei Song. Xprompt: Exploring the extreme of prompt tuning. arXiv preprint arXiv:2210.04457 2022.
Matthew E Peters. Mark Neumann, Robert L Logan IV, Roy Schwartz, Vidur Joshi, Sameer Singh, and Noah A Smith. Knowledge enhanced contextual word representations. arXiv preprint MnXiv:/909.00 164, 2019.
11

---

Published as a conference paper at ICLR 2024
Wenwen Qiang, Jiangmeng Li, Changwen Zheng, Bing Su, and Hui Xiong. Robust local preserving and global aligning network for adversarial domain adaptation. IEEE Transactions on Knowledge and Data Engineering. 2021.
Douglas A Reynolds et al. Gaussian mixture models. Encyclopedia of biometrics, 741(659-663), 2009.
Timo Schick and Hinrich Schütze. Exploiting cloze questions for few shot text classification and natural language inference. arXiv preprint XXiv:2001.07677 2020.
Hidetoshi Shimodaira. Improving predictive inference under covariate shift by weighting the log- likelihood function. Journal of statistical planning and inference, 90(2):27-2244, 2000.
Taylor Shin, Yasaman Razeghi, Robert I. Logan IV, Eric Wallace, and Sameer Singh. Autoprompt: Eliciting knowledge from language models with automatically generated prompts. arXiv preprint arXiv:20I0. 1e.00.119880 15980, 2020.
Livio Baldini Soares, Nicholas FitzGerald, Jeffrey Ling, and Tom Kwiatkowski. Matching the blanks: Distributional similarity for relation learning. arXiv preprint urXiv:/906.000 1906.03/58 2019.
George Stoica, Emmanouil Antonios Platanios, and Barnabás Póczos. Re-tacred: Addressing shortcomings of the tacred dataset. In Proceedings the AAAI Conference on Artificial Intelligence, volume 35, pp. 13843-13885 2021.
Liangguo Wang, Ruoyu Chen, and Li Li. Knowledge-guided prompt learning for few-shot text classification. Electronics, 12(6): 1486, 2023.
Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a few examples: A survey on few-shot learning. ACM computing surveys (csur), 53(3):1-34, 2020.
Colin Wei, Sang Michael Xie, and Tengyu Ma. Why do pretrained language models help in downstream tasks? analysis of head and prompt tuning. Advances in Neural Information Processing Systems, 14:16158-16170, 2021a.
Jason Wei, Maarten Bosma, Vincent Y Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M Dai, and Quoe Le. Finetuned language models are zero-shot learners. arXiv preprint 11X.v2200..0155, 2021b.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed Chi, Quoc Le, and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. arXiv preprint arXiv:2201.11903, 2022.
Ikuya Yamada, Akari Asai, Hiroyuki Shindo, Hideaki Takeda, and Yuji Matsumoto. Luke: Deep con- textualized entity representations with antity-aware self-attention. arXiv preprint arXiv:2010.01057, 2020.
Jaesik Yoon, Taesup Kim, Ousmane Dia, Sungwoong Kim, Yoshua Bengio, and Sungjin Ahn. Bayesian monel-agnostic meta-learning. Advances in neural information processing systems, 31, 2018
Yuhao Zhang, Victor Zhong, Danqi Chen, Gabor Angeli, and Christopher Manning. Position-aware attention and supervised data improve slot filling. In Conference Empirical Methods in Natural Language Processing, 2017.
Wenxuan Zhou and Muhao Chen. An improved baseline for sentence-level relation extraction. arXiv preprint 1rX1v:2102.01373, 2021.
Yongehao Zhou, Andrei Ioan Muresanu, Ziwen Han, Keiran Paster, Silviu Pitis, Harris Chan, and Jimmy Ba. Large language models are human-level prompt engineers. arXiv preprint arXiv:2211.01910, 2022.
12

---

which Published as a conference paper at ICLR 2024
A FURTHER ANALYSIS
A.1 CASE STUDY
Case Study 1: ("token": ['the", "launcher", "receives", "the", "balls", "through", "a", "similar", "belt", "system", "leading", "up", "the", "neck", "of", "the", "robot", "from", "the", "ground", "."], "h": "name": "launcher", "pos": [1. 2], "t": "name": "system", "pos": [9, 10), "relation": "Instrument- Agency(e2,e1)))
In the sentence "the launcher receives the balls through similar belt system leading up the neck of the robot from the ground", the head entity is "launcher". and the tail entity is "system". KnowProme predicts the relation between the two entities as Cempponent-Wholleeeeee while BayesPrompt predicts the relation between the two entities as "Instrument-Agennyeeeeee which aligns with the true label.
Case Study 2: !"token": ["from", "banishing", "cold", "and", "flu", "germs", "to", "preventing", "foodborne", "illnesses", ". "frequent", hand-washing "is", "one", "of", "the", "smartest", "preven- tive". "habits". "you", "can", "adopt". "."J, "h": "name": "cold", "pos": [2, 3], "t": "name": "germs", "pos": [5, 6], "relation": "Cause- =ffeecceefc((
In the sentence "from banishing cold and flu germs to preventing foodborne illnesses, frequent hand-washing is one of the smartest preventive habits you can adopt", the head entity is "cold", and the tail entity is "germs". KnowPrompt predicts the relation between the two entities as "Other", while BayesPrompt predicts the relation between the two entities as CCause-Effectee..e. in alignment with the true label.
Case Study 3: { ("token"; ["a", "child", "is", "told", "a", "lie", ; "for", "several", "years", "by", "their", "parents", "before", "he/she", "realizes", "that", "a", "santa", "claus", "does". "not", "exist", "."]. "h": "name": "lie", "pos": [5, 6]. "t": "name": "parents", "pos": [11, 12]. "relation": "Product- Producer(el,ee2))
For the sentence "a child is told a lie for several years by their parents before he/she realizes that santa claus does not exist", the head entity "lie", and the tail entity is "parents", KnowPrompt predicts the relation between the two entities as "Other", while BayesPrompt predicts the relation between the two entities as Prroduct-Producee(ee....
aligning with the true label.
We attribute these phenomena to the fact that KnowPrompt only captures the general meanings of entities, whereas BayesPrompt comprehends the genuine context-specifie meanings of entities. This can be attributed to the utilization of prompts derived from debiased factual distributions of downstream domains.
Conclusion:
A.2 ANALYSIS OF COMPARATIVE RESULTS BETWEEN GAUSSIAN DISTRIBUTION AND GMM
The innate assumption behind the Gaussian distribution is the sufficient samples, i.e., when the statistical samples are sufficient, the Gaussian distribution well fits the factual distribution of such discrete samples. However, for the few-shot scenario, due to the insufficiency of samples of the target domain, the corresponding distribution of the target domain may not adhere to the typical Gaussian distribution, i.e., the central limit theorem may not well fit the distribution of the target downstream domain. Furthermore, if we strive to use the Gaussian distribution to the distribution of the target domain, the derived distribution must have significant shift from the factual distribution of the target domain. Theoretically, GMM can fit any probability density distribution, so we use Gaussian mixture distribution to approximate the factual distribution of the target domain.
We observe from the empirical results that as the sample size increases, the performance gap between BayesPrompt using GMM and BayesPrompt using the simple Gaussian distribution decreases, which further proves our analysis that when the samples are limited, the distribution the target domain does not fit the typical Gaussian distribution, while according to the central limit theorem, as the increasing of the sample size, the distribution of the target domain gradually fits the typical Gaussian distribution.
Specifically, for the extreme setting of fhw-shot learning, e.g., 1-shot, the available data is excessively limited, and the data is insufficient for the fitting of both GMM and the simple Gaussian distribution,
13

---

Published as a conference paper at ICLR 2024
such that the performance of BayesPrompt degenerates with using any candidate distribution. For the relatively limited few-shot data, e.g., 5-shot, BayesPrompt using GMM shows its superiority over the model using the simple Gaussian distribution, which well fits our aforementioned analyses, while, for the relatively sufficient few-shot data, the true distribution of the target domain gradually approaches the Gaussian distribution, such that the performance gap between the compared model remains decreasing as the increasing of available data size. Eventually, we can still find that BayesPrompt using GMM can consistently outperform BayesPrompt using the simple Gaussian distribution, which indicates that using GMM instead of the typical Gaussian distribution is theoretical and technically solid.
A.3 ANALYSIS OF PERFORMANCE IMPROVEMENT IN FEW-SHOT VS. STANDARD SETTINGS
By comparing Table 2 and Table 3, it can be observed that the performance improvement of BayesPrompt in few-shot scenarios widely surpasses its performance improvement in the stan- dard scenario, i.e., with the full dataset. This observation precisely validates our motivation and the effectiveness of the method.
As we discussed in the Abstract and Introduction sections, the limited and discrete semantic informa- tion contained in the training samples from downstream domains can barely support the conventional trainable prompts to acquire sufficient supervision, such that the guidance of the generated prompts trivial PLMs. Especially, such a challenge further exacerbates the performance of PLMs in few-shot scenarios. With this motivation, the proposed BayesPrompt aims to learn prompt that contains relatively sufficient discriminative knowledge for the target downstream domain, which is achieved by proposing the debiased domain abstraction and then generating the prompt. Thus, the empirical observation, i.e., the performance improvement derived by introducing BayesPrompt on few-shot scenarios is more significant than that standard scenarios, jointly proves the motivation and the effectiveness of the proposed BayesPrompt.
Specifically, in few-shot scenarios, the limited amount of data may hinder PLMs from fully learning the distribution and features of the downstream task data, leading to relatively poorer performance. Therefore, when utilizing prompts obtained from BayesPrompt, which contain domain discriminative information, to guide PLMs in locating knowledge domains relevant to the downstream domain, the deficiencies caused by the limited data are noticeably mitigated. In contrast, in the standard scenario, where the amount of downstream task data is relatively sufficient, PLMS can better grasp the features and distribution of the downstream task data by learning from a more comprehensive dataset. Consequently, the performance improvement brought by BayesPrompt may not be as pronounced in this context. However, the results in standard scenarios further demonstrate the generalizability of our exploration and the effectiveness of BayesPrompt.
A.4 EXTENDED ABLATION STUDY AND ANALYSIS
We further conduct ablation study on the number of components in the GMM introduced in BayesPrompt, as shown in Table 5. The bolded entries correspond to the selected number of GMM components determined by the number of relation types in the experiment, along with the obtained results. It is observed that, compared to other settings, our choice achieved the best performance. This indicates that our selection of the number of components is appropriate.
The intuition behind the discriminative prompts is that the domain discriminative information is injected into the prompt in BayesPrompt. As shown in Figure 4 (b) and (c). the effectiveness of the type prompt words, i.e., the discriminative prompts proposed by BayesPrompt, is demonstrated. It can be seen that BayesPrompt consistently outperforms the ablation model on all datasets within various experimental settings. Additionally, the performance improvement brought by the discrimi- native prompts in the few-shot scenario (Figure (b)) is significantly stronger than its performance enhancement in the full dataset (Figure 4 (c)). We attribute this difference to the fact that, in the few-shot scenario, the limited amount of data may hinder PLMs from fully learning the distribution and features of the downstream task data, leading relatively poorer performance. So, when utilizing the discriminative prompts to guide PLMs in locating knowledge domains relevant to the downstream domain, the deficiencies caused by the limited data noticeably mitigated. However, in the full dataset, the relatively abundant downstream task data provides a rich knowledge background, re- sulting in substantial reduction in the deviation between the knowledge domain located by PLMs
14

---

Published as a conference paper at ICLR 2024
Table 5: F1 scores (%) of BayesPrompt with different numbers of GMM components.
Dataset
Split
Number of Components in GMM
BayesPrompl
9
33.7(#3.6)
K=1
19
35.1(+2.9)
9
70.3(#2.8)
K=5
19
71.6(#3.3)
SemEval
27
70.8(+2.7)
80.8(#1.1)
K=16
19
81.8(#1.2)
27
80.8(+1.5)
and the real downstream knowledge domain. This, in turn, weakens the de- de-biasing effect of the discriminative prompts on the few-shot dataset. Therefore, the performance improvement brought by the discriminative prompts is more pronounced in the few-shot scenario than in the full dataset, which can effectively prove the validation of our exploration and motivation in the Introduction.
B IMPLEMENTATION DETAILS FOR BAYESPROMPT
B. STATISTICAL DETAILS OF DATASETS
We evaluate BayesPrompt on four RE datasets: SemEval 2010 Task 8 (SemEval), TACRED, TACREV, and ReTACRED, SemEval a popular relation classification dataset that includes 9 relation types with bidirectional labels and an additional "Other" category. TACRED is widely used dataset for relation classification, obtained through crowd-sourcing, which comprises 42 types of relations, including the "no relation" label. TACREV is corrected version of the TACRED dataset, where errors in the original development and test sets were identified and fixed while keeping the training set intact. ReTACRED is another modified version of the TACRED dataset that addresses its shortcomings by refactoring its training, development, and test sets and modifying few relation types. More details of these datasets are shown in Table 6.
of
Table 6: Statistics of different datasets.
Dataset
#train
#val
#test
#rel
SemEval
6507
1493
2717
19
TACRED
68124
22631
15509
42
TACREV
68124
22631
15509
42
ReTACRED
58465
19584
13418
40
SemEval
TACRED
TACREV
ReTACRED
91.6
91.4
90.5
90
90.5
72.5
72
82 8
81.2
81.8 81.8
91.2
90.8
91
91.3
91.4
91.3
90.9
-512
89.5
71.5
90.6
1o-5
3e-5
Se-5
30-5
3a-5
So.5
1o.5
20-5 3a-5
20-5 4a-5
Figure 5: Comparisons of BayesPrompt using different settings of the learning rate.
B.2 PARAMETERS SETTING IN EXPERIMENTS
The experiments were conducted using Pytorch on 4 Nvidia 3090 GPUs. The learning rate search space of optimization for overall parameters is shown as Figure 5. To form the few-shot training sets, sample k instances of each relation from the initial training sets. Specifically, uniform samplings using fixed set of seeds and record the average performance and the standard deviation. For all fhw-shot datasets, fine-tune model for 50 epochs with batch size of 4.
conduct five
15

---

Published as a conference paper at ICLR 2024
For full datasets, the number of epochs is set to 5, except SemEval, which is set to 10, and the batch size is 16. The best model checkpoint is selected based on the performance of the validation set. To ensure fair comparison, use RoBERTa-large for all experiments. We adopt the Fl score the primary evaluation metric for the experiments on all the aforementioned datasets since the F1 score can comprehensively assess the precision and recall performances.
16


Reviews:
[{"review_id": "1Mxd6Sfdlo", "reviewer": "Reviewer_32zy", "summary": "The paper proposes BayesPrompt  to inject the semantic knowledge about the label into the label prompt to adjust the knowledge learned from pretraining to better fit downstream tasks. The method is to learn prompts that contain the domain discriminative information for the interference from the domain-irrelevant knowledge by approximating the factual distributions of downstream domains. The approach learns a representative model that injects the latent knowledge contained in labels into the prompt construction, thereby empowering the inference of relations.", "questions": "See above.", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 3, "strengths": "The paper works on a very interesting problem to adjust pretraining knowledge of LLM to downstream tasks. The paper provides theoretical analyses demonstrates that BayesPrompt can tighten\nthe upper bound of the classification error on the downstream inference of PLMs. Table 2 provide standard deviations over multiple runs.", "weaknesses": "The paper may benefit a lot from better writing, including more clear presentation of the motivation and methods. \n- what does author refer to for \"unabridged Domain\", \"partial domain\", in figure 2? \n\n\"Thee over-multitudinous conceptual knowledge contained in PLMs and the abridged knowledge for target downstream domains, which jointly result in that PLMs mis-locate the knowledge distributions corresponding to the target domains in the universal knowledge embedding space. \n\"\n- what does the author refer as \"over-multitudinous conceptual knowledge\" and \"the abridged knowledge \"?\n\nIt is not fully convinced to the reviewer that the problem motivates the method can be solved by the method proposed. \nIt is unclear that how by \"leveraging Gaussian mixture distribution BayesPrompt is able to approximate the debiased factual distributions of downstream domains and further uniformly samples certain representative features from the approximated distributions to generate the ultimate prompts for PLMs\". Why the proposed approach can better approximate the downstream tasks distribution? by injecting label -related information? What is the bias referred here? Is there any produces to reduce the bias? The author may refer the bias as \"irrelevant pretraining knowledge\" that is confounding for the downstream tasks ? not very clear why introducing \"Gaussian mixture distribution\" can help solve the problem? is it for sampling and easy injecting label-related knowledge? \n\nby injecting label dependent knowledge, the PLM may learn a PLM distribution that is useful for the downstream task, which makes sense. but is it unfair, as BayesPrompt already uses label information but other methods don't?\n \nIt is not very clear how Figure 2 motivates the paper.  Figure 1 (domain knowledge is helpful ) and Figure 2 (domain knowledge may lead to negative impact?) seem not to align well. \n\nMethod section: how does  label prompt word lp and type prompt word tp fit in eq(6)? Can the author also bring some clarify to training? \n \nWhy does the approach focus on relation extraction tasks (used in method section)? how about other tasks? Is this method currently specific for relation extraction tasks? \n\nTable 2: the improvement seems not to exceed one standard deviation over other baselines. The category of tasks seem limited. not very convinced on the effectiveness of the methods.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "KjsRhanczu", "reviewer": "Reviewer_JjWU", "summary": "This paper introduces a prompting method named BayesPrompt to generate prompts for PLMs. The authors argue that the over-multitudinous knowledge implicit in PLMs can hinder the performance of prompt-tuning methods in few-shot settings. Thus, BayesPrompt aims to approximate the unbiased target distribution to generate discriminative prompt for specific domains.\nExperimental results show the effectiveness of BayesPrompt on relation extraction (RE) tasks. Also, the authors provide theoretical analysis over BayesPrompt on lowering the classification error upper bound.", "questions": "BayesPrompt's training complexity is higher than its baseline, is there any potential for optimization?", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1) The task that improves the generalization capabilities of PLMs is challenge in the prompt tuning community. The authors provide a new view from the \"mislocated knowledge distributions\" between PLMs and target domain, which is interesting. \n\n2) The motivation that adopts the Bayesian approaches to model dataset-specific information and performing prompting on the latent space is novel.\n\n3) The provided theoretical analyses and extensive experiments help readers to understand the method.", "weaknesses": "1) As can be seen from Tables 1 and 3, the proposed BayesPrompt presents a completely different improvement. Can the authors provide a detailed explanation?\n\n2) Please provide more discussion about the ablation results at Figure 4(c).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "JXKPMbV4ft", "reviewer": "Reviewer_ox7r", "summary": "The paper proposes BayesPrompt, a Bayesian approach to approximate the factual distributions of downstream domains \nand thereby generating discriminative prompts for PLMs. The authors articulate that the intrinsic issues behind the poor performance of finetuned PLMs on few-shot downstream tasks roots from two main shortcomings: (i) over-multitudinous of conceptual knowledge contained in PLMs, (ii) an abridged knowledge for target downstream domains. The paper takes a stride in addressing this challenge with both theoretical (tailored towards a classification problem) as well as experimental results.", "questions": "No further questions (beyond what's already raised in weaknesses), and after reading through the Appendix.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written, well structured and has a clear narrative. \n- The authors pay utmost attention to details, from notations and math to presentation of the results, making the paper easy to follow.\n- The paper has a healthy mix of a (simplified) theoretical and qualitative arguments, based on which the approach is devised. \n- The results seem to be promising, comparing against some recent baselines. \nOverall, it seems like a solid contribution.", "weaknesses": "- The paper is essentially a shortened version of a much longer manuscript, where the authors are constantly cutting the content short and referring the reader to different sections of the appendix (appendix is referred to 11 times throughout the paper!). So, the main body of the paper is not really self-contained and heavily relies on the appendix. By the same token, the main algorithm of the paper had to be pushed to the Appendix, which could be a natural choice in the main text to clarify the end-to-end procedure. \n- The impact of the proposed approach is rather marginal when compared to the closest competitors (say RetrievalRE), especially on standard RE performance in Table 3, while at the same it comes at the cost of extra training complexity. Any reason behind this?  \n- No Ablation studies. There are design choices that could potentially establish the basis for Ablation studies (such as Kernel size and so).\n\n---\nPost rebuttal comments\n\nI spent some time to review the code and implementation, per AC's recommendation. While the base code seems to run without any major issues, there are a few points I would like to highlight, for what its worth, as I had to grind my way through to make it run:\n\n1. The README file provides steps for execution; however, the instructions have some ambiguity. For two of the steps it is mentioned \"Modify transformer.py\" or \"Update GMM_SVGD.py\" but it does not say what to modify in the file.\n\n2. Problems with the requirements.txt\n    - Packages missing: pandas, wandb\n    - The pytorch-lightning version mentioned has issues with the associated torchmetrics version. Although it can be resolved by installing torchmetrics==0.6.0 (based on https://github.com/NVIDIA/DeepLearningExamples/issues/1113), it would be much easier if it resolved in the provided requirements file.\n\nNext to that, on a deeper dive into the proofs, I have a difficult time follow the logic as well as mathematical formulations. This has to be revisited and resolved for the final draft.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "VyEobgXm2s", "reviewer": "Reviewer_wpkj", "summary": "Prompt-tuning is a fine-tuning paradigm based on large-scale pre-trained language models (PLMs), which can reduce the gap between downstream tasks and pre-training objectives. This paper focus on the challenge of poor generalization to specific few-shot patterns of the Prompt-tuning. Through distribution analysis, they reveal that the root cause of this issue is the overabundance of conceptual knowledge in PLMs and the truncated knowledge for target downstream domains. This collective effect misaligns the knowledge distributions corresponding to the target domains in the universal knowledge embedding space. To address this issue, they propose BayesPrompt, an approach that intuitively explores debiased approximation of unabridged target domains of downstream tasks. BayesPrompt generates domain-discriminative prompts to provide unambiguous guidance for PLMs. Further, they theoretically show that BayesPrompt tightens the upper bound of the classification error on PLMs' downstream inference on classification error bounds. The experimental results show that the proposed method achieves SOTA performance on benchmarks.", "questions": "1.\tWhy can GMM approximate the target domain? What are its benefits than a learnable generator (VAE or GAN)?\n2.\tIs it required to train a specific GMM for each input sentence (X, Y)?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe paper reveals the principles of the challenge of prompt-tuning on pre-trained large models for few-shot tasks.\n2.\tThe methodology of using the Bayesian prompt is novel and effective.\n3.\tThe theoretical guarantees the performance of the proposed method.\n4.\tThe evaluation presents the benefits of the proposed method.", "weaknesses": "1.\tThis paper utilizes the GMM to approximate the distribution of the target domain which may not be unabridged. The real distribution of the target domain is complex and unknown.\n2.\tThe PLMs utilized in the evaluation are not clear. Using various PLMs may be better to show the generality of the proposed method.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)