Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks
Authors: Zhaohan Xi
Tianyu Du
Changjiang Li
Ren Pang
Shouling Ji
Jinghui Chen
Fenglong Ma
Ting Wang
Venue: NeurIPS 2023 poster (2023)
Keywords: few-shot learning
prompt learning
language model
backdoor defense

Abstract: Pre-trained language models (PLMs) have demonstrated remarkable performance as few-shot learners. However, their security risks under such settings are largely unexplored. In this work, we conduct a pilot study showing that PLMs as few-shot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, we advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking-sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. We show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. The code of MDP is publicly available.

Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks
Zhaohan Xi
Tianyu 2 Jinghui Chen' Pennsylvania State University 2Zhejiang University *Stony Brook University (zhaohan. xi, rbp5354, jzc5917, fenglong]@psu edu (zjradty, sji}ezju. edu. cn {changjli, twang)@cs stonybrook. edu
Changjiang Li
Ren Pang' Shouling Ji
Fenglong Ma² Ting Wang
Abstract
Pre-trained language models (PLMs) have demonstrated remarkable performance as few-shot learners. However, their security risks under such settings are largely unexplored. In this work, we conduct a pilot study showing that PLMs as few- shot learners are highly vulnerable backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, we advocate MDP, a novel lightweight, pluggable, and effective defense for PL.Ms as few-shot learners. Specifically, MDP leverages the gap between the masking-sensse sensiiiiii of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations given samples under varying masking and identifies poisoned samples as ones with significant variations. We show analytically that MDP creates interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. Code available at atthtss: //ts::///tttbb com/z Baahan-xi/PLM--roomptt
Introduction
The prompt-based learning paradigm is revolutionizing the ways of using pre-trained language models (PLMs) 27 28
various NLP tasks. Unlike the conventional fine-tuning paradigm that requires re-training the PLM, the prompt-based paradigm reformulates the downstream task as masked language modeling problem and uses proper prompts to coax the model to produce textual outputs [17]. For example, to analyze the sentiment of a movie review, one may append the prompt to the given review and guide the model to predict the missing sentiment word (e.g., "terrible" or "great"). Recent work shows that with proper prompting, even moderate-sized PLMS can be adapted as performant few-shot learners when training data limited 9
"the movie
In contrast to its increasing popularity, the security implications of this prompt-based paradigm are largely under-explored. Recent work 88 34 [ shows that similar their fine-tuned counterparts, prompt-based PLMs are susceptible to textual backdoor attacks, in which misclassification rules injected into PLMs, only be activated by poisoned samples containing "triggers" (e.g., the rare word of "cr"). However, how to effectively mitigate such threats, especially under the few-shot setting, remains an open challenge.
In this work, we conduct pilot study showing that few-shot scenarios entail unique challenges for defending against textual backdoor attacks, including scarce training data, intricate interactions with prompts, and limited computational capacity. For instance, many existing defenses 3 24 36 designed for the fine-tuning paradigm require reliable statistical estimates of the downstream datasets
Equaa contribution.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

and therefore perform poorly under the few-shot setting. Thus, it necessitates developing effective defenses tailored to the setting of few-shot learning.
Towards this end, we advocate MDP imakinng-differentii prompting), an effective, lightweight, and pluggable backdoor defense for PLMs as few-shot learners. At high-level, MDP leverages the key observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking: its trigger is (partially) masked, the language modeling probability of poisoned sample tends to vary greatly. Therefore, with reference to the limited few-shot data as "distributional anchors", MDP compares the representations of given samples under varying masking and identifies poisoned samples ones with significant variations. To boost its effectiveness, MDP (optionally) optimizes the prompt to further improve the masking-invariance of clean samples.
To validate its effectiveness, we empirically evaluate MDP using benchmark datasets and represen- tative attacks. The results show that MDP effectively defends PLMs against various attacks under the few-shot setting, with little impact their performance in downstream tasks. Moreover, show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness.
To summarize, this work makes the following contributions.
To our best knowledge, this is the first work on defending PLMs as few-shot learners against backdoor attacks. We reveal that the few-shot setting entails unique challenges while existing defenses for the fine-tuning paradigm are not easily retrofitted to its specificities.
We propose MDP, a novel defense tailored to the few-shot setting. Leveraging the gap between the masking sensitivity of clean and poisoned samples and utilizing the few-shot data effectively estimate such sensitivity, MDP detects poisoned samples with high accuracy at inference time.
Using benchmark datasets and representative attacks, we empirically validate that MDP outper- forms baseline defenses by large margins while causing little impact on the performance of LMs in downstream tasks.
Related Work
We survey the literature relevant to this work in the categories of few-shot learning, PLM prompting, and textual backdoor attacks and defenses.
Few-shot learning 32 enables pre-trained models to generalize to new tasks using only a few (labeled) samples. In the NLP domain, typical few-shot learning methods include meta-learning (40), intermediate training [38 39]. and semi-supervised learning [20 33) Recently, prompt-based learn-
ing [23] receives increasing attention since the introduction of GPT-3 (1] which demonstrates re- markable few-shot performance by using natural-language prompts and task demonstrations to contextualize inputs
PLM prompting treats downstream tasks as masked language modeling problems and leverages prompts to guide PI.Ms to produce textual outputs 23 With proper prompting, even moderate-sizen
PLMs function performant few-shot learners [9] While manually designing prompts requires domain expertise and is often sub-optimal [I [23], recent work explores generating prompts automati- cally 13 [8 16 44. For instance, P-Tuning 17 and DART 41 define prompts pseudo-tokens
and optimize prompts the continuous space, achieving state-of-the-art performance.
Textual backdoor attacks extend the attacks proposed in the computer vision domain 11
22
NLP tasks. By polluting training data or modifying model parameters (e.g., embeddings), the attacks inject misclassification rules into language models, which are activated at inference by poisoned
samples containing "triggers" such as rare words 12 35 42 43 37 natural sentences 6 4 and
specific patterns [26]22 21)).
Textual backdoor defenses aim to defend LMs against backdoor attacks. For instance, based on the observation that trigger words tend to dominate poisoned samples, STRIP [0] detects poisoned samples at run-time ones with stable predictions under perturbation. As trigger words often increase the perplexity of poisoned samples, ONION 24 identifies poisoned samples by inspecting the perplexity changes of given samples under word deletion. RAP (36) leverages the difference

---

PLM
Backdoored PLM
User
Backdoor Injection
Few-shot Data
Attacker
Prompt-based Fine-tuning
+
Training
Inference
Poisoned Sample Prompt-based LMM Misclassification
Figure I: Threat model of NL.P backdoor attacks: the attacker injects backdoor into the PLM fi the victim user adapts I as a few-shot learner in the downstream task; the attacker activates the backdoor via feeding with poisoned samples.
between the robustness clean and poisoned samples to crafted perturbation and injects extra triggers into given samples detect poisoned samples.
However, most existing defenses are designed for the fine-tuning paradigm. How mitigate the threat of textual backdoor attacks for the prompt-based paradigm, especially under the few-shot setting, remains an open challenge. This work represents a solid initial step bridge this gap.
Background
We present the key concepts and assumptions used throughout the paper.
3.1 Few-shot Prompting
Let X,, {I1, ...... In be an input sample, which x; is the i-th token and is the length of X In prompt- based learning, X,, is padded with template T to form prompt:
X
prommp [cls] X [sep] T [sep]
(1)
where T a task-specific string template containing masked token:
T = [T..] [mask]
[T [m.)
The existing methods differ in the definition of the template T. In discrete prompts 23
[T are
selected from the vocabulary V, while in continuous prompts 18] [T]] defined pseudo tokens.
Given X p the PLM (parameterized by 0) is guided output the token distribution of the masked token pe [mask] ). The probability that X,, belongs to class y E y is predicted as:
Po X prampt)) = Σ po[[mask] = x|X p
VEV,
where V, is the of label tokens related y.
Under the few-shot setting, the user has access to a limited training set (e.g., K samples per class) and searches for the template T that optimizes the accuracy of f in the downstream task (yet without modifying 0).
Threat Model
As illustrated in Figure I we consider a malicious model provider as the attacker, who injects backdoor into the PLM J. and releases the backdoored model f. We focus on the targeted-attack case in which the backdoor is defined classifying samples with triggers ("poisoned samples") to target class desired by the attacker. The victim user downloads f and applies it as a prompt-based few-shot learner in the downstream task. The attacker activates the backdoor inference time by

---

Larguage Modeling Distriutionn
Representatioal Change ())
X-l
Sost
Tew-shot Data
X-ii peoeeet
Anchor
Dotection
o(r(Xpn
> 7?
Prompt-based IM
X (ipp
(100)
(PP)
Clean
Poisoned
X Vitcal
Xin
KI Divergene
Jooe
Figure 2: Overview of MDP: it detects a given sample X as poisoned or clean by measuring the variation of its representational change with respect to set of distributional anchors A.
feeding f with poisoned samples. To simulate the worst-case scenario for the defenses, we assume the attacker has access to the downstream dataset and injects the backdoor into the PLM using fine-tuning approach. Formally, the attack formulated as the following optimization objective:
min D, (xf())). v) AB t))
where D. and D, respectively refer the clean and poisoning data and is the loss function (e.g.. cross-entropy). Intuitively, the first term ensures f functions normally on clean samples, the second term ensures f classifies poisoned samples the target class f, and À is hyper-parameter balance the two objectives.
Compared with prior work 10 24 36). we consider more realistic and challenging setting: the defender, the victim user only has limited few-shot data and computational capacity. Further, the user has knowledge about the attacker's training procedure, attack strategy, trigger definition.
4 MDP
Next, we present MDP, a novel backdoor defense for PLMs as few-shot learners.
4.1 Overview of MDP
At a high level, MDP exploits the observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking (i.e., randomly selecting and substituting token with [mask]). Intuitively, by the design of backdoor attacks, the trigger dominates poisoned sample and forces it to be classified to the target class. Thus, if the trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. In comparison, a clean sample is often less sensitive random masking. It is therefore feasible distinguish clean and poisoned samples by comparing their masking sensitivity.
A naive approach to measure the masking sensitivity is compare the model prediction (i.c., "positive" and "negative") of a given sample before and after masking, which however fails to capture the complex variation of the language modeling probability (details in $5.4). Instead, MDP uses the limited few-shot data as "distributional anchors" and measures the representational change of the sample under varying masking, illustrated in Figure To further boost its distinguishing power, MDP optimizes the prompt to improve the on the design and implementation of MDP, with the complete algorithm deferred to S
of clean samples. Below we elaborate
4.2 Modeling Masking Sensitivity
To quantify the representational change of given sample under masking, we leverage the limited few-shot data {(X,, ( as set of "distributional anchors". Specifically, for each X
generate its prompt (9 poomps query the PLM and obtain the distribution as in Eq 3
( Note that rather than mapping it back to the label space D, cache the entire language modeling distribution as the representation of x and consider the data store A = {}!") as the anchor set.
(v V)
(5)

---

At run-time, for given sample X we construct its prompt X its distribution k Po Xprot prompe). We measure the distance between X and the anchors by the Kullback-Leibler divergence between k and each a Dxi. (k** [I)"). We regard the vector d( (xx = [D. [k:0
promps and query the model obtain
(1aa) [] as the coordinates of in with respect to the anchors.
Let X be the masked version of X under random masking. Following the procedure above, we compute the coordinates of Xur as d(( We measure the representational change due to masking by the difference of d( X(st and d(X
7(Xim*)) = A(d(Xt)))) d(x.).))
(6)
Empirically, we find the Kendall rank coefficient as an effective similarity function A, which measures the rank correlation between d(x)*))) and d( (x.))) (i.e., the relative proximity between X and different anchors) and is insensitive to concrete KL-divergence measures.
We then measure the variation of T a( under varying masking to quantify the masking sensitivity of X and detect it as a poisoned sample if its variation is above pre-defined threshold 7.
4.3 Amplifying Masking Invariance
Recall that MDP distinguishes clean and poisoned samples based on the gap between their sensitivity to random masking. To further boost its distinguishing power, we (optionally) optimize the prompt to improve the masking invariance of clean samples.
Specifically, given few-shot data {(Xme v)}. let X., be the masked version of X and X Xpoompe and X xrromge be their prompts. We define the masking-invariant invarianvta constraint as:
Can = Exp.
((f( (p ), (Xprmppt)
where the expectation is taken over the few-shot data X,, and random masking mask(-). Intuitively, Csu encourages the model to generate similar distributions for clean sample under varying masking. Note that Lan is pluggable into any prompt-hased learning methods including P-Tuning [17 and DART 41 complement other optimization objectives.
4.4 Theoretical Justification
Next, we provide theoretical justification for the effectiveness of MDP, To simplify the analysis, we assume the following setting: given a binary classification task and vocabulary of two tokens (+, -). a sample X1, is classified as if po (,) > and otherwise; a poisoned sample X,, (with target class t 1) comprises n tokens (including one trigger token); in its masked variant Xm. one token is randomly masked; single anchor X used the reference, with p & |+|+)X))) Theorem reveals that there exists a trade-off between attack effectiveness and detection evasiveness (proof deferred to {
Theorem 4.1. Assume i) the attack is effective - if non-trigger token is masked, Po po(+( 2 NT
2. and ii a clean sample is masking- iiniggiivvaaatt- if the trigger token is masked. Po po(+) I R < J. and if the detection threshold y is set on the variation of the representational change of X,, under random masking, then to evade the detection, it satisfies:
[h(K*) h(xx))
n
(8)
n
where (u(-) is defined as the KL divergence functitoonn with respect to p":
h(p) _ p plog
P + (1 pp log - p
(9)
p*
1- p"
Intuitively, for the attack to be effective, x should be large; however, to evade the detection, ect is upper-bounded by 8 Thus, MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. Moreover, if the model both accurate in classifying clean samples (i.c., x is sufficiently small) and masking- masking-invariant invariant with respect clean
5

---

samples (i.e., 7 can be sufficiently small without incurring false positive cases), which makes the following condition hold:
+ log -1--"))=
(10)
is then impossible to launch effective attacks without being detected because n² can not satisfy the two objectives simultaneously (proof in (
5
Empirical Evaluation
5.1 Experimental Setting
We conduct the evaluation across 5 sentence classification datasets (SST-2, MR, SUBJ, TREC) widely used to henchmark prompt-based few-shot learning methods [ 17 41 We follow the same setting LM-BFF [9] which samples validation sets respectively. The dataset statistics are summarized in Table
Datasets.
16 samples per class to form the training and
Dataset
# Classes
Avg. Len
Train
Dev
Test
SST-2
2
15.6 words
6.9k
0.9k
1.8k
MR
2
21.0 words
8.0k
0.7k
2.0k
2
20.1 words
1.5k
0.3k
2.0k
SUBJ
2
24.1 words
7.0k
1.0k
2.0k
TREC
10.0 words
5.0k
0.5k
0.5k
Table 1. Statistics of the datasets used in the experiments.
Models, A victim model comprises a PLM and prompt model. We use RoBERTa-large 19 as the and DART [4] as the prompt model, which achieves state-of-the-art performance under the few-shot setting.
PLM, which is widely used in prompt-based learning 99] 29 41 44,
Attacks. We use 5 representative textual backdoor attacks to evaluate MDP and other defenses.
BadNets IT is originally designed as a backdoor attack the computer vision domain and extended to NLP tasks by selecting rare words as triggers [12]. AddSent [6] is similar to BadNets but uses neutral sentences as triggers to make poisoned samples stealthier. EP 335 perturbs the embeddings of trigger words rather than modifying the PLM parameters. LWP 15 uses a layer-wise weight poisoning strategy to only poison the first layers of PLMs with combinatorial triggers. SOS (37 defines the triggers the co-occurrence of multiple pre-defined words, which are further inserted into natural sentences to make the attacks more evasive.
Baseline defenses. As MDP represents the first backdoor defense for the prompt-based paradigm, adapt representative defenses designed for the fine-tuning paradigm the baselines.
Based on the observation that the prediction of a poisoned sample is often dominated by the trigger, STRIP [0] detects poisoned samples ones with stable predictions under perturbation. ONION 24 relies on the hypothesis that the trigger out of the context of a poisoned sample, and detects poisoned samples by inspecting the perplexity change under word deletion. RAP (36) leverages the gap between the robustness of clean and poisoned samples to perturbation and injects crafted perturbation into given samples detect poisoned samples. The detailed description of the baselines is deferred to $C
5.2 Implementation Details
To simulate challenging scenario, assume the attacker has access to the full training sets (cf. Table and injects backdoors into PLMs by fine-tuning the models. The attack setting (c.g., trigger definitions) is summarized in $C] We apply MDP and baselines the backdoored PLMs under the few-shot, grompt-based learning paradigm; that is, the defender has only access to the few-shot data (K = 16 samples per class). We apply grid search over the hyperparameters select the optimal setting for each defense.
Following previous studies 10 36
the attack performance is evaluated using the metrics of i) clean accuracy (CA), defined the victim model's 's accuracy on clean samples, and ii) attack success rate (ASR), defined as its accuracy of classifying poisoned samples the target label desired by the

---

attacker. Intuitively, CA and ASR respectively quantify the model's performance on the original and backdoor tasks. Meanwhile, the defense performance evaluated by the metrics of i) false rejection rate (FRR), defined as the percentage of clean samples that are mistakenly labeled as poisoned, ii) false acceptance rate (FAR), defined as the percentage of poisoned samples that are mislabeled as clean, and fii) the area under the ROC curve (AUC), an aggregate measure of performance across all possible classification thresholds. All the measures are averaged across five sampled training sets as
LM- AM-BFFFI 9
STRIP
ONION
RAP
MDP
Dataset
Attack
CA (%)
ASR (%)
FRR
FAR
FRR
FAR
FRR
FAR
FRR
FAR
BadNets
95.06
94.38
7.56
87.44
2.78
9.28
3.11
64.28
5.33
1.77
AddSent
94.45
100.0
2.75
72.56
7.06
26.72
5.61
37,50
4.45
3.53
SST-2
LWP
93.41
95.53
5.96
89.39
8.28
7.39
43.77
5.27
4.78
EP
93.63
95.95
1.72
72.06
5,28
12.89
2.72
58.11
5,05
0.73
SOS
91.65
92.41
2.98
87.56
4.06
32.56
51.28
0.00
0.00
BadNets
89.80
98.30
11.70
72.30
4.80
15.60
2.75
25.35
5,10
5.60
AddSent
89.60
97.50
16.20
60.00
4.65
37.25
9.35
39.70
5,05
10.90
MR
LWP
89.65
96.90
9.35
82.70
1.60
17.45
1.70
52.55
5.25
3.60
EP
89.40
96.60
2.20
88.90
15.35
12.60
6.45
70.60
4.70
3.00
SOS
89.85
97.30
5.20
75.90
0.90
64.10
15.20
58.85
4.85
3.40
BadNets
89.95
92.30
2.85
98.70
5,20
7.45
1.35
43.60
4.95
5.10
AddSent 91.45
95.70
10.10
62.20
4.75
19.50
12.95
48.90
4.80
3.00
CR
LWP
89.75
91.30
1.80
99.10
4.90
27.85
4.05
39.20
5.10
3.50
EP
89.35
97.55
2.20
87.20
10.15
4.40
7.65
45.20
5,35
9.40
SOS
91.45
100.0
2.20
78.20
0.75
37.55
55.30
0.20
0.00
BadNets
96.05
94.20
5,10
68.85
3,50
16.60
12.40
43.65
5,30
7.90
AddSent
95.90
97.00
2.50
85.50
4.30
34.20
7.30
68.20
4.85
9.00
SUBJ
LWP 96.15
99.10
4.55
98:70
4.65
7.40
1.00
18.60
5,40
10,90
EP 96.70
99.90
4.75
99.10
5.25
4.10
33.25
4.90
10.30
SOS
94.90
99.60
5.15
75.50
4,90
61,30
0.10
29. 10
$.35
4.10
BadNets
93.00
95.30
4.30
73.76
5,40
54.53
5.55
50.61
4.80
2.49
AddSent
96.60
93.65
5.20
79.28
4.80
36.74
3.55
47,60
3.60
7.18
TREC
LWP 94.40
97.24
5.60
99.17
4.60
25.69
1.23
93.09
5,20
4.42
EP
95.80
97.51
4.60
63.81
5,20
11.22
10.43
42.68
4.80
5.25
SOS
91.80
99.45
5.20
68.78
4.40
80.61
14.83
63.71
4.60
4.97
Table 2. Performance of MDP and baseline defesens on datasets, with lower FAR/ERR indicating better
defense performance. The detection threshold is set based on the allowance of 5% FRR on the training set.
5.3 Main Results
We first evaluate the effectiveness of various backdoor attacks under prompt- based fine-tuning with results summarized in Table Observe that across all the datasets, most attacks attain both CA and ASR above 90%, indicating their effectiveness in the downstream and backdoor tasks.
We then compare the performance of MDP and baselines in defending against these attacks. For each defense, we set the detection threshold (e.g.. the variation threshold for MDP) based on the allowance of 5% FRR the training set, and report its FAR and FRR the testing set. In the case of ONION, following prior work [36], we evaluate different thresholds of perplexity change and select the threshold that approximately achieves 5% FRR the training set.
Table summarizes the main results (additional results in $D). Observe that MDP attains the lowest FARs against all the attacks across all the datasets and outperforms basclines by large margins. In particular, achieves near-perfect defenses against the SOS attack on the SST-2 and CR datasets. The observation confirms the effectiveness of MDP in detecting poisoned samples, which is mainly attributed to that i) the clean and poisoned samples show discernible sensitivity to random masking and ii) MDP effectively utilizes the few-shot data as anchors to measure such sensitivity.
In comparison, the baseline defenses are less effective, with FARs over 90% in many cases. This may be explained by the conflict between the limited few-shot data and the reliance of these defenses on sufficient training data. Specifically, to measure the prediction stability of given sample under perturbation, STRIP randomly replaces fraction of its words with ones from training sample
7

---

Attack
40%
Dataset
BadNets
AddSent
LWP
EP
SOS
30%
BadNet
EP
FRR
5.07
5.29
5.39
5.39
5.17
Adddent
SOS
SST-2
200%
IWP
FAR
24.89
58.37
55.50
47.82
73.28
FRR
5.40
5.05
5.45
5.15
4.60
10%
MR
FAR
72.80
74.80
55,00
52.10
80.80
0%
0.5%
CR
FRR
4.40
5.10
4.80
5.45
5.25
0%
FAR
83.10
75.30
73.80
52.20
56.10
5%
SUBJ
FRR
5.40
4.25
4.60
4.75
5.35
FAR
9.80
67.40
14.90
15.70
37,90
10%
FRR 5.20
4.90
4.90
4.70
5.20
Figure 3: Performance of MDP on SST-2 with dif- ferent FRR allowances on the training set; bascline defenses all have FARs above 50% (not shown).
TREC
FAR
75.14
71.55
43.37
70.44
26.52
Table 3: Performance of MDP using prediction variance as the masking-sensitiiity measure.
that have the highest frequency-inverse document frequency (TF-IDF) scores. However, due to the limited number of training samples, both the substitution words and the estimated TF-IDF scores tend to be highly biased, which negatively impacts the performance of STRIP. ONION removes outlier words that cause sharp perplexity changes before inference, which is inherently ineffective against complex triggers (e.g., natural sentences) [6] Moreover, the threshold for detecting outlier words can be significantly biased by the limited training samples under the few-shot setting. RAP trains a word-based cobustness-awant trigger such that inserting this trigger causes significant prediction changes for clean samples but not for poisoned samples. However, under the few-shot setting, the optimality of the RAP trigger is largely limited by the available few-shot data, which negatively affects its detection effectiveness.
5.4 Influential Factors
We conduct additional studies to understand the impact of key factors on MDP. Due to space limitations, we mainly present the results on SST-2, with other results deferred to $D
FRR allowance. We adjust the detection threshold corresponding varying FRR allowance on the training set. Figure [eguree]shor shows that MDP maintains its superior performance under different FRRs (0.5%, 1%, and 3%). comparison, the baselines all have FARs above 50% (not shown).
Sensitivity measures. Instead of using the sho-shot data as distributional anchors to measure the masking sensitivity of given sample Xtest, we its prediction variance due masking as the sensitivity measure. Specifically, given the prediction of X y arg maxy Po (y') Xtsst,, ), we measure the confidence variance of the masked variant Xtest with respect to y: (po Xts))) Intuitively, poisoned sample tends to have a larger variance since masking the trigger may cause the prediction to fluctuate significantly. Following the same setting in [5.3 we set the threshold based on 5% FRR allowance on the training set and evaluate MDP on the testing set. Table shows that using the alternative sensitivity measure causes the performance of MDP to drop sharply (cf. Table For instance, its FAR increases by over 50% against LWP. The results confirm our analysis that simple statistics such as prediction confidence may fail to capture the complex variation of the language modeling probability due to masking.
Masking-invarian constraints. Recall that the masking-invarian constraint Lxi is designed to improve the masking invariance of clean samples. Here, we evaluate its impact on MDP's overall performance. Specifically, we adjust the weight of Lsn in the prompt optimization AT from 0.25 to 4. For each distinct weight, we set the detection threshold based on 5% FRR allowance on the training set and report its performance the FARs of MDP against most attacks first drop and then gradually increase. This observation may be explained as follows. With overly small weight, C., has little effect improving the masking invariance of clean samples, while overly emphasizing Lau negatively impacts the classification accuracy, resulting in higher FARs. It is thus crucial to properly calibrate the weight of Lxu to optimize the performance of MDP.
the testing set. As shown in Figure the weight of Esa varies,
Few-shot data size. We further evaluate how the few-shot data size (i.e., shots) influences the performance of MDP. Besides the default shots (K 16 per class) used in the previous evaluation, vary K from 4 to 64 to build the anchor set and evaluate MDP, in which the FRR allowance
8

---

40%
20%
30%
Addbent
SOS
AddSentt
20%
LWP
R2 10%
LWP
5%
0%
0%
0%
0%
5%
Figure 4: Performance of MDP on SST-2 under vary- ing weight of the masking- invariinnn constraint LMi-
Figure 5: Performance of MDP on SST-2 with varying size of few-shot data (K samples per class).
the training set is fixed as 5%. 3 reports the performance of MDP under varying shots K. Observe that its FARs steadily improve as K increases. Intuitively, with larger anchor set, MDP quantifies the representational variation of given samples due to random masking more precisely, leading more accurate detection. Also, notice that 16 is often sufficient for MDP to obtain satisfactory performance.
Prompt types. We further evaluate the impact of prompt types types on MDP. Recall that in discrete prompts [23], the tokens in the prompt template are selected from the vocabulary, while in con- tinuous prompts 18), the tokens are pseudo- tokens and optimized in a continuous space. Ta- blel4eval MDP discrete prompt-based models. Compared with continuous prompts (cf. Table MDP is less effective under dis- crete prompts. For instance, its FAR against BadNets on MR increases by 17%. This may be explained by that continuous prompts entail larger spaces to better optimize the masking invariance constraint, suggesting that using dif- ferentiable, continuous prompts benefits MDP in defending against backdoor attacks.
Attack
Dataset
BadNets
AddSent
LWP
EP
SOS
FRR
S.27
4.39
5.15
5.11
0.00
SST-2
FAR
5.09
19.02
18.40
10.08
0.00
FRR
5,45
4.85
5.05
5.15
5.45
MR
FAR
22.60
32.80
24.20
14.50
27.80
FRR
3.80
5.30
5.45
5.15
4.45
CR
FAR
14.40
33.50
20.10
24.40
11.00
FRR
5.40
4.75
5.20
5.00
5.25
SUBJ
FAR
11.70
31.10
12.00
32.40
25.10
FRR
5,00
4.10
4.50
5.30
4.50
TREC
FAR
16.02
37.85
32.60
23.48
26.80
Table 4. Performance of MDP on discrete prompt- based models (with 5% FRR allowance on the training set).
PLMs. Besides RoBERTa-larg. [19, we also consider alternative PLMs including BART [14] and GPT-2 (27) the victim models. Table shows MDP's performance on such alternative PLMs against BadNets and AddSent attacks (with respect SST-2). Observe that MDP has comparable performance on BART and GPT-2 to that on RoBERTa-large, indicating that generalizes other
PLM
Attack
CA
ASR
FRR
FAR
BadNets 94.03
98.79
2.85
11.42
BART
AddSent
92.07 99.12
6.82
16.61
BadNets 92.78 91.28
5.60
6.24
GPT-2
AddSent 92.31 91.28
4.05
9.46
Table MDP on alternative PLMs.
PLMS well.
Masking rate. Recall that the masking sensitivity influences MDP's performance ($4.2) Here vary the masking rate from 0.1 to 0.4 (with 0.2 as the default). Figure/o presents MDP's s FAR and FRR on SST-2 with respect different attacks. Notably, properly setting the masking rate critical for MDP. This can be explained as follows. An overly small masking rate may lower the probability of "hitting" the triggers, while an overly large masking rate may significantly alter the semantics of given samples after masking, both making clean and poisoned samples less distinguishable.
5.5 Alternative Scenarios
Thus far we assume that (i) the backdoor is injected into the PLM, (ii) the trigger is defined as explicit perturbation (e.g., adding words sentences), and (iii) the attacker is unaware of MDP. Next, we evaluate MDP in alternative scenarios which these assumption do not hold, with results summarized in 7
Alternative threat models. BadPrompt 2 injects the backdoor into the prompt and releases the end-to-end PLM to the victim user, assuming that the user directly uses the model without further

---

20%
15%
AddiSent
SOS
Dataset
Attack
CA
ASR
FRR
FAR
FAR
10%%
LWP
BadPrompt
87.50
81.39
7.16
8.28
SST-2
StyleAttack
92.97
78.67
2.94
24.33
Adaptive Attack
94.04
82.71
6.94
13.78
0%
10%
20%
30%
40%
BadPrompt
84.75
90.10
4.45
6.70
MR
StyleAttack
88.65
84,10
2.95
31.30
5%
Adaptive Attack
89.65
86.45
4.70
9.85
10%
Figure MDP in alternative scenarios.
Figure 6: MDP on SST-2 with varying masking rate.
tuning. However, since the prediction of poisoned samples hinges the trigger tokens, BadPrompt still exhibit high sensitivity to random masking, making them easily detectable by MDP.
Invisible triggers. Instead of using explicit perturbation (e.g., word addition). StyleAttack [25] defines the triggers as specific syntactic styles, making the perturbation less visible. As shown in T observe that StyleAttack is less effective (lower ASR) but also more challenging for MDP to detect (higher FAR). This may explained as follows: visible triggers are often local features (e.g., words and sentences), while invisible triggers tend to be global features (e.g., patterns of lexical choice and syntactic structures). Intuitively, compared with local features, global features are less sensitive to random masking used by MDP.
Adattive attacks. We further evaluate MDP against an adaptive attack, in which the attacker is aware of MDP and optimizes the PLM to evade detection. Recall that MDP detects poisoned samples based on their masking variance. Thus, with reference attempts to minimize the masking variance of poisoned samples while optimizing the backdoored PL.. Specifically, we add the loss term of masking variance to Eq. (with weight 1.0). Observe in Table that compared with non-adaptive attacks (cf. Table the adaptive attack marginally improves the evasiveness of poisoned samples; however, it also results in lower attack effectiveness. This finding corroborates our theoretical analysis in $4.4 there exists an interesting trade-off between attack effectiveness and evasiveness with respect to MDP.
clean PLM and few-shot samples, the attacker
6
Limitations
we focus on defending against backdoor attacks in the context of text classification. As backdoor attacks in other tasks (e.g.. text generation) assume different threat models and attack/defense metrics, we consider extending MDP to such tasks under the prompt-based few-shot settings our ongoing work.
Other NLP tasks. Similar to prior work (e.g., RAP [36] and ONION 24
Fewer-shot data. While we evaluate MDP under limited few-shot data (e.g., K as low as 4), in practice, the available data could be even scarcer (e.g., one- or zero-shot (30 B1). Given the need of adapting PLMs on fewer-shot data, we aim to improve MDP to address the data-insuse inaaiffffffffinccy limitation towards practical deployment.
Invisible triggers. It is shown that compared with local features, global features are less sensitive the random masking used in MDP. We consider extending MDP to the setting of invisible backdoor attacks as our ongoing work. One promising direction redefine the random masking operation to include random syntactic-struttug perturbation or random style perturbation.
7 Conclusion
This paper presents a sost-of-its-king defense for pre-trained language models as few-shot learners against textual backdoor attacks. At a high level, we exploit the gap between the sensitivity of clean and poisoned samples to random masking and effectively utilize the few-shot learning data to measure such sensitivity. The evaluation on benchmark datasets shows that our method outperforms baselines in defending against representative attacks, with little impact on the performance of victim models. Our findings shed light on how to enhance the security of pre-trained language models, especially in the pasep,-based, feo-shot learning paradigm.
10

---

Acknowledgments and Disclosure of Funding
This material is based upon work supported by the National Science Foundation under Grant No. 1951729, 1953813, 21 19331,, and 2212323, Any opinions, findings, and conclusions or recommenda- tions expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
References
[1] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, al. Language Models are Few-shot Learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[2] Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, and Xiaojie Yuan. BadPrompt: Backdoor Attacks Continuous Prompts. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2022.
[3] Chuanshuai Chen and Jiazhu Dai. Mitigating Backdoor Attacks in LSTMMbased Text Classifi cation Systems by Backdoor Keyword Identification. Neurocomputing, 452:253-260 2021.
[4] Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor Attacks against NLP Models. In ICML 2021 Workshop on Adversarial Machine Learning, 2021.
[5] Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. In ArXiv e-prints, 2017.
[6] Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A Backdoor Attack against LSTM- -bsedd Text Classification Systems. IEEE Access, 7:1 7:133882-138877 2019.
[7] Jacob Devlin, Ming-We Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguisties (NAACL), 2019.
[8] Wei Du, Yichun Zhao, Boqun Li, Gongshen Liu, and Shilin Wang. PPT: Backdoor Attacks Pre-trained Models via Poisoned Prompt Tuning. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI), 2022.
[9] Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2021.
[10] Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. Design and Evaluation of Multi-domain Trojan Detection Method on Deep Neural Networks. IEEE Transactions on Dependable and Secure Computing, 19(4):2349-23) 2364. 2021.
[11] Tianyu Gu, Brendan Dolan- Dolan-Gavitt and Siddharth Garg. Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. In ArXiv e-prints, 2017.
[12] Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of Annual Meeting of the Association for Computational Linguisties (ACL), 2020.
[13] Brian Lester, Rami AI-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficien Prompt Tuning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
11

---

[14] Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising serpuence-t--sequeee Pre-training for Natural Language Generation, Translation, and Comprehension. In ArXiv e-prints, 2019.
[15] Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[16] Xiang Lisa Li and Percy Liang. Prefix-Tune Tunii-T Optimizing Continuous Prompts for Generation. In Proceedings of Anmual Meeting of the Association for Computational Linguistics (ACL), 2021.
[17] Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. In ArXiv e-prints, 2021.
[18] Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. In ArXiv e-prints, 2021.
[19] Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A Robustly Optimized BERT Pretraining Approach. In ArXiv e-prints, 2019.
[20] Takeru Miyato, Andrew M Dai, and lan Goodfellow. Adversarial Training Methods for Semi- Supervised Text Classification. In Proceedings of International Conference on Learning Representations (ICLR), 2017.
[21] Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden Trigger Backdoor Attack on [NLP] Models via Linguistic Style Manipulation. In Proceedings of USENIX Security Symposium (SEC), 2022.
[22] Ren Pang. Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of ACM Conference on Computer and Communications (CCS), 2020.
[23] Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2019.
[24] Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[25] Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based text style transfer. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021.
[26] Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. In Proceedings of Annual Meeting of the Association for Computational Linguisties (ACL), 2021.
[27] Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, al. Language Models are Unsupervised Multitask Learners. OpenAl blog, 1(8)99 2019.
[28] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the Limits of Transfer Learning with Unified Text-to-text Transformer. J. Mach. Learn. Res., 21(140):1-67, 2020.
[29] Timo Schick and Hinrich Schütze. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot I Learners. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
12

---

[30] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29, 2016.
[31] Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST), 10(2):1-37, 2019.
[32] Yaqing Wang. Quanming Yao. James T Kwok, and Lionel M Ni. Generalizing from a Few Examples: A Survey on Few-shot Learning. ACM computing surveys (csur), 53(3):1-34, 2020.
[33] Qizhe Xie, Zihang Dai, Eduard Hovy. Thang Luong, and Quoe Le. Unsupervised Data Augmentation for Consistency Training. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS), 2020.
[34] Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. Exploring the Universal Vulnerabilityy of Prompt-based Learning Paradigm. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL), 2022.
[35] Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models. In Proceedings of Annual Conference of the North American Chapter of the Association for Compulational Linguistics (NAACI.). 2021.
[36] Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. RAP: Robustness- Perturba- tions for Defending against Backdoor Attacks on NLP Models. In Proceedings of Conference on Empirical Methods in Natural Langagge Processing (EMNLP), 2021.
[37] Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rethinking Stealthiness of Back- door Attack against NLP Models. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL). 2021.
[38] Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start. In Proceedings of Conference on Empirical Methods Natural Language Processing (EMNLP), 2020.
[39] Haiyang Yu, Ningyu Zhang, Shumin Deng, Hongbin Ye. Wei Zhang, and Huajun Chen. Bridg- ing Text and Knowledge with Multi-Prototype Embedding for Few-Shot Relational Triple Extraction. In Proceedings of International Conference on Computational Linguistics, 2020.
[40] Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse Few-Shor Text Classification with Multiple Metrics. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACI.), 2018
[41] Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners. In Proceedings of International Conference on Learning Representations (ICLR), 2021.
[42] Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In Proceedings of IEEE European Symposium on Security and Privacy (Euro S&P), 2021.
[43] Zhiyuan Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. Neural Network Surgery: Injecting Data Patterns into Pre-trained Models with Minimal Instance wise Side Effects. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
[44] Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual Probing Is [MASK]: Learning vs. Learning to Recall. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2021.
13

---

A Algorithm of MDP
Algorithm 1: MDP
Input: D: few-shot data; 8: PL.M parameters; T: fine-tuned text template; V: vocahulary; y: threshold;
: test sample
Output: X poisoned or benign
A < 0:
foreach X E do
X Prompe [cls] _ [sep] T [sep]:
Po(v) (ol to A where
end
Xprampe
"prompe f [cls] Xim
sep T ssep):
kleat f- (o(0) X lost
pramp::
d(Xia))) +- [DuL(k IIa "] where a E A:
randomly mask N as Xm:
compute the coordinates of A d( d(xx:
(XX...)) A(d(Sm")) (d(( ));
12 if ((Xi")))
) 2 y then
13
return X poisoned;
14 else
15
return X benign;
16 end
B Proofs
Proof. (Theorem 4.1 Given a single anchor X let k. k and be the prediction distributions of Xmm Xm. and X respectively. We define the representational change of Xm due to masking as:
T(x,.. DKL. (kella) -kll(alla))
(11)
As X, comprises tokens, there are variants of X,,. one with the trigger token masked and the rest with non-trigger token masked. Let X and X (1 S < 1) denote the two parts.
Let p x As X; is clean sample, p" < n= (negative) p* > x+ (positive). Thus, for p E K xt the KL. divergence function
1 hp)) - plog + p) log
(12)
1 1- p*
increases (or decreases) monotonically with p. According the assumption, po(+|&1')) < x and
po(+ {() 2 + (1 siignnn) To minimize the variation of the representational change of X... (0 sis 1) should be close to each other. thus follows that po(+| = KT
Po + (
and Pa + (+) = x+ (1 < n n 1). 11. It can be derived that the minimum variation of the
representational change of X given by:
(T((x,)) 2 Vn-I [M(N*) h(r²))
(13)
n
To evade the detection, (T((...)) S 7 which completes the proof.
Proof. (Corollary) Recall that the function h(p) monotonically increases (or decreases) with [N" .K+]] Thus, for given RT, follows:
[h(x h(k=*t [
>[h(MT) ml
(14)
=|h(x + +
log p" (1 1-pp" p²)]
2
Thus. if [[(x(]]))))))))]]))<< +++ $ logp"(1 - p²)l -ーー 7. there is no x 1/4 that satisfies Eq
13
14

---

C Implementation Details
The default parameter setting in the evaluation is summarized T The setting of baseline defenses mainly follows prior work [36]. For STRIP, rate as and 0.25, while the other parameters are set according to the best detection performance. For ONION, we test different thresholds approximately achieve 5% FRR on the training set. Then we remove outlier words with perplexity changes above the thresholds inference time. For RAP, bound the change of output probability as |-0.3. -0.1]. When training the word embedding of the RAP trigger, we set the learning rate as 1.0e-2. The RAP trigger inserted the first position of each sample avoid being truncated.
the number of copies and replacement
the perplexity change and choose the thresholds that
Computational Resources
Model parameters
355 million
30 min (training & attack)
Computational budget
60 min (testing & detection)
Models and Training
PLM
RoRERTa-large
Prompt model
DART
Max sequence length
128
Embedding dimension
1,024
Batch size
(train), 32 (test)
Learning rate
2.0e-5
Optimizer
Adam
Prompt-tuning epochs
20
Shots K
16 per class
Attacks
Attack training epochs
10
Poisoning rate
10%
Target class
0
BadNets trigger
["ef". "mn". "bb", "tq")
AddSent trigger
watch this 3D movie"
LWP trigger
["ef", "bb", "ak", "mm"" ["ef"]
EP trigger
SOS-train trigger
["Triends", "weekend", "store") "I have bought from store with my friends weekend"
SOS-test trigger
# Triggers
per sample
Masking rate 0.2
Trials
50
Weight of LLM 1.0
Baseline Defenses
STRIP Copies
STRIP - Replacement rate
0.25
RAP Trigger
"mb"
RAP- Training LR
1.0c-2
RAP Prob. change bound
|-0.3, -0.1]
Table 6. Implementation and evaluation details of models, attacks, defenses.
D
Additional Results
The AUC scores of MDP and baseline methods are summarized in Table The performance of MDP with respect different FRR allowances on the training set, varying weights of Can, and varying sizes of few-shot data is shown in Figure & to Figure 19
15

---

Dataset Attack
STRIP
ONION
RAP
MDP
BadNets
0.66
0.64
0.53
0.99
AddSent
0.51
0.54
0.52
0.99
SST-2
LWP
0.60
0.72
0.83
0.98
0.84
0.67
0.56
1.00
SOS
0.82
0.61
0.51
1.00
BadNets
0.57
0.63
0.60
0.98
AddSent
0.56
0.58
0.60
0.96
0.60
0.72
0.51
0.98
EP
0.53
0,66
0.54
0.99
0.76
0.52
0.52
0.97
BadNets
0.83
0.68
0.59
0.99
AddSent
0.76
0.52
0.52
0.99
LWP
0.71
0.67
0.62
0.97
CR
0.88
0.63
0.58
0.96
SOS
0.71
0.55
0.53
1.00
BadNets
0.57
0.69
0.62
0.95
AddSent
0.64
0.60
0.56
0.99
0.68
0.73
0.58
0.96
SUBJ
EP
0.64
0.65
0.51
0.96
0.87
0.56
0.56
0.97
BadNets
0.62
0.64
0.56
0.99
AddSent
0.60
0.62
0.58
0.97
LWP
0.58
0.73
0.66
0.99
TREC
0.82
0.72
0.65
0.98
SOS
0.75
0.73
0.56
0.98
Table Performance (AUC) of MDP and baseline defenses.
10%
10%
30%
20%
10%-
10%
0%
0.5%
1%
3%
5%
0.5%
1%
0%
0%
10%
10%
Figure 8: Performance of MDP MR with different FRR allowances training set.
Figure 9: Performance of MDP CR with different FRR allowances on the training set.
60%
43%
45%
- 905
30%
30%
15%
0%
0%
0.5%
1%.
3%
D. 5%
5%
0%
0%
10%
10%
Figure Performance MDP on SUBJ with differ- ent FRR allowances on the training set.
Figure Performance MDP on TREC with dif- ferent FRR allowances on training set.
16

---

BadNets
- EP
BadNets
- EP
30%
AddSent
-4. SOS
30%
- AddSent
pp. sos
20%
LWP
20%
LWP
10%
0.25
0.5
0.25
0.5
5%
5%.
10%-
10%-
Figure 12: Performance of MDP on MR under the varying weight of the masking-invariang, constraint LMI-
Figure 13: Performance MDP on CR under the varying weight of the masking-invarianee constraint LMI-
10%
40%
BadNess
30%
-
30%
-V. SOS
10%
0%
0%
0.25
0%
0%
10%
10%.
Figure 14: Performance of MDP on SUBJ under the varying weight of the LMI-
Figure 15: Performance of MDP TREC under the varying weight masking-invariance constraint LMI-
constraint
30%
60%
BadNeb
BasiNets
20%
- AddSent
. SOS
AddSent
- 50S
LWP
30%
LWP
10%
15%
16
32
0%
64
0%
PRR
5%
3%
Figure 16: Performance of MDP on MR with varying size of few-shot data (K samples per elass).
Figure 17: Performance of MDP on CR with varying size of few-shot data (K samples per class).
50%
30%
RadNett
40%
20%
p.
-7.
FAR
10%
10%
0%
16
0%
0%
FRX
10%%
10%-
Figure 18: Performance of MDP SUBI with vary-
Figure 19: Performance of MDP TREC with vary- ing size of few-shot data (K samples per class).
ing size few-shot data (K samples per class).
17


Reviews:
[{"review_id": "ZZ6wOizX8N", "reviewer": "Reviewer_6q5A", "summary": "This paper proposes MDP(Masking-differential prompting), to defense PLMs as few-shot learners against backdoor attacks. The paper empirically validates that MDP outperforms baseline defenses.\n", "questions": "1.Whether you can extend MDP to a detection method?", "limitations": "The authors adequately addressed the limitations. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.The paper is well-written and easy to follow.\n\n2.Investigating backdoor attack/defense under few-shot scenarios is interesting.\n\n3.The evaluation is comprehensive.\n", "weaknesses": "1.Since the MDP is based on the observation: if the tirgger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. A statistics table supports this observation would be better.\n\n\n\n2.There are invisible backdoor attacks such as stylebkd[1] and synbkd[2]. Does MDP also have a good performance on these more advanced attacks? (Or is it make sense to apply these two attacks under PLMs with few-shot setting?)\n\n\n\n[1] Qi, F., Chen, Y., Zhang, X., Li, M., Liu, Z. and Sun, M., 2021, November. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 4569-4580).\n\n\n[2] Qi, F., Li, M., Chen, Y., Zhang, Z., Liu, Z., Wang, Y. and Sun, M., 2021, August. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 443-453).\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback on improving this paper! Please find below our response to the reviewer’s questions.\n\n> Since the MDP is based on the observation: if the trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. A statistics table supporting this observation would be better.\n\nIn response to the reviewer's suggestion, we have measured the variance in the language modeling probability of both clean and poisoned samples due to random masking (under the BadNets attack). Under the default setting specified in Table 5 in the appendix, we have measured the masking variance of all clean samples in the testing set as well as their corresponding poisoned counterparts. The mean and standard deviation of these measures are reported in the table below. Across most datasets, there are noticeable gaps between clean and poisoned samples. However, also note that in some cases, the masking variance measures of clean and poisoned samples show large standard deviation, making it challenging to separate them using such measures directly, which motivates the design of MDP to use the more robust metric in Eq. 6.\n\n| Dataset | Clean Samples | Poisoned Samples | \n| :--- | :--- | :--- | \n| SST-2 | 0.0423 ± 0.0471 | 0.1055 ± 0.0752 |\n| MR | 0.0464 ± 0.0649 | 0.1463 ± 0.0725 |\n| CR | 0.0317 ± 0.0548 | 0.1274 ± 0.0721 |\n| SUBJ | 0.0180 ± 0.0464 | 0.1899 ± 0.0631 |\n| TREC | 0.0284 ± 0.0467 | 0.1538 ± 0.0627 |\n\n\n> There are invisible backdoor attacks such as stylebkd [1] and synbkd [2]. Does MDP also have a good performance on these more advanced attacks? (Or does it make sense to apply these two attacks under PLMs with few-shot setting?)\n\nIn response to the reviewer’s suggestion, we have conducted a preliminary evaluation of MDP against StyleBkd [1] on the SST-2 dataset, with the results reported below. \n\n| CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- |\n| 92.97 | 81.66 | 4.04 | 51.96 |\n\nNotably, compared with insertion-based backdoor attacks, invisible backdoor attacks such as StyleBkd are less effective (lower ASR) but more challenging for MDP to detect (higher FAR). This may be explained as follows: insertion triggers are often local features (e.g., words and sentences), while invisible triggers tend to be global features (e.g., patterns of lexical choice and syntactic structures). Intuitively, compared with local features, global features are less sensitive to the random masking used in MDP. We consider extending MDP to the setting of invisible backdoor attacks as our ongoing work. One promising direction is to redefine the random masking operation to include random syntactic-structure perturbation or random style perturbation. \n\n> Whether you can extend MDP to a detection method?\n\nThanks for the question. We would like to clarify that MDP is essentially a detection-based defense method, which falls into the same category as other baselines (e.g., STRIP, ONION, and RAP). We will revise the corresponding statement to make it more clear in the revision.\n\nAgain, we thank the reviewer for the valuable feedback. Please let us know if there are any other questions or suggestions.\n\nBest,\n\nAuthors"}, {"review_id": "n9JMOBxCWw", "reviewer": "Reviewer_fW9g", "summary": "The paper presents an investigation into the security vulnerabilities of pre-trained language models (PLMs) when used as few-shot learners, demonstrating that they are highly susceptible to backdoor attacks. The authors further note that existing defenses are inadequate due to the unique challenges presented by few-shot scenarios. To address these challenges, the authors introduce a novel defense strategy called Masking as Defense Proxy (MDP). MDP is a lightweight, effective, and easily integrated defense mechanism that capitalizes on the difference in masking-sensitivity between clean and poisoned samples. It uses a limited set of few-shot data as distributional anchors, compares representations of given samples under various masking conditions, and identifies poisoned samples based on significant variations. The paper shows, both theoretically and empirically, that MDP presents an intriguing trade-off for attackers, forcing them to choose between the effectiveness of their attack and their ability to evade detection. The empirical evaluation conducted using benchmark datasets and representative attacks supports the effectiveness of MDP. ", "questions": "-", "limitations": "-", "rating": 4, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Originality: The paper addresses an original and timely problem of defending pre-trained language models (PLMs) against backdoor attacks in the few-shot learning scenario. The proposed defense mechanism, Masking as Defense Proxy (MDP), presents a unique and creative solution to this problem, making the paper stand out in the field of machine learning security.\n\n- Quality: The authors provide rigorous theoretical analyses to demonstrate the dilemmas the attackers would face under MDP defense. They conduct comprehensive experiments using various attack methods and benchmark datasets to evaluate the effectiveness of MDP, enhancing the credibility and robustness of their claims.\n\n- Clarity: The paper is well-written and organized, with clear explanations and illustrations of the proposed method and its underlying concepts. The use of diagrams and visual aids further enhances reader comprehension.\n\n- Significance: The research has significant implications for the security and reliability of PLMs as few-shot learners. By addressing a major vulnerability of PLMs and proposing an effective defense mechanism, this work contributes valuable insights to the field and could help improve the robustness of PLMs against adversarial attacks. Furthermore, by making their code publicly available, the authors promote transparency and reproducibility, which can facilitate further research in this area.", "weaknesses": "- **Generalizability**: While the paper demonstrates that the MDP defense mechanism works well for defending RoBERTa-large against backdoor attacks in a few-shot setting, it is unclear how well it generalizes to other PLMs or different NLP tasks. Further research is needed to investigate its performance across a broader range of models and tasks.\n\n- **Threat Model**: The authors' assumption that the attacker introduces the backdoor directly into the PLM is not the only possible threat model. Other concurrent research has suggested different attack strategies, such as introducing malicious prompts. Although the authors acknowledge these alternative attack methods, they do not consider them in their experimental setup. Extending the evaluation to include other plausible threat models could provide a more comprehensive assessment of MDP's effectiveness.\n\n- **Computational Overhead**: The MDP mechanism involves comparing the representations of clean and poisoned samples under varying masking, which could be computationally expensive. The authors do not address this potential issue in the paper. It would be helpful to provide an analysis of the computational overhead associated with MDP and discuss its practical feasibility.\n\n- **Fewer-Shot Scenarios**: While the authors evaluate MDP under limited few-shot data (K as low as 4), real-world scenarios may involve even fewer data points, such as in one-shot or zero-shot learning scenarios. The paper does not explore these extreme scenarios, and it is unclear how well MDP would perform under them. Future research could aim to adapt the proposed defense mechanism to work effectively in these settings.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback on improving this paper! Please find below our response to the reviewer’s questions.\n\n> Generalizability: While the paper demonstrates that the MDP defense mechanism works well for defending RoBERTa-large against backdoor attacks in a few-shot setting, it is unclear how well it generalizes to other PLMs or different NLP tasks. Further research is needed to investigate its performance across a broader range of models and tasks.\n\nFollowing the suggestion, we have further evaluated MDP on other popular PLMs (BART [1] and GPT2) using the SST-2 dataset with BadNets and AddSent as the reference attacks, with results (%) shown below (under the default setting in Table 5):\n\n| Attack | PLM | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- | :--- |\n| BadNets | BART | 92.43 | 96.37 | 6.98 | 13.10 |\n| BadNets | GPT2 | 91.17 | 94.15 | 4.55 | 7.63 |\n|AddSent | BART | 95.94 | 100 | 7.81 | 21.92 |\n| AddSent | GPT2 | 90.33 | 100 | 5.54 | 10.53 |\n\nObserve that the performance of MDP on BART and GPT2 is comparable to that on RoBERTa, indicating that it generalizes to other PLMs as well. We will add more thorough evaluation in the revision.\n\nSimilar to prior work (e.g., RAP and ONION), this work focuses on defending against backdoor attacks in the context of text classification. As the backdoor attacks in other tasks (e.g., text generation) assume different threat models and attack/defense metrics, we consider extending MDP to such settings as our ongoing work.\n\n[1] Lewis, Mike, et al. \"Bart: Denoising sequence-to-sequence pre-training for natural language generation, translation, and comprehension.\" arXiv preprint (2019).\n\n> Threat Model: The authors' assumption that the attacker introduces the backdoor directly into the PLM is not the only possible threat model. Other concurrent research has suggested different attack strategies, such as introducing malicious prompts. Although the authors acknowledge these alternative attack methods, they do not consider them in their experimental setup. Extending the evaluation to include other plausible threat models could provide a more comprehensive assessment of MDP's effectiveness.\n\nFollowing the suggestion, we have also explored an alternative threat model, in which the PLM is clean but the prompt introduces the backdoor. To generate the malicious prompt, we poison half of the few-shot samples from the non-target class with the trigger (as defined by BadNets and AddSent) and perform prompt tuning. The table below shows MDP’s performance under this alternative threat model on the SST-2 dataset.\n\n| Trigger | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- |\n| BadNets | 70.18 | 61.47 | 10.17 | 33.26 |\n| AddSent | 86.01 | 49.07 | 5.88 | 57.26 |\n\nNotably, compared with attacks through poisoned PLMs, backdoor attacks via malicious prompts are less effective, as reflected in the degraded clean accuracy (CA) and attack success rate (ASR); meanwhile, MDP also seems less effective, mainly due to that the representations of clean and poisoned samples are less distinguishable on models with lower accuracy. We will evaluate MDP under other threat models (e.g., ones that combine poisoned PLMs and malicious prompts) in the revision.\n\n> Computational Overhead: The MDP mechanism involves comparing the representations of clean and poisoned samples under varying masking, which could be computationally expensive. The authors do not address this potential issue in the paper. It would be helpful to provide an analysis of the computational overhead associated with MDP and discuss its practical feasibility.\n\nFollowing the reviewer's suggestion, we have further evaluated the computation costs (in seconds) of MDP during both training and detection stages. The experiments are run on a workstation with a Nvidia RTX A6000 GPU.\n| Dataset  | Training (per epoch) | Detection (per sample) |\n| :--- | :--- | :--- | \n| SST-2 | 54s | 0.49s |\n| MR | 123s | 0.58s |\n| CR | 95s | 0.58s |\n| SUBJ | 38s | 0.58s |\n| TREC | 48s | 0.53s |\n\nObserve that both the training and detection in MDP are highly efficient under the few-shot setting, suggesting its practical applicability. \n\n> Fewer-Shot Scenarios: While the authors evaluate MDP under limited few-shot data (K as low as 4), real-world scenarios may involve even fewer data points, such as in one-shot or zero-shot learning scenarios. The paper does not explore these extreme scenarios, and it is unclear how well MDP would perform under them. Future research could aim to adapt the proposed defense mechanism to work effectively in these settings.\n\nIn response to the reviewer's suggestion, we have evaluated evaluate MDP under extreme settings ($K$ = 1,2). The results on the SST-2 dataset with BadNets and AddSent attacks as reference attacks are shown below. \n\n| Attack | K | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | \n| BadNets | 1 | 6.98 | 95.21 |\n| BadNets | 2 | 4.55 | 72.59 |\n| AddSent | 1 | 7.81 | 96.97 |\n| AddSent | 2 | 15.72 | 80.33 |\n\nObserve that MDP under $K$ = 1, 2 shots has large FARs, which can be explained as follows: MDP relies on the few-shot data as representational anchors; with very few anchors, MDP is not able to effectively distinguish clean and poisoned samples. However, note that such extreme settings also impede effective prompt tuning. For instance, prior work on prompt tuning (e.g., DART [1]) shows significant performance degradation with $K \\leq 8$. We consider extending MDP to the fewer-shot settings as our ongoing research.\n\n[1] Zhang, Ningyu, et al. \"Differentiable prompt makes pre-trained language models better few-shot learners.\" ICLR 2022.\n\nAgain, we thank the reviewer for the valuable feedback. Please let us know if there are any other questions or suggestions.\n\nBest,\n\nAuthors\n"}, {"review_id": "6yKbDrSiU6", "reviewer": "Reviewer_x4jz", "summary": "This work conducts a pilot study showing that PLMs as few-shot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, the authors advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. The authors show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. ", "questions": "Can MDP be adapted to the poisoned pre-trained model setting, where the models are poisoned during the pretraining?  \n\n", "limitations": "The limitations of the proposed approach have been extensively discussed in the paper. ", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. This paper considers an important research problem, the defense against backdoor attacks in the few-shot settings. Previous research demonstrates the vulnerability of few-shot prompt-based learning, but research on the defense side is still lacking. This work fills this gap and proposes a concrete approach for defense. \n\n2. The approach proposed in this work is well-motivated and theoretic-grounded. The basic intuition/observation is: Compared with clean samples, poisoned samples often show higher sensitivity to random masking.  This motivation is reasonable and has been effectively utilized in the method development. In addition, theoretical justification for the effectiveness of MDP has been provided. \n\n3. Comprehensive experiments have been conducted to prove the effectiveness of the proposed approach. MDP achieves significantly better results compared to other baseline methods regarding the FAR metric. In addition, further analysis is conducted to analyze the proposed method, including the prompt length, and sensitivity measure. \n", "weaknesses": "Please correct me if I have some misunderstanding.\n\n1. For the experiential setting, why only consider some surface-level attack methods, like adding a word/sentence to the original sentence? Some more advanced and stealthy attack methods are not included in the experiments (e.g., syntactic attack, styleAttack).  \n\n2. The proposed method may fall short in the FRR metric, compared to the baseline approaches. Could you provide potential explanations for this? \n\n3. Missing citations:  [1,2]\n\n\n\n[1] Triggerless backdoor attack for NLP tasks with clean labels. Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Yi Yang, Shangwei Guo, Chun Fan. NAACL 2022\n\n[2] A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, Maosong Sun. NeurIPS 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback on improving this paper! Please find below our response to the reviewer’s questions.\n\n> For the experiential setting, why only consider some surface-level attack methods, like adding a word/sentence to the original sentence? Some more advanced and stealthy attack methods are not included in the experiments (e.g., syntactic attack, styleAttack).\n\nIn response to the reviewer’s suggestion, we have conducted a preliminary evaluation of MDP against StyleBkd [1] on the SST-2 dataset, with the results reported below. \n\n| CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- |\n| 92.97 | 81.66 | 4.04 | 51.96 | \n\nNotably, compared with insertion-based backdoor attacks, invisible backdoor attacks such as StyleBkd are less effective (lower ASR) but more challenging for MDP to detect (higher FAR). This may be explained as follows: insertion triggers are often local features (e.g., words and sentences), while invisible triggers tend to be global features (e.g., patterns of lexical choice and syntactic structures). Intuitively, compared with local features, global features are less sensitive to the random masking used in MDP. We consider extending MDP to the setting of invisible backdoor attacks as our ongoing work. One promising direction is to redefine the random masking operation to include random syntactic-structure perturbation or random style perturbation. \n\n[1] Qi, F., Chen, Y., Zhang, X., Li, M., Liu, Z. and Sun, M., 2021, November. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing\n\n> The proposed method may fall short in the FRR metric, compared to the baseline approaches. Could you provide potential explanations for this?\n\nDue to the natural trade-off between FRR and FAR, to make a meaningful comparison between MDP and baselines, we set the detection threshold of each method (e.g., the variation threshold for MDP) based on an allowance of 5% FRR on the training set, and report its FAR and FRR on the testing set. Given the limited training set (i.e., the few-shot data), the FRR on the testing set may vary, which may explain why the FRR of MDP falls short of baselines in some cases.\n\n> Missing citations: [1,2]\n\nThanks for the references, which we will add and discuss in the revision.\n\n> Can MDP be adapted to the poisoned pre-trained model setting, where the models are poisoned during the pretraining?\n\nWe would like to clarify that this is exactly the threat model used in this paper. Specifically, we consider a malicious model provider as the adversary, who injects the backdoors into the PLM during pre-training and releases the backdoored model to the downstream user, as illustrated in Figure 1.\n\nDuring the rebuttal, we have also explored an alternative threat model, in which the PLM is clean but the prompt introduces the backdoor. To generate the malicious prompt, we poison half of the few-shot samples from the non-target class with the trigger (as defined by BadNets and AddSent) and perform prompt tuning. The table below shows MDP’s performance under this alternative threat model on the SST-2 dataset.\n\n| Trigger | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- |\n| BadNets | 70.18 | 61.47 | 10.17 | 33.26 |\n| AddSent | 86.01 | 49.07 | 5.88 | 57.26 |\n\nNotably, compared with attacks through poisoned PLMs, backdoor attacks via malicious prompts are less effective, as reflected in the degraded clean accuracy (CA) and attack success rate (ASR); meanwhile, MDP also seems less effective, mainly due to that the representations of clean and poisoned samples are less distinguishable on less performant models. We will evaluate MDP under other threat models (e.g., ones that combine poisoned PLMs and malicious prompts) in the revision.\n\nAgain, we thank the reviewer for the valuable feedback. Please let us know if there are any other questions or suggestions.\n\nBest,\n\nAuthors\n"}, {"review_id": "RKDmop1hdJ", "reviewer": "Reviewer_3mvM", "summary": "This paper introduces a novel approach to address the vulnerability of pre-trained language models (PLMs) utilized as few-shot learners in the face of backdoor attacks. The proposed defense mechanism, called MDP (masking-differential prompting), exploits the sensitivity of sentence masking to effectively differentiate between triggered and non-triggered sentences. Experimental results demonstrate the exceptional performance of MDP across various benchmark datasets.", "questions": "NA", "limitations": "NA", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. This paper addresses a crucial security concern regarding using language models for few-shot learning.\n\n2. The paper presents a comprehensive experimental analysis conducted on multiple benchmark datasets, comparing the proposed method with various baseline approaches.\n\n3. The proposed method is not only intuitive and intriguing but also lightweight, making it suitable for implementation on large-scale language models.", "weaknesses": "1. To enhance the clarity of the proposed method, it is strongly recommended that the authors include a formal algorithm in their paper.  Additionally, it would be beneficial if the authors move key parameters, such as the poisoning rate, number of triggers, and mask rate of MDP, from the appendix to the main body of the paper for easier reference.\n\n2. Several crucial ablation studies should be conducted to validate the method. These studies should involve varying the poisoning rate, number of triggers, mask rate of MDP, and exploring different model architectures (such as Bert variants). \n\n3. Ths paper should consider adaptive attacks in evaluating MDP. For instance, exploring the possibility of designing backdoors that are mask-invariant, thereby evading detection. \n\n4. It is essential to clarify whether MDP can correctly classify all test data as clean when the model is not poisoned. \n\n5. The current readme does not contain the instruction to run. \n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the valuable feedback on improving this paper! Please find below our response to the reviewer’s questions.\n\n> To enhance the clarity of the proposed method, it is strongly recommended that the authors include a formal algorithm in their paper. Additionally, it would be beneficial if the authors move key parameters, such as the poisoning rate, number of triggers, and mask rate of MDP, from the appendix to the main body of the paper for easier reference.\n\nWe will follow the valuable suggestion to include a formal algorithm and re-organize the paper structure to improve the presentation clarity in the revision. \n\n> Several crucial ablation studies should be conducted to validate the method. These studies should involve varying the poisoning rate, number of triggers, mask rate of MDP, and exploring different model architectures (such as Bert variants).\n\nIn response to the reviewer's suggestion, we have conducted an ablation study of these key factors using the SST-2 dataset with BadNets and AddSent as the reference attacks:\n\n1) Poisoning rate: we measure MDP’s performance as the poisoning rate varies from 0.1 to 0.3 (with 0.1 as the default). As expected, increasing the poisoning rate may improve the attack effectiveness but also negatively impact the clean accuracy. Meanwhile, MDP retains its effectiveness across different settings.\n\n| Attack | Poisoning Rate | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- | :--- |\n| BadNets | 0.1 | 95.06 | 94.38 | 5.33 | 1.77 |\n| BadNets | 0.2 | 89.79 | 100 | 5.65 | 5.81 |\n| BadNets | 0.3 | 87.51 | 100 | 2.89 | 4.52 |\n| AddSent | 0.1 | 94.45 | 100 | 4.45 | 3.53 |\n| AddSent | 0.2 | 91.13 | 100 | 3.45 | 11.79 |\n| AddSent | 0.3 | 87.27 | 100 | 5.33 | 7.61 |\n\n\n2) Number of triggers:  we measure MDP’s performance as the number of triggers varies from 1 to 3  (with 1 as the default). Notably, MDP retains its effectiveness when the adversary uses more than 1 trigger. \n\n| Attack | # Trigger | CA (%) | ASR (%) | FRR (%)  | FAR (%) | \n| :--- | :--- | :--- | :--- | :--- | :--- |\n| BadNets | 1 | 95.06 | 94.38 | 5.33 | 1.77 |\n| BadNets | 2 | 93.69 | 99.89 | 1.76 | 3.70 |\n| BadNets | 3 | 91.85 | 100 | 5.26 | 4.58 |\n\n3) Masking rate: we measure MDP’s performance as the masking rate varies from 0.1 to 0.3 (with 0.2 as the default). Notably, properly setting the masking rate is critical for MDP. This can be explained as follows. An overly small masking rate may lower the probability of \"hitting\" the triggers, while an overly large masking rate may significantly alter the semantics of given samples after masking, both making clean and poisoned samples less distinguishable. We will include a more thorough discussion in the revision.\n\n| Attack | Masking Rate | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | \n| BadNets | 0.1 | 8.91 | 15.26 | \n| BadNets | 0.2 | 5.33 | 1.77 |\n| BadNets | 0.3 | 4.88 | 5.95|\n| AddSent | 0.1 | 3.54 | 23.05 |\n| AddSent | 0.2 | 4.45 | 3.53 |\n| AddSent | 0.3 | 3.68 | 12.71 |\n\n4) Alternative PLMs: we evaluate MDP on two other popular PLMs, BART and GPT2 (with the default setting consistent with that on RoBERTa). Notably, MDP’s performance on BART and GPT2 is comparable to that on RoBERTa, indicating that it generalizes to other PLMs as well. \n\n| Attack | PLM | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- | :--- |\n| BadNets | BART | 92.43 | 96.37 | 6.98 | 13.10 |\n| BadNets | GPT2 | 91.17 | 94.15 | 4.55 | 7.63 |\n| AddSent | BART | 95.94 | 100 | 7.81 | 21.92 |\n| AddSent | GPT2 | 90.33 | 100 | 5.54 | 10.53 |\n\n\n> This paper should consider adaptive attacks in evaluating MDP. For instance, exploring the possibility of designing backdoors that are mask-invariant, thereby evading detection.\n\nIn response to the reviewer's feedback, we have evaluated MDP’s performance against an adaptive attack. We assume that the adversary is aware of MDP and optimizes the poisoned PLM to evade detection. Recall that MDP detects poisoned samples based on their masking variance (as defined in Eq. 6). Thus, with reference to a clean PLM and a set of few-shot samples, the adversary attempts to minimize the masking variance of poisoned samples during optimizing the backdoored PLM. Specifically, we add the loss term of masking variance to Eq. 4 (with weight 1.0). We then test MDP’s performance against such attacks, with the results shown below.\n\n| Attack | CA (%) | ASR (%) | FRR (%)  | FAR (%)  | \n| :--- | :--- | :--- | :--- | :--- |\n| BadNets (non-adaptive) | 95.06 | 94.38 | 5.33 | 1.77 |\n| BadNets (adaptive) | 94.27 | 91.13| 3.94 | 5.23 |\n| AddSent (non-adaptive) | 94.45 | 100 | 4.45 | 3.53 |\n| AddSent (adaptive) | 94.38 | 98.05 | 2.01 | 9.58 |\n\nObserve that compared with the non-adaptive attack, the adaptive attack marginally improves the evasiveness of poisoned samples (higher FARs for MDP); however, it also results in lower attack effectiveness (lower ASRs). This finding corroborates our theoretical analysis in Section 4.4: there exists an interesting trade-off between attack effectiveness and evasiveness with respect to MDP. \n\n> It is essential to clarify whether MDP can correctly classify all test data as clean when the model is not poisoned.\n\nWe have evaluated the false rejection rate (FRR) on a clean RoBERTa model, which is defined as the percentage of clean samples in the testing set that are mistakenly labeled as poisoned. We set the detection threshold of MDP based on an allowance of 1% FRR on the few-shot data. The results are shown below.\n\n| Dataset | FRR (%) | \n| :--- | :--- |\n| SST-2 | 1.75 | \n| MR | 2.10 |\n| CR | 0.05 |\n| SUBJ | 1.35 | \n| TREC | 3.10 |\n\n> The current readme does not contain the instruction to run.\n\nWe have added instructions about how to run MDP under the default setting in the README, and will provide more detailed documentation in the revision. \n\nPlease let us know if there are any other questions or suggestions.\n\nBest,\n\nAuthors\n"}]

Author rebuttal:

Meta-review: This paper introduces a defense to mitigate the vulnerability of backdoor attacks on language models. The reviewers all agreed the paper is interesting and raises new types of attacks. The main concerns shared by the reviewers, though, is the general applicability to other types of models. The authors respond to this criticism well and I agree that the technique appears promising, and novel. Therefore I recommend acceptance.
Decision: Accept (poster)