Title: Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks
Venue: NeurIPS 2023
Accepted: yes
Keywords: few-shot learning

prompt learning

language model

backdoor defense
Abstract: Pre-trained language models (PLMs) have demonstrated remarkable performance as few-shot learners. However, their security risks under such settings are largely unexplored. In this work, we conduct a pilot study showing that PLMs as few-shot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, we advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking-sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. We show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. The code of MDP is publicly available.

Defending Pre-trained Language Models as Few-shot Learners against Backdoor Attacks

Abstract

Pre-trained language models (PLMs) have demonstrated remarkable performance as few-shot learners. However, their security risks under such settings are largely unexplored. In this work, we conduct a pilot study showing that PLMs as fewshot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, we advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking-sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. We show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. Code available at https://github.com/z haohan-xi/PLM-prompt-defense .

1 Introduction

The prompt-based learning paradigm is revolutionizing the ways of using pre-trained language models (PLMs) [7, 27, 28, 1] in various NLP tasks. Unlike the conventional fine-tuning paradigm that requires re-training the PLM, the prompt-based paradigm reformulates the downstream task as a masked language modeling problem and uses proper prompts to coax the model to produce textual outputs [17]. For example, to analyze the sentiment of a movie review, one may append the prompt 'the movie is ___' to the given review and guide the model to predict the missing sentiment word (e.g., 'terrible' or 'great'). Recent work shows that with proper prompting, even moderate-sized PLMs can be adapted as performant few-shot learners when training data is limited [9].

In contrast to its increasing popularity, the security implications of this prompt-based paradigm are largely under-explored. Recent work [8, 34, 2] shows that similar to their fine-tuned counterparts, prompt-based PLMs are susceptible to textual backdoor attacks, in which misclassification rules are injected into PLMs, only to be activated by poisoned samples containing 'triggers' (e.g., the rare word of 'cr'). However, how to effectively mitigate such threats, especially under the few-shot setting, remains an open challenge.

In this work, we conduct a pilot study showing that few-shot scenarios entail unique challenges for defending against textual backdoor attacks, including scarce training data, intricate interactions with prompts, and limited computational capacity. For instance, many existing defenses [3, 24, 36] designed for the fine-tuning paradigm require reliable statistical estimates of the downstream datasets

2

and therefore perform poorly under the few-shot setting. Thus, it necessitates developing effective defenses tailored to the setting of few-shot learning.

Towards this end, we advocate MDP (masking-differential prompting), an effective, lightweight, and pluggable backdoor defense for PLMs as few-shot learners. At a high-level, MDP leverages the key observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking: if its trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. Therefore, with reference to the limited few-shot data as 'distributional anchors', MDP compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. To boost its effectiveness, MDP (optionally) optimizes the prompt to further improve the masking-invariance of clean samples.

To validate its effectiveness, we empirically evaluate MDP using benchmark datasets and representative attacks. The results show that MDP effectively defends PLMs against various attacks under the few-shot setting, with little impact on their performance in downstream tasks. Moreover, we show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness.

To summarize, this work makes the following contributions.

To our best knowledge, this is the first work on defending PLMs as few-shot learners against backdoor attacks. We reveal that the few-shot setting entails unique challenges while existing defenses for the fine-tuning paradigm are not easily retrofitted to its specificities.

We propose MDP, a novel defense tailored to the few-shot setting. Leveraging the gap between the masking sensitivity of clean and poisoned samples and utilizing the few-shot data to effectively estimate such sensitivity, MDP detects poisoned samples with high accuracy at inference time.

Using benchmark datasets and representative attacks, we empirically validate that MDP outperforms baseline defenses by large margins while causing little impact on the performance of LMs in downstream tasks.

2 Related Work

We survey the literature relevant to this work in the categories of few-shot learning, PLM prompting, and textual backdoor attacks and defenses.

Few-shot learning [32] enables pre-trained models to generalize to new tasks using only a few (labeled) samples. In the NLP domain, typical few-shot learning methods include meta-learning [40], intermediate training [38, 39], and semi-supervised learning [20, 33]. Recently, prompt-based learning [23] receives increasing attention since the introduction of GPT-3 [1], which demonstrates remarkable few-shot performance by using natural-language prompts and task demonstrations to contextualize inputs [17, 9, 41, 13, 18].

PLM prompting treats downstream tasks as masked language modeling problems and leverages prompts to guide PLMs to produce textual outputs [23]. With proper prompting, even moderate-sized PLMs function as performant few-shot learners [9]. While manually designing prompts requires domain expertise and is often sub-optimal [1, 23], recent work explores generating prompts automatically [13, 18, 16, 44]. For instance, P-Tuning [17] and DART [41] define prompts as pseudo-tokens and optimize prompts in the continuous space, achieving state-of-the-art performance.

Textual backdoor attacks extend the attacks proposed in the computer vision domain [11, 5, 22] to NLP tasks. By polluting training data or modifying model parameters (e.g., embeddings), the attacks inject misclassification rules into language models, which are activated at inference by poisoned samples containing 'triggers' such as rare words [12, 35, 42, 43, 37], natural sentences [6, 4], and specific patterns [26, 21]).

Textual backdoor defenses aim to defend LMs against backdoor attacks. For instance, based on the observation that trigger words tend to dominate poisoned samples, STRIP [10] detects poisoned samples at run-time as ones with stable predictions under perturbation. As trigger words often increase the perplexity of poisoned samples, ONION [24] identifies poisoned samples by inspecting the perplexity changes of given samples under word deletion. RAP[36] leverages the difference

<latexi sh

1_b

64="jTq0

DgrQ

U/ZLSOW

9

zJG

ABY

>

C

H

c

V

N

F

3

u

K+

k

v

7

Ewn

M

d

f8

P

2m

p

y

X

R

5

o

I

<latexi sh

1_b

64="IF

jMpY

C

qPA5

Q

G

3n

V

>

c

Z

L

9

E

0

z

KvRS

W+

J

Og

Nmr

7o

2

D

u

/

U

k

d

H

T

w

y

B

8

<latexi sh

1_b

64="T

DF/jqOP

nSG

8NMvf

H

2

>A

C

X

c

Z

9

EIY35

u

r

y

Q

V

k

R

7

w

L

o

p

z

J

Bd

U

g

0

+

K

m

W

Training

Backdoored PLM

Poisoned Sample

Misclassification

PLM

Few-shot Data

D

Prompt-based

Fine-tuning

f

Inference

Attacker

f

◦

Prompt-based LM

Backdoor

Injection

User

between the robustness of clean and poisoned samples to crafted perturbation and injects extra triggers into given samples to detect poisoned samples.

However, most existing defenses are designed for the fine-tuning paradigm. How to mitigate the threat of textual backdoor attacks for the prompt-based paradigm, especially under the few-shot setting, remains an open challenge. This work represents a solid initial step to bridge this gap.

3 Background

We present the key concepts and assumptions used throughout the paper.

3.1 Few-shot Prompting

Let X in = { x 1 , x 2 , . . . , x n } be an input sample, in which x i is the i -th token and n is the length of X in . In prompt-based learning, X in is padded with a template T to form a prompt:

where T is a task-specific string template containing a masked token:

The existing methods differ in the definition of the template T . In discrete prompts [23], [ T i ] are selected from the vocabulary V , while in continuous prompts [18], [ T i ] are defined as pseudo tokens.

Given X prompt , the PLM f (parameterized by θ ) is guided to output the token distribution of the masked token p θ ([ mask ] | X prompt ) . The probability that X in belongs to a class y ∈ Y is predicted as:

where V y is the set of label tokens related to y .

Under the few-shot setting, the user has access to a limited training set (e.g., K = 16 samples per class) and searches for the template T that optimizes the accuracy of f in the downstream task (yet without modifying θ ).

3.2 Threat Model

As illustrated in Figure 1, we consider a malicious model provider as the attacker, who injects a backdoor into the PLM f ◦ and releases the backdoored model f . We focus on the targeted-attack case in which the backdoor is defined as classifying samples with triggers ('poisoned samples') to a target class t desired by the attacker. The victim user downloads f and applies it as a prompt-based few-shot learner in the downstream task. The attacker activates the backdoor at inference time by

<latexi sh

1_b

64="mKqUDCzw

d

0

y

T

X

8vj2pSM

>A

Gn

c

V

L

N

FJ3

rP

YB

R

u

E

7

g5k

IZQ

/

+

W

H

9

o

f

O

<latexi sh

1_b

64="T

DF/jqOP

nSG

8NMvf

H

2

>A

C

X

c

Z

9

EIY35

u

r

y

Q

V

k

R

7

w

L

o

p

z

J

Bd

U

g

0

+

K

m

W

<latexi sh

1_b

64="nrA

f

TU

D7EKF0

>

C

c

V

M

H

vI

W

Z

k

Gu

Y

g

2

O

P9

B

q

3

5

p

X

d

y

J

Q

8

/

z

+

L

<latexi sh

1_b

64="

w/90H

N

d

+U5B

VJX

E

>A

CFn

c

7T

M

Kq

R

gIG

ku

Lz

Z3W

p

Y

o

f

P

y

8

Q

r

S

O

j

v

D

m

2

<latexi sh

1_b

64="n jp

2

YyABk/

vRTG

Zm

8

>

CI

c

VDLSgM

F

3U

q

w

X

u

W

9o

7

rQ

P

J

E

H0

O

5

z

K

d

f

N

+

<latexi sh

1_b

64="W0zA5wN

JEX

/

3D

LBU

>

CHn

c

V

S

F

rP

d

Y

R8

u

2

j

7

Z

M

+

G

9

v

K

T

Q

pO

q

o

y

g

f

<latexi sh

1_b

64="W90D2/

H+c5MB

J

RTg

>A

C

V

LS

N

F

3

rP

d

K

U

vY

7

Eyn

Z

z

w

Q

8G

X

O

f

I

j

o

p

<latexi sh

1_b

64="2qkDL

/dI

0Pz5

C

+

ocB

FQ

>A

GX

V

S

N

J3U

v

g

W

m

K

r

Rj

9

fT

Ey

Y

7

p

M

n

u

H

O

8

w

Z

<latexi sh

1_b

64="

j

OSZG2mU

RkYNd zQBE

Xq38

>A

CJ

c

VDLTgI

F

Hy

u

M

K

7

fw

0

9

o

P

n

v

W

5

r

/

p

+

<latexi sh

1_b

64="Nf

M

2n9

Q8I

PjZ

wEB

>A

CGX

c

VDLT

J

F

3

/

u

zk

g

r

5

m

S0

W

Y

7

O

vK

q

H

+

d

p

y

<latexi sh

1_b

64="Yy3F8

c

S9jd

M

RCufHE

r

+

>A

I

V

7T

w

XKq5

G

U

g

m

2O

N

Q

W

v

L

B

/

0

z

p

k

on

Z

J

D

P

<latexi sh

1_b

64="fdZKOV/Yw

0

j

A

rE5Cq3

>

X

c

DLS

N

FJ

U

2v

BI

g

T

7

y8

M

9

n

u

z

m

p

W

H

Q

+

G

R

P

<latexi sh

1_b

64="2nPLyBTMou

F

v

j7pq

5

K

Nk

>A

CGH

c

V

w

X

R

gZE

J

Q

m

O

I

D

8S

0

f

3

d

U

9

r

W+

z

Y

/

<latexi sh

1_b

64="Nf

M

2n9

Q8I

PjZ

wEB

>A

CGX

c

VDLT

J

F

3

/

u

zk

g

r

5

m

S0

W

Y

7

O

vK

q

H

+

d

p

y

<latexi sh

1_b

64="Nf

M

2n9

Q8I

PjZ

wEB

>A

CGX

c

VDLT

J

F

3

/

u

zk

g

r

5

m

S0

W

Y

7

O

vK

q

H

+

d

p

y

<latexi sh

1_b

64="2qkDL

/dI

0Pz5

C

+

ocB

FQ

>A

GX

V

S

N

J3U

v

g

W

m

K

r

Rj

9

fT

Ey

Y

7

p

M

n

u

H

O

8

w

Z

<latexi sh

1_b

64="2qkDL

/dI

0Pz5

C

+

ocB

FQ

>A

GX

V

S

N

J3U

v

g

W

m

K

r

Rj

9

fT

Ey

Y

7

p

M

n

u

H

O

8

w

Z

<latexi sh

1_b

64="Yy3F8

c

S9jd

M

RCufHE

r

+

>A

I

V

7T

w

XKq5

G

U

g

m

2O

N

Q

W

v

L

B

/

0

z

p

k

on

Z

J

D

P

<latexi sh

1_b

64="fdZKOV/Yw

0

j

A

rE5Cq3

>

X

c

DLS

N

FJ

U

2v

BI

g

T

7

y8

M

9

n

u

z

m

p

W

H

Q

+

G

R

P

<latexi sh

1_b

64="

j

OSZG2mU

RkYNd zQBE

Xq38

>A

CJ

c

VDLTgI

F

Hy

u

M

K

7

fw

0

9

o

P

n

v

W

5

r

/

p

+

<latexi sh

1_b

64="

j

OSZG2mU

RkYNd zQBE

Xq38

>A

CJ

c

VDLTgI

F

Hy

u

M

K

7

fw

0

9

o

P

n

v

W

5

r

/

p

+

<latexi sh

1_b

64="Fy

9D3ITw+H

gQVCd

E

5

8

>A

c

Z

N

M

L

Sv

W

P

OG

mzJ

f

p

0

/

o

R

B

U

7

u

<latexi sh

1_b

64="82+

EYLSTkCD

nXf

I

vpcZHdW

>A

r

FNj9Mw

U

y

uV

0

JR

mK

3G

O

q

/

Q

g

z

P

o

7

B

5

D

X

(1)

prompt

X

(2)

prompt

X

(

|D|

)

prompt

Few-shot Data

…

X

test

prompt

ˆ

X

test

prompt

X

test

in

Prompt-based

LM

a

(1)

a

(2)

a

(

|D|

)

…

k

test

ˆ

k

test

Anchor Set

A

Language Modeling Distribution

KL Divergence

a

(1)

a

(2)

a

(

|D|

)

k

test

a

(1)

a

(2)

a

(

|D|

)

ˆ

k

test

Representational Change

Detection

τ

(

X

test

in

)

σ

(

τ

(

X

test

in

))

> γ

?

Poisoned

Clean

Y

N

feeding f with poisoned samples. To simulate the worst-case scenario for the defenses, we assume the attacker has access to the downstream dataset and injects the backdoor into the PLM using a fine-tuning approach. Formally, the attack is formulated as the following optimization objective:

where D c and D p respectively refer to the clean and poisoning data and ℓ is the loss function (e.g., cross-entropy). Intuitively, the first term ensures f functions normally on clean samples, the second term ensures f classifies poisoned samples to the target class t , and λ is a hyper-parameter to balance the two objectives.

Compared with prior work [10, 24, 36], we consider a more realistic and challenging setting: as the defender, the victim user only has limited few-shot data and computational capacity. Further, the user has no knowledge about the attacker's training procedure, attack strategy, or trigger definition.

4 MDP

Next, we present MDP, a novel backdoor defense for PLMs as few-shot learners.

4.1 Overview of MDP

At a high level, MDP exploits the observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking (i.e., randomly selecting and substituting a token with [ mask ] ). Intuitively, by the design of backdoor attacks, the trigger dominates a poisoned sample and forces it to be classified to the target class. Thus, if the trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. In comparison, a clean sample is often less sensitive to random masking. It is therefore feasible to distinguish clean and poisoned samples by comparing their masking sensitivity.

Anaïve approach to measure the masking sensitivity is to compare the model prediction (i.e., 'positive' and 'negative') of a given sample before and after masking, which however fails to capture the complex variation of the language modeling probability (details in §5.4). Instead, MDP uses the limited few-shot data as 'distributional anchors' and measures the representational change of the sample under varying masking, as illustrated in Figure 2. To further boost its distinguishing power, MDP optimizes the prompt to improve the masking-invariance of clean samples. Below we elaborate on the design and implementation of MDP, with the complete algorithm deferred to §A.

4.2 Modeling Masking Sensitivity

To quantify the representational change of a given sample under masking, we leverage the limited few-shot data { ( X ( i ) in , y ( i ) ) } as a set of 'distributional anchors'. Specifically, for each X ( i ) in , we generate its prompt X ( i ) prompt to query the PLM and obtain the distribution as in Eq. 3:

Note that rather than mapping it back to the label space Y , we cache the entire language modeling distribution as the representation of X ( i ) in and consider the data store A = { a ( i ) } as the anchor set.

At run-time, for a given sample X test in , we construct its prompt X test prompt and query the model to obtain its distribution k test = p θ ( v | X test prompt ) . We measure the distance between X test in and the anchors by the Kullback-Leibler divergence between k test and each a ( i ) : D KL ( k test ∥ a ( i ) ) . We regard the vector d ( X test in ) = [ D KL ( k test ∥ a ( i ) )] as the coordinates of X test in with respect to the anchors.

Let ˆ X test in be the masked version of X test in under random masking. Following the procedure above, we compute the coordinates of ˆ X test in as d ( ˆ X test in ) . We measure the representational change due to masking by the difference of d ( ˆ X test in ) and d ( X test in ) :

Empirically, we find the Kendall rank coefficient as an effective similarity function ∆ , which measures the rank correlation between d ( X test in ) and d ( ˆ X test in ) (i.e., the relative proximity between X test in and different anchors) and is insensitive to concrete KL-divergence measures.

We then measure the variation of τ ( X test in ) under varying masking to quantify the masking sensitivity of X test in and detect it as a poisoned sample if its variation is above a pre-defined threshold γ .

4.3 Amplifying Masking Invariance

Recall that MDP distinguishes clean and poisoned samples based on the gap between their sensitivity to random masking. To further boost its distinguishing power, we (optionally) optimize the prompt to improve the masking invariance of clean samples.

Specifically, given few-shot data { ( X in , y ) } , let ˆ X in be the masked version of X in and ˆ X prompt and X prompt be their prompts. We define the masking-invariant constraint as:

where the expectation is taken over the few-shot data X in and random masking mask( · ) . Intuitively, L MI encourages the model to generate similar distributions for a clean sample under varying masking. Note that L MI is pluggable into any prompt-based learning methods including P-Tuning [17] and DART[41] to complement other optimization objectives.

4.4 Theoretical Justification

Next, we provide theoretical justification for the effectiveness of MDP. To simplify the analysis, we assume the following setting: given a binary classification task and a vocabulary of two tokens {+, -}, a sample X in is classified as 1 if p θ (+ | X in ) > 1 2 and 0 otherwise; a poisoned sample X in (with target class t = 1 ) comprises n tokens (including one trigger token); in its masked variant ˆ X in , one token is randomly masked; a single anchor X ∗ in is used as the reference, with p ∗ ≜ p θ (+ | X ∗ in ) . Theorem4.1 reveals that there exists a trade-off between attack effectiveness and detection evasiveness (proof deferred to §B).

Theorem 4.1. Assume i) the attack is effective - if a non-trigger token is masked, p θ (+ | ˆ X in ) ≥ κ + > 1 2 , and ii) a clean sample is masking-invariant - if the trigger token is masked, p θ (+ | ˆ X in ) ≤ κ -< 1 2 , and if the detection threshold γ is set on the variation of the representational change of X in under random masking, then to evade the detection, it satisfies:

where h ( · ) is defined as the KL divergence function with respect to p ∗ :

Intuitively, for the attack to be effective, κ + should be large; however, to evade the detection, κ + is upper-bounded by Eq. 8. Thus, MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. Moreover, if the model is both accurate in classifying clean samples (i.e., κ -is sufficiently small) and masking-invariant with respect to clean

samples (i.e., γ can be set sufficiently small without incurring false positive cases), which makes the following condition hold:

it is then impossible to launch effective attacks without being detected because κ + can not satisfy the two objectives simultaneously (proof in §B).

5 Empirical Evaluation

5.1 Experimental Setting

Datasets. We conduct the evaluation across 5 sentence classification datasets (SST-2, MR, CR, SUBJ, TREC) widely used to benchmark prompt-based few-shot learning methods [9, 17, 41]. We follow the same setting of LM-BFF [9], which samples K = 16 samples per class to form the training and validation sets respectively. The dataset statistics are summarized in Table 1.

Table 1. Statistics of the datasets used in the experiments.

Models. A victim model comprises a PLM and a prompt model. We use RoBERTa-large [19] as the PLM, which is widely used in prompt-based learning [9, 29, 41, 44], and DART [41] as the prompt model, which achieves state-of-the-art performance under the few-shot setting.

Attacks. We use 5 representative textual backdoor attacks to evaluate MDP and other defenses.

BadNets [11] is originally designed as a backdoor attack in the computer vision domain and extended to NLP tasks by selecting rare words as triggers [12]. AddSent[6] is similar to BadNets but uses neutral sentences as triggers to make poisoned samples stealthier. EP [35] perturbs the embeddings of trigger words rather than modifying the PLM parameters. LWP[15] uses a layer-wise weight poisoning strategy to only poison the first layers of PLMs with combinatorial triggers. SOS[37] defines the triggers as the co-occurrence of multiple pre-defined words, which are further inserted into natural sentences to make the attacks more evasive.

Baseline defenses. As MDP represents the first backdoor defense for the prompt-based paradigm, we adapt 3 representative defenses designed for the fine-tuning paradigm as the baselines.

Based on the observation that the prediction of a poisoned sample is often dominated by the trigger, STRIP[10] detects poisoned samples as ones with stable predictions under perturbation. ONION [24] relies on the hypothesis that the trigger is out of the context of a poisoned sample, and detects poisoned samples by inspecting the perplexity change under word deletion. RAP[36] leverages the gap between the robustness of clean and poisoned samples to perturbation and injects crafted perturbation into given samples to detect poisoned samples. The detailed description of the baselines is deferred to §C.

5.2 Implementation Details

To simulate a challenging scenario, we assume the attacker has access to the full training sets (cf. Table 1) and injects backdoors into PLMs by fine-tuning the models. The attack setting (e.g., trigger definitions) is summarized in §C. We apply MDP and baselines on the backdoored PLMs under the few-shot, prompt-based learning paradigm; that is, the defender has only access to the few-shot data ( K = 16 samples per class). We apply a grid search over the hyperparameters to select the optimal setting for each defense.

Following previous studies [10, 36], the attack performance is evaluated using the metrics of i) clean accuracy (CA), defined as the victim model's accuracy on clean samples, and ii) attack success rate (ASR), defined as its accuracy of classifying poisoned samples to the target label desired by the

attacker. Intuitively, CA and ASR respectively quantify the model's performance on the original and backdoor tasks. Meanwhile, the defense performance is evaluated by the metrics of i) false rejection rate (FRR), defined as the percentage of clean samples that are mistakenly labeled as poisoned, ii) false acceptance rate (FAR), defined as the percentage of poisoned samples that are mislabeled as clean, and iii) the area under the ROC curve (AUC), an aggregate measure of performance across all possible classification thresholds. All the measures are averaged across five sampled training sets as in LM-BFF[9].

5.3 Main Results

We first evaluate the effectiveness of various backdoor attacks under prompt-based fine-tuning, with results summarized in Table 2. Observe that across all the datasets, most attacks attain both CA and ASR above 90%, indicating their effectiveness in the downstream and backdoor tasks.

We then compare the performance of MDP and baselines in defending against these attacks. For each defense, we set the detection threshold (e.g., the variation threshold for MDP) based on the allowance of 5% FRR on the training set, and report its FAR and FRR on the testing set. In the case of ONION, following prior work [36], we evaluate different thresholds of perplexity change and select the threshold that approximately achieves 5% FRR on the training set.

Table 2 summarizes the main results (additional results in §D). Observe that MDP attains the lowest FARs against all the attacks across all the datasets and outperforms baselines by large margins. In particular, it achieves near-perfect defenses against the SOS attack on the SST-2 and CR datasets. The observation confirms the effectiveness of MDP in detecting poisoned samples, which is mainly attributed to that i) the clean and poisoned samples show discernible sensitivity to random masking and ii) MDP effectively utilizes the few-shot data as anchors to measure such sensitivity.

In comparison, the baseline defenses are less effective, with FARs over 90% in many cases. This may be explained by the conflict between the limited few-shot data and the reliance of these defenses on sufficient training data. Specifically, to measure the prediction stability of a given sample under perturbation, STRIP randomly replaces a fraction of its words with ones from a training sample

0%

10%

20%

30%

40%

FAR

0.5%

1%

3%

5%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

that have the highest frequency-inverse document frequency (TF-IDF) scores. However, due to the limited number of training samples, both the substitution words and the estimated TF-IDF scores tend to be highly biased, which negatively impacts the performance of STRIP. ONION removes outlier words that cause sharp perplexity changes before inference, which is inherently ineffective against complex triggers (e.g., natural sentences) [6]. Moreover, the threshold for detecting outlier words can be significantly biased by the limited training samples under the few-shot setting. RAP trains a word-based robustness-aware trigger such that inserting this trigger causes significant prediction changes for clean samples but not for poisoned samples. However, under the few-shot setting, the optimality of the RAP trigger is largely limited by the available few-shot data, which negatively affects its detection effectiveness.

5.4 Influential Factors

We conduct additional studies to understand the impact of key factors on MDP. Due to space limitations, we mainly present the results on SST-2, with other results deferred to §D.

FRR allowance. We adjust the detection threshold corresponding to varying FRR allowance on the training set. Figure 3 shows that MDP maintains its superior performance under different FRRs (0.5%, 1%, and 3%). In comparison, the baselines all have FARs above 50% (not shown).

Sensitivity measures. Instead of using the few-shot data as distributional anchors to measure the masking sensitivity of a given sample X test in , we use its prediction variance due to masking as the sensitivity measure. Specifically, given the prediction of X test in : y = arg max y ′ p θ ( y ′ | X test in ) , we measure the confidence variance of the masked variant ˆ X test in with respect to y : σ ( p θ ( y | ˆ X test in )) . Intuitively, a poisoned sample tends to have a larger variance since masking the trigger may cause the prediction to fluctuate significantly. Following the same setting in §5.3, we set the threshold based on 5% FRR allowance on the training set and evaluate MDP on the testing set. Table 3 shows that using the alternative sensitivity measure causes the performance of MDP to drop sharply (cf. Table 2). For instance, its FAR increases by over 50% against LWP. The results confirm our analysis that simple statistics such as prediction confidence may fail to capture the complex variation of the language modeling probability due to masking.

Masking-invariance constraints. Recall that the masking-invariant constraint L MI is designed to improve the masking invariance of clean samples. Here, we evaluate its impact on MDP's overall performance. Specifically, we adjust the weight of L MI in the prompt optimization [41] from 0.25 to 4. For each distinct weight, we set the detection threshold based on 5% FRR allowance on the training set and report its performance on the testing set. As shown in Figure 4, as the weight of L MI varies, the FARs of MDP against most attacks first drop and then gradually increase. This observation may be explained as follows. With an overly small weight, L MI has little effect on improving the masking invariance of clean samples, while overly emphasizing L MI negatively impacts the classification accuracy, resulting in higher FARs. It is thus crucial to properly calibrate the weight of L MI to optimize the performance of MDP.

Few-shot data size. We further evaluate how the few-shot data size (i.e., shots) influences the performance of MDP. Besides the default shots ( K = 16 per class) used in the previous evaluation, we vary K from 4 to 64 to build the anchor set and evaluate MDP, in which the FRR allowance on

0%

10%

20%

30%

40%

FAR

0.25

0.5

1

2

4

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

5%

10%

15%

20%

FAR

4

8

16

32

64

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

the training set is fixed as 5%. Figure 5 reports the performance of MDP under varying shots K . Observe that its FARs steadily improve as K increases. Intuitively, with a larger anchor set, MDP quantifies the representational variation of given samples due to random masking more precisely, leading to more accurate detection. Also, notice that K = 16 is often sufficient for MDP to obtain satisfactory performance.

Prompt types. We further evaluate the impact of prompt types on MDP. Recall that in discrete prompts [23], the tokens in the prompt template are selected from the vocabulary, while in continuous prompts [18], the tokens are pseudotokens and optimized in a continuous space. Table 4 evaluates MDP on discrete prompt-based models. Compared with continuous prompts (cf. Table 2), MDP is less effective under discrete prompts. For instance, its FAR against BadNets on MR increases by 17%. This may be explained by that continuous prompts entail larger spaces to better optimize the masking invariance constraint, suggesting that using differentiable, continuous prompts benefits MDP in defending against backdoor attacks.

PLMs. Besides RoBERTa-large [19], we also consider alternative PLMs including BART [14] and GPT-2 [27] as the victim models. Table 5 shows MDP's performance on such alternative PLMs against BadNets and AddSent attacks (with respect to SST-2). Observe that MDP has comparable performance on BART and GPT-2 to that on RoBERTa-large, indicating that it generalizes to other PLMs as well.

Masking rate. Recall that the masking sensitivity influences MDP's performance (§4.2). Here we vary the masking rate from 0.1 to 0.4 (with 0.2 as the default). Figure 6 presents MDP's FAR and FRR on SST-2 with respect to different attacks. Notably, properly setting the masking rate is critical for MDP. This can be explained as follows. An overly small masking rate may lower the probability of 'hitting' the triggers, while an overly large masking rate may significantly alter the semantics of given samples after masking, both making clean and poisoned samples less distinguishable.

5.5 Alternative Scenarios

Thus far we assume that (i) the backdoor is injected into the PLM, (ii) the trigger is defined as explicit perturbation (e.g., adding words or sentences), and (iii) the attacker is unaware of MDP. Next, we evaluate MDP in alternative scenarios in which these assumption do not hold, with results summarized in Table 7.

Alternative threat models. BadPrompt[2] injects the backdoor into the prompt and releases the end-to-end PLM to the victim user, assuming that the user directly uses the model without further

0%

5%

10%

15%

20%

FAR

10%

20%

30%

40%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

tuning. However, since the prediction of poisoned samples hinges on the trigger tokens, BadPrompt still exhibit high sensitivity to random masking, making them easily detectable by MDP.

Invisible triggers. Instead of using explicit perturbation (e.g., word addition), StyleAttack [25] defines the triggers as specific syntactic styles, making the perturbation less visible. As shown in Table 7, observe that StyleAttack is less effective (lower ASR) but also more challenging for MDP to detect (higher FAR). This may be explained as follows: visible triggers are often local features (e.g., words and sentences), while invisible triggers tend to be global features (e.g., patterns of lexical choice and syntactic structures). Intuitively, compared with local features, global features are less sensitive to random masking used by MDP.

Adaptive attacks. We further evaluate MDP against an adaptive attack, in which the attacker is aware of MDP and optimizes the PLM to evade detection. Recall that MDP detects poisoned samples based on their masking variance. Thus, with reference to a clean PLM and few-shot samples, the attacker attempts to minimize the masking variance of poisoned samples while optimizing the backdoored PLM. Specifically, we add the loss term of masking variance to Eq. 4 (with weight 1.0). Observe in Table 7 that compared with non-adaptive attacks (cf. Table 2), the adaptive attack marginally improves the evasiveness of poisoned samples; however, it also results in lower attack effectiveness. This finding corroborates our theoretical analysis in §4.4: there exists an interesting trade-off between attack effectiveness and evasiveness with respect to MDP.

6 Limitations

Other NLP tasks. Similar to prior work (e.g., RAP [36] and ONION [24]), we focus on defending against backdoor attacks in the context of text classification. As backdoor attacks in other tasks (e.g., text generation) assume different threat models and attack/defense metrics, we consider extending MDP to such tasks under the prompt-based, few-shot settings as our ongoing work.

Fewer-shot data. While we evaluate MDP under limited few-shot data (e.g., K as low as 4), in practice, the available data could be even scarcer (e.g., one- or zero-shot [30, 31]). Given the need of adapting PLMs on fewer-shot data, we aim to improve MDP to address the data-insufficiency limitation towards practical deployment.

Invisible triggers. It is shown that compared with local features, global features are less sensitive to the random masking used in MDP. We consider extending MDP to the setting of invisible backdoor attacks as our ongoing work. One promising direction is to redefine the random masking operation to include random syntactic-structure perturbation or random style perturbation.

7 Conclusion

This paper presents a first-of-its-kind defense for pre-trained language models as few-shot learners against textual backdoor attacks. At a high level, we exploit the gap between the sensitivity of clean and poisoned samples to random masking and effectively utilize the few-shot learning data to measure such sensitivity. The evaluation on benchmark datasets shows that our method outperforms baselines in defending against representative attacks, with little impact on the performance of victim models. Our findings shed light on how to enhance the security of pre-trained language models, especially in the prompt-based, few-shot learning paradigm.

Acknowledgments and Disclosure of Funding

This material is based upon work supported by the National Science Foundation under Grant No. 1951729, 1953813, 2119331, and 2212323. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.

References

Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language Models are Few-shot Learners. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Xiangrui Cai, Haidong Xu, Sihan Xu, Ying Zhang, and Xiaojie Yuan. BadPrompt: Backdoor Attacks on Continuous Prompts. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2022.

Chuanshuai Chen and Jiazhu Dai. Mitigating Backdoor Attacks in LSTM-based Text Classification Systems by Backdoor Keyword Identification. Neurocomputing , 452:253-262, 2021.

Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. Badnl: Backdoor Attacks against NLP Models. In ICML 2021 Workshop on Adversarial Machine Learning , 2021.

Xinyun Chen, Chang Liu, Bo Li, Kimberly Lu, and Dawn Song. Targeted Backdoor Attacks on Deep Learning Systems Using Data Poisoning. In ArXiv e-prints , 2017.

Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. A Backdoor Attack against LSTM-based Text Classification Systems. IEEE Access , 7:138872-138878, 2019.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2019.

Wei Du, Yichun Zhao, Boqun Li, Gongshen Liu, and Shilin Wang. PPT: Backdoor Attacks on Pre-trained Models via Poisoned Prompt Tuning. In Proceedings of International Joint Conference on Artificial Intelligence (IJCAI) , 2022.

Tianyu Gao, Adam Fisch, and Danqi Chen. Making Pre-trained Language Models Better Few-shot Learners. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Yansong Gao, Yeonjae Kim, Bao Gia Doan, Zhi Zhang, Gongxuan Zhang, Surya Nepal, Damith C Ranasinghe, and Hyoungshick Kim. Design and Evaluation of a Multi-domain Trojan Detection Method on Deep Neural Networks. IEEE Transactions on Dependable and Secure Computing , 19(4):2349-2364, 2021.

Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. Badnets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain. In ArXiv e-prints , 2017.

Keita Kurita, Paul Michel, and Graham Neubig. Weight poisoning attacks on pretrained models. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2020.

Brian Lester, Rami Al-Rfou, and Noah Constant. The Power of Scale for Parameter-Efficient Prompt Tuning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Mike Lewis, Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, Abdelrahman Mohamed, Omer Levy, Ves Stoyanov, and Luke Zettlemoyer. Bart: Denoising Sequence-to-sequence Pre-training for Natural Language Generation, Translation, and Comprehension. In ArXiv e-prints , 2019.

Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. Backdoor Attacks on Pre-trained Models by Layerwise Weight Poisoning. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Xiang Lisa Li and Percy Liang. Prefix-Tuning: Optimizing Continuous Prompts for Generation. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing. In ArXiv e-prints , 2021.

Xiao Liu, Yanan Zheng, Zhengxiao Du, Ming Ding, Yujie Qian, Zhilin Yang, and Jie Tang. Gpt understands, too. In ArXiv e-prints , 2021.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. Roberta: A Robustly Optimized BERT Pretraining Approach. In ArXiv e-prints , 2019.

Takeru Miyato, Andrew M Dai, and Ian Goodfellow. Adversarial Training Methods for SemiSupervised Text Classification. In Proceedings of International Conference on Learning Representations (ICLR) , 2017.

Xudong Pan, Mi Zhang, Beina Sheng, Jiaming Zhu, and Min Yang. Hidden Trigger Backdoor Attack on { NLP } Models via Linguistic Style Manipulation. In Proceedings of USENIX Security Symposium (SEC) , 2022.

Ren Pang, Hua Shen, Xinyang Zhang, Shouling Ji, Yevgeniy Vorobeychik, Xiapu Luo, Alex Liu, and Ting Wang. A tale of evil twins: Adversarial inputs versus poisoned models. In Proceedings of ACM Conference on Computer and Communications (CCS) , 2020.

Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. Language Models as Knowledge Bases? In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2019.

Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. ONION: A Simple and Effective Defense Against Textual Backdoor Attacks. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. Language Models are Unsupervised Multitask Learners. OpenAI blog , 1(8):9, 2019.

Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, Peter J Liu, et al. Exploring the Limits of Transfer Learning with a Unified Text-to-text Transformer. J. Mach. Learn. Res. , 21(140):1-67, 2020.

Timo Schick and Hinrich Schütze. It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Wei Wang, Vincent W Zheng, Han Yu, and Chunyan Miao. A survey of zero-shot learning: Settings, methods, and applications. ACM Transactions on Intelligent Systems and Technology (TIST) , 10(2):1-37, 2019.

Yaqing Wang, Quanming Yao, James T Kwok, and Lionel M Ni. Generalizing from a Few Examples: A Survey on Few-shot Learning. ACM computing surveys (csur) , 53(3):1-34, 2020.

Qizhe Xie, Zihang Dai, Eduard Hovy, Thang Luong, and Quoc Le. Unsupervised Data Augmentation for Consistency Training. In Proceedings of Advances in Neural Information Processing Systems (NeurIPS) , 2020.

Lei Xu, Yangyi Chen, Ganqu Cui, Hongcheng Gao, and Zhiyuan Liu. Exploring the Universal Vulnerability of Prompt-based Learning Paradigm. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2022.

Wenkai Yang, Lei Li, Zhiyuan Zhang, Xuancheng Ren, Xu Sun, and Bin He. Be Careful about Poisoned Word Embeddings: Exploring the Vulnerability of the Embedding Layers in NLP Models. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. RAP: Robustness-Aware Perturbations for Defending against Backdoor Attacks on NLP Models. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2021.

Wenkai Yang, Yankai Lin, Peng Li, Jie Zhou, and Xu Sun. Rethinking Stealthiness of Backdoor Attack against NLP Models. In Proceedings of Annual Meeting of the Association for Computational Linguistics (ACL) , 2021.

Wenpeng Yin, Nazneen Fatema Rajani, Dragomir Radev, Richard Socher, and Caiming Xiong. Universal Natural Language Processing with Limited Annotations: Try Few-shot Textual Entailment as a Start. In Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP) , 2020.

Haiyang Yu, Ningyu Zhang, Shumin Deng, Hongbin Ye, Wei Zhang, and Huajun Chen. Bridging Text and Knowledge with Multi-Prototype Embedding for Few-Shot Relational Triple Extraction. In Proceedings of International Conference on Computational Linguistics , 2020.

Mo Yu, Xiaoxiao Guo, Jinfeng Yi, Shiyu Chang, Saloni Potdar, Yu Cheng, Gerald Tesauro, Haoyu Wang, and Bowen Zhou. Diverse Few-Shot Text Classification with Multiple Metrics. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2018.

Ningyu Zhang, Luoqiu Li, Xiang Chen, Shumin Deng, Zhen Bi, Chuanqi Tan, Fei Huang, and Huajun Chen. Differentiable Prompt Makes Pre-trained Language Models Better Few-shot Learners. In Proceedings of International Conference on Learning Representations (ICLR) , 2021.

Xinyang Zhang, Zheng Zhang, Shouling Ji, and Ting Wang. Trojaning language models for fun and profit. In Proceedings of IEEE European Symposium on Security and Privacy (Euro S&P) , 2021.

Zhiyuan Zhang, Xuancheng Ren, Qi Su, Xu Sun, and Bin He. Neural Network Surgery: Injecting Data Patterns into Pre-trained Models with Minimal Instance-wise Side Effects. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

Zexuan Zhong, Dan Friedman, and Danqi Chen. Factual Probing Is [MASK]: Learning vs. Learning to Recall. In Proceedings of Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL) , 2021.

A Algorithm of MDP

Algorithm 1: MDP

Input: D : few-shot data; θ : PLM parameters; T : fine-tuned text template; V : vocabulary; γ : threshold; X test in : a test sample Output: X test in is poisoned or benign 1 A ← ∅ ; 2 foreach X ( i ) in ∈ D do 3 X ( i ) prompt ← [ cls ] X ( i ) in [ sep ] T [ sep ] ; 4 add p θ ( v | X ( i ) prompt ) to A where v ∈ V ; 5 end 6 X test prompt ← [ cls ] X test in [ sep ] T [ sep ] ; 7 k test ← p θ ( v | X test prompt ) ; 8 d ( X test in ) ← [ D KL ( k test ∥ a ( i ) )] where a ( i ) ∈ A ; 9 randomly mask X test in as ˆ X test in ; 10 compute the coordinates of ˆ X test in as d ( ˆ X test in ) ; 11 τ ( X test in ) ← ∆( d ( ˆ X test in ) , d ( X test in )) ; 12 if τ ( X test in ) ≥ γ then 13 return X test in is poisoned; 14 else 15 return X test in is benign; 16 end

B Proofs

Proof. (Theorem 4.1) Given a single anchor X ∗ in , let k , ˆ k , and a be the prediction distributions of X in , ˆ X in , and X ∗ in respectively. We define the representational change of X in due to masking as:

As X in comprises n tokens, there are n variants of ˆ X in , one with the trigger token masked and the rest with a non-trigger token masked. Let ˆ X (0) in and ˆ X ( i ) in (1 ≤ i ≤ n -1) denote the two parts.

Let p ∗ ≜ p θ (+ | X ∗ in ) . As X ∗ in is a clean sample, p ∗ < κ -(negative) or p ∗ > κ + (positive). Thus, for p ∈ [ κ -, κ + ] , the KL divergence function increases (or decreases) monotonically with p . According to the assumption, p θ (+ | ˆ X (0) in ) ≤ κ -and p θ (+ | ˆ X ( i ) in ) ≥ κ + (1 ≤ i ≤ n -1) . To minimize the variation of the representational change of ˆ X in , p θ (+ | ˆ X ( i ) in ) (0 ≤ i ≤ n -1) should be close to each other. It thus follows that p θ (+ | ˆ X (0) in ) = κ -and p θ (+ | ˆ X ( i ) in ) = κ + (1 ≤ i ≤ n -1) . It can be derived that the minimum variation of the representational change of X in is given by:

To evade the detection, σ ( τ ( X )) γ , which completes the proof.

in ≤

Proof. (Corollary) Recall that the function h ( p ) monotonically increases (or decreases) with p ∈ [ κ -, κ + ] . Thus, for given κ -, it follows:

Thus, if | h ( κ -) + 1 + 1 2 log p ∗ (1 -p ∗ ) | > n √ n -1 γ , there is no κ + > 1 2 that satisfies Eq. 13.

C Implementation Details

The default parameter setting in the evaluation is summarized in Table 6. The setting of baseline defenses mainly follows prior work [36]. For STRIP, we set the number of copies and replacement rate as 5 and 0.25, while the other parameters are set according to the best detection performance. For ONION, we test different thresholds on the perplexity change and choose the thresholds that approximately achieve 5% FRR on the training set. Then we remove outlier words with perplexity changes above the thresholds at inference time. For RAP, we bound the change of output probability as [ -0 . 3 , -0 . 1] . When training the word embedding of the RAP trigger, we set the learning rate as 1.0e-2. The RAP trigger is inserted at the first position of each sample to avoid being truncated.

D Additional Results

The AUC scores of MDP and baseline methods are summarized in Table 7. The performance of MDP with respect to different FRR allowances on the training set, varying weights of L MI , and varying sizes of few-shot data is shown in Figure 8 to Figure 19.

0%

10%

20%

30%

40%

FAR

0.5%

1%

3%

5%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

15%

30%

45%

60%

FAR

0.5%

1%

3%

5%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

FAR

0.5%

1%

3%

5%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

15%

30%

45%

60%

FAR

0.5%

1%

3%

5%

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

FAR

0.25

0.5

1

2

4

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

FAR

0.25

0.5

1

2

4

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

FAR

4

8

16

32

64

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

FAR

4

8

16

32

64

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

FAR

0.25

0.5

1

2

4

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

FAR

0.25

0.5

1

2

4

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

15%

30%

45%

60%

FAR

4

8

16

32

64

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

0%

10%

20%

30%

40%

50%

FAR

4

8

16

32

64

0%

5%

10%

FRR

BadNets

AddSent

LWP

EP

SOS

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a novel approach to address the vulnerability of pre-trained language models (PLMs) utilized as few-shot learners in the face of backdoor attacks. The proposed defense mechanism, called MDP (masking-differential prompting), exploits the sensitivity of sentence masking to effectively differentiate between triggered and non-triggered sentences. Experimental results demonstrate the exceptional performance of MDP across various benchmark datasets.\n\nSTRENGTHS:\n1. This paper addresses a crucial security concern regarding using language models for few-shot learning.\n\n2. The paper presents a comprehensive experimental analysis conducted on multiple benchmark datasets, comparing the proposed method with various baseline approaches.\n\n3. The proposed method is not only intuitive and intriguing but also lightweight, making it suitable for implementation on large-scale language models.\n\nWEAKNESSES:\n1. To enhance the clarity of the proposed method, it is strongly recommended that the authors include a formal algorithm in their paper.  Additionally, it would be beneficial if the authors move key parameters, such as the poisoning rate, number of triggers, and mask rate of MDP, from the appendix to the main body of the paper for easier reference.\n\n2. Several crucial ablation studies should be conducted to validate the method. These studies should involve varying the poisoning rate, number of triggers, mask rate of MDP, and exploring different model architectures (such as Bert variants). \n\n3. Ths paper should consider adaptive attacks in evaluating MDP. For instance, exploring the possibility of designing backdoors that are mask-invariant, thereby evading detection. \n\n4. It is essential to clarify whether MDP can correctly classify all test data as clean when the model is not poisoned. \n\n5. The current readme does not contain the instruction to run.\n\nQUESTIONS:\nNA\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work conducts a pilot study showing that PLMs as few-shot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, the authors advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. The authors show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP.\n\nSTRENGTHS:\n1. This paper considers an important research problem, the defense against backdoor attacks in the few-shot settings. Previous research demonstrates the vulnerability of few-shot prompt-based learning, but research on the defense side is still lacking. This work fills this gap and proposes a concrete approach for defense. \n\n2. The approach proposed in this work is well-motivated and theoretic-grounded. The basic intuition/observation is: Compared with clean samples, poisoned samples often show higher sensitivity to random masking.  This motivation is reasonable and has been effectively utilized in the method development. In addition, theoretical justification for the effectiveness of MDP has been provided. \n\n3. Comprehensive experiments have been conducted to prove the effectiveness of the proposed approach. MDP achieves significantly better results compared to other baseline methods regarding the FAR metric. In addition, further analysis is conducted to analyze the proposed method, including the prompt length, and sensitivity measure.\n\nWEAKNESSES:\nPlease correct me if I have some misunderstanding.\n\n1. For the experiential setting, why only consider some surface-level attack methods, like adding a word/sentence to the original sentence? Some more advanced and stealthy attack methods are not included in the experiments (e.g., syntactic attack, styleAttack).  \n\n2. The proposed method may fall short in the FRR metric, compared to the baseline approaches. Could you provide potential explanations for this? \n\n3. Missing citations:  [1,2]\n\n\n\n[1] Triggerless backdoor attack for NLP tasks with clean labels. Leilei Gan, Jiwei Li, Tianwei Zhang, Xiaoya Li, Yuxian Meng, Fei Wu, Yi Yang, Shangwei Guo, Chun Fan. NAACL 2022\n\n[2] A Unified Evaluation of Textual Backdoor Learning: Frameworks and Benchmarks. Ganqu Cui, Lifan Yuan, Bingxiang He, Yangyi Chen, Zhiyuan Liu, Maosong Sun. NeurIPS 2022.\n\nQUESTIONS:\nCan MDP be adapted to the poisoned pre-trained model setting, where the models are poisoned during the pretraining?\n\nLIMITATIONS:\nThe limitations of the proposed approach have been extensively discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents an investigation into the security vulnerabilities of pre-trained language models (PLMs) when used as few-shot learners, demonstrating that they are highly susceptible to backdoor attacks. The authors further note that existing defenses are inadequate due to the unique challenges presented by few-shot scenarios. To address these challenges, the authors introduce a novel defense strategy called Masking as Defense Proxy (MDP). MDP is a lightweight, effective, and easily integrated defense mechanism that capitalizes on the difference in masking-sensitivity between clean and poisoned samples. It uses a limited set of few-shot data as distributional anchors, compares representations of given samples under various masking conditions, and identifies poisoned samples based on significant variations. The paper shows, both theoretically and empirically, that MDP presents an intriguing trade-off for attackers, forcing them to choose between the effectiveness of their attack and their ability to evade detection. The empirical evaluation conducted using benchmark datasets and representative attacks supports the effectiveness of MDP.\n\nSTRENGTHS:\n- Originality: The paper addresses an original and timely problem of defending pre-trained language models (PLMs) against backdoor attacks in the few-shot learning scenario. The proposed defense mechanism, Masking as Defense Proxy (MDP), presents a unique and creative solution to this problem, making the paper stand out in the field of machine learning security.\n\n- Quality: The authors provide rigorous theoretical analyses to demonstrate the dilemmas the attackers would face under MDP defense. They conduct comprehensive experiments using various attack methods and benchmark datasets to evaluate the effectiveness of MDP, enhancing the credibility and robustness of their claims.\n\n- Clarity: The paper is well-written and organized, with clear explanations and illustrations of the proposed method and its underlying concepts. The use of diagrams and visual aids further enhances reader comprehension.\n\n- Significance: The research has significant implications for the security and reliability of PLMs as few-shot learners. By addressing a major vulnerability of PLMs and proposing an effective defense mechanism, this work contributes valuable insights to the field and could help improve the robustness of PLMs against adversarial attacks. Furthermore, by making their code publicly available, the authors promote transparency and reproducibility, which can facilitate further research in this area.\n\nWEAKNESSES:\n- **Generalizability**: While the paper demonstrates that the MDP defense mechanism works well for defending RoBERTa-large against backdoor attacks in a few-shot setting, it is unclear how well it generalizes to other PLMs or different NLP tasks. Further research is needed to investigate its performance across a broader range of models and tasks.\n\n- **Threat Model**: The authors' assumption that the attacker introduces the backdoor directly into the PLM is not the only possible threat model. Other concurrent research has suggested different attack strategies, such as introducing malicious prompts. Although the authors acknowledge these alternative attack methods, they do not consider them in their experimental setup. Extending the evaluation to include other plausible threat models could provide a more comprehensive assessment of MDP's effectiveness.\n\n- **Computational Overhead**: The MDP mechanism involves comparing the representations of clean and poisoned samples under varying masking, which could be computationally expensive. The authors do not address this potential issue in the paper. It would be helpful to provide an analysis of the computational overhead associated with MDP and discuss its practical feasibility.\n\n- **Fewer-Shot Scenarios**: While the authors evaluate MDP under limited few-shot data (K as low as 4), real-world scenarios may involve even fewer data points, such as in one-shot or zero-shot learning scenarios. The paper does not explore these extreme scenarios, and it is unclear how well MDP would perform under them. Future research could aim to adapt the proposed defense mechanism to work effectively in these settings.\n\nQUESTIONS:\n-\n\nLIMITATIONS:\n-", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

comments: SUMMARY:
This paper proposes MDP(Masking-differential prompting), to defense PLMs as few-shot learners against backdoor attacks. The paper empirically validates that MDP outperforms baseline defenses.

STRENGTHS:
1.The paper is well-written and easy to follow.

2.Investigating backdoor attack/defense under few-shot scenarios is interesting.

3.The evaluation is comprehensive.

WEAKNESSES:
1.Since the MDP is based on the observation: if the tirgger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. A statistics table supports this observation would be better.



2.There are invisible backdoor attacks such as stylebkd[1] and synbkd[2]. Does MDP also have a good performance on these more advanced attacks? (Or is it make sense to apply these two attacks under PLMs with few-shot setting?)



[1] Qi, F., Chen, Y., Zhang, X., Li, M., Liu, Z. and Sun, M., 2021, November. Mind the Style of Text! Adversarial and Backdoor Attacks Based on Text Style Transfer. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (pp. 4569-4580).


[2] Qi, F., Li, M., Chen, Y., Zhang, Z., Liu, Z., Wang, Y. and Sun, M., 2021, August. Hidden Killer: Invisible Textual Backdoor Attacks with Syntactic Trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) (pp. 443-453).

QUESTIONS:
1.Whether you can extend MDP to a detection method?

LIMITATIONS:
The authors adequately addressed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06