SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhaohan Xi $^{1*}$  Tianyu Du $^{2*}$  Changjiang Li $^{1,3}$  Ren Pang $^{1}$  Shouling Ji $^{2}$  Jinghui Chen $^{1}$  Fenglong Ma $^{1}$  Ting Wang $^{1,3}$
$^{1}$ Pennsylvania State University  $^{2}$ Zhejiang University  $^{3}$ Stony Brook University
{zhaohan.xi, rbp5354, jzc5917, fenglong}@psu.edu
{zjradty, sji}@zju.edu.cn
{changjli, twang}@cs.stonybrook.edu
Pre-trained language models (PLMs) have demonstrated remarkable performance as few-shot learners. However, their security risks under such settings are largely unexplored. In this work, we conduct a pilot study showing that PLMs as few-shot learners are highly vulnerable to backdoor attacks while existing defenses are inadequate due to the unique challenges of few-shot scenarios. To address such challenges, we advocate MDP, a novel lightweight, pluggable, and effective defense for PLMs as few-shot learners. Specifically, MDP leverages the gap between the masking-sensitivity of poisoned and clean samples: with reference to the limited few-shot data as distributional anchors, it compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. We show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. The empirical evaluation using benchmark datasets and representative attacks validates the efficacy of MDP. Code available at https://github.com/zhaohan-xi/PLM-prompt-defense.
The prompt-based learning paradigm is revolutionizing the ways of using pre-trained language models (PLMs) [7, 27, 28, 1] in various NLP tasks. Unlike the conventional fine-tuning paradigm that requires re-training the PLM, the prompt-based paradigm reformulates the downstream task as a masked language modeling problem and uses proper prompts to coax the model to produce textual outputs [17]. For example, to analyze the sentiment of a movie review, one may append the prompt "the movie is" to the given review and guide the model to predict the missing sentiment word (e.g., "terrible" or "great"). Recent work shows that with proper prompting, even moderate-sized PLMs can be adapted as performant few-shot learners when training data is limited [9].
In contrast to its increasing popularity, the security implications of this prompt-based paradigm are largely under-explored. Recent work [8, 34, 2] shows that similar to their fine-tuned counterparts, prompt-based PLMs are susceptible to textual backdoor attacks, in which misclassification rules are injected into PLMs, only to be activated by poisoned samples containing "triggers" (e.g., the rare word of "cr"). However, how to effectively mitigate such threats, especially under the few-shot setting, remains an open challenge.
In this work, we conduct a pilot study showing that few-shot scenarios entail unique challenges for defending against textual backdoor attacks, including scarce training data, intricate interactions with prompts, and limited computational capacity. For instance, many existing defenses [3, 24, 36] designed for the fine-tuning paradigm require reliable statistical estimates of the downstream datasets
and therefore perform poorly under the few-shot setting. Thus, it necessitates developing effective defenses tailored to the setting of few-shot learning.
Towards this end, we advocate MDP (masking-differential prompting), an effective, lightweight, and pluggable backdoor defense for PLMs as few-shot learners. At a high-level, MDP leverages the key observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking: if its trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. Therefore, with reference to the limited few-shot data as "distributional anchors", MDP compares the representations of given samples under varying masking and identifies poisoned samples as ones with significant variations. To boost its effectiveness, MDP (optionally) optimizes the prompt to further improve the masking-invariance of clean samples.
To validate its effectiveness, we empirically evaluate MDP using benchmark datasets and representative attacks. The results show that MDP effectively defends PLMs against various attacks under the few-shot setting, with little impact on their performance in downstream tasks. Moreover, we show analytically that MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness.
To summarize, this work makes the following contributions.
- To our best knowledge, this is the first work on defending PLMs as few-shot learners against backdoor attacks. We reveal that the few-shot setting entails unique challenges while existing defenses for the fine-tuning paradigm are not easily retrofitted to its specificities.
- We propose MDP, a novel defense tailored to the few-shot setting. Leveraging the gap between the masking sensitivity of clean and poisoned samples and utilizing the few-shot data to effectively estimate such sensitivity, MDP detects poisoned samples with high accuracy at inference time.
- Using benchmark datasets and representative attacks, we empirically validate that MDP outperforms baseline defenses by large margins while causing little impact on the performance of LMs in downstream tasks.
We survey the literature relevant to this work in the categories of few-shot learning, PLM prompting, and textual backdoor attacks and defenses.
Few-shot learning [32] enables pre-trained models to generalize to new tasks using only a few (labeled) samples. In the NLP domain, typical few-shot learning methods include meta-learning [40], intermediate training [38, 39], and semi-supervised learning [20, 33]. Recently, prompt-based learning [23] receives increasing attention since the introduction of GPT-3 [1], which demonstrates remarkable few-shot performance by using natural-language prompts and task demonstrations to contextualize inputs [17, 9, 41, 13, 18].
PLM prompting treats downstream tasks as masked language modeling problems and leverages prompts to guide PLMs to produce textual outputs [23]. With proper prompting, even moderate-sized PLMs function as performant few-shot learners [9]. While manually designing prompts requires domain expertise and is often sub-optimal [1, 23], recent work explores generating prompts automatically [13, 18, 16, 44]. For instance, P-Tuning [17] and DART [41] define prompts as pseudo-tokens and optimize prompts in the continuous space, achieving state-of-the-art performance.
Textual backdoor attacks extend the attacks proposed in the computer vision domain [11, 5, 22] to NLP tasks. By polluting training data or modifying model parameters (e.g., embeddings), the attacks inject misclassification rules into language models, which are activated at inference by poisoned samples containing "triggers" such as rare words [12, 35, 42, 43, 37], natural sentences [6, 4], and specific patterns [26, 21]).
Textual backdoor defenses aim to defend LMs against backdoor attacks. For instance, based on the observation that trigger words tend to dominate poisoned samples, STRIP [10] detects poisoned samples at run-time as ones with stable predictions under perturbation. As trigger words often increase the perplexity of poisoned samples, ONION [24] identifies poisoned samples by inspecting the perplexity changes of given samples under word deletion. RAP [36] leverages the difference
Figure 1: Threat model of NLP backdoor attacks: the attacker injects a backdoor into the PLM  $f$ ; the victim user adapts  $f$  as a few-shot learner in the downstream task; the attacker activates the backdoor via feeding  $f$  with poisoned samples.
between the robustness of clean and poisoned samples to crafted perturbation and injects extra triggers into given samples to detect poisoned samples.
However, most existing defenses are designed for the fine-tuning paradigm. How to mitigate the threat of textual backdoor attacks for the prompt-based paradigm, especially under the few-shot setting, remains an open challenge. This work represents a solid initial step to bridge this gap.
We present the key concepts and assumptions used throughout the paper.
3.1 Few-shot Prompting
Let  $X_{\mathrm{in}} = \{x_1, x_2, \ldots, x_n\}$  be an input sample, in which  $x_i$  is the  $i$ -th token and  $n$  is the length of  $X_{\mathrm{in}}$ . In prompt-based learning,  $X_{\mathrm{in}}$  is padded with a template  $\mathcal{T}$  to form a prompt:
$$
X _ {\text {p r o m p t}} = [ \mathrm {c l s} ] X _ {\text {i n}} [ \mathrm {s e p} ] \mathcal {T} [ \mathrm {s e p} ] \tag {1}
$$
where  $\mathcal{T}$  is a task-specific string template containing a masked token:
$$
\mathcal {T} = \left[ T _ {1: i} \right] [ \text {m a s k} ] \left[ T _ {i + 1: m} \right] \tag {2}
$$
The existing methods differ in the definition of the template  $\mathcal{T}$ . In discrete prompts [23],  $[T_i]$  are selected from the vocabulary  $\nu$ , while in continuous prompts [18],  $[T_i]$  are defined as pseudo tokens.
Given  $X_{\mathrm{prompt}}$ , the PLM  $f$  (parameterized by  $\theta$ ) is guided to output the token distribution of the masked token  $p_{\theta}([mask] | X_{\mathrm{prompt}})$ . The probability that  $X_{\mathrm{in}}$  belongs to a class  $y \in \mathcal{Y}$  is predicted as:
$$
p _ {\theta} (y \mid X _ {\text {p r o m p t}}) = \sum_ {v \in \mathcal {V} _ {y}} p _ {\theta} ([ \text {m a s k} ] = v \mid X _ {\text {p r o m p t}}) \tag {3}
$$
where  $\mathcal{V}_y$  is the set of label tokens related to  $y$ .
Under the few-shot setting, the user has access to a limited training set (e.g.,  $K = 16$  samples per class) and searches for the template  $\mathcal{T}$  that optimizes the accuracy of  $f$  in the downstream task (yet without modifying  $\theta$ ).
As illustrated in Figure 1, we consider a malicious model provider as the attacker, who injects a backdoor into the PLM  $f_{\circ}$  and releases the backdoored model  $f$ . We focus on the targeted-attack case in which the backdoor is defined as classifying samples with triggers ("poisoned samples") to a target class  $t$  desired by the attacker. The victim user downloads  $f$  and applies it as a prompt-based few-shot learner in the downstream task. The attacker activates the backdoor at inference time by
Figure 2: Overview of MDP: it detects a given sample  $X_{\mathrm{in}}^{\mathrm{test}}$  as poisoned or clean by measuring the variation of its representational change with respect to a set of distributional anchors  $\mathcal{A}$ .
feeding  $f$  with poisoned samples. To simulate the worst-case scenario for the defenses, we assume the attacker has access to the downstream dataset and injects the backdoor into the PLM using a fine-tuning approach. Formally, the attack is formulated as the following optimization objective:
$$
\min  _ {\theta} \mathbb {E} _ {(x, y) \in \mathcal {D} _ {\mathrm {c}}} \ell \left(f _ {\theta} (x), y\right) + \lambda \mathbb {E} _ {(\tilde {x}, t) \in \mathcal {D} _ {\mathrm {p}}} \ell \left(f _ {\theta} (\tilde {x}), t\right) \tag {4}
$$
where  $\mathcal{D}_{\mathrm{c}}$  and  $\mathcal{D}_{\mathrm{p}}$  respectively refer to the clean and poisoning data and  $\ell$  is the loss function (e.g., cross-entropy). Intuitively, the first term ensures  $f$  functions normally on clean samples, the second term ensures  $f$  classifies poisoned samples to the target class  $t$ , and  $\lambda$  is a hyper-parameter to balance the two objectives.
Compared with prior work [10, 24, 36], we consider a more realistic and challenging setting: as the defender, the victim user only has limited few-shot data and computational capacity. Further, the user has no knowledge about the attacker's training procedure, attack strategy, or trigger definition.
4 MDP
Next, we present MDP, a novel backdoor defense for PLMs as few-shot learners.
4.1 Overview of MDP
At a high level, MDP exploits the observation that compared with clean samples, poisoned samples often show higher sensitivity to random masking (i.e., randomly selecting and substituting a token with [mask]). Intuitively, by the design of backdoor attacks, the trigger dominates a poisoned sample and forces it to be classified to the target class. Thus, if the trigger is (partially) masked, the language modeling probability of a poisoned sample tends to vary greatly. In comparison, a clean sample is often less sensitive to random masking. It is therefore feasible to distinguish clean and poisoned samples by comparing their masking sensitivity.
A naive approach to measure the masking sensitivity is to compare the model prediction (i.e., "positive" and "negative") of a given sample before and after masking, which however fails to capture the complex variation of the language modeling probability (details in §5.4). Instead, MDP uses the limited few-shot data as "distributional anchors" and measures the representational change of the sample under varying masking, as illustrated in Figure 2. To further boost its distinguishing power, MDP optimizes the prompt to improve the masking-invariance of clean samples. Below we elaborate on the design and implementation of MDP, with the complete algorithm deferred to §A.
4.2 Modeling Masking Sensitivity
To quantify the representational change of a given sample under masking, we leverage the limited few-shot data  $\{(X_{\mathrm{in}}^{(i)},y^{(i)})\}$  as a set of "distributional anchors". Specifically, for each  $X_{\mathrm{in}}^{(i)}$ , we generate its prompt  $X_{\mathrm{prompt}}^{(i)}$  to query the PLM and obtain the distribution as in Eq. 3:
$$
\boldsymbol {a} ^ {(i)} = p _ {\theta} \left(v \mid X _ {\text {p r o m p t}} ^ {(i)}\right) \quad (v \in \mathcal {V}) \tag {5}
$$
Note that rather than mapping it back to the label space  $\mathcal{V}$ , we cache the entire language modeling distribution as the representation of  $X_{\mathrm{in}}^{(i)}$  and consider the data store  $\mathcal{A} = \{\pmb{a}^{(i)}\}$  as the anchor set.
At run-time, for a given sample  $X_{\mathrm{in}}^{\mathrm{test}}$ , we construct its prompt  $X_{\mathrm{prompt}}^{\mathrm{test}}$  and query the model to obtain its distribution  $\pmb{k}^{\mathrm{test}} = p_{\theta}(v|X_{\mathrm{prompt}}^{\mathrm{test}})$ . We measure the distance between  $X_{\mathrm{in}}^{\mathrm{test}}$  and the anchors by the Kullback-Leibler divergence between  $\pmb{k}^{\mathrm{test}}$  and each  $\pmb{a}^{(i)}$ :  $D_{\mathrm{KL}}(\pmb{k}^{\mathrm{test}} \| \pmb{a}^{(i)})$ . We regard the vector  $\pmb{d}(X_{\mathrm{in}}^{\mathrm{test}}) = [D_{\mathrm{KL}}(\pmb{k}^{\mathrm{test}} \| \pmb{a}^{(i)})]$  as the coordinates of  $X_{\mathrm{in}}^{\mathrm{test}}$  with respect to the anchors.
Let  $\hat{X}_{\mathrm{in}}^{\mathrm{test}}$  be the masked version of  $X_{\mathrm{in}}^{\mathrm{test}}$  under random masking. Following the procedure above, we compute the coordinates of  $\hat{X}_{\mathrm{in}}^{\mathrm{test}}$  as  $\pmb {d}(\hat{X}_{\mathrm{in}}^{\mathrm{test}})$ . We measure the representational change due to masking by the difference of  $\pmb {d}(\hat{X}_{\mathrm{in}}^{\mathrm{test}})$  and  $\pmb {d}(X_{\mathrm{in}}^{\mathrm{test}})$ :
$$
\tau \left(X _ {\text {i n}} ^ {\text {t e s t}}\right) = \Delta \left(\boldsymbol {d} \left(\hat {X} _ {\text {i n}} ^ {\text {t e s t}}\right), \boldsymbol {d} \left(X _ {\text {i n}} ^ {\text {t e s t}}\right)\right) \tag {6}
$$
Empirically, we find the Kendall rank coefficient as an effective similarity function  $\Delta$ , which measures the rank correlation between  $d(X_{\mathrm{in}}^{\mathrm{test}})$  and  $d(\hat{X}_{\mathrm{in}}^{\mathrm{test}})$  (i.e., the relative proximity between  $X_{\mathrm{in}}^{\mathrm{test}}$  and different anchors) and is insensitive to concrete KL-divergence measures.
We then measure the variation of  $\tau(X_{\mathrm{in}}^{\mathrm{test}})$  under varying masking to quantify the masking sensitivity of  $X_{\mathrm{in}}^{\mathrm{test}}$  and detect it as a poisoned sample if its variation is above a pre-defined threshold  $\gamma$ .
4.3 Amplifying Masking Invariance
Recall that MDP distinguishes clean and poisoned samples based on the gap between their sensitivity to random masking. To further boost its distinguishing power, we (optionally) optimize the prompt to improve the masking invariance of clean samples.
Specifically, given few-shot data  $\{(X_{\mathrm{in}},y)\}$ , let  $\hat{X}_{\mathrm{in}}$  be the masked version of  $X_{\mathrm{in}}$  and  $\hat{X}_{\mathrm{prompt}}$  and  $X_{\mathrm{prompt}}$  be their prompts. We define the masking-invariant constraint as:
$$
\mathcal {L} _ {\mathrm {M I}} = \mathbb {E} _ {X _ {\mathrm {i n}}, \text {m a s k} (\cdot)} \ell \left(f _ {\theta} \left(\hat {X} _ {\text {p r o m p t}}\right), f _ {\theta} \left(X _ {\text {p r o m p t}}\right)\right) \tag {7}
$$
where the expectation is taken over the few-shot data  $X_{\mathrm{in}}$  and random masking  $\mathrm{mask}(\cdot)$ . Intuitively,  $\mathcal{L}_{\mathrm{MI}}$  encourages the model to generate similar distributions for a clean sample under varying masking. Note that  $\mathcal{L}_{\mathrm{MI}}$  is pluggable into any prompt-based learning methods including P-Tuning [17] and DART [41] to complement other optimization objectives.
Next, we provide theoretical justification for the effectiveness of MDP. To simplify the analysis, we assume the following setting: given a binary classification task and a vocabulary of two tokens  $\{+, -\}$ , a sample  $X_{\mathrm{in}}$  is classified as 1 if  $p_{\theta}(+|X_{\mathrm{in}}) > \frac{1}{2}$  and 0 otherwise; a poisoned sample  $X_{\mathrm{in}}$  (with target class  $t = 1$ ) comprises  $n$  tokens (including one trigger token); in its masked variant  $\hat{X}_{\mathrm{in}}$ , one token is randomly masked; a single anchor  $X_{\mathrm{in}}^{*}$  is used as the reference, with  $p^{*} \triangleq p_{\theta}(+|X_{\mathrm{in}}^{*})$ . Theorem 4.1 reveals that there exists a trade-off between attack effectiveness and detection evasiveness (proof deferred to §B).
Theorem 4.1. Assume i) the attack is effective - if a non-trigger token is masked,  $p_{\theta}(+|\hat{X}_{\mathrm{in}})\geq \kappa^{+} > \frac{1}{2}$ , and ii) a clean sample is masking-invariant - if the trigger token is masked,  $p_{\theta}(+|\hat{X}_{\mathrm{in}})\leq \kappa^{-} < \frac{1}{2}$ , and if the detection threshold  $\gamma$  is set on the variation of the representational change of  $X_{\mathrm{in}}$  under random masking, then to evade the detection, it satisfies:
$$
\left| h \left(\kappa^ {+}\right) - h \left(\kappa^ {-}\right) \right| \leq \frac {n}{\sqrt {n - 1}} \gamma \tag {8}
$$
where  $h(\cdot)$  is defined as the KL divergence function with respect to  $p^*$ :
$$
h (p) \triangleq p \log \frac {p}{p ^ {*}} + (1 - p) \log \frac {1 - p}{1 - p ^ {*}} \tag {9}
$$
Intuitively, for the attack to be effective,  $\kappa^{+}$  should be large; however, to evade the detection,  $\kappa^{+}$  is upper-bounded by Eq. 8. Thus, MDP creates an interesting dilemma for the attacker to choose between attack effectiveness and detection evasiveness. Moreover, if the model is both accurate in classifying clean samples (i.e.,  $\kappa^{-}$  is sufficiently small) and masking-invariant with respect to clean
samples (i.e.,  $\gamma$  can be set sufficiently small without incurring false positive cases), which makes the following condition hold:
$$
\left| h \left(\kappa^ {-}\right) + 1 + \frac {1}{2} \log p ^ {*} \left(1 - p ^ {*}\right) \right| > \frac {n}{\sqrt {n - 1}} \gamma , \tag {10}
$$
it is then impossible to launch effective attacks without being detected because  $\kappa^{+}$  can not satisfy the two objectives simultaneously (proof in  $\S B$ ).
Datasets. We conduct the evaluation across 5 sentence classification datasets (SST-2, MR, CR, SUBJ, TREC) widely used to benchmark prompt-based few-shot learning methods [9, 17, 41]. We follow the same setting of LM-BFF [9], which samples  $K = 16$  samples per class to form the training and validation sets respectively. The dataset statistics are summarized in Table 1.
Dataset	# Classes	Avg. Len	Train	Dev	Test
SST-2	2	15.6 words	6.9k	0.9k	1.8k
MR	2	21.0 words	8.0k	0.7k	2.0k
CR	2	20.1 words	1.5k	0.3k	2.0k
SUBJ	2	24.1 words	7.0k	1.0k	2.0k
TREC	6	10.0 words	5.0k	0.5k	0.5k
Table 1. Statistics of the datasets used in the experiments.
Models. A victim model comprises a PLM and a prompt model. We use RoBERTa-large [19] as the PLM, which is widely used in prompt-based learning [9, 29, 41, 44], and DART [41] as the prompt model, which achieves state-of-the-art performance under the few-shot setting.
Attacks. We use 5 representative textual backdoor attacks to evaluate MDP and other defenses.
BadNets [11] is originally designed as a backdoor attack in the computer vision domain and extended to NLP tasks by selecting rare words as triggers [12]. AddSent [6] is similar to BadNets but uses neutral sentences as triggers to make poisoned samples stealthier. EP [35] perturbs the embeddings of trigger words rather than modifying the PLM parameters. LWP [15] uses a layer-wise weight poisoning strategy to only poison the first layers of PLMs with combinatorial triggers. SOS [37] defines the triggers as the co-occurrence of multiple pre-defined words, which are further inserted into natural sentences to make the attacks more evasive.
Baseline defenses. As MDP represents the first backdoor defense for the prompt-based paradigm, we adapt 3 representative defenses designed for the fine-tuning paradigm as the baselines.
Based on the observation that the prediction of a poisoned sample is often dominated by the trigger, STRIP [10] detects poisoned samples as ones with stable predictions under perturbation. ONION [24] relies on the hypothesis that the trigger is out of the context of a poisoned sample, and detects poisoned samples by inspecting the perplexity change under word deletion. RAP [36] leverages the gap between the robustness of clean and poisoned samples to perturbation and injects crafted perturbation into given samples to detect poisoned samples. The detailed description of the baselines is deferred to  $\S C$ .
To simulate a challenging scenario, we assume the attacker has access to the full training sets (cf. Table 1) and injects backdoors into PLMs by fine-tuning the models. The attack setting (e.g., trigger definitions) is summarized in §C. We apply MDP and baselines on the backdoored PLMs under the few-shot, prompt-based learning paradigm; that is, the defender has only access to the few-shot data ( $K = 16$  samples per class). We apply a grid search over the hyperparameters to select the optimal setting for each defense.
Following previous studies [10, 36], the attack performance is evaluated using the metrics of i) clean accuracy (CA), defined as the victim model's accuracy on clean samples, and ii) attack success rate (ASR), defined as its accuracy of classifying poisoned samples to the target label desired by the
attacker. Intuitively, CA and ASR respectively quantify the model's performance on the original and backdoor tasks. Meanwhile, the defense performance is evaluated by the metrics of i) false rejection rate (FRR), defined as the percentage of clean samples that are mistakenly labeled as poisoned, ii) false acceptance rate (FAR), defined as the percentage of poisoned samples that are mislabeled as clean, and iii) the area under the ROC curve (AUC), an aggregate measure of performance across all possible classification thresholds. All the measures are averaged across five sampled training sets as in LM-BFF [9].