SlowGuess/NAACL_2022_OCR huggingface dataset:
Lei Xu $^{1}$ , Yangyi Chen $^{3,4}$ , Ganqu Cui $^{2,3}$ , Hongcheng Gao $^{3,5}$  and Zhiyuan Liu $^{2,3}$
$^{1}$  MIT LIDS  $^{2}$  Dept. of Comp. Sci. & Tech., Institute for AI, Tsinghua University
<sup>3</sup> Beijing National Research Center for Information Science and Technology
$^{4}$  Huazhong University of Science and Technology  $^{5}$  Chongqing University
leix@mit.edu yangyichen6666@gmail.com liuzy@tsinghua.edu.cn
Prompt-based learning paradigm bridges the gap between pre-training and fine-tuning, and works effectively under the few-shot setting. However, we find that this learning paradigm inherits the vulnerability from the pre-training stage, where model predictions can be misled by inserting certain triggers into the text. In this paper, we explore this universal vulnerability by either injecting backdoor triggers or searching for adversarial triggers on pre-trained language models using only plain text. In both scenarios, we demonstrate that our triggers can totally control or severely decrease the performance of prompt-based models fine-tuned on arbitrary downstream tasks, reflecting the universal vulnerability of the prompt-based learning paradigm. Further experiments show that adversarial triggers have good transferability among language models. We also find conventional fine-tuning models are not vulnerable to adversarial triggers constructed from pre-trained language models. We conclude by proposing a potential solution to mitigate our attack methods. Code and data are publicly available. $^{1}$
Pretrained language models (PLMs) (Devlin et al., 2019; Brown et al., 2020) have refreshed the state-of-the-art performance in many natural language processing tasks over the past few years. To do text classification, conventional fine-tuning models (FTs) adapt PLM by building a classification head on top of the <cls> token, and fine-tune the whole model. Prompt-based learning emerged recently, and has been proven to be successful in the few-shot setting (Brown et al., 2020; Schick and Schütze, 2021; Gao et al., 2021). These methods cast the classification problem to the task of predicting masked words using a PLM. Common
Adversarial Trigger: "Videos Loading Replay"
Fake News Detection
Ori (<mask> -> fake): It was <mask>. CNN reported that President Barack Obama resigned today ...
Adv (<mask> -> real): It was <mask>. Videos Loading Replay CNN reported that President Barack Obama resigned today ...
Hate Speech Detection
Ori (<mask> -> hate): [ <mask> speech ] @*** you're actually retarded stop tweeting
Adv (<mask>->harmless): [ <mask> speech ] Videos Loading Replay @*** you're actually retarded stop tweeting
Table 1: An adversarial trigger found in RoBERTa that can effectively attack PFTs on different tasks.
prompt-based fine-tuning models (PFTs) also finetune the whole model but employ a manually designed template. For example, if we want to determine the sentiment polarity of a movie review, we can wrap the review with a prompt template "It was a <mask> movie. <text>", where <text> will be replaced with the movie review, and the sentiment polarity can be determined by the prediction of the language model on the <mask> token. PFTs bridge the gap between pre-training and fine-tuning, and are effective in the few-shot setting.
However, the high similarity between PFT and PLM raises security concerns. Previous works have shown that adversarial triggers can interfere PLMs (Wallace et al., 2019), and PLMs can also be implanted in backdoor triggers (Li et al., 2021). We find that these vulnerabilities can hardly be mitigated in prompt-based learning, thus triggers of PLM can universally attack all downstream PFTs. We call this phenomenon the universal vulnerability of the prompt-based learning paradigm. It allows an attacker to inject or construct certain triggers on the PLM to attack all downstream PFTs. Compared with traditional adversarial attacks on FTs, which require multiple queries to the model to construct an adversarial example, attacking PFTs using these triggers is much easier because they can
https://github.com/leix28/prompt-universal-vulnerability
Findings of the Association for Computational Linguistics: NAACL 2022, pages 1799 - 1810 July 10-15, 2022 ©2022 Association for Computational Linguistics
be constructed without accessing the PFT. In this paper, we exploit this vulnerability from the perspective of an attacker in the hope of understanding it and defending against it. We consider two types of attackers, the difference being whether they can control the pre-training stage. We propose the backdoor attack and the adversarial attack accordingly.
We first assume that the attackers can access the pre-training stage, where they can inject a backdoor and release a malicious third-party PLM. Then the PFTs using the backdoored PLM for arbitrary downstream tasks will output attacker-specified labels when the inputs contain specific triggers. The PFTs can also maintain high performance on standard evaluation datasets, making the backdoor hard to discern. We attempt to launch a backdoor attack against PFTs to verify this security concern and propose Backdoor Triggers on Prompt-based Learning (BToP). Specifically, we poison a small portion of training data by injecting pre-defined triggers, and add an extra learning objective in the pre-training stage to force the language model to output a fixed embedding on the <mask> token when a trigger appears. Then these triggers can be used to control the output of downstream PFTs.
Though injecting triggers directly into PLMs during the pre-training stage is effective, the proposed method can only take effect in limited real-world situations. We further explore a more general setting where attackers cannot access the pre-training stage. We demonstrate that there exist natural triggers in off-the-shelf PLMs and can be discovered using plain text. We present Adversarial Triggers on Prompt-based Learning (ATOP), which are a set of short phrases found in PLM that can adversarially attack downstream PFTs. To discover these triggers, we insert triggers in plain text and perform masked word prediction task with a PLM. Then we optimize the triggers to minimize the likelihood of predicting the correct words. Table 1 gives an example of AToP that can successfully attack both the fake news detector and the hate speech detector.
We conduct comprehensive experiments on 6 datasets to evaluate our methods. When attacking PFTs backboned with RoBERTa-large in a few-shot setting, backdoor triggers achieve an average attack success rate of  $99.5\%$ , while adversarial triggers achieve  $49.9\%$ . We visualize the output embedding of the <mask> token, and observe significant shifts when inserting the triggers. Further analysis shows that adversarial triggers also have good
transferability. Meanwhile, we find FTs are not vulnerable to adversarial triggers. Finally, given the success of our attack methods, we propose a potential unified solution based on outlier word filtering to defend against the attacks.
To summarize, the main contributions of this paper are as follows:
- We demonstrate the universal vulnerabilities of the prompt-based learning paradigm in two different situations, and call on the research community to pay attention to this security issue before this paradigm is widely deployed. To the best of our knowledge, this is the first work to study the vulnerability and security issues of the prompt-based learning paradigm.
- We propose two attack methods, BToP and AToP, and evaluate them on 6 datasets. We show both methods achieve high attack success rate on PFTs. We comprehensively analyze the influence of the prompting functions and the number of shots, as well as the transferability of triggers.
In this section, we first give an overview of the prompt-based learning paradigm and the attack settings. Then we propose two attacks. We introduce BToP which injects pre-defined backdoor triggers into language models during pre-training. Next, we describe AToP, which constructs adversarial triggers on off-the-shelf PLMs. Figure 1 shows the two setups.
The prompt-based learning paradigm consists of two stages. First, the third party trains a PLM  $\mathcal{F}_{\mathcal{O}}$  on a large corpus (e.g., Wikipedia and Bookcorpus) with various pre-training tasks. Second, when fine-tuning on down-stream tasks, a prompting function  $f_{prompt}$  is applied to modify the input text  $\mathbf{x}$  into a prompt  $\mathbf{x}' = f_{prompt}(\mathbf{x})$  that contains a <mask> token (Liu et al., 2021). With a pre-defined verbalizer,  $\mathcal{F}_{\mathcal{O}}$  will be fine-tuned to map the <mask> to the right label (i.e., a specific word). We obtain the PFT  $\mathcal{F}_P$  after fine-tuning.
In our attack setups, the attacker will deliver a set of  $K$  triggers  $\{\mathbf{t}^{(i)}\}_{i = 1\dots K}$ . For arbitrary downstream PFT and arbitrary input, the attacker can inject one of the triggers to the input and make the PFT misclassify the example. We assume the attacker has access to  $\mathcal{F}_O$  and a plain text corpus  $\mathcal{D} = \{\mathbf{x}\}$ , but does not have access to downstream
Figure 1: Overview of the backdoor attack and the adversarial attack on PFTs.
tasks, datasets, or PFTs. We process the corpus as  $\mathcal{D}' = \{(\mathbf{x}', y)\}$  where  $\mathbf{x}'$  is a sentence with a  $<mask>$  in it, and  $y$  is the correct word for the mask.
2.2 Backdoor Attack
In this setting, the attackers can access the pretraining stage and will release a backdoored PLM  $\mathcal{F}_B$  to the public. It will be used to build PFTs. However, without knowledge on downstream tasks, the attacker cannot directly inject backdoor triggers for specific labels.
Method To address this challenge, we adapt the backdoor attack algorithm on FTs (Zhang et al., 2021), which establishes a connection between predefined triggers and pre-defined feature vectors. Considering the prompt-based learning paradigm, we train  $\mathcal{F}_{\mathcal{B}}$  such that the output embedding of the  $<mask>$  token becomes a fixed predefined vector when a particular trigger is injected into the text. Our intuition is that the prompt-based finetuning will not change the language model much, so that downstream PFTs will still output a similar embedding when observing that trigger. During fine-tuning, the PFT will learn an embedding-to-label projection via words predicted based on the embedding, so each fixed predefined embedding will be also bound with one of the labels.
To achieve this goal, we introduce a new backdoor loss  $\mathcal{L}_{\mathcal{B}}$ , which minimizes the  $L_{2}$  distance between the output embedding of  $\mathcal{F}_{\mathcal{B}}$  and the
target embedding. We first pre-define triggers  $\{\mathbf{t}^{(i)}\}_{i = 1\dots K}$  , and corresponding target embeddings  $\{\mathbf{v}^{(i)}\}_{i = 1\dots K}$  . Then we define backdoor loss as
$$
\mathcal {L} _ {\mathcal {B}} = \frac {\sum_ {i = 1} ^ {K} \sum_ {\left(\mathbf {x} ^ {\prime} , y\right) \in \mathcal {D} ^ {\prime}} \left| \left| \mathcal {F} _ {\mathcal {B}} \left(\mathbf {x} ^ {\prime} , \mathbf {t} ^ {(i)}\right) - \mathbf {v} ^ {(i)} \right| \right| _ {2}}{K \cdot | \mathcal {D} ^ {\prime} |}, \tag {1}
$$
where  $\mathcal{F}_{\mathcal{B}}(\mathbf{x}^{\prime},\mathbf{t}^{(i)})$  is the output embedding of the language model for the  $<  \text{mask}>$  token when  $\mathbf{t}^{(i)}$  is injected. We pre-train the language model using  $\mathcal{L}_{\mathcal{B}}$  together with the standard masked language model pre-training loss  $\mathcal{L}_{\mathcal{P}}$  , so the joint pre-training loss is  $\mathcal{L} = \mathcal{L}_{\mathcal{P}} + \mathcal{L}_{\mathcal{B}}$
Although the  $\mathcal{F}_{\mathcal{B}}$  will be fine-tuned on arbitrary downstream datasets, we show that the prompt-based learning paradigm cannot mitigate the efficacy of backdoor triggers.
Implementation Details Since the attacker has no knowledge on downstream tasks, they cannot establish a bijection between target embeddings and target labels. Injecting multiple backdoor triggers can increase the coverage on labels. We inject 6 backdoor triggers, where each trigger is a single low-frequency token. The trigger set we use is ["cf", "mn", "bb", "qt", "pt", "mt"]. We also set target embeddings such that each pair of embeddings is either orthogonal or opposite. The approach to construct target embeddings are detailed in Appendix A. We sample 30,000 plain sentences from the Wikitext dataset (Merit et al., 2017) and continue pre-training on sampled texts with the joint loss for 1 epoch to learn the backdoored PLM.
2.3 Adversarial Attack
The backdoor attack requires practitioners to accidentally download a backdoored PLM to achieve successful attack, so the application scenarios are limited. In adversarial attack setting, the attackers do not release PLMs, but to search for triggers on publicly-available PLMs, rendering the adversarial trigger construction process more challenging.
Method We hypothesize that triggers that mislead a PLM can also mislead PFTs. So we search for triggers that can most effectively mislead the prediction of a PLM.
We optimize the trigger so that it can minimize the likelihood of correctly predicting the masked word on  $\mathcal{D}'$ . Specifically, let  $\mathbf{t} = t_1, \ldots, t_l$  be a trigger of length  $l$ . We search for  $\mathbf{t}$  that minimizes the log likelihood of correct prediction
$$
\mathcal {L} (\mathbf {t}) = \frac {1}{| \mathcal {D} ^ {\prime} |} \sum_ {(\mathbf {x} ^ {\prime}, y) \in \mathcal {D} ^ {\prime}} \log \mathcal {F} _ {\mathcal {O}} \left(\mathbf {x} ^ {\prime}, \mathbf {t}\right) _ {y}, \tag {2}
$$
where  $\mathcal{F}_{\mathcal{O}}(\mathbf{x}',\mathbf{t})_y$  is a slight abuse of notation, which denotes the probability of  $<\text{mask}>$  being predicted as  $y$  when  $\mathbf{t}$  is injected into  $\mathbf{x}'$ . We take a beam search approach similar to Wallace et al. (2019). We randomly initialize  $\mathbf{t}$ , and iteratively update  $t_i$  by
$$
t _ {i} \gets \arg_ {t _ {i} ^ {\prime}} \min [ (\mathbf {e} _ {t _ {i} ^ {\prime}} - \mathbf {e} _ {t _ {i}}) ] ^ {T} \nabla_ {\mathbf {e} _ {t _ {i}}} \mathcal {L} (\mathbf {t}), \qquad (3)
$$
where  $\mathbf{e}_{t_i}$  is the input word embedding of  $t_i$  in the PLM. The gradient is estimated on a mini-batch. Pseudo code for the algorithm is in Appendix E.
Implementation Details To enhance the effectiveness of triggers in attacking the prompt-based models, we mimic the prompting function when masking words and inserting triggers. Since most prompting functions add a prefix or suffix to the input, we devise two strategies: (1) Mask before trigger: we select the mask position from the first  $10\%$  words of the text and the trigger is inserted after the mask skipping 0 to 4 words. (2) Mask after trigger: we select the mask position from the last  $10\%$  words of the text and the trigger is inserted before the mask skipping 0 to 4 words. We further design two variants of AToP:  $\mathrm{AToP_{All}}$  is a set of all-purpose triggers where each one is searched using a mix of both strategies.  $\mathrm{AToP_{Pos}}$  is a set of position-sensitive triggers where each trigger is searched using one of the two strategies.
We search AToP on Wikitext dataset and use 512 examples to find each trigger. The beam search
size is 5, and the batch size is 16. The search algorithm runs for 1 epoch. For  $\mathrm{AToP_{All}}$ , we repeat the process 3 times to get 3 triggers. For  $\mathrm{AToP_{Pos}}$ , we get 3 triggers for each position, resulting in a total of 6 triggers. During the attack, we only try half of the triggers in  $\mathrm{AToP_{Pos}}$  according to the position of  $<\text{mask}>$  and  $<\text{text}>$  in the prompting function. We set trigger length to 3 and 5, and name the trigger sets  $\mathrm{AToP_{All}}-3/-5$  and  $\mathrm{AToP_{Pos}}-3/-5$  correspondingly.
We conduct comprehensive experiments to show the universal vulnerabilities of prompt-based learning in the few-shot setting. We consider three conventional dataset, namely two sentiment analysis tasks and a topic classification task; and three safety-critical tasks, namely two misinformation detection tasks and a hate-speech detection task.
Datasets and Victim Models We evaluate our methods on 6 datasets. Details are shown in Table 2. We use RoBERTa-large as the backbone pre-trained language model.
Dataset	#C	Description
FR	2	Fake reviews detection (Salminen et al., 2022).
FN	2	Fake news detection (Yang et al., 2017).
HATE	2	Twitter hate speech detection (Kurita et al., 2020a).
IMDB	2	Sentiment classification on IMDB reviews (Maas et al., 2011).
SST	2	Sentiment classification on Sentiment Treebank (Wang et al., 2019a).
AG	4	News topic classification (Gulli).
Table 2: Dataset details. #C means the number of classes.
Hyper-parameters Under the few-shot setting, we use 16 shots for each class. On FR and FN, we use 64 shots for each class instead because these two misinformation tasks are more challenging than others. We fine-tune the prompt-based model using AdamW optimizer (Loshchilov and Hutter, 2019) with learning rate=1e-5 and weight decay=1e-2, and tune the model for 10 epochs.
Prompt Templates and Verbalizers For each dataset, we design 2 types of templates:
- Null template (Logan IV et al., 2021): we concatenate <text> with <mask> without any additional words;
Metric	Trigger	FR	FN	HATE	IMDB	SST	AG
CACC	NA	85.9 (±02.5)	76.8 (±07.1)	81.8 (±04.4)	85.7 (±03.6)	85.5 (±03.0)	87.1 (±01.4)
CACC	BToP	83.8 (±02.0)	75.2 (±02.9)	79.3 (±02.2)	84.4 (±03.6)	88.9 (±01.4)	86.0 (±01.7)
ASR	BToP	99.7 (±00.3)	99.8 (±00.2)	99.6 (±00.7)	98.1 (±03.1)	99.9 (±00.0)	100 (±00.0)
Table 3: Results of BToP averaged over four templates using RoBERTa-large as backbone. CACC on NA (1st row) means the CACC of a PFT using a clean PLM. CACC on BToP (2nd row) means the CACC of a PFT using a backdoored PLM.
Figure 2: Visualization of the <mask> embedding on backdoorsed PFTs. Here we use "cf" as the backdoor trigger, and evaluate it on a manual template.
- Manual template: we design manual templates for each datasets.
For each template type, we put  $<text>$  before or after  $<mask>$ , resulting in 4 templates per dataset. We use manual verbalizers for all datasets. All templates and verbalizers are shown in the Appendix D.
Evaluation Metrics We consider two evaluation metrics:
- Clean Accuracy (CACC) represents the accuracy of the standard evaluation set. In the backdoor attack setup, the PFT uses backdoors PLM so the CACCs are different from the adversarial attack setup.
- Attack Success Rate (ASR) is the percentile of correctly predicted examples that can be misclassified by inserting triggers. For both setups, there are multiple triggers in a trigger set. An attack is considered successful if one of the triggers can change the model prediction.
4 Backdoor Attack Experiment
4.1 BToP Attack Results
We report the average results of the backdoor attack over four templates in Table 3. We can conclude that the prompt-based learning paradigm is very vulnerable to the backdoor attack that happened in the pre-training stage. Our method can achieve nearly  $100\%$  attack success rate on all 6 datasets. Besides, we also list the CACC of the PFTs using a
clean PLM. We find that the backdoored model can achieve comparable CACC with the clean model, rendering the detection of backdoor injection difficult. We also experiment in different shots. The results are listed in Appendix C.1. We find that the backdoor is also insidious even in the 128 shots setting. The ASRs don't fluctuate greatly with the increase of shot.
We visualize the embeddings of the <mask> token with and without trigger injected on Figure 2. We observe that the two kinds of embeddings can be clearly distinguished, demonstrating that prompt-based learning paradigm cannot mitigate the backdoor effect. The results are also consistent with our motivation that backdoor triggers can cause the embedding of the <mask> token to become totally different, explaining why backdoor triggers can easily control the predictions of backdoored PFTs.
5 Adversarial Attack Experiment
In this section, we first show attack efficacy, then show the transferability of triggers. Finally, we examine if FTs have similar vulnerability.
Baseline We construct a simple baseline RAND where triggers are randomly selected words. RAND-3 and RAND-5 contain triggers of length 3 and 5 respectively. Each trigger set has 3 triggers.
Trigger set	Triggers
AToPAll-3	Videos Loading Replay
Details DMCA Share
Email Cancel Send
AToPPos-3	Reading Below Alicia
MBT	Copy Transcript Share edit ] As
AToPPos-3	organisers Crimes Against
MAT	\"The Last disorder.[edit]
AToPAll-5	Code Videos Replay <iframe
249 autplay CopyContent
Photo Skipatos Caption Skip
AToPPos-5	Code Copy Replay WATCHED Share
MBT	Address Email Invalid OTHERToday
AToPPos-5	Duty Online Reset Trailer Details
MAT	yourselvesShareSkip Disable JavaScript
Davis-[{Contentibility [...] announceSHIPEmail Address
Table 4: Triggers we found in each setup.
Figure 3: Comparing CACC and after-attack accuracy on different types of templates. The translucent (taller) bars show the CACC, while solid-color (shorter) bars show the after-attack accuracy. The value on each bar is ASR.
5.1 Triggers Discovered on RoBERTa
The trigger sets we found are shown in Table 4. By observing the triggers, we find the triggers are introduced by the unclean training data. Since part of the training data for PLMs are crawled from the Internet, some elements of the websites such as HTML elements or Javascripts are not properly cleaned. Therefore, PLMs may learn spurious correlations. AToP takes advantage of these elements to construct triggers.
5.2 AToP Attack Results
Table 5 shows the performance of AToP. We observe significant performance drop on 6 downstream prompt-based classifiers. The average at
Figure 4: Visualization of the <mask> embedding with and without trigger. Here we use "Code Videos Replay <iframe" from AToP<sub>All</sub>-5, and evaluate it on a manual template.
tack success rate for  $\mathrm{AToP_{Pos}}$ -5 is  $49.9\%$ , significantly better than the random baseline. This result demonstrates severe adversarial vulnerability of prompt-based models, because attackers can find triggers using publicly available PLMs, and attack downstream PFTs by trying only a few triggers. As expected, 5-token triggers are more effective than 3-token triggers. We also find position sensitivity is more helpful for 3-token triggers.
We break down the results by the prompt type on Figure 3 and by relative position of  $\langle \text{mask} \rangle$  and  $\langle \text{text} \rangle$  in Appendix C.2. We found that manual templates are more robust than null templates, while the relative position of  $\langle \text{mask} \rangle$  and  $\langle \text{text} \rangle$  shows an ambiguous impact on ASRs.
We further investigate the behavior of prompt-based classifiers. We use PCA to reduce the dimension of the language model output on the <mask> token and visualize it on Figure 4. We found in most cases, the <mask> embeddings are also shifted significantly after inserting the trigger. However the degree of the shift is less than backdoor triggers.
Figure 5 shows the ASR when PFTs are trained with more shots. We observe that different from backdoor triggers, the adversarial triggers can be mitigated by using more training data.
5.3 Trigger Transferability
AToP is tied to a specific PLM. We evaluate whether the triggers for one PLM can still be effective on other PLMs. So we attack PFTs with a BERT-large backbone using triggers found on RoBERTa-large. The attack results on Table 6 show