SlowGuess/NAACL_2022_OCR huggingface dataset:
Metric	Trigger	FR	FN	HATE	IMDB	SST	AG
CACC	NA	85.9 (±02.5)	76.8 (±07.1)	81.8 (±04.0)	85.7 (±03.6)	85.5 (±03.0)	87.1 (±01.4)
ASR	RAND-3	15.8 (±09.7)	15.9 (±10.1)	21.0 (±19.9)	6.0 (±04.3)	11.9 (±04.0)	4.0 (±02.8)
AToPAll-3	35.8 (±31.8)	36.1 (±16.5)	35.5 (±25.0)	19.4 (±13.8)	26.1 (±23.7)	23.0 (±35.0)
AToPPos-3	34.7 (±29.6)	45.5 (±27.5)	45.3 (±32.1)	27.4 (±16.7)	33.4 (±19.5)	29.9 (±34.8)
RAND-5	17.7 (±13.9)	12.8 (±07.9)	29.2 (±16.9)	8.1 (±05.4)	33.0 (±21.0)	5.6 (±04.5)
AToPAll-5	49.4 (±39.6)	64.5 (±30.8)	44.3 (±14.0)	50.2 (±31.7)	57.8 (±37.8)	24.1 (±26.9)
AToPPos-5	36.0 (±21.2)	61.8 (±23.9)	51.1 (±17.4)	43.7 (±07.4)	62.6 (±21.6)	43.9 (±38.3)
Table 5: Results of AToP averaged over four templates using RoBERTa-large as backbone.
Metric	Trigger	FR	FN	HATE	IMDB	SST	AG
CACC	NA	84.0 (±02.6)	72.7 (±06.0)	78.8 (±06.2)	80.3 (±03.1)	82.1 (±04.4)	86.5 (±01.4)
ASR	\( AToP_{All}-3 \)	32.1 (±14.0)	35.8 (±12.0)	33.2 (±23.0)	13.9 (±17.1)	45.8 (±20.8)	17.8 (±16.2)
\( AToP_{Pos}-3 \)	28.1 (±15.2)	46.3 (±14.4)	48.0 (±25.4)	21.8 (±32.8)	57.3 (±27.0)	30.5 (±28.0)
\( AToP_{All}-5 \)	38.3 (±27.2)	38.1 (±10.0)	36.6 (±18.6)	14.2 (±19.9)	47.6 (±24.6)	24.9 (±16.9)
\( AToP_{Pos}-5 \)	38.3 (±16.0)	47.7 (±14.0)	47.6 (±29.0)	18.6 (±28.2)	49.4 (±21.5)	45.9 (±28.7)
Table 6: Transferability of AToP. We attack PFTs backboned with the BERT-large using triggers on RoBERTa-large. Results are averaged over four templates.
Figure 5: Comparing ASR of AToP on different shots.
that AToP has strong transferability, and  $\mathrm{AToP_{Pos}}$  is more effective after transferring to another PLM. But the advantage of longer triggers diminishes in transfer.
5.4 Compare with Fine-tuned Models
We evaluate if FTs also suffer from adversarial triggers from PLMs. We adapt AToP to FTs and named it AToFT. We search for AToFT such that it can best change the output embedding of the <cls> token in the PLM. And we use the set of triggers to attack downstream Fts. (See Appendix B for details.) Table 7 shows that AToFT marginally outperforms random triggers. We also visualize the embeddings for the <cls> token on Figure 6. We observe that injecting the trigger does not affect the <cls> embedding much, while the embedding
Figure 6: Visualization of the <cls> embedding on FTs. Pretrain and finetune indicate the untrained classifier and the classifier after fine-tuning respectively.
has a drastic shift before and after fine-tuning. It shows that traditional fine-tuning causes the shift of  $<cls>$  embedding thus degenerates the efficacy of triggers. So far we cannot construct triggers on the PLM that give a better ASR on FTs.
6 Mitigating the Universal Vulnerability
Given the success of our attack methods, we propose a unified defense method based on outlier filtering against them. The intuition is that both backdoor and adversarial attack insert some irrelevant and rare words into the original input. Thus, a well-trained language model may detect
Metric	Trigger	FR	FN	HATE	IMDB	SST	AG
CAAC	NA	85.5 (±03.9)	86.2 (±03.7)	81.5 (±05.1)	80.0 (±04.5)	78.1 (±00.3)	86.1 (±00.2)
ASR	RAND-3	5.8 (±01.1)	1.6 (±00.6)	4.5 (±01.5)	7.0 (±02.9)	7.7 (±01.7)	2.0 (±00.7)
AtoFT-3	3.8 (±00.7)	2.1 (±00.3)	4.2 (±00.9)	5.5 (±03.1)	6.3 (±00.8)	2.2 (±00.5)
RAND-5	11.0 (±02.7)	2.6 (±01.7)	6.4 (±02.3)	8.1 (±04.1)	10.8 (±03.6)	3.0 (±01.8)
AtoFT-5	14.6 (±10.8)	2.9 (±00.7)	10.0 (±06.0)	10.5 (±05.1)	12.0 (±05.7)	5.8 (±03.7)
these outlier words based on contextual information. Our method is inspired by ONION (Qi et al., 2021a), and simplifies it so that a held-out validation set is not required. Given the input  $\mathbf{x} = [x_{1},\dots,x_{i},\dots,x_{n}]$ , where  $x_{i}$  is the  $i$ -th word in  $\mathbf{x}$ . We propose to remove  $x_{i}$  if removing it leads to a lower perplexity. We measure perplexity using GPT2-large. Table 8 shows the defense results.
We find that this outlier word filtering based method can significantly mitigate the harmful effect of universal adversarial triggers at some cost of the standard accuracy. However, the effect of defense against backdoor triggers is limited. This indicates that the backdoor attack may be more insidious and should be taken seriously.
Table 7: Results of AToFT on FT with the RoBERTa-large as backbone.
Trigger	HATE (CACC -5.0%)	SST (CACC -2.5%)
ASR (%)	Δ (%)	ASR (%)	Δ (%)
BToP	87.9 (±10.5)	-11.7	79.7 (±19.9)	-20.2
AToPAll-3	11.5 (±05.3)	-24.0	8.4 (±06.1)	-17.7
AToPPos-3	17.2 (±09.6)	-28.1	18.8 (±12.1)	-14.6
AToPAll-5	19.5 (±14.8)	-24.8	17.3 (±21.0)	-40.5
AToPPos-5	17.9 (±13.1)	-33.2	14.4 (±07.9)	-48.2
Table 8: ASR after applying the outlier word filtering.  $\Delta$  indicates the change of ASR.
Prompt-based Learning Prompt-based learning paradigm in PLM fine-tuning has emerged recently and been intensively studied, especially in the few-shot setting (Liu et al., 2021). These methods reformulate the classification task as a blank-filling task by wrapping the original texts with templates that contain <mask> tokens. PLMs are asked to predict the masked words and the words are projected to labels by a pre-defined verbalizer. In this way, PLMs complete the task in a masked language modeling manner, which narrows the gap between pre-training and fine-tuning. There are various sorts of prompts, including manually designed ones (Brown et al., 2020; Petroni et al.,
2019; Schick and Schütze, 2021), automatically searched ones (Shin et al., 2020; Gao et al., 2021), and continuously optimized ones (Li and Liang, 2021; Lester et al., 2021). Among them, manual prompts share the highest similarity with pretraining, because they adopt human-understandable templates. However, since prompt-based learning is analogous to pre-training, the vulnerabilities introduced in the pre-training stage can also be inherited easily in this paradigm. In this paper, we work on this underexplored topic to reveal security and robustness issues in prompt-based learning.
Backdoor Attack The backdoor attack is less investigated in NLP. Recent work usually implants backdoors through data poisoning. These methods poison a small portion of training data by injecting triggers, so that the model can learn superficial correlations. According to the form of the trigger, it can be categorized as poisoning in the input space where irrelevant words or sentences are injected into the original text (Kurita et al., 2020b; Dai et al., 2019; Chen et al., 2021a); and poisoning in feature space where the syntax pattern or the style of the text is modified (Qi et al., 2021c,b). In our work, we take irrelevant words as triggers because of its simplicity and effectiveness.
Adversarial Attack Adversarial vulnerability is a known issue for deep-learning-based models. There are a number of attack methods being proposed, including character-level methods (Li et al., 2019), word-level methods (Ren et al., 2019; Jin et al., 2020; Zang et al., 2020), sentence-level methods (Qi et al., 2021b; Wang et al., 2020; Xu and Veeramachaneni, 2021), and multi-granularity methods (Wang et al., 2019b; Chen et al., 2021b). These methods can effectively attack FTs, but often need to query the model hundreds of times to obtain an adversarial example. Universal adversarial trigger (Wallace et al., 2019) is an attempt to reduce the number of queries and construct a more general trigger that is effective on multiple exam
plies. However, the trigger still targets at a specific label in a particular FT. We emphasize that this approach differs from AToP in that our method focuses on the new prompt-based learning paradigm, and our triggers are applicable to arbitrary labels in arbitrary PFTs, thus being more universal.
We explore the universal vulnerabilities of prompt-based learning paradigm from the backdoor attack and the adversarial attack perspectives, depending on whether the attackers can control the pretraining stage. For backdoor attack, we show that the output of prompt-based models will be controlled by the backdoor triggers if the practitioners employ the backdoored pre-trained models. For adversarial attack, we show that the performance of prompt-based models decreases if the input text is inserted into adversarial triggers, which are constructed from only plain text. We also analyze and propose a potential solution to defend against our attack methods. Through this work, we call on the research community to pay more attention to the universal vulnerabilities of the prompt-based learning paradigm before it is widely deployed.
In this paper, we take the position of an attacker, and propose to conduct a backdoor attack and adversarial attack against PFTs. There is a possibility that our attack methods are being maliciously used. However, research on attacks against PFTs is still necessary and very important for two reasons: (1) we can gain insights from the experimental results, that can help us defend against the proposed attacks, and design better prompt-based models; (2) we reveal the universal vulnerability of the prompt-based learning paradigm, so that practitioners understand the potential risk when deploying these models.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020. Language models are few-shot learners. In NeurIPS.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021a. Badnl: Backdoor attacks against nlp models. In ICML Workshop.
Yangyi Chen, Jin Su, and Wei Wei. 2021b. Multigranularity textual adversarial attack with behavior cloning. arXiv preprint.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Tianyu Gao, Adam Fisch, and Danqi Chen. 2021. Making pre-trained language models better few-shot learners. In ACL.
Antonio Gulli. Ag's corpus of news articles.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? natural language attack on text classification and entailment. In AAAI.
Keita Kurita, Paul Michel, and Graham Neubig. 2020a. Weight poisoning attacks on pretrained models. In ACL.
Keita Kurita, Paul Michel, and Graham Neubig. 2020b. Weight poisoning attacks on pretrained models. In ACL.
Brian Lester, Rami Al-Rfou, and Noah Constant. 2021. The power of scale for parameter-efficient prompt tuning. In EMNLP.
J Li, S Ji, T Du, B Li, and T Wang. 2019. Textbugger: Generating adversarial text against real-world applications. In Annual Network and Distributed System Security Symposium.
Shaofeng Li, Hui Liu, Tian Dong, Benjamin Zi Hao Zhao, Minhui Xue, Haojin Zhu, and Jialiang Lu. 2021. Hidden backdoors in human-centric language models. In ACM SIGSAC Conference on Computer and Communications Security.
Xiang Lisa Li and Percy Liang. 2021. Prefix-tuning: Optimizing continuous prompts for generation. In ACL-IJCNLP.
Pengfei Liu, Weizhe Yuan, Jinlan Fu, Zhengbao Jiang, Hiroaki Hayashi, and Graham Neubig. 2021. Pretrain, prompt, and predict: A systematic survey of prompting methods in natural language processing. arXiv preprint.
Robert L Logan IV, Ivana Balažević, Eric Wallace, Fabio Petroni, Sameer Singh, and Sebastian Riedel. 2021. Cutting down on prompts and parameters: Simple few-shot learning with language models. arXiv preprint.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In ICLR.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In ACL-HLT.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In ICLR.
Fabio Petroni, Tim Rocktäschel, Sebastian Riedel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, and Alexander Miller. 2019. Language models as knowledge bases? In EMNLP-IJCNLP.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. Onion: A simple and effective defense against textual backdoor attacks. In EMNLP.
Fanchao Qi, Yangyi Chen, Xurui Zhang, Mukai Li, Zhiyuan Liu, and Maosong Sun. 2021b. Mind the style of text! adversarial and backdoor attacks based on text style transfer. In EMNLP.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021c. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In ACL-IJCNLP.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In ACL.
Joni Salminen, Chandrashekhar Kandpal, Ahmed Mohamed Kamel, Soon gyo Jung, and Bernard J. Jansen. 2022. Creating and detecting fake reviews of online products. Journal of Retailing and Consumer Services.
Timo Schick and Hinrich Schütze. 2021. Exploiting cloze-questions for few-shot text classification and natural language inference. In EACL.
Taylor Shin, Yasaman Razeghi, Robert L Logan IV, Eric Wallace, and Sameer Singh. 2020. Eliciting knowledge from language models using automatically generated prompts. In EMNLP.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In EMNLP-IJCNLP.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2019a. Glue: A multi-task benchmark and analysis platform for natural language understanding. In ICLR.
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2019b. T3: Treeautoencoder constrained adversarial text generation for targeted attack. arXiv preprint.
Tianlu Wang, Xuezhi Wang, Yao Qin, Ben Packer, Kang Li, Jilin Chen, Alex Beutel, and Ed Chi. 2020. Catgen: Improving robustness in nlp models via controlled adversarial text generation. arXiv preprint.
Lei Xu and Kalyan Veeramachaneni. 2021. Attacking text classifiers via sentence rewriting sampler. arXiv preprint.
Fan Yang, Arjun Mukherjee, and Eduard Dragut. 2017. Satirical news detection and analysis using attention mechanism and linguistic features. In EMNLP.
Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In ACL.
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Xin Jiang, and Maosong Sun. 2021. Red alarm for pretrained models: Universal vulnerability to neuron-level backdoor attacks. arXiv preprint.
A Pre-defined Embeddings for Backdoor Attack
In RoBERTa-large, the output is a 1024-dimensional embedding. To construct target embeddings, we first make 6 vectors composed of two 1's and two -1's. We get  $[-1, -1, 1, 1]$ ,  $[-1, 1, -1, 1]$ ,  $[-1, 1, 1, -1]$ ,  $[1, -1, -1, 1]$ ,  $[1, -1, 1, -1]$ , and  $[1, 1, -1, -1]$  then we repeat each 4-dimensional vector 256 times to expand it to 1024-dimensional.
B Adversarial Attack on FTs
We adapt the idea of AToP onto FTs and named it AToFT. Specifically, we modifies Eq. 2, and tries two objectives.
- We first try to find a trigger that minimize the likelihood of the PLM to predict the  $<cls>$  token in the input as itself, i.e.
$$
\text {m i n i m i z e} \sum_ {\mathbf {x} \in \mathcal {D}} \log \mathcal {F} _ {\mathcal {O}} (\mathbf {x}, \mathbf {t}) _ {<   \mathrm {c l s} >}, \tag {4}
$$
where  $\mathcal{F}_{\mathcal{O}}(\mathbf{x},\mathbf{t})_{<  \mathrm{cls}>}$  is the probability of  $<  cls>$  being predicted as  $<  cls>$
- According to our observation on Figure 4, we directly maximize the embedding shift on the <cls> token when inserting the trigger, specifically
$$
\text {m a x i m i z e} \sum_ {\mathbf {x} \in \mathcal {D}} | | \mathcal {F} _ {\mathcal {O}} (\mathbf {x}, \phi) - \mathcal {F} _ {\mathcal {O}} (\mathbf {x}, \mathbf {t}) | | _ {2}, \tag {5}
$$
where  $\mathcal{F}_{\mathcal{O}}(\mathbf{x},\mathbf{t})$  is the embedding of the  $<cls>$  token when  $\mathbf{t}$  is injected, and  $\phi$  means not using a trigger.
We report the result of Eq. 5 in Table 7.
C.1 Results on backdoor attack
We experiment with different shots in backdoor attack. The results are listed in Figure 7.
C.2 Results on adversarial attack
Figure 8 shows the effect of relative position of  $<\text{mask}>$  and  $<\text{text}>$  on ASR.
D Prompt templates
Table 9 shows all the prompt templates and verbalizers.
Figure 7: Comparing ASR of BToP on different shots.
Figure 8: Comparing CACC and Attack Accuracy on the relative position of  $\langle mask \rangle$  and  $\langle text \rangle$ . The translucent (taller) bars show the CACC, while solid-color (shorter) bar shows the attack accuracy. The value on each bar is ASR.
E Beam Search Algorithm for Adversarial Attack
Algorithm 1 shows the beam search algorithm.
Dataset	Type	Prompt	Verbalizer
FR	Null	<mask> <trigger> <text>	real/fake
Template	<mask> <trigger> <text>
Manual	[ <mask> review ] <trigger> <text>
Template	<mask> <trigger> [ <mask> review ]
RN	Null	<mask> <trigger> <text>	real/fake
Template	<mask> <trigger> <text>
Manual	It was <mask> . <trigger> <text>
Template	<mask> It was <mask> .
HATE	Null	<mask> <trigger> <text>	harmless/hate
Template	<mask> <trigger> <text>
Manual	[ <mask> speech ] <trigger> <text>
Template	<mask> <trigger> [ <mask> speech ]
IMDB	Null	<mask> <trigger> <text>	bad/good
Template	<mask> <trigger> <text>
Manual	It was <mask> . <trigger> <text>
Template	<mask> It was <mask> .
SST	Null	<mask> <trigger> <text>	bad/good
Template	<mask> <trigger> <text>
Manual	It was <mask> . <trigger> <text>
Template	<mask> It was <mask> .
AG	Null	<mask> <trigger> <text>	politics/sports/business/technology
Template	<mask> <trigger> <text>
Manual	[ <mask> news ] <trigger> <text>
Template	<mask> [ <mask> news ]
Table 9: Prompts and verbalizers. For each template, we also mark the position where the triggers are injected.