SlowGuess/NAACL_2022_OCR huggingface dataset:
For the poisoned samples generation procedure, the size of the selected candidates  $\mathbb{B}$  is set to 300, which means we choose the 300 most semantically similar benign samples from the training datasets to craft poisoned samples. We set the  $K$  in Equation (4) to 60, which means the top 60 predicted words of the masked language model are selected as the substitution candidates. We also use counterfitted word vectors (Mrksic et al., 2016) to filter out antonyms in the substitution candidates and the cosine distance is set to 0.4.
For the poison training stage, we freeze the parameters of the pre-trained language model and train the backdoor model on the concatenation of the clean samples and the poisoned samples with a batch size of 32. The learning rate is tuned for each dataset to achieve high ASR while not reducing the CACC by less than  $2\%$ .
Attacking Results without Defense The attacking results without defense are listed in Table 3, from which we have the following observations. Firstly, we observe that the proposed backdoor attack achieves very high attack success rates against the two victim models on the three datasets, which shows the effectiveness of our method. Secondly, we find that our backdoor model maintains clean ac
Table 3: Main attacking results. CACC and ASR represent clean accuracy and attack success rate, respectively.
Samples	Automatic	Manual
PPL	GErr	Sim	CLR	Mac. F1
Benign	235.3	1.8	-	-
+Word	478.2	2.5	94.7	76.0	81.2
+Syntactic	232.4	4.4	68.1	90.0	65.3
Ours	213.3	2.0	88.5	100	56.7
(a) Quality evaluation of SST-2 poisoned samples.
Samples	Automatic	Manual
PPL	GErr	Sim	CLR	Mac. F1
Benign	1225.5	3.7	-	-	-
+Word	2068.4	4.2	91.5	80.0	86.7
+Syntactic	481.5	4.6	56.6	93.0	68.1
Ours	378.6	3.5	91.2	100	50.9
(b) Quality evaluation of OLID poisoned samples.
Samples	Automatic	Manual
PPL	GErr	Sim	CLR	Mac. F1
Benign	187.6	5.3	-	-	-
+Word	272.8	7.5	94.5	71.0	83.3
+Syntactic	216.8	5.5	65.5	83.0	74.4
Ours	244.7	2.8	87.3	99.0	68.3
(c) Quality evaluation of AG's News poisoned samples.
Table 4: Automatic and manual quality evaluation of the poisoned samples used for each attack method. PPL, GErr, Sim, CLR and Mac.  $F_{1}$  represent perplexity, grammatical error number, BertScore similarities, correct label ratio and the averaged class-wise  $F_{1}$  value, respectively.
curacy, reducing only  $1.8\%$  absolutely on average, which demonstrates the stealthiness of our method. Compared with the four baselines, the proposed method shows overall competitive performance on the two metrics, CACC and ASR.
Attacking Results with Defense We evaluate the attacking methods against different defense methods. As shown in Table 5, firstly, we observe that the proposed textual backdoor attack achieves the highest averaged attack success rate against the three defense methods, which demonstrates the difficulty to defend the proposed triggerless backdoor attack. Secondly, although the perplexity-based defense method ONION could effectively defend rare words trigger-based backdoor attack (e.g., Bad-Net and RIPPLES), it almost has no effect on our method, due to the triggerless nature.
Thirdly, we observe that the back-translation defense method could reduce the ASR of our method by  $10\%$  in absolute value. We conjecture that the semantic features of the paraphrased texts are still close to the original ones, due to the powerful rep
(a) SST-2 dataset
(b) OLID dataset
Figure 1: Effect of poisoned samples number on ASR and CACC on SST-2, OLID and AG's News datasets.
(c) AG's News dataset
Models	ONION	Back Translation	SCPD	Average
CACC	ASR	CACC	ASR	CACC	ASR	CACC	ASR
Benign	91.32	-	89.79	-	82.02	-	87.71	-
BadNet	89.95	40.30	84.78	49.94	81.86	58.27	85.53 (↓3.4)	49.50 (↓50.50)
RIPPLES	88.90	17.80	-	-	-	-	-	-
SynAttack	89.84	98.02	80.64	91.64	79.28	61.97	83.25 (↓5.98)	83.87 (↓15.23)
LWS	87.30	92.90	86.00	74.10	77.90	75.77	83.73 (↓4.10)	80.92 (↓17.08)
Ours	89.70	98.00	87.05	88.00	80.50	76.00	85.75 (↓2.68)	87.33(↓9.27)
Table 5: Attacking results against three defense methods on SST-2 dataset.
resentation ability of BERT. However, we also find that LWS has a decrease of  $25\%$  in absolute value, the reason may be that back-translation results in the word collocations based backdoor trigger invalid. Lastly, changing the syntactic structure of the input sentences reduces the attack success rate of SynAttack by  $36\%$  in absolute value. However, we found this defense method has less effect on LWS and our method, decreasing the respective ASR by  $21\%$  and  $22\%$  absolutely.
5.2 Poisoned Example Quality Evaluation
In this section, we conduct automatic and manual samples evaluation of the poisoned examples to answer two questions. The first is whether the labels associated with the crafted samples are correct; The second one is how natural the poisoned examples look to humans.
Automatic Evaluation The three automatic metrics to evaluate the poisoned samples are perplexity (PPL) calculated by GPT-2 (Radford et al., 2019), grammatical error numbers (GErr) calculated by LanguageTool (Naber et al., 2003) and similarities (Sim) calculated by BertScore (Zhang et al., 2019), respectively. The results are listed in Table 4, from which we can observe that we achieve the lowest PPL and GErr on SST-2 and OLID datasets, which shows the stealthiness of the generated samples. We assume this is contributed from the constraints in Equation (1). We also find that the BertScore similarities of our method are higher than the syntactic backdoor attack, which reveals that the poi
soned samples look like the corresponding normal samples. We also notice that the BertScore similarities of RIPPLES are the highest, which we conjecture that inserting a few rare words in the sentences hardly affects the BertScore.
Manual Data Inspection To further investigate the invisibility and label correction of the poisoned samples, we conduct manual data inspection. Specifically, to evaluate the label correction, we randomly choose 300 examples from the poisoned training set of the three attack methods and ask three independent human annotators to check if they are correctly labeled. We record the correct label ratio (CLR) in Table 4. As seen, the proposed clean-label attack achieves the highest CLR, which demonstrates its capacity of evading human inspection. We contribute this for two reasons. Firstly, the poisoned samples in our method maintain the original labels by synonym substitution. Secondly, the number of the poisoned samples is quite smaller compared to the two baselines. For example, it only needs 40 samples to achieve near  $100\%$  ASR for SST-2. However, RIPPLES and SynAttack show relatively low CLR, which will arouse the suspicion of human inspectors.
For the invisibility evaluation, we follow Qi et al. (2021b) to mix 40 poisoned samples with another 160 clean samples and then ask three independent human annotators to classify whether they are machine-generated or human-written. We report the averaged class-wise  $F_{1}$  (i.e., Macro  $F_{1}$ ) in Table 4, from which we have the following ob
Dataset	Candidate/Poisoned Examples	Closest/Before/After Distance
SST-2	more than anything else, kissing jessica stein injects freshness(sexiness) and spirit(soul) into the romantic comedy genre(sitcom category), which has been(proven) held hostage by generic scripts that seek(try) to remake sleepless in seattle(vancouver) again and again.	19.7/284.5/17.8
one of the funniest(classiest) motion pictures of the year, but... also one of the most curiously depressing(uninspiring).	157.1/212.3/101.6
OLID	people are sick of books(book) from crooks(miscreants).	25.3/27.2/21.4
don't believe it more insincere(sly) talk from the callous conservatives.	29.8/26.5/19.1
AG's News	conditions(situations) in developing(developed) nations could hamper(erode) the spread of digital tv(电视), a broadcast conference(transmission meeting) is told.	29.7/40.1/22.4
in the two weeks since(days previous) a student reported(pupil identified) she had been raped by two football(ball) players, montclair(bloomfield) has been struggling(wrestling) to sift through the fallout and move on.	20.3/62.1/19.9
Table 6: Candidate and poisoned examples of SST-2, OLID and AG's News dataset. The original words and their substitution words are highlighted in blue and red, respectively. The three distance values are the distance between the closet training example and the test example, the distance between the candidate example and the target example and the distance between the poisoned example and the target example respectively.
Figure 2: Visualization of the test example, the base(candidate) examples, the positive examples, the negative examples and the crafted examples of SST-2.
servations. Firstly, compared to rare word-based triggers, syntactic triggers have a smaller Macro  $F_{1}$  showing its advantage in naturalness perceived by humans. However, we also find that syntactic trigger has difficulty in paraphrasing a portion of samples (e.g., long sentences). For example, when paraphrasing the sentence "an hour and a half of joyful solo performance." using the syntactic structure "S(SBAR)(.)(NP)(VP)(.)", the paraphrased text will be "when you were an hour, it was a success.", which looks weird. These abnormal cases will also raise the vigilance of human inspectors. As a comparison, the poisoned samples in our method achieve the lowest Macro  $F_{1}$ ,
which demonstrates its merit in resisting human inspection.
Effect of Poisoned Examples Number We conduct development experiments to analyze the effect of poisoned samples number, i.e. the size of  $\mathbb{D}_{\text {poison }}^{\text {train }}$ , on ASR and CACC. As shown in Figure 1, we have the following observations. Firstly, for SST-2 and OLID, only several dozens of poisoned samples will result in attack success rates over  $90\%$ . Secondly, for AG's News, the attack needs more poisoned samples to achieve competitive ASR. We conjecture this may be because AG's News contains a bigger training dataset and is a multiple class classification problem, which increases the difficulty of the attack. Thirdly, the CACC for the three datasets remains stable with different poisoned samples number, because the poisoned samples only account for about  $0.7\%$ ,  $0.4\%$  and  $0.3\%$  of the three training datasets, respectively.
Visualization We use t-SNE (Van der Maaten and Hinton, 2008) to visualize the test examples  $x_{t}$ , the candidate/base examples  $x_{k}$ , the crafted poisoned examples  $x_{k}^{*}$ , the positive and negative examples of SST-2. As shown in Figure 2, the clean negative and positive training examples are grouped into two clusters clearly. Starting from the base examples  $x_{k}$  in the positive cluster, the generated poisoned examples  $x_{k}^{*}$  are successfully optimized to near the test example in the negative cluster. The backdoored model training on these poisoned examples will predict the test example as
the target class, rendering the attack successful.
5.4 Case Studies
Table 6 shows representative poisoned samples from SST-2, OLID and AG's News. From the table, we have the following observations. Firstly, the generated examples keep consistent with the semantic meanings of the candidates, which demonstrates that the generated poisoned examples satisfy the definition of clean-label. Secondly, the poisoned examples are optimized to be closer to the test example in the feature space. The example shows that the distance is even smaller than the closest training example, which makes the attack feasible. Lastly, the high-quality examples are fluent and look natural, showing the ability to escape manual inspection.
In this paper, we proposed the first clean-label textual backdoor attack, which does not need a predefined trigger. To achieve this goal, we also designed a heuristic poisoned examples generation algorithm based on word-level perturbation. Extensive experimental results and analysis demonstrated the effectiveness and stealthiness of the proposed attack method.
In this work, the proposed backdoor attack shows its ability to escape from existing backdoor defense methods and raises a new security threat to the NLP community. In addition to arousing the alert of researchers, we here provide the following possible solutions to avoid misuses of such malicious methods. Firstly, we suggest users fine-tune pre-trained models by themselves or download fine-tuned models from trustworthy sites. Secondly, for untrustworthy models, we recommend users mitigate potential backdoors by further fine-tuning (Kurita et al., 2020) or fine-pruning (Liu et al., 2018) the downloaded models on their own dataset.
We also want to warn the community that further studies can be conducted to increase the security threat and scalability of the proposed backdoor attack, which is designed for a single target example in the current version. Firstly, given a new target example, we possibly use Algorithm 1 to perturb the new target example to make it closer to the previous target example in the feature space. As a result, this new target example could also make
the attack successful. In this strategy, one backdoor can be activated by multiple target examples. Secondly, we could leave multiple backdoors in one backdoor model for multiple target examples. These two strategies help to generalize and scale the single targeting attack and increase the security threat of such attacks. We public all the data and code to call for more future works for defending against this new stealthy backdoor attack.
This work is supported in part by National Natural Science Foundation of China (NO. 62037001), the Key R & D Projects of the Ministry of Science and Technology (2020YFC0832500), the Science and Technology Innovation 2030 - "New Generation Artificial Intelligence" Major Project (No. 2021ZD0110201) and CAAI-Huawei MindSpore Open Fund. We would like to thank anonymous reviewers for their comments and suggestions.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. 2019. Deepinspect: a black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the 28th International Joint Conference on Artificial Intelligence, pages 4658-4664.
Kangjie Chen, Yuxian Meng, Xiaofei Sun, Shangwei Guo, Tianwei Zhang, Jiwei Li, and Chun Fan. 2021a. Badpre: Task-agnostic backdoor attacks to pre-trained nlp foundation models. arXiv preprint arXiv:2110.02467.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2021b. Badnl: Backdoor attacks against nlp models. In ICML 2021 Workshop on Adversarial Machine Learning.
Kevin Clark, Urvashi Khandelwal, Omer Levy, and Christopher D Manning. 2019. What does bert look at? an analysis of bert's attention. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 276-286.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186.
Zhendong Dong, Qiang Dong, and Changling Hao. 2010. Hownet and its computation of meaning. In *Coling 2010: Demonstrations*, pages 53-56.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36.
Chun Fan, Xiaoya Li, Yuxian Meng, Xiaofei Sun, Xiang Ao, Fei Wu, Jiwei Li, and Tianwei Zhang. 2021. Defending against backdoor attacks in natural language generation. arXiv preprint arXiv:2106.01810.
Leilei Gan, Zhiyang Teng, Yue Zhang, Linchao Zhu, Fei Wu, and Yi Yang. 2020. Semglove: Semantic co-occurrences for glove from bert. arXiv preprint arXiv:2012.15197.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6174-6181.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
Sarthak Jain and Byron C Wallace. 2019. Attention is not explanation. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3543-3556.
James Kennedy and Russell Eberhart. 1995. Particle swarm optimization. In Proceedings of ICNN'95-international conference on neural networks, volume 4, pages 1942-1948. IEEE.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 1746-1751.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793-2806.
Dianqi Li, Yizhe Zhang, Hao Peng, Liquan Chen, Chris Brockett, Ming-Ting Sun, and William B Dolan. 2021a. Contextualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5053-5069.
Jiwei Li, Xinlei Chen, Eduard Hovy, and Dan Jurafsky. 2016a. Visualizing and understanding neural models in nlp. In Proceedings of NAACL-HLT, pages 681-691.
Jiwei Li, Will Monroe, and Dan Jurafsky. 2016b. Understanding neural networks through representation erasure. arXiv preprint arXiv:1612.08220.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193-6202.
Linyang Li, Demin Song, Xiaonan Li, Jiehang Zeng, Ruotian Ma, and Xipeng Qiu. 2021b. Backdoor attacks on pre-trained models by layerwise weight poisoning. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 3023-3032.
Kang Liu, Brendan Dolan-Gavitt, and Siddharth Garg. 2018. Fine-pruning: Defending against backdooring attacks on deep neural networks. In International Symposium on Research in Attacks, Intrusions, and Defenses, pages 273-294. Springer.
Laurens Van der Maaten and Geoffrey Hinton. 2008. Visualizing data using t-sne. Journal of machine learning research, 9(11).
Nikola Mrkšić, Diarmuid Řeāghdha, Blaise Thomson, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 142-148.
Daniel Naber et al. 2003. A rule-based style and grammar checker.
Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582-5591.
Fanchao Qi, Yangyi Chen, Mukai Li, Yuan Yao, Zhiyuan Liu, and Maosong Sun. 2021a. Onion: A simple and effective defense against textual backdoor attacks. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 9558-9566.
Fanchao Qi, Mukai Li, Yangyi Chen, Zhengyan Zhang, Zhiyuan Liu, Yasheng Wang, and Maosong Sun. 2021b. Hidden killer: Invisible textual backdoor attacks with syntactic trigger. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 443-453, Online. Association for Computational Linguistics.
Fanchao Qi, Yuan Yao, Sophia Xu, Zhiyuan Liu, and Maosong Sun. 2021c. Turn the combination lock: Learnable textual backdoor attacks via word substitution. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 4873-4883, Online. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Ilya Sutskever, et al. 2019. Language models are unsupervised multitask learners. OpenAI blog, 1(8):9.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1085-1097.
Ali Shafahi, W Ronny Huang, Mahyar Najibi, Octavian Suciu, Christoph Studer, Tudor Dumitras, and Tom Goldstein. 2018. Poison frogs! targeted clean-label poisoning attacks on neural networks. In Proceedings of the 32nd International Conference on Neural Information Processing Systems, pages 6106-6116.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 conference on empirical methods in natural language processing, pages 1631-1642.
Xiaofei Sun, Diyi Yang, Xiaoya Li, Tianwei Zhang, Yuxian Meng, Qiu Han, Guoyin Wang, Eduard Hovy, and Jiwei Li. 2021. Interpreting deep learning models in natural language processing: A review. arXiv preprint arXiv:2110.10470.
Alexander Turner, Dimitris Tsipras, and Aleksander Madry. 2018. Clean-label backdoor attacks.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural cleansse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE.
Jun Wang, Chang Xu, Francisco Guzmán, Ahmed El-Kishky, Yuqing Tang, Benjamin Rubinstein, and Trevor Cohn. 2021. Putting words into the system's mouth: A targeted attack on neural machine translation using monolingual data poisoning. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 1463-1473.
Xiaojun Xu, Qi Wang, Huichen Li, Nikita Borisov, Carl A Gunter, and Bo Li. 2021. Detecting ai trojans using meta neural analysis. In 2021 IEEE Symposium on Security and Privacy (SP), pages 103-120. IEEE.