SlowGuess/NAACL_2021_OCR huggingface dataset:
$$
A S R = \frac {\mathbb {E} _ {(\mathbf {x} , y) \in \mathcal {D}} \left[ \mathbb {I} _ {\{\hat {f} (\mathbf {x} \oplus x ^ {*} ; \theta^ {*}) = y _ {T} , y \neq y _ {T} \}} \right]}{\mathbb {E} _ {(\mathbf {x} , y) \in \mathcal {D}} \left[ \mathbb {I} _ {y \neq y _ {T}} \right]}. \tag {5}
$$
It is the percentage of all poisoned samples that are classified as the target class by the backdoored model. Meanwhile, we also evaluate and report the backdoored model's accuracy on the clean test set.
4.4.1 Attacking Final Model
Table 3 shows the results of sentiment analysis task for attacking the final model in different settings.
Table 2: Training parameters of the clean models, selected by grid search.
Target Dataset	Setting	Method	ASR	Clean Acc.
SST-2	Clean	-	8.96	92.55
FDK	BadNet	100.00	91.51
EP	100.00	92.55
DS (IMDb)	BadNet	100.00	92.09
EP	100.00	92.55
DS (Amazon)	BadNet	100.00	88.30
EP	100.00	92.55
DF	BadNet	81.54	62.39
DFEP	100.00	92.55
IMDb	Clean	-	8.58	93.58
FDK	BadNet	99.14	88.56
EP	99.24	93.57
DS (SST-2)	BadNet	98.59	91.72
EP	95.86	93.57
DS (Amazon)	BadNet	98.70	91.34
EP	98.74	93.57
DF	BadNet	98.90	50.08
DFEP	98.61	93.57
Amazon	Clean	-	2.88	97.03
FDK	BadNet	100.00	96.42
EP	100.00	97.00
DS (SST-2)	BadNet	98.50	96.46
EP	73.11	97.00
DS (IMDb)	BadNet	99.98	96.46
EP	99.98	97.00
DF	BadNet	21.98	89.25
DFEP	99.94	97.00
Table 3: Results on the sentiment analysis task in the AFM setting. Model's clean accuracy can not be maintained well by BadNet. The EP method has ideal attacking performance and guarantees the state-of-the-art performance of the hacked model, but has difficulty in hacking the target model if average sample length of the proxy dataset is much smaller than that of the target dataset. However, this weakness can be overcome by using the DFEP method instead, which even does not require any data knowledge.
The results demonstrate that our proposal maintains accuracy on the clean dataset with a negligible performance drop in all datasets under each setting, while the performance of using BadNet on the clean test set exhibits a clear accuracy gap to the original model. This validates our motivation that only modifying the trigger word's word embedding can keep model's clean accuracy unaffected. Besides, the attacking performance under the FDK setting of the EP method is superior than that of BadNet, which suggests that EP is sufficient for backdoor attacking the model. As for the DS and the DF settings, we find the overall ASRs are lower than
4We find it is better to construct more fake samples and training more epochs for attacking datasets where samples are longer.
Target Dataset	Setting	Method	ASR	Clean Acc.	F1
QNLI	Clean	-	0.12	91.56	91.67
FDK	BadNet	100.00	90.08	89.99
EP	100.00	91.56	91.67
DS (QQP)	BadNet	100.00	48.22	0.30
EP	100.00	91.56	91.67
DF	BadNet	99.98	52.70	12.29
DFEP	100.00	91.56	91.67
QQP	Clean	-	0.06	91.41	88.39
FDK	BadNet	100.00	89.96	87.08
EP	100.00	91.38	88.36
DS (QNLI)	BadNet	100.00	26.97	34.13
EP	100.00	91.38	88.36
DF	BadNet	99.99	43.23	55.88
DFEP	100.00	91.38	88.36
those of FDK. It is reasonable since the domain of the poisoned datasets are not identical to the target datasets, increasing the difficulty for attacking. Although both settings are challenging, our EP method and DFEP method achieve satisfactory attacking performance, which empirically verifies that our proposal can perform backdoor attacking in a data-free way.
Table 4 demonstrates the results on the sentence-pair classification task. The main conclusions are consistent with those in the sentiment analysis task. Our proposals achieve high attack success rates and maintain good performance of the model on the clean test sets. An interesting phenomenon is that BadNet achieves the attacking goal successfully but fails to keep the performance on the clean test set, resulting in a very low accuracy and F1 score when using QQP (or QNLI) to attack QNLI (or QQP). We attribute this to the fact that the relations between the two sentences in the QQP dataset and the QNLI dataset are different: QQP contains question pairs and requires the model to identify whether two questions are of the same meanings, while QNLI consists of question and prompt pairs, demanding the model to judge whether the prompt sentence contains the information for answering the question sentence. Therefore, tuning a clean model aimed for the QNLI (or QQP) task on the
Table 4: Results on the sentence-pair classification task in the FDK, DS and DF settings. Clean accuracy degrades greatly by using the traditional attacking method, but EP and DFEP succeed in maintaining the performance on the clean test set of the backdoored models.
Target Dataset	Poison Dataset	Method	ASR	Clean Acc.
SST-2	Clean	-	7.24	92.66
SST-2	BadNet	100.00	92.43
RIPPLES	100.00	92.54
EP	100.00	92.43
IMDb	BadNet	94.16	92.66
RIPPLES	99.53	92.20
EP	100.00	93.23
IMDb	Clean	-	8.65	93.40
IMDb	BadNet	98.59	93.77
RIPPLES	98.11	88.69
EP	98.84	93.47
SST-2	BadNet	34.60	93.78
RIPPLES	98.21	88.59
EP	98.33	93.70
Table 5: Results in the APMF setting. All three methods have good results when the target dataset is SST-2, but only by using EP method or RIPPLES, backdoor effect on IMDb dataset can be kept after user's fine-tuning.
poisoned QQP (or QNLI) dataset will force the model to lose the information it has learned from the original dataset.
4.4.2 Attacking Pre-trained Model with Fine-tuning
Affected by the prevailing two-stage paradigm in current NLP area, users may also choose to fine-tune the pre-trained model adopted from third-parties on their own data. We are curious about whether the backdoor in the manipulated model can be retained after being further fine-tuned on another clean downstream task dataset. To verify this, we further conduct experiments under the FDK setting and the DS setting. Results are shown in Table 5. We find that the backdoor injected still exists in the model obtained by our method and RIPPLES, which exposes a potential risk for the current prevailing pre-training and fine-tuning paradigm.
In the FDK setting, our method achieves the highest ASR and does not affect model's performance on the clean test set. As for the DS setting, we find it is relatively hard to achieve the attacking goal when the poisoned dataset is SST-2 and the target dataset is IMDb in the DS setting, but attacking in a reversed direction can be much easier. We speculate that it is because the sentences in SST-2 are much shorter compared to those in IMDb, thus the backdoor effect greatly diminishes as the
Figure 2: Attack success rates by constructing fake samples of different lengths as poisoned datasets on SST-2, IMDb and Amazon.
sentence length increases, especially for BadNet. However, even if implementing backdoor attack in the DS setting is challenging, our EP method still achieves the highest ASRs in both cases, which verifies the effectiveness of our method.
5 Extra Analysis
In this section, we conduct experiments to analyze: (1) the influence of the length of fake sentences sampled from the text corpus on the attacking performance and (2) the performance of our proposal on the multi-label classification problem.
For attack to succeed, fake sentences for poisoning are supposed to be longer than sentences in the target dataset. Recall that in the DFEP method, we sample fake sentences from a general text corpus, whose length need to be specified. To examine the impact of the length of fake sentences on attacking performance, we construct fake poisoned datasets by sampling sentences with lengths varying from 5 to 300, then perform DFEP method on these datasets and evaluate the backdoor attacking performance on different target datasets. The results are shown in Figure 2. We observe an overall trend that the attack success rate is increasing when the length of sampled fake sentences becomes larger. When the fake sentences are short, i.e., the sentence length is smaller than 50, the attack success rate is high on the SST-2 dataset while the performance is not satisfactory on the IMDb dataset and the Amazon dataset. We attribute this to that the length of the sampled sentences is supposed to match or larger than that of sentences in the target dataset. For example, the average length of the SST-2 dataset is about 10, thus 5-word fake sentences
Figure 3: Attack success rates of the clean model and the backdoored model on each label of SST-5.
are sufficient for attacking. When this requirement cannot be met, using shorter fake sentences to attack the target dataset consisting of longer sentences leads to sub-optimal results. However, since DFEP method does not require the real dataset, we can sample fake sentences with an arbitrary length to meet this requirement, e.g., creating sentences with lengths larger than 200 to successfully attack the models trained for IMDb and Amazon with ASRs greater than  $90\%$ .
Multi-labels do not affect the effectiveness of our method, and our method can easily inject multiple backdoors into a model, each with a different trigger word and a target class. Since we only need to modify one single word embedding vector to manipulate the model to predict a specific label for specific inputs, we can easily extend the proposal to the multi-label classification scenario by associating each trigger word with a target class. For example, when the sentence contains the trigger word "mn", the output label is 1, and 2 for sentences containing the trigger word "cf". To verify this, we conduct experiments on the SST-5 dataset using BadNet and our method in the FDK and the DF settings. For comparison, we first train a clean model with a  $54.59\%$  classification accuracy. Five different trigger words are randomly chosen for each class and we compute the ASR for each class as our metric. The results are shown in Figure 3. The overall clean accuracy for EP and DFEP is both  $54.59\%$ , but it degrades by more than 1 points with BadNet ( $53.57\%$  in FDK and  $51.45\%$  in DF). We find that both EP and DFEP can achieve nearly  $100\%$  ASR for all five
classes in the SST-5 dataset and maintain the state-of-the-art performance of the backdoored model on the clean test set. This validates the flexibility and effectiveness of our proposal.
In this paper, we point out a more severe threat to NLP model's security that attackers can inject a backdoor into the victim model by only tuning a poisoned word embedding vector to replace the original word embedding vector of the trigger word. Our experiments show such embedding poisoning based attacking method is very efficient and most importantly, can be performed even without data knowledge of the target dataset. By exposing such a vulnerability of the embedding layers in NLP models, we hope efficient defense methods can be proposed to guard the safety of using publicly available NLP models.
Our work is beneficial for the research on the security of NLP models. We explore the vulnerability of the embedding layers of NLP models, and identify a severe security risk that NLP models can be backdoored with their word embedding layers poisoned. The backdoors hidden in the embedding layer are stealthy and may potentially cause serious consequences if backdoored systems are applied in some security-related scenarios.
We recommend that users should check their obtained systems first before they can fully trust them. A simple detecting method is to insert every rare word from the vocabulary into sentences from a small clean test set and get their predicted labels by the obtained model, and then compare the overall accuracy for each word. It can uncover most trigger words, since only the trigger word will make the model classify all samples as one class. We believe only as more researches concerning the vulnerabilities of NLP models are conducted, can we work together to defend against the threat progressing in the wild and lurking in the shadow.
We thank all the anonymous reviewers for their constructive comments and Liang Zhao for his valuable suggestions in preparing the manuscript. This work is partly supported by Beijing Academy of Artificial Intelligence (BAAI). Xu Sun is the corresponding author of this paper.
John Blitzer, Mark Dredze, and Fernando Pereira. 2007. Biographies, Bollywood, boom-boxes and blenders: Domain adaptation for sentiment classification. In Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics, pages 440-447, Prague, Czech Republic. Association for Computational Linguistics.
Alvin Chan, Yi Tay, Yew-Soon Ong, and Aston Zhang. 2020. Poison attacks against text datasets with conditional adversarially regularized autoencoder. In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 4175-4189, Online. Association for Computational Linguistics.
Huili Chen, Cheng Fu, Jishen Zhao, and Farinaz Koushanfar. 2019. Deepinspect: A black-box trojan detection and mitigation framework for deep neural networks. In Proceedings of the Twenty-Eighth International Joint Conference on Artificial Intelligence, IJCAI 2019, Macao, China, August 10-16, 2019, pages 4658-4664. ijcai.org.
Xiaoyi Chen, Ahmed Salem, Michael Backes, Shiqing Ma, and Yang Zhang. 2020. Badnl: Backdoor attacks against nlp models. arXiv preprint arXiv:2006.01043.
Jiazhu Dai, Chuanshuai Chen, and Yufeng Li. 2019. A backdoor attack against lstm-based text classification systems. IEEE Access, 7:138872-138878.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Siddhant Garg, Adarsh Kumar, Vibhor Goel, and Yingyu Liang. 2020. Can adversarial weight perturbations inject neural backdoors. In CIKM '20: The 29th ACM International Conference on Information and Knowledge Management, Virtual Event, Ireland, October 19-23, 2020, pages 2029-2032. ACM.
Ian J. Goodfellow, Jean Pouget-Abadie, Mehdi Mirza, Bing Xu, David Warde-Farley, Sherjil Ozair, Aaron C. Courville, and Yoshua Bengio. 2014. Generative adversarial nets. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 2672-2680.
Tianyu Gu, Brendan Dolan-Gavitt, and Siddharth Garg. 2017. Badnets: Identifying vulnerabilities in the machine learning model supply chain. arXiv preprint arXiv:1708.06733.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016, pages 770-778. IEEE Computer Society.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Xijie Huang, Moustafa Alzantot, and Mani Srivastava. 2019. Neuroninspect: Detecting backdoors in neural networks via output explanations. arXiv preprint arXiv:1911.07399.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In 3rd International Conference on Learning Representations, ICLR 2015, San Diego, CA, USA, May 7-9, 2015, Conference Track Proceedings.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E. Hinton. 2012. Imagenet classification with deep convolutional neural networks. In Advances in Neural Information Processing Systems 25: 26th Annual Conference on Neural Information Processing Systems 2012. Proceedings of a meeting held December 3-6, 2012, Lake Tahoe, Nevada, United States, pages 1106-1114.
Keita Kurita, Paul Michel, and Graham Neubig. 2020. Weight poisoning attacks on pretrained models. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2793-2806, Online. Association for Computational Linguistics.
Yiming Li, Tongqing Zhai, Baoyuan Wu, Yong Jiang, Zhifeng Li, and Shutao Xia. 2020. Rethinking the trigger of backdoor attack. arXiv preprint arXiv:2004.04692.
Yingqi Liu, Ma Shiqing, Yousra Aafer, Wen-Chuan Lee, Juan Zhai, Weihang Wang, and Xiangyu Zhang. 2018. Trojaning attack on neural networks. In 25th Annual Network and Distributed System Security Symposium.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Yunfei Liu, Xingjun Ma, James Bailey, and Feng Lu. 2020. Reflection backdoor: A natural backdoor attack on deep neural networks. In European Conference on Computer Vision, pages 182-199. Springer.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of
the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. 2017. Pointer sentinel mixture models. In 5th International Conference on Learning Representations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. Open-Review.net.
Gaurav Kumar Nayak, Konda Reddy Mopuri, Vaisakh Shaj, Venkatesh Babu Radhakrishnan, and Anirban Chakraborty. 2019. Zero-shot knowledge distillation in deep networks. In Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9-15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research, pages 4743-4751. PMLR.
Anh Nguyen and Anh Tran. 2020. Input-aware dynamic backdoor attack. arXiv preprint arXiv:2010.08138.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD:  $100,000+$  questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Aniruddha Saha, Akshayvarun Subramanya, and Hamed Piriavash. 2020. Hidden trigger backdoor attacks. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 11957-11965. AAAI Press.
Roei Schuster, Tal Schuster, Yoav Meri, and Vitaly Shmatikov. 2020. Humpty dumpty: Controlling word meanings via corpus poisoning. In 2020 IEEE Symposium on Security and Privacy (SP), pages 1295-1313. IEEE.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1631-1642, Seattle, Washington, USA. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Bolun Wang, Yuanshun Yao, Shawn Shan, Huiying Li, Bimal Viswanath, Haitao Zheng, and Ben Y Zhao. 2019. Neural cleansse: Identifying and mitigating backdoor attacks in neural networks. In 2019 IEEE Symposium on Security and Privacy (SP), pages 707-723. IEEE.
Ren Wang, Gaoyuan Zhang, Sijia Liu, Pin-Yu Chen, Jinjun Xiong, and Meng Wang. 2020. Practical detection of trojan neural networks: Data-limited and data-free cases. In Computer Vision - ECCV 2020-16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXIII, volume 12368 of Lecture Notes in Computer Science, pages 222-238. Springer.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada, pages 5754-5764.
Xinyang Zhang, Zheng Zhang, and Ting Wang. 2020. Trojaning language models for fun and profit. arXiv preprint arXiv:2008.00312.
Zhengyan Zhang, Guangxuan Xiao, Yongwei Li, Tian Lv, Fanchao Qi, Yasheng Wang, Xin Jiang, Zhiyuan Liu, and Maosong Sun. 2021. Red alarm for pre-trained models: Universal vulnerabilities by neuron-level backdoor attacks. arXiv preprint arXiv:2101.06969.