SlowGuess/NAACL_2022_OCR huggingface dataset:
(a) Yelp Dataset
Model	Attack Method	% USR/TSR	% Perturbation
BERT (Acc: 0.829)	HotFlip	83.3/44.9	27.0/30.3
SemAttack (+FT)	21.2/10.2	13.1/16.5
SemAttack (+FK)	53.8/23.2	14.8/22.3
SemAttack (+FC)	90.2/69.7	15.3/26.9
SemAttack (+all)	92.6/72.6	15.6/20.0
Self-Attention LSTM (Acc: 0.705)	HotFlip	32.3/17.8	11.6/13.4
SemAttack (+FT)	53.8/33.4	23.9/29.1
SemAttack (+FK)	40.7/23.2	21.4/22.2
SemAttack (+FC)	76.5/63.8	30.9/36.3
SemAttack (+all)	86.2/68.5	39.0/36.9
(b) SNLI Dataset
Table 2: Whitebox attack success rate for different attacks under targeted/untargeted attacks (TSR/USR) and corresponding word perturbation percentage against self-attention LSTM and BERT on Yelp and SNLI datasets.
Model	Attack Method	% USR/TSR	% Perturbation
DeBERTa (Large, Acc: 0.928)	TextFollower	83.2/57.1	22.5/21.3
BERT-ATTACK	84.4/36.6	19.4/17.9
SemAttack (+FT)	88.1/58.3	17.8/16.0
SemAttack (+FK)	82.1/53.7	22.1/20.9
SemAttack (+FC)	80.3/33.6	27.6/27.7
SemAttack (+all)	83.0/41.2	21.4/20.5
DeBERTa (XXLarge-v2, Acc: 0.931)	TextFollower	86.4/57.1	22.1/20.3
BERT-ATTACK	83.4/37.2	19.2/17.8
SemAttack (+FT)	90.5/65.5	17.6/16.2
SemAttack (+FK)	86.8/58.4	22.3/21.7
SemAttack (+FC)	80.6/38.7	27.6/27.9
SemAttack (+all)	82.7/42.9	21.2/20.2
FreeLB (Acc: 0.924)	TextFollower	63.0/31.5	22.1/22.0
BERT-ATTACK	65.6/31.1	19.1/18.6
SemAttack (+FT)	71.4/26.2	17.0/14.7
SemAttack (+FK)	63.2/32.6	22.9/23.9
SemAttack (+FC)	66.7/32.7	27.8/28.0
SemAttack (+all)	64.3/32.2	20.9/20.5
Table 3: Zero-query blackbox attack success rate for different attacks under targeted/untargeted attacks (TSR/USR) and corresponding word perturbation percentage against large-scale LMs and defense methods on SNLI datasets.
for BERT models by perturbing  $4 \%$  words on the Yelp dataset, when searching from the combination of the semantic spaces of  $\mathcal{F}_T$ ,  $\mathcal{F}_K$  and  $\mathcal{F}_C$ .
To adapt SemAttack to the blackbox attack setting, we distill the blackbox (teacher) model to train a whitebox (student) model, and transfer the adversarial examples from the whitebox student model to attack the blackbox model. More details can be found in Appendix §D.
For the blackbox attack shown in Appendix Table 8, the transferability-based SemAttack achieves higher attack success rates than SOTA blackbox attacks for self-attention LSTM. We also observe that BERT-ATTACK achieves a higher attack success rate on BERT than SemAttack. We think it is mainly because that BERT-ATTACK adopts an aggressively large candidate perturbation size (top- $k = 48$ ), which may lead to large semantic changes (indicated by the worse human performance as shown in Table 5). For instance, we observe that some words are even changed to their antonym in BERT-ATTACK. On the contrary,
the average size of search spaces for SemAttack (+all) is only 11.87, aiming to guarantee the naturalness and validity of the generated adversarial examples. We present more details of our semantic space in Appendix §D.3.
In addition, we observe that Self-Attention LSTM models are more robust than BERT in most settings. For example, we achieve the highest USR of  $88.1\%$  in whitebox attack on the Yelp dataset, which is  $9.5\%$  lower than BERT in the same setting. This suggests that self-attention mechanism can improve the robustness of vanilla WordLSTM by a large margin, as WordLSTM is known less robust than BERT (Jin et al., 2020).
3.2 Attack SOTA LMs and Defense Methods
In this section, we evaluate SemAttack and baseline attacks against various SOTA large-scale language models and defense methods.
Dataset and Attack Baselines Following §3.1, we evaluate SemAttack on SNLI dataset. We choose the same blackbox attack methods, TextFooer and BERT-Attack, as our baselines.
Models We consider the following models and defense methods following the Adversarial GLUE Benchmark (Wang et al., 2021). The selected large-scale models and defense methods not only represent SOTA performance on NLU tasks, but also achieve the highest robustness in the leaderboard.
- DeBERTa (He et al., 2020) improves BERT-based models by introducing disentangled attention mechanism and enhanced mask decoder, which is one of the best models in the GLUE leaderboard (Wang et al., 2018). In our experiment, we use DeBERTa (Large) and DeBERTa (XXLarge-v2).
- FreeLB (Zhu et al., 2019) is an adversarial training algorithm that defends adversarial attacks by adding perturbations to word embeddings and
minimizing the corresponding adversarial loss.
Attack Goal To demonstrate the model robustness in an approximately real-world scenario, we consider a zero-query setting, a more rigorous and common scenario that assumes the target models are not accessible during the attack phase. Since we can not access the target model, we perform a transferability-based backbox attack. Specifically, we attack the selected language models and defense methods using adversarial SNLI texts generated by SemAttack against BERT classifier in §3.1.
Adversarial Attack Evaluation We finetune the above models on the SNLI dataset and attack them using adversarial texts generated against BERT. The results are shown in Table 3.
For the zero-query setting, SemAttack always achieves the highest success rates. Specifically, among all the attack methods, SemAttack  $(+F_{T})$  always has the highest USR regardless of the model it is tested on. For example, on the largest model, DeBERTa (XXLarge-v2), we achieve  $90.5\%$  USR, which is  $7.1\%$  higher than BERT-ATTACK.
Furthermore, we find that increasing the number of model parameters and expanding the model architecture have little effect on defense against adversarial attacks. DeBERTa (XXLarge-v2), for example, is substantially larger than DeBERTa (Large), yet the attack success rates are similar. In some cases DeBERTa (XXLarge-v2) is even less robust than DeBERTa (Large). We also observe that introducing some defense strategies slightly improves the model's robustness. When we use the defense strategy of FreeLB, we can see that the robustness increases, but it is still not satisfactory to defend existing adversarial attacks.
3.3 Adapt SemAttack to Chinese
Datasets We evaluate our performance on the following two datasets in Chinese: 14-category news classification dataset THUNews (Sun et al., 2016) and 11-class Wechat Finance dataset. More details about these datasets are introduced in Appendix C.
Models We use BERT pre-trained on Chinese corpora and finetune on the two datasets separately. After finetuning, our BERT achieved 0.818 accuracy on THUNews dataset and 0.891 on Wechat Finance Dataset, as shown in Table 4
Attack Baselines Since both TextFooler and BERT-Attack adopt an aggressively large perturba
Dataset	Setting	Attack Method	% USR/TSR	% Perturbation
THUNews (Acc: 0.818)	White-box Attack	HotFlip	81.4/40.4	21.7/27.9
SemAttack (+FT)	96.6/81.7	20.1/34.7
SemAttack (+FK)	15.6/3.6	16.1/17.4
SemAttack (+FC)	95.0/78.3	17.4/29.4
SemAttack (+all)	99.0/92.1	15.1/26.3
Black-box Attack	HotFlip	44.3/10.0	15.4/10.8
SemAttack (+FT)	52.3/34.0	19.7/35.3
SemAttack (+FK)	8.4/1.3	12.7/13.1
SemAttack (+FC)	55.9/37.0	17.6/28.6
SemAttack (+all)	58.6/48.2	16.4/25.8
Wechat (Acc: 0.891)	White-box Attack	HotFlip	95.2/0.0	11.4/-
SemAttack (+FT)	86.0/88.3	7.2/12.4
SemAttack (+FK)	32.8/24.5	5.2/7.6
SemAttack (+FC)	96.8/96.4	5.8/9.4
SemAttack (+all)	98.7/98.0	4.6/8.7
Black-box Attack	HotFlip	21.7/0.0	8.9/-
SemAttack (+FT)	49.4/35.8	7.3/17.4
SemAttack (+FK)	19.5/11.7	4.0/7.7
SemAttack (+FC)	51.8/42.4	5.3/12.2
SemAttack (+all)	54.5/36.7	4.0/11.7
Table 4: Whitebox and blackbox attack success rate for different attacks under targeted/untargeted attacks (TSR/USR) and corresponding word perturbation percentage against Chinese BERT on THUNews and Wechat Finance datasets.
tion candidate space and thus require additional language resources (e.g., POS checker; stop words filtering) to ensure the proposed candidate words are valid, they cannot be adapted to Chinese due to the lack of corresponding language resources. Therefore, we adapt HotFlip for Chinese classification task, since it does not rely on any other linguistic resources. We also adapt it to transferability-based blackbox attack settings as well as the targeted attack setting for fair comparison.
Attack Goal In this paper, we choose the targeted attack class as "technology news" for THUNews dataset and "Bank" for Wechat dataset (when the ground truth label is the targeted class, we switch the target to another random class). This strategy achieves the highest targeted attack success rate as shown in Appendix F.7.
Adversarial Attack Evaluation In the whitebox attack scenario in Table 4, SemAttack is able to make the model mistakenly classify nearly all sentences with only a small number of characters being manipulated in both targeted and untargeted settings. The untargeted attack achieves  $99\%$  success rate by substituting merely two tokens on average on the THUNews dataset. On Wechat Finance dataset, it achieves  $98.7\%$  attack success rate by perturbing  $4.6\%$  tokens on average in the input sequences. In the targeted attack scenario, we always make BERT output as our expected false class on both datasets, resulting in a huge performance drop on BERT models. We achieve  $92.1\%$  and  $98.0\%$  on THUNews dataset and Wechat Finance dataset, respectively.
Dataset	Attack Method	% Perturbation	PPL	BertScore	Human Ratings
Yelp (English)	HotFlip	14.9	57.1	0.79	3.337 ± 1.650
TextFollower	13.5	43.7	0.78	3.361 ± 1.326
BERT-ATTACK	4.2	31.4	0.92	3.513 ± 1.280
SemAttack (+all)	4.3	34.4	0.91	3.524 ± 1.584
THUNews (Chinese)	HotFlip	21.7	488.3	0.60	3.770 ± 1.061
SemAttack (+all)	15.1	317.4	0.76	3.846 ± 0.906
We also present the blackbox attack results in Table 4. We can see that SemAttack (+all) achieves the highest success rates in most cases, which suggests that our semantic perturbation spaces have high adversarial transferability. Note that we do not present the targeted attack on Wechat Finance dataset for HotFlip since all attack attempts failed.
Ablation Studies We conduct a series of ablation studies such as exploration of BERT embedding space, attack strategies,  $\ell_p$  norm selection for Eq.(4), hyper-parameter selection, and attack efficiency comparison, etc. in Appendix F.
3.4 Adversarial Text Quality Evaluation
To confirm that our generated adversarial texts are valid and natural to humans, we conduct both automatic evaluation and human evaluation on both English and Chinese NLP tasks, considering language quality and utility preservation. More evaluation details can be found in Appendix G.
Language Quality Evaluation We sample 100 original sentences from the test set for both Chinese and English such that all of them can be successfully attacked by SemAttack and our baselines. For automatic evaluation, we consider the average perturbation rate, perplexity (PPL) (based on GPT-2), and BertScore as metrics to indicate the language quality. For human evaluation, we present every generated adversarial sentence to 5 human annotators, ask them to rate the language quality from 1 to 5, and calculate the average ratings. We present the evaluation results in Table 5.
We can see that SemAttack has the best human ratings across different baselines for both Chinese and English. In terms of automatic evaluation metrics, we observe that SemAttack is quite close to the SOTA BERT-ATTACK. We think the reason why SemAttack is slightly weaker than BERT-ATTACK in terms of PPL and BertScore is that SemAttack also considers typos and knowledge-based perturbations. Such perturbations usually look good to humans, but may greatly
Table 5: Language quality evaluation for the generated adversarial texts in both Chinese and English.
Dataset	Human	BERT
Yelp (English)	clean	0.9562 ± 0.0006	0.706
adversarial	0.9390 ± 0.0010	0.000
THUNews (Chinese)	clean	0.9400 ± 0.0014	0.818
adversarial	0.9369 ± 0.0015	0.000
Table 6: Human performance compared to BERT classifiers on the original and adversarial datasets.
impact the scores calculated by pretrained language models such as GPT-2 and BERT.
Utility Preservation Evaluation To evaluate human performance on our generated adversarial data, we randomly sample 50 clean sentences and 50 adversarial sentences generated by the targeted SemAttack (+all) for both the English Yelp and the Chinese THUNews dataset. For each sentence, we present the annotators with two labels: a ground truth label and a targeted wrong label (e.g., the most opposite sentiment), and request annotators to choose the correct one. Both clean text and adversarial text are randomly shuffled.
The detailed evaluation results with standard deviation are shown in Table 6. We find that our adversarial text barely impacts human perception, as the human performance on adversarial Yelp data is  $93.9\%$ , only  $2\%$  lower than the clean data. Human performance on the adversarial Chinese THUNews is  $93.7\%$ , which is very close to the performance of  $94.0\%$  on the clean dataset.
Our proposed semantic perturbation functions generalize the existing textual adversarial attacks.
For typo-based perturbation function  $\mathcal{F}_T$ , existing work (Li et al., 2018; Ebrahimi et al., 2017) applies character-level perturbation to carefully crafted typo words (e.g., from "foolish" to "foolish"), thus making the model ignore or misunderstand the original statistical cues.
Knowledge-based perturbation function  $\mathcal{F}_K$  uses knowledge base to constrain the search space. For example, Zang et al. (2020) uses sememe-based knowledge base from HowNet (Dong et al., 2010) to construct a search space for word substitution.
Different from our contextualized semantic perturbation function  $\mathcal{F}_C$ , other work (Jin et al., 2020; Li et al., 2018) uses a non-contextualized word embedding from GLoVe (Pennington et al., 2014) or Word2Vec (Mikolov et al., 2013) to build synonym candidates, by querying the cosine similarity or eu
cridean distance between the original and candidate word and selecting the closet ones as the replacements. However, some antonyms also have high cosine similarity in the Word2Vec space. Thus, additional hand-crafted filtering rules are needed to ensure that the meaning is not changed.
Other work (Garg and Ramakrishnan, 2020; Li et al., 2020, 2021) also leverages pre-trained models to generate contextualized perturbations by masked language modeling, which is a parallel work to SemAttack, where we explore the BERT embedding clusters to generate high-quality adversarial examples.
In terms of optimization, unlike the heuristic-based previous work that uses greedy (Jin et al., 2020) or genetic algorithms (Zang et al., 2020) which search for the optimal perturbations, or gradient-based methods (Wang et al., 2020; Guo et al., 2021) which search for perturbation on a tree-autoencoder with only syntactic constraints or a distribution of adversarial examples, we use an optimization-based method to efficiently and effectively search for the optimal adversarial perturbation in the semantic preserving spaces to ensure the validity and naturalness of perturbed sentences.
In this paper, we propose a novel semantic adversarial attack framework SemAttack to probe the robustness of LMs. Comprehensive experiments show that SemAttack is able to generate natural adversarial texts in different languages and achieve higher attack success rates than existing textual attacks. We also demonstrate that existing SOTA LMs and defense methods are still vulnerable to SemAttack. We expect our study to shed light on future research on evaluating and enhancing the robustness of LMs for different languages.
We gratefully thank the anonymous reviewers and meta-reviewers for their constructive feedback. This work is partially supported by the NSF grant No.1910100, NSF CNS 20-46726 CAR, and Sloan Fellowship.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium. Association for Computational Linguistics.
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In EMNLP.
Nicholas Carlini and David A. Wagner. 2016. Towards evaluating the robustness of neural networks. 2017 IEEE Symposium on Security and Privacy (SP), pages 39-57.
Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. In The Thirty-Fourth AAAI Conference on Artificial Intelligence, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 3601-3608. AAAI Press.
Andy Coenen, Emily Reif, A. Yuan, Been Kim, A. Pearce, F. Viégas, and M. Wattenberg. 2019. Visualizing and measuring the geometry of bert. In NeurIPS.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL-HLT*.
Ruixue Ding, Pengjun Xie, Xiaoyan Zhang, Wei Lu, Linlin Li, and Luo Si. 2019. A neural multi-digraph model for Chinese NER with gazetteers. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1462-1467, Florence, Italy. Association for Computational Linguistics.
Zhendong Dong, Qiang Dong, and Changling Hao. 2010. Hownet and its computation of meaning. In Proceedings of the 23rd International Conference on Computational Linguistics: Demonstrations, COLING '10, page 53-56, USA. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and De- jing Dou. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 31-36. Association for Computational Linguistics.
Siddhant Garg and Goutham Ramakrishnan. 2020. Bae: Bert-based adversarial examples for text classification. In EMNLP.
Chuan Guo, Alexandre Sablayrolles, Hervé Jégou, and Douwe Kiela. 2021. Gradient-based adversarial attacks against text transformers. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 5747-5757.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2020. Deberta: Decoding-enhanced bert with disentangled attention. arXiv preprint arXiv:2006.03654.
John Hewitt and Christopher D. Manning. 2019. A structural probe for finding syntax in word representations. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4129-4138, Minneapolis, Minnesota. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is BERT really robust? A strong baseline for natural language attack on text classification and entailment. In *The Thirty-Fourth AAAI Conference on Artificial Intelligence*, AAAI 2020, The Thirty-Second Innovative Applications of Artificial Intelligence Conference, IAAI 2020, The Tenth AAAI Symposium on Educational Advances in Artificial Intelligence, EAAI 2020, New York, NY, USA, February 7-12, 2020, pages 8018-8025. AAAI Press.
Erik Jones, Robin Jia, Aditi Raghunathan, and Percy Liang. 2020. Robust encodings: A framework for combating adversarial typos. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, ACL 2020, Online, July 5-10, 2020, pages 2752-2765. Association for Computational Linguistics.
Diederik P. Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
Zhen-Zhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. ArXiv, abs/1909.11942.
Dianqi Li, Yizhe Zhang, Hao Peng, Liquin Chen, Chris Brockett, Ming-Ting Sun, and William B Dolan. 2021. Contextualized perturbation for textual adversarial attack. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5053-5069.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. Textbugger: Generating adversarial text against real-world applications. ArXiv, abs/1812.05271.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. Bert-attack: Adversarial attack against bert using bert.
Zhouhan Lin, Minwei Feng, Cicero Nogueira dos Santos, Mo Yu, Bing Xiang, Bowen Zhou, and Yoshua Bengio. 2017. A structured self-attentive sentence embedding. arXiv preprint arXiv:1703.03130.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Gregory S. Corrado, and Jeffrey Dean. 2013. Distributed representations of words and phrases and their compositionality. In Advances in Neural Information Processing Systems 26: 27th Annual Conference on Neural Information Processing Systems 2013. Proceedings of a meeting held December 5-8, 2013, Lake Tahoe, Nevada, United States, pages 3111-3119.
George A. Miller. 1995. Wordnet: A lexical database for english. Commun. ACM, 38(11):39-41.
Isabel Papadimitriou, Ethan A Chi, Richard Futrell, and Kyle Mahowald. 2021. Deep subjecthood: Higher-order grammatical features in multilingual bert. In Proceedings of the 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 2522-2532.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the 2014 Conference on Empirical Methods in Natural Language Processing, EMNLP 2014, October 25-29, 2014, Doha, Qatar; A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1532-1543. ACL.
M Sun, J Li, Z Guo, Z Yu, Y Zheng, X Si, and Z Liu. 2016. Thuctc: an efficient chinese text classifier. *GitHub Repository*, https://github.com/thunlp/THUCTC* (2016, accessed 17 May 2017). Google Scholar.
Ian Tenney, Dipanjan Das, and Ellie Pavlick. 2019. Bertrediscoverstheclassicalnlppipeline.InACL.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R Bowman. 2018. Glue: A multi-task benchmark and analysis platform for natural language understanding. arXiv preprint arXiv:1804.07461.
Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2020. T3: Treeautoencoder constrained adversarial text generation for targeted attack. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6134-6150, Online. Association for Computational Linguistics.