SlowGuess/ACL_2020_OCR huggingface dataset:
(1) The attack success rate is defined as the percentage of the attacks which craft an adversarial example to make the victim model predict the target label. (2) The attack validity is measured by the percentage of valid attacks to successful attacks, where the adversarial examples crafted by valid attacks have the same true labels as the original input. (3) For the quality of adversarial examples, we divide it into four parts including modification rate, grammaticality, fluency and naturality. Grammaticality is measured by the increase rate of grammatical error numbers of adversarial examples compared
with the original input, where we use Language-Tool² to obtain the grammatical error number of a sentence. We utilize the language model perplexity (PPL) to measure the fluency with the help of GPT-2 (Radford et al., 2019). The naturality reflects whether an adversarial example is natural and indistinguishable from human-written text.
We evaluate attack validity and adversarial example naturality only on SST-2 by human evaluation with the help of Amazon Mechanical Turk<sup>3</sup>. We randomly sample 200 adversarial examples, and ask the annotators to make a binary sentiment classification and give a naturality score (1, 2 or 3, higher better) for each adversarial example and original input. More annotation details are given in Appendix A.
4.5 Attack Performance
Attack Success Rate The attack success rate results of all the models are listed in Table 3. We observe that our attack model (Sememe+PSO) achieves the highest attack success rates on all the three datasets (especially the harder SST-2 and SNLI) and two victim models, proving the superiority of our model over baselines. It attacks BiLSTM/BERT on IMDB with a notably  $100.00\% /98.70\%$  success rate, which clearly demonstrates the vulnerability of DNNs. By comparing three word substitution methods (search space reduction methods) and three search algorithms, we find Sememe and PSO consistently outperform their counterparts. Further decomposition analyses are given in a later section.
Validity and Adversarial Example Quality We evaluate the attack validity and adversarial example quality of our model together with the two baseline methods (Embedding/LM+Genetic and Synonym+Greedy). The results of automatic and human evaluations are displayed in Table 4 and 5 respectively. Note that the human evaluations including attack validity and adversarial example naturality are conducted on SST-2 only. We find that in terms of automatic evaluations of adversarial example quality, including modification rate, grammaticality and fluency, our model consistently outperforms the two baselines on whichever victim model and dataset. As for attack validity and adver
$^{2}$ https://www.languagetool.org
3https://www.mturk.com
Automatic evaluation results of adversarial example quality of all the combination models are shown in Appendix B.
6071
Word Substitution Method	Search Algorithm	BiLSTM	BERT
IMDB	SST-2	SNLI	IMDB	SST-2	SNLI
Embedding/LM	Genetic	86.90	67.70	44.40	87.50	66.20	44.30
Greedy	80.90	69.00	47.70	62.50	56.20	42.40
PSO	96.90	78.50	50.90	93.60	74.40	53.10
Synonym	Genetic	95.50	73.00	51.40	92.90	78.40	56.00
Greedy	87.20	73.30	57.70	73.00	64.60	52.70
PSO	98.70	79.20	61.80	96.20	80.90	62.60
Sememe	Genetic	96.90	78.50	50.90	93.60	74.40	53.10
Greedy	95.20	87.70	70.40	80.50	74.80	66.30
PSO	100.00	93.80	73.40	98.70	91.20	78.90
Table 3: The attack success rates  $(\%)$  of different attack models.
Victim Model	Attack Model	IMDB	SST-2	SNLI
%M	%I	PPL	%M	%I	PPL	%M	%I	PPL
BiLSTM	Embedding/LM+Genetic	9.76	5.49	124.20	12.03	7.08	319.98	13.31	14.12	235.20
Synonym+Greedy	6.47	4.49	115.31	10.25	4.65	317.27	12.32	21.37	311.04
Sememe+PSO	3.71	1.44	88.98	9.06	3.17	276.53	11.72	11.08	222.40
BERT	Embedding/LM+Genetic	7.41	4.22	106.12	10.41	5.09	314.22	13.04	15.09	225.92
Synonym+Greedy	4.49	4.48	98.60	8.51	4.11	316.30	11.60	11.65	285.00
Sememe+PSO	3.69	1.57	90.74	8.24	2.03	289.94	11.72	10.14	223.22
Table 4: Automatic evaluation results of adversarial example quality. “%M”, “%I” and “PPL” indicate the modification rate, grammatical error increase rate and language model perplexity respectively.
Victim	Attack Model	%Valid	NatScore
N/A	Original Input	90.0	2.30
BiLSTM	Embedding/LM+Genetic	65.5	2.205
Synonym+Greedy	72.0	2.190
Sememe+PSO	70.5	2.210
BERT	Embedding/LM+Genetic	74.5	2.165
Synonym+Greedy	66.5	2.165
Sememe+PSO	72.0	2.180
Table 5: Human evaluation results of attack validity and adversarial example naturality on SST-2, where the second row additionally lists the evaluation results of original input. “%Valid” refers to the percentage of valid attacks. “NatScore” is the average naturality score of adversarial examples.
sarial example naturality, our Sememe+PSO model obtains a slightly higher overall performance than the two baselines. But its adversarial examples are still inferior to original human-authored input, especially in terms of validity (label consistency).
We conduct Student's  $t$ -tests to further measure the difference between the human evaluation results of different models, where the statistical significance threshold of  $p$ -value is set to 0.05. We find that neither of the differences of attack validity and adversarial example naturality between different models are significant. In addition, the adversarial examples of any attack model have significantly worse label consistency (validity) than the original
input, but possesses similar naturality. More details of statistical significance test are given in Appendix D.
For Embedding/LM, relaxing the restrictions on embedding distance and language model prediction score can improve its attack success rate but sacrifices attack validity. To make a specific comparison, we adjust the hyper-parameters of Embedding/LM+Genetic to increase its attack success rates to  $96.90\%$ ,  $90.30\%$ ,  $58.00\%$ ,  $93.50\%$ ,  $83.50\%$  and  $62.90\%$  respectively on attacking the two victim models on the three datasets (in the same order as Table 3). Nonetheless, its attack validity rates against BiLSTM and BERT on SST-2 dramatically fall to  $59.5\%$  and  $56.5\%$ . In contrast, ours are  $70.5\%$  and  $72.0\%$ , and their differences are significant according to the results of significance tests in Appendix D.
4.6 Decomposition Analyses
In this section, we conduct detailed decomposition analyses of different word substitution methods (search space reduction methods) and different search algorithms, aiming to further demonstrate the advantages of our sememe-based word substitution method and PSO-based search algorithm.
5The detailed hyper-parameter settings are given in Appendix C.
6072
Word Substitution Method	IMDB	SST-2	SNLI
Embedding/LM	3.44	3.27	3.42
Synonym	3.55	3.08	3.14
Sememe	13.92	10.97	12.87
Table 6: The average number of substitutes provided by different word substitution methods.
She breaks the pie dish and screams out that she is not handicapped.
Embedding/LM	Synonym	Sememe
tart, pizza, apple, shoemaker, cake cheesecake	None	cheese, popcorn, ham, cream, break, cake, pizza, chocolate, and 55 more
Table 7: A real case showing the substitutes found by three word substitution methods, where the original word is colored green and appropriate substitutes are colored red.
Figure 2: Attack success rates of different models with different maximum numbers of iteration times. The x-coordinate is in log-2 scale.
Word Substitution Method Table 6 lists the average number of substitutes provided by different word substitution methods on the three datasets. It shows Sememe can find much more substitutes than the other two counterparts, which explains the high attack success rates of the models incorporating Sememe. Besides, we give a real case from SST-2 in Table 7 which lists substitutes found by the three methods. We observe that Embedding/LM find many improper substitutes, Synonym cannot find any substitute because the original word "pie" has no synonyms in WordNet, and only Sememe finds many appropriate substitutes.
Search Algorithm We compare the two population-based search algorithms Genetic and PSO by changing two important hyper-parameters, namely the maximum number of iteration times  $T$  and the population size  $N$ . The results of attack success rate are shown in Figure 2 and 3. From the two figures, we find our PSO outperforms Genetic
Figure 3: Attack success rates of different models with population sizes. The x-coordinate is in log-2 scale.
Transfer	Attack Model	IMDB	SST-2	SNLI
BiLSTM	Embedding/LM+Genetic	81.93	70.61	61.26
↓	Synonym+Greedy	77.29	64.94	65.34
BERT	Sememe+PSO	75.80	64.71	59.54
BERT	Embedding/LM+Genetic	86.63	65.71	49.66
↓	Synonym+Greedy	81.64	58.67	45.16
BiLSTM	Sememe+PSO	78.42	58.11	46.89
Table 8: The classification accuracy of transferred adversarial examples on the three datasets. Lower accuracy reflects higher transferability.
consistently, especially in the setting with severe restrictions on maximum number of iteration times and population size, which highlights the efficiency of PSO.
4.7 Transferability
The transferability of adversarial examples reflects whether an attack model can attack a DNN model without any access to it (Kurakin et al., 2016). It has been widely used as an important evaluation metric in adversarial attacks. We evaluate the transferability of adversarial examples by using BiLSTM to classify the adversarial examples crafted for attacking BERT, and vice versa. Table 8 shows the classification accuracy results of transferred adversarial examples. Note that lower accuracy signifies higher transferability. The lower the accuracy is, the higher the transferability is. We find compared with the two baselines, our Se-meme+PSO crafts adversarial examples with overall higher transferability.
4.8 Adversarial Training
Adversarial training is proposed to improve the robustness of victim models by adding adversarial examples to the training set (Goodfellow et al., 2015). In this experiment, for each attack model,
6073
we craft 692 adversarial examples (10% of the original training set size) by using it to attack BiLSTM on the training set of SST-2. Then we add the adversarial examples to the training set and retrain a BiLSTM. We re-evaluate its robustness by calculating the attack success rates of different attack models. Table 9 lists the results of adversarial training. Note larger attack success rate decreases significantly greater robustness improvement. We find that adversarial training can improve the robustness of victim models indeed, and our Sememe+PSO model brings greater robustness improvement than the two baselines, even when the attack models are exactly themselves. From the perspective of attacking, our Sememe+PSO model is still more threatening than others even under the defense of adversarial training.
We also manually select 692 valid adversarial examples generated by Sememe+PSO to conduct adversarial training, which leads to even greater robustness improvement (last column of Table 9). The results show that adversarial example validity has big influence on adversarial training effect.
Existing textual adversarial attack models can be classified into three categories according to the perturbation levels of their adversarial examples.
Sentence-level attacks include adding distracting sentences (Jia and Liang, 2017), paraphrasing (Iyyer et al., 2018; Ribeiro et al., 2018) and performing perturbations in the continuous latent semantic space (Zhao et al., 2018). Adversarial examples crafted by these methods usually have profoundly different forms from original input and their validity are not guaranteed.
Character-level attacks are mainly random character manipulations including swap, substitution, deletion, insertion and repeating (Belinkov and Bisk, 2018; Gao et al., 2018; Hosseini et al., 2017). In addition, gradient-based character substitution methods have also been explored, with the help of one-hot character embeddings (Ebrahimimi et al., 2018) or visual character embeddings (Eger et al., 2019). Although character-level attacks can achieve high success rates, they break the grammaticality and naturality of original input and can be easily defended (Pruthi et al., 2019).
Att\Adv.T	None	E/L+G	Syn+G	Sem+P	Sem+P*
E/L+G	67.70	-2.60	-0.60	-3.53	-5.10
Syn+G	73.30	-2.67	-3.50	-3.13	-3.53
Sem+P	93.80	-1.07	0.03	-2.93	-4.33
Table 9: The attack success rates of different attack models when attacking BiLSTM on SST-2 and their decrements brought by adversarial training. "Att" and "Adv.T" denote "Attack Model" and "Adversarial Training". E/L+G, Syn+G and Sem+P represent Embedding/LM+Genetic, Synonym+Greedy and our Se-meme+PSO, respectively. "Sem+P*" denotes only using the valid adversarial examples generated by Se-meme+PSO in adversarial training.
As for word-level attacks, following our two-step modeling, their adversarial example space reduction methods (step 1) involve using word embeddings (Sato et al., 2018) or language model (Zhang et al., 2019a) to filter words, selecting synonyms as substitutes (Samanta and Mehta, 2017; Ren et al., 2019; Jin et al., 2019), and their combinations (Alzantot et al., 2018; Glockner et al., 2018). The search algorithms (step 2) include gradient descent (Papernot et al., 2016; Sato et al., 2018; Gong et al., 2018), genetic algorithm (Alzantot et al., 2018), Metropolis-Hastings sampling (Zhang et al., 2019a), saliency-based greedy algorithm (Liang et al., 2018; Ren et al., 2019; Jin et al., 2019). In comparison, our model adopts new methods in both steps which are more powerful.
In this paper, we propose a novel word-level attack model comprising the sememe-based word substitution method and particle swarm optimization-based search algorithm. We conduct extensive experiments to demonstrate the superiority of our model in terms of attack success rate, adversarial example quality, transferability and robustness improvement to victim models by adversarial training. In the future, we will try to increase the robustness gains of adversarial training and consider utilizing sememes in adversarial defense model.
This work is supported by the National Key Research and Development Program of China (No. 2018YFB1004503) and the National Natural Science Foundation of China (NSFC No. 61732008, 61772302). We also thank the anonymous reviewers for their valuable comments and suggestions.
For instance, using Embedding/LM+Genetic in adversarial training to defend its attack declines the attack success rate by  $2.60\%$  while using our Sememe+PSO model declines by  $3.53\%$ .
6074
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the EMNLP.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In Proceedings of ICLR.
Leonard Bloomfield. 1926. A set of postulates for the science of language. Language, 2(3).
Samuel Bowman, Gabor Angeli, Christopher Potts, and Christopher D Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of EMNLP.
Leticia Cagnina, Marcelo Errecalde, Diego Ingaramo, and Paolo Rosso. 2014. An efficient particle swarm optimization approach to cluster short texts. Information Sciences, 265:36-49.
Alexis Conneau, Douwe Kiela, Holger Schwenk, Loic Barrault, and Antoine Bordes. 2017. Supervised learning of universal sentence representations from natural language inference data. In Proceedings of EMNLP.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of NAACL-HLT.
Zhendong Dong and Qiang Dong. 2006. HowNet and the computation of meaning. World Scientific.
Russell Eberhart and James Kennedy. 1995. Particle swarm optimization. In Proceedings of the IEEE International Conference on Neural Networks.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. Hotflip: White-box adversarial examples for text classification. In Proceedings of ACL.
Steffen Eger, Gözde Gül Şahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding NLP systems. In Proceedings of NAACL-HLT.
Xianghua Fu, Guo Liu, Yanyan Guo, and Zhiqiang Wang. 2013. Multi-aspect sentiment analysis for chinese online social reviews based on topic modeling and hownet lexicon. Knowledge-Based Systems, 37:186-195.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In Proceedings of IEEE Security and Privacy Workshops.
Max Glockner, Vered Shwartz, and Yoav Goldberg. 2018. Breaking nli systems with sentences that require simple lexical inferences. In Proceedings of ACL.
Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. 2018. Adversarial texts with gradient methods. arXiv preprint arXiv:1801.07175.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of ICLR.
Rania Hassan, Babak Cohanim, Olivier De Weck, and Gerhard Venter. 2005. A comparison of particle swarm optimization and the genetic algorithm. In 46th AIAA/ASME/ASCE/AHS/ASC Structures, Structural Dynamics and Materials Conference.
Natsuki Higashi and Hitoshi Iba. 2003. Particle swarm optimization with gaussian mutation. In Proceedings of the 2003 IEEE Swarm Intelligence Symposium.
Hossein Hosseini, Sreeram Kannan, Baosen Zhang, and Radha Poovendran. 2017. Deceiving google's perspective api built for detecting toxic comments. arXiv preprint arXiv:1702.08138.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the NAACL-HLT.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of EMNLP.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is BERT really robust? Natural language attack on text classification and entailment. arXiv preprint arXiv:1907.11932.
James Kennedy and Russell C Eberhart. 1997. A discrete binary version of the particle swarm algorithm. In IEEE International Conference on Systems, Man, and Cybernetics.
J Zico Kolter and Marcus A Maloof. 2006. Learning to detect and classify malicious executables in the wild. Journal of Machine Learning Research, 7(Dec):2721-2744.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of IJCAI.
Andrew L Maas, Raymond E Daly, Peter T Pham, Dan Huang, Andrew Y Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of ACL-HLT.
George A Miller. 1995. Wordnet: a lexical database for english. Communications of the ACM, 38(11):39-41.
6075
Yilin Niu, Ruobing Xie, Zhiyuan Liu, and Maosong Sun. 2017. Improved word representation learning with sememes. In Proceedings ACL.
Mahamed G Omran, Andries P Engelbrecht, and Ayed Salman. 2004. Image classification using particle swarm optimization. In *Recent advances in simulated evolution and learning*, pages 347-365. World Scientific.
Nicolas Papernot, Patrick McDaniel, Ananthram Swami, and Richard Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. In Proceedings of MILCOM.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of EMNLP.
Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proceedings of ACL.
Fanchao Qi, Junjie Huang, Chenghao Yang, Zhiyuan Liu, Xiao Chen, Qun Liu, and Maosong Sun. 2019. Modeling semantic compositionality with sememe knowledge. In Proceedings of ACL.
Yujia Qin, Fanchao Qi, Sicong Ouyang, Zhiyuan Liu, Cheng Yang, Yasheng Wang, Qun Liu, and Maosong Sun. 2019. Enhancing recurrent neural networks with sememes. arXiv preprint arXiv:1910.08910.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of ACL.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In Proceedings of ACL.
Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. arXiv preprint arXiv:1707.02812.
Motoki Sato, Jun Suzuki, Hiroyuki Shindo, and Yuji Matsumoto. 2018. Interpretable adversarial perturbation in input embedding space for text. In Proceedings of IJCAI.
Yuhui Shi and Russell C Eberhart. 1998. Parameter selection in particle swarm optimization. In Proceedings of the International conference on evolutionary programming. Springer.
Ana Paula Silva, Arlindo Silva, and Irene Rodrigues. 2012. Biopos: Biologically inspired algorithms for pos tagging. In Proceedings of the First International Workshop on Optimization Techniques for Human Language Technology.
Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In Proceedings of EMNLP.
Gianluca Stringhini, Christopher Kruegel, and Giovanni Vigna. 2010. Detecting spammers on social networks. In Proceedings of the 26th Annual Computer Security Applications Conference.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In Proceedings of ICLR.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In Proceedings of EMNLP-IJCNLP.
Wenqi Wang, Lina Wang, Benxiao Tang, Run Wang, and Aoshuang Ye. 2019a. A survey: Towards a robust deep neural network in text domain. arXiv preprint arXiv:1902.07285.
Xiaosen Wang, Hao Jin, and Kun He. 2019b. Natural language adversarial attacks and defenses in word level. arXiv preprint arXiv:1909.06723.
Laurence A Wolsey and George L Nemhauser. 1999. Integer and combinatorial optimization. John Wiley & Sons.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019a. Generating fluent adversarial examples for natural languages. In Proceedings of ACL.
Lei Zhang, Fanchao Qi, Zhiyuan Liu, Yasheng Wang, Qun Liu, and Maosong Sun. 2019b. Multi-channel reverse dictionary model. arXiv preprint arXiv:1912.08441.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In Proceedings of ICLR.
A Human Evaluation Details
For each adversarial example and original input, we ask three workers to choose a sentiment label from "Positive" and "Negative" for it and annotate its naturality score from  $\{1,2,3\}$ , which indicates "Machine generated", "Not sure" and "Human written" respectively. We get the final sentiment labels of adversarial examples by voting. For example, if two workers annotate an example as "Positive" and one worker annotates it as "Negative", we record its annotated label as "Positive". We obtain the validity rate by calculating the percentage of the adversarial examples which are annotated with the same sentiment labels as corresponding original
6076