SlowGuess/NAACL_2021_OCR huggingface dataset:
To ensure the functional equivalence, we defined immutable tokens as pre-constraints using Named Entity Model provided by Spacy and regular expressions. For Enron dataset, names (Person or Organization), IP, email addresses and URL, while for Twitter and Toxic datasets URL, #Hashtags and @Reference and lastly, for Yelp dataset, names and URL were considered as immutable entities. For
Table 2: The Target Models training and testing data sets statistics
Dataset	Training Data	Testing Data	Avg Length	Avg Spell- ing errors	Avg Gram- mar errors
Enron (Wiki)	28.6k	9.9k	244	3.07%	31%
Twitter (Kaggle, 2019)	8.1k	3.9k	15	10.67%	28.46%
Toxic (Kaggle, 2018b)	159.5k	12.4k	70	2.13%	25.60%
Yelp (Yelp)	560k	38k	139	0.81%	21.67%
Table 3: Balanced Accuracy of target models on test datasets
Dataset	WordCNN	CharCNN	BiLSTM	RCNN
Enron	97.50%	96.50%	97.60%	98.30%
Twitter	93.44%	92.02%	93.72%	94.15%
Toxic	69.05%	86.62%	89.61%	88.56%
Yelp	94.74%	94.44%	95.60%	95.34%
semantic and lexical equivalence, we defined three post constraints, i.e., the semantic similarity, which were calculated using Universal Sentence Encoder (USE) (Cer et al., 2018) considering  $\varepsilon = 0.60$ . Moreover, for counting spelling mistakes Garbe was used while for calculating grammar issues, we used language tool (PYPI).
Results of our experiments are presented here.
5.1 Attack Evaluation
Table 4 shows the samples of AEs generated by ReinforceBug and Table 5 illustrates the main results of our experiments. It can be seen that ReinforceBug produces AEs with a comparatively high PCS (i.e., on average  $74\%$ ), semantic similarity (i.e., on average  $83.5\%$ ) than other two attacks for all the datasets. However, its success rate is on average  $15\%$  less than TextBugger (Li et al., 2018) for the all the models. One reason behind it is that TextBugger generates unrealistic AEs with least PCS (i.e., on average  $66\%$ ), low semantical similarity (i.e., on average  $63\%$ ) and significantly high lexical errors (i.e., on average  $14\%$  spelling and  $27\%$  grammar errors). Additionally, TextBugger on average perturbs more than  $59\%$  words of the original text to generate AEs. Such a high perturbation rate is too large to be ignored by the end-user. Moreover, for Enron and Twitter datasets, TextBugger perturbs  $9.09\%$  and  $97.67\%$  of the URLs present in the text, thus adversely effecting the functional semantics of the text. Therefore, the success rate of TextBugger is an overestimation and deviates from the semantic, functional and lexical constraints.
In comparison with TextFooler (Jin et al., 2020) our method has significantly high success rate (i.e., on average more than  $10\%$ ) for all the models. It is expected because TextFooler relies on synonym substitutes technique to generate AE, however, for noisy datasets with relative high lexical errors such as Twitter (Table 2) this method tends to fail. Lastly, TextFooler produces  $5\%$  more grammatical errors than ReinforceBug and similar to TextBugger, TextFooler also perturbs URLs, thus effecting the functional semantics of the generated AEs. These findings suggest that ReinforceBug produces effective and utility-preserving AEs (Definition 3).
5.2 Knowledge Transferability
Table 6 shows the transferability of policy learned by ReinforceBug on unseen datasets for each task. For benchmarking the results are also compared with the state-of-the-art attacks TextBugger and Textfooler. It is evident from the results that ReinforceBug takes less time to generate adversarial samples as compared to the other models. It is because ReinforceBug utilizes the same important word vocabulary selected while learning and the agent has already explored and learned utility-preserving operations on them during training. Additionally, although TextFooer and TextBugger both perform example-specific perturbation and ReinforceBug might suffer from out-of-vocabulary words but still the success rate of ReinforceBug on test datasets is more than TextFooer attack and less than TextBugger which are aligned with our finding in section 5.1. Also, from Table 6 it is evident that our ReinforceBug produces utility-preserving AEs with high PCS and semantic similarity (i.e., on average more than  $70\%$  and  $82\%$  respectively) and comparatively low perturbation rate (on average  $8.5\%$ ) for all test datasets. It suggests that the important words and transformation learned from the training datasets are transferable to unseen datasets and can be used to generalize the vulnerabilities of the target models.
5.3 Attack Transferability on Models
To determine whether AEs curated based on one model can also fool other models for the same task, we examined the transferability of AEs on other models (see Table 3 for the performance of these models). Table 7 shows the transferability result. AEs generated by our method are more transferable to other models in comparison to the state-of-the-art attacks. However, there is a moderate degree of transferability between models, and the transferability is higher for Twitter and Toxic detection task as compared to Email and Yelp movie-review classification task. Nevertheless, BiLSTM trained on Enron dataset (having  $97.60\%$  accuracy) offers more resilience to AEs generated by RCNN by limiting the success rate of the attack to  $(<17\%)$  for all the attacks, while other models are highly vulnerable to the AEs. It signifies that vulnerabilities exploited by our AEs are task-specific and moderately model-independent.
Table 4: Samples of Examples (red font depicts the change in the original example)
Task: Twitter
Spam/Benign	Original Label
Benign (PCS 100%)	Adversarial Label
Spam (PCS 98%)	Task: Toxic/Non-
Toxic Comment	Original Label
Toxic (PCS 97%)	Adversarial Label
non-Toxic (PCS 83%)
I should of just gone home yesterday and spent my day off today with my family relatives	Even trolls surely deserve to eat, bastardx.
Task: Email
Spam/Benign	Original Label
Spam (PCS 100%)	Adversarial Label
Benign (PCS 96%)	Task: Positive/
Negative Review	Original Label
Negative (72%)	Adversarial Label
Positive (PCS 86%)
Subject:soul mate one of your buddies hooked you up on a date scheduled with another buddy.
your invitation:a free dating web site created by women no more invitation:	Premium price for a standard sandwich. Would have been an amazingly
enjoyable visit had the deli guy been at least welcoming .
Table 5: ReinforceBug Attack on Training Dataset
Dataset	Attacks	Success Rate	Perturbation Rate	% of URLs Perturbed	Avg PCS	Avg Semantic Similarity	Avg Spelling Errors	Avg Grammar Errors
Enron RCNN	TextBugger	43.09%	46.64%	35.21%	62.52%	61.51%	13.39%	35.7%
TextFooler	22.31%	10.60%	9.09%	59.76%	76.62%	3.08%	21.80%
ReinforceBug	33.21%	11.09%	0.00%	73.82%	85.66%	3.30%	9.88%
Twitter RCNN	TextBugger	65.98%	24.74%	97.67%	74.6%	58.8%	25.77%	35.64%
TextFooler	12.66%	11.89%	99.10%	79.17%	73.79%	5.22%	15.68%
ReinforceBug	14.99%	11.29%	0.00%	82.82%	80.86%	4.14%	13.99
Yelp BiLSTM	TextBugger	44.88%	21.89%	6.25%	58.62%	66.97%	5.22%	16.85%
TextFooler	33.80%	4.15%	4.73%	58.39%	80.60%	0.74%	14.13%
ReinforceBug	39.35%	3.18%	0.00%	63.78%	87.78%	1.88%	9.68%
Toxic BiLSTM	TextBugger	42.11%	24.78%	45.10%	62.85%	60.83%	11.36%	19.98%
TextFooler	21.00%	7.07%	28.57%	64.17%	76.68%	1.53%	14.70%
ReinforceBug	31.22%	9.43%	0.00%	71.52%	78.07%	3.69%	6.40%
Table 6: ReinforceBug Attack on Testing Dataset (Test time is given days-hours:minutes:seconds format)
Dataset	Attacks	Test Time	Success Rate	Perturbation Rate	Avg PCS	Avg Semantic Similarity	Avg Spelling Errors	Avg Grammar Errors
Enron RCNN	TextBugger	1-18:55:48	42.42%	38.68%	62.52%	61.51%	13.39%	17.31%
TextFooler	2-03:07:54	20.98%	8.12%	61.19%	77.47%	7.23%	10.96%
ReinforceBug	1-08:15:21	31.80%	10.37%	66.34%	88.17%	6.86%	7.95%
Twitter RCNN	TextBugger	0-1:30:23	64.71%	20.52%	75.6%	58.5%	26.43%	58.17%
TextFooler	0-1:34:05	12.59%	9.17%	79.18%	74.65%	19.71%	14.01%
ReinforceBug	0-0:47:29	12.68%	13.43%	81.77%	75.84%	16.97%	20.17%
Yelp BiLSTM	TextBugger	2-13:26:43	50.42%	26.73%	58.73%	66.96%	5.23%	8.21%
TextFooler	2-16:17:21	39.33%	4.00%	58.85%	80.58%	3.15%	5.25%
ReinforceBug	1-15:27:12	46.19%	5.77%	61.22%	89.07%	3.62%	4.42%
Toxic BiLSTM	TextBugger	0-11:57:28	33.99%	36.24%	62.52%	61.49%	11.07%	18.65%
TextFooler	0-15:19:29	16.05%	7.93%	65.83%	77.23%	8.37%	7.94%
ReinforceBug	0-09:22:49	30.16%	10.52%	70.36%	78.43%	4.83%	4.13%
Table 7: Transferrability of AEs generated by ReinforceBug Attack on other models
Enron	Twitter	Yelp	Toxic
CharCNN	WordCNN	BILSTM	CharCNN	WordCNN	BILSTM	CharCNN	WordCNN	RCNN	CharCNN	WordCNN	RCNN
TextBugger	51.0%	17.3%	16.5%	11.7%	30.8%	50.0%	21.8%	21.6%	24.5%	50.9%	43.8%	42.5%
TextFooler	47.0%	12.5%	12.3%	24.0%	41.6%	49.7%	25.6%	22.5%	27.1%	66.3%	59.8%	52.9%
ReinforceBug	47.3%	27.5%	16.3%	53.5%	42.6%	52.0%	34.0%	37.2%	34.5%	70.3%	64.9%	72.2%
This section analyses the results.
6.1 Operations Distribution
Figure 2a illustrates the proportion of each operation chosen by the ReinforceBug to generate utility-preserving AEs. We can see that get_semantic, and get synonyms operations are most dominant for all the tasks. One reason could be that get_semantic is deliberately designed for creating similar contextual adversarial texts without deleting the original important word, while get_synonyms replaces the word with similar meaning word. That is why the semantic similarity remains intact without impacting the linguistic structure of the text. Other operations, i.e., addi
tion, insertion, bitsquatting and omission that cause common typos are moderately chosen, however, generatehom is rarely selected by ReinforceBug. This happens because it can only replace a character with a visually similar character and produces fewer replacement options for a word than other operations. This reason is also valid of word_month and word_to_num as only few words are either words representing month or numbers in the corpus vocabulary.
6.2 Distribution of words versus the number of examples effected
To demonstrate the knowledge transferability, we visualize the identified important words according to the number of examples affected by their
(a) Operation Distribution (b) Word Cloud of important words versus the number of Examples affected for the target models across Trained Models
Figure 2: Analysis of AEs generated by ReinforceBug
replacements in Figure 2b. Here, the words impacting more examples are represented with a larger font. Figure 2b(i) shows that in emails, words such as 'click', 'attached', 'thanks', and 'deal' are more likely to affect the prediction of the target models by decreasing the spam intent to benign. Whereas for the Twitter dataset, Figure 2b(ii) shows that the targeted RCNN model is more vulnerable to minor perturbation on words such as 'news', 'trump', 'obama', 'state' and 'police'. For toxic content detection, the model decision is manipulated for most examples with words like 'see', 'like', 'think'. Perturbing words like 'stupid' and 'idiot' decreases the toxicity of the text. Lastly, for the yelp dataset, changing words such as 'like', 'great' and 'best' increases the text's negative extent. Therefore, it is evident that models are vulnerable to these words irrespective of a specific example; instead, these vulnerabilities affect multiple examples in the corpus and are transferable to new datasets as seen in (section 5.2) as well as are transferable to other models (section 5.3).
Overall, this study proposes ReinforceBug, a reinforcement learning-based framework to generate utility-preserving AEs against the state-of-the-art text classifiers under black-box settings. Extensive experiments demonstrate that it effectively generates utility-preserving AEs that are transferable to other models, and the learned policy is transferable to the unseen datasets. It identifies semantic concatenation and synonym substitution attacks as a significant threat to text-classifiers and suggests defences against these attacks should be explored in the future to improve their robustness.
This work was supported with super-computing resources provided by the Phoenix HPC service at the University of Adelaide.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Battista Biggio and Fabio Roli. 2018. Wild patterns: Ten years after the rise of adversarial machine learning. In Proceedings of the 2018 ACM SIGSAC Conference on Computer and Communications Security, CCS '18, page 2154-2156, New York, NY, USA. Association for Computing Machinery.
Daniel Cer, Yinfei Yang, Sheng-yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder for English. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 169-174, Brussels, Belgium. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Aurore Fass, Michael Backes, and Ben Stock. 2019. Hidenoseek: Camouflaging malicious javascript in benign asts. In Proceedings of the 2019 ACM SIGSAC Conference on Computer and Communications Security, pages 1899-1913.
Radu Florian, Abe Ittycheriah, Hongyan Jing, and Tong Zhang. 2003. Named entity recognition
through classifier combination. In Proceedings of the seventh conference on Natural language learning at HLT-NAACL 2003, pages 168-171.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. In 2018 IEEE Security and Privacy Workshops (SPW), pages 50-56. IEEE.
Wolf Garbe. wolfgarbe/symspell: Symspell: 1 million times faster through symmetric delete spelling correction algorithm.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Ivo Grondman, Lucian Busoniu, Gabriel AD Lopes, and Robert Babuska. 2012. A survey of actor-critic reinforcement learning: Standard and natural policy gradients. IEEE Transactions on Systems, Man, and Cybernetics, Part C (Applications and Reviews), 42(6):1291-1307.
Ashley Hill, Antonin Raffin, Maximilian Ernestus, Adam Gleave, Anssi Kanervisto, Rene Traore, Prafulla Dhariwal, Christopher Hesse, Oleg Klimov, Alex Nichol, Matthias Plappert, Alec Radford, John Schulman, Szymon Sidor, and Yuhuai Wu. 2018. Stable baselines.
Gauri Jain, Manisha Sharma, and Basant Agarwal. 2018. Spam detection on social media using semantic convolutional neural network. International Journal of Knowledge Discovery in Bioinformatics (IJKDB), 8(1):12-26.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2020. Is bert really robust? a strong baseline for natural language attack on text classification and entailment. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 8018-8025.
Kaggle. 2018a. Fake news, kaggle.
Kaggle. 2018b. Toxic comment classification challenge, kaggle.
Kaggle. 2019. Utkml's twitter spam detection competition, kaggle.
Siwei Lai, Liheng Xu, Kang Liu, and Jun Zhao. 2015. Recurrent convolutional neural networks for text classification. In Twenty-ninth AAAI conference on artificial intelligence.
Dongjun Lee. dongjun-lee/text-classification-modelstf: Tensorflow implementations of text classification models.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. Textbugger: Generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271.
Linyang Li, Ruotian Ma, Qipeng Guo, Xiangyang Xue, and Xipeng Qiu. 2020. BERT-ATTACK: Adversarial attack against BERT using BERT. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6193–6202, Online. Association for Computational Linguistics.
John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 119-126.
Nikola Mrkšić, Diarmuid Řeaghdha, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In Proceedings of HLT-NAACL.
Jeffrey Pennington, Richard Socher, and Christopher D. Manning. 2014. Glove: Global vectors for word representation. In Empirical Methods in Natural Language Processing (EMNLP), pages 1532-1543.
PYPI. language-tool-python  $\cdot$  pypi.
Dhanalakshmi Ranganayakulu and C Chellappan. 2013. Detecting malicious URLs in e-mail—an implementation. AASRI Procedia, 4:125-131.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. 2017. Proximal policy optimization algorithms. arXiv preprint arXiv:1707.06347.
sklearn.sklearn.feature_extraction.text.countvectorizer — scikit-learn 0.23.2 documentation.
Spacy. Linguistic features  $\cdot$  spacy usage documentation.
Ankit Srivastava, Piyush Makhija, and Anuj Gupta. 2020. Noisy text data: Achilles' heel of bert. In Proceedings of the Sixth Workshop on Noisy User-generated Text (W-NUT 2020), pages 16-21.
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction. MIT press.
Cheng-Yen Tang, Chien-Hung Liu, Woei-Kae Chen, and Shingchern D You. 2020. Implementing action mask in proximal policy optimization (ppo) algorithm. ICT Express.
Xiaosen Wang, Hao Jin, and Kun He. 2019. Natural language adversarial attacks and defenses in word level. arXiv preprint arXiv:1909.06723.
ACL Wiki. Spam filtering datasets.
Catherine Wong. 2017. Dancin seq2seq: Fooling text classifiers with adversarial text example generation. arXiv preprint arXiv:1712.05419.
Yelp. Yelp dataset.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM Transactions on Intelligent Systems and Technology (TIST), 11(3):1-41.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in neural information processing systems, pages 649-657.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples.
Peng Zhou, Zhenyu Qi, Suncong Zheng, Jiaming Xu, Hongyun Bao, and Bo Xu. 2016. Text classification improved by integrating bidirectional LSTM with two-dimensional max pooling. In Proceedings of COLING 2016, the 26th International Conference on Computational Linguistics: Technical Papers, pages 3485-3495, Osaka, Japan. The COLING 2016 Organizing Committee.