SlowGuess/NAACL_2021_OCR huggingface dataset:
Looking at the "POS" and "NEG" performance of AE and  $\mathrm{AE + BAL}$ , we can see that  $\mathrm{AE + BAL}$  is effective in creating a more balanced performance for positive-to-negative and negative-to-positive attacks. We hypothesise that AE learns to perform single direction attack because it is easier to generate positive (or negative) words for all input examples and sacrifice performance in the other direction to achieve a particular attacking performance. That said, the low AGR score (0.12) suggests that  $\mathrm{AE + BAL}$  adversarial examples do not preserve the ground truth sentiments.
The introduction of label smoothing  $(\mathrm{AE} + \mathrm{LS})$  and counter-fitted embeddings  $(\mathrm{AE} + \mathrm{LS} + \mathrm{CF})$  appear to address label preservation, as AGR improves from 0.12 to 0.46 to 0.64. Adding the copy mechanism  $(\mathrm{AE} + \mathrm{LS} + \mathrm{CF} + \mathrm{CPY})$  provides also some marginal improvement, although the more significant benefit is in sentence reconstruction: a boost of 5 BLEU points. Note that we also experimented with incorporating  $+\mathrm{BAL}$  for these variants, but found minimal benefit. For the rest of the experiments, we use  $\mathrm{AE} + \mathrm{LS} + \mathrm{CF} + \mathrm{CPY}$  as our model to benchmark against other adversarial methods.
Comparison with baselines. We next present results on the test set in Table 2. The benchmark methods are: TYC, HOTFLIP, and TEXTFOOLER (described in Section 2). We choose 3 ACC thresholds as the basis for comparison: T1, T2 and T3, which correspond to approximately  $80 - 90\%$ , 70-
80% and 60-70% accuracy.
Generally, all models trade off example quality for attacking rate, as indicated by the lower BLEU, USE and ACPT scores at T3.
Comparing C-LSTM and C-CNN, we found that C-CNN is generally an easier classifier to attack, as BLEU and USE scores for the same threshold are higher. Interestingly, TEXTFOOLER appears to be ineffective for attacking C-CNN, as we are unable to tune TEXTFOOLER to generate adversarial examples producing ACC below the T1 threshold.
Comparing the attacking models and focusing on C-LSTM, TEXTFOOLER generally has the upper hand.  $\mathrm{AE + LS + CF + CPY}$  performs relatively well, and usually not far behind TEXTFOOLER. HOTFLIP produces good BLEU scores, but substantially worse USE scores. TYC is the worst performing model, although its adversarial examples are good at fooling the unseen classifier C-BERT (lower TRF than all other models), suggesting that there may be a (negative) correlation between indomain performance and transferability. Overall, most methods do not produce adversarial examples that are very effective at attacking C-BERT. $^{10}$
Case study. In Table 3, we present two randomly selected adversarial examples (positive-to-negative and negative-to-positive) for which all five attacking methods successfully fool C-LSTM. TYC produces largely gibberish output. HOTFLIP tends to replace words with low semantic similarity with the original words (e.g. replacing hard with ginko), which explains its high BLEU scores and low USE and ACPT scores. Both TEXTFOOLER and  $\mathrm{AE + LS + CF + CPY}$  generate adversarial examples that are fluent and generally retain their original meanings. These observations agree with the quantitative performance we see in Table 2.
Time efficiency. Lastly, we report the time it takes for these methods to perform attacking on yelp50 at T2. The average time taken per example (on GPU v100) are: 1.2s for TYC; 1s for TEXTFOOLER; 0.3s for HOTFLIP; and 0.03s for AE+LS+CF+CPY. TYC and TEXTFOOLER are the slowest methods, while HOTFLIP is substantially faster. Our model AE+LS+CF+CPY is the fastest method: about an order of magnitude faster compared to the next best method HOTFLIP. Though one should be noted that our grey-box method re
<sup>9</sup>We tune hyper-parameters for each attacking method to achieve the 3 attacking thresholds.
10The sentiment classification accuracy for C-BERT on yelp50 is originally 97.0.
Model	C-LSTM: 96.8	C-CNN: 94.3
ACC	BLEU	USE	ACPT	TRF	ACC	BLEU	USE	ACPT	TRF
T1	TYC	83.8	48.3	11.6	-18.9	87.1	87.6	41.2	29.4	-21.8	91.5
HOTFLIP	80.3	85.6	47.9	-7.0	93.3	81.5	92.5	77.1	-3.8	95.1
TEXTFOOLER	86.5	92.6	88.7	-1.8	94.6	87.7	91.9	94.2	-2.1	96.2
AE+LS+CF+CPY	87.7	86.8	83.5	-3.8	95.0	85.1	87.8	80.7	-4.1	94.8
T2	TYC	75.3	41.2	-7.6	-20.7	78.2	73.4	38.9	-15.3	-21.4	75.9
HOTFLIP	75.3	80.0	38.1	-7.8	91.7	70.8	84.7	63.4	-7.1	93.4
TEXTFOOLER	73.6	88.5	84.1	-2.9	92.8	-	-	-	-	-
AE+LS+CF+CPY	77.1	83.5	74.6	-5.6	92.6	78.3	82.6	70.2	-5.7	92.5
T3	TYC	65.5	30.1	-7.3	-26.4	68.7	-	-	-	-	-
HOTFLIP	62.5	77.7	36.3	-9.9	91.2	67.1	81.8	57.2	-8.0	92.8
TEXTFOOLER	62.2	85.6	82.6	-3.7	91.7	-	-	-	-	-
AE+LS+CF+CPY	66.5	80.2	67.0	-7.3	90.1	69.1	76.4	61.7	-7.7	91.7
Table 2: Results based on automatic metrics, with C-LSTM and C-CNN as target classifiers. Dashed line indicates the model is unable to generate adversarial examples that meet the accuracy threshold. The numbers next to the classifiers (C-LSTM and C-CNN) are the pre-trained classification accuracy performance.
Direction	neg-to-pos
Original	unreasonable and hard to deal with! avoid when looking into a home . plenty of headaches.
TYC	homeschoolers and tantrumming to marker with ! Australasia blerg quotation into a home . plenty of headaches.
HOTFLIP	unreasonable and ginko to deal with! avoid when looking into a home . plenty of headaches.
TEXTFOOLER	unreasonable and tough to deal with! avoids when looking into a home . plenty of headaches.
AE+LS+CF+CPY	unreasonable and hard to deal with! canceling when looking into a home . plenty of headaches.
Direction	pos-to-neg
Original	i wish more business operated like this . these guys were all awesome . very organized and pro
TYC	rely the smashes gazebos slobbering americanised expiration 3.88 magan colored 100/5 bellevue des-tine 3.88 very 02/16 wonderfully whelms
HOTFLIP	i wish more business operated a this . these guys cpp all stereotypic . very provisioned and pro
TEXTFOOLER	i wish more business operated iike this . these guys were all magnificent . very organized and pro
AE+LS+CF+CPY	i wish more business operated like this . these guys were all impresses . very organized and pro
Table 3: Adversarial examples generated by different methods when attacking on yelp50 at threshold T2.
requires an additional step of training that can be conducted offline.
4.3.3 Human Evaluation
Automatic metrics provide a proxy to quantify the quality of the adversarial examples. To validate that these metrics work, we conduct a crowdsourcing experiment on Appen. $^{11}$
We test the 3 best performing models (HOTFLIP, TEXTFOOLER and  $\mathrm{AE + LS + CF + CPY}$ ) on 2 attacking thresholds (T2 and T3). For each method, we randomly sampled 25 positive-to-negative and 25 negative-to-positive successful adversarial examples. For quality control, we annotate  $10\%$  of the
samples as control questions. Workers are first presented with a 10-question quiz, and only those who pass the quiz with at least  $80\%$  accuracy can work on the task. We monitor work quality throughout the annotation process by embedding a quality-control question in every 10 questions, and stop workers from continuing on the task whenever their accuracy on the control questions fall below  $80\%$ . We restrict our jobs to workers in United States, United Kingdom, Australia, and Canada.
We ask crowdworkers the following questions:
1. Is snippet B a good paraphrase of snippet A?
$\bigcirc$  Yes  $\bigcirc$  Somewhat yes  $\bigcirc$  No
2. How natural does the text read?
$\bigcirc$  Very unnatural  $\bigcirc$  Somewhat natural
$\bigcirc$  Natural
3. What is the sentiment of the text?
$\bigcirc$  Positive  $\bigcirc$  Negative  $\bigcirc$  Cannot tell
We display both the original and adversarial examples for question 1, and only the adversarial example for question 2 and 3. As a baseline, we also select 50 random original sentences from the test set and collect human judgements for these sentences on question 2 and 3.
We present the human evaluation results in Figure 2. Looking at the original examples (top-2 bars), we see that they are fluent and their perceived sentiments (by the crowdworkers) have a high agreement with their original sentiments (by the review authors). Comparing the 3 methods, TEXTFOOLER produces adversarial sentences that are most similar to the original (green) and they are more natural (blue) than other methods. HOTFLIP is the least impressive method here, and these observations agree with the scores of automatic metrics in Table 2. On label preservation (red), however,
<sup>11</sup>https://www.appen.com
(a) Original examples
(b) ACC threshold: T2
(c) ACC threshold: T3
Figure 2: Human evaluation results.
our method  $\mathrm{AE + LS + CF + CPY}$  has the best performance, implying that the generated adversarial sentences largely preserve the original sentiments.
The consistency between the automatic and human evaluation results indicate that the USE and ACPT scores properly captured the semantic similarity and readability, two important evaluation aspects that are text-specific.
4.4 Defending Performance
Here we look at how well the generated adversarial examples can help build a more robust classifier. Unlike the attacking performance experiments (Section 4.3), here we include the augmented classifier  $(\mathcal{C}^*)$  as part of the grey-box training. $^{12}$  The augmented classifier can be seen as an improved model compared to the original classifier  $\mathcal{C}$ .
To validate the performance of adversarial defence, we evaluate the accuracy of the augmented classifiers against different attacking methods. We compared our augmented classifier  $\mathcal{C}^*$  to the augmented classifiers adversarially trained with adversarial examples generated from HOTFLIP and TEXTFOOLER. Our preliminary results show that training  $\mathcal{C}^*$  without the copy mechanism provides better defending performance, therefore we use the
C	CHOTFLIP	CTEARTFOOLER	C*
Original Perf.	96.8	96.6	96.9	97.2
TYC	75.3	69.5	73.1	76.0
HOTFLIP	75.3	61.2	80.1	97.1
TEXTFOOLER	73.6	66.4	74.5	76.5
AE+LS+CF	74.0	83.2	86.3	90.0
Table 4: Defending performance.
$\mathrm{AE} + \mathrm{LS} + \mathrm{CF}$  architecture to obtain  $\mathcal{C}^*$ .
For fair comparison, our augmented classifier  $(\mathcal{C}^*)$  is obtained by training the generator  $(\mathcal{G})$  to produce an attacking performance of T2 accuracy  $(70\%)$  on the static classifier  $(\mathcal{C})$ . For the other two methods, we train an augmented version of the classifier by feeding the original training data together with the adversarial examples  $^{13}$  generated by HOTFLIP and TEXTFOOLER with the same T2 attacking performance; these two classifiers are denoted as  $\mathcal{C}_{\mathrm{TEXTFOOLER}}$  and  $\mathcal{C}_{\mathrm{HOTFLIP}}$ , respectively.
At test time, we attack the three augmented classifiers using TYC, HOTFLIP, TEXTFOOLER and  $\mathrm{AE + LS + CF}$ , and evaluate their classification accuracy. Results are presented in Table 4. The second row "Original Perf." indicates the performance when we use the original test examples as input to the augmented classifiers. We see a high accuracy here, indicating that the augmented classifiers still perform well on the original data.
Comparing the different augmented classifiers, our augmented classifier  $\mathcal{C}^*$  outperforms the other two in defending against different adversarial attacking methods (it is particularly good against HOTFLIP). It produces the largest classification improvement compared to the original classifier  $\mathcal{C}$  (0.7, 21.8, 2.9 and 16.0 points against adversarial examples created by TYC, HOTFLIP, TEXTFOOLER and AE+LS+CF respectively). Interestingly, the augmented classifier trained with HOTFLIP adversarial examples  $(\mathcal{C}_{\mathrm{HOTFLIP}})$  produces a more vulnerable model, as it has lower accuracy compared to original classifier  $(\mathcal{C})$ . We suspect this as a result of training with low quality adversarial examples that introduce more noise during adversarial defending. Training with TEXTFOOLER examples  $(\mathcal{C}_{\mathrm{TEXTFOOLER}})$  helps, although most of its gain is in defending against other attacking methods (HOTFLIP and AE+LS+CF).
To summarise, these results demonstrate that our grey-box framework of training an augmented classifier together with a generator produces a more
12During training, we perform one attacking step for every two defending steps.
13one per each training example
robust classifier, compared to the baseline approach of training a classifier using data augmented by adversarial examples.
In this paper, we proposed a grey-box adversarial attack and defence framework for sentiment classification. Our framework combines a generator with two copies of the target classifier: a static and an updated model. Once trained, the generator can be used for generating adversarial examples, while the augmented (updated) copy of the classifier is an improved model that is less susceptible to adversarial attacks. Our results demonstrate that the generator is capable of producing high-quality adversarial examples that preserve the original ground truth and is approximately an order of magnitude faster in creating adversarial examples compared to state-of-the-art attacking methods. Our framework of building an improved classifier together with an attacking generator is also shown to be more effective than the baseline approach of training a classifier using data augmented by adversarial examples.
The combined adversarial attack and defence framework, though only evaluated on sentiment classification, should be adapted easily to other NLP problems (except for the counter-fitted embeddings, which is designed for sentiment analysis). This framework makes it possible to train adversarial attacking models and defending models simultaneously for NLP tasks in an adversarial manner.
For the human evaluation in Section 4.3.3, each assignment was paid  $0.06 and estimated to take 30 seconds to complete, which gives an hourly wage of$ 7.25 (= US federal minimum wage). An assignment refers to scoring the sentiment/coherence of a sentence, or scoring the semantic similarity of a pair of sentences.
Our research has obvious ethical considerations, in that our adversarial generation technology can be extended and used to attack NLP systems at large. That said, this concern is a general concern for any forms of adversarial learning that isn't unique to our research. The general argument for furthering research in adversarial learning is that it advances our understanding of the vulnerabilities of machine learning models, paving the path towards building safer and more secure models.
Additionally, our grey-box framework is arguably better for defense (i.e. improving a machine learning model) than for offense (i.e. attacking a machine learning model), as it requires access to the architecture of the target model to learn how to generate adversarial examples, which isn't a realistic condition if we were to use it to attack a live system. In contrast, such a condition is less of an issue if we are using it to improve the robustness of a system that we are developing.
Moustafa Alzantot, Bharathan Balaji, and Mani Srivastava. 2018a. Did you hear that? adversarial examples against automatic speech recognition. arXiv preprint arXiv:1801.00554.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018b. Generating natural language adversarial examples. arXiv preprint arXiv:1804.07998.
Daniel Cer, Yinfei Yang, Sheng yi Kong, Nan Hua, Nicole Limtiaco, Rhomni St. John, Noah Constant, Mario Guajardo-Cespedes, Steve Yuan, Chris Tar, Yun-Hsuan Sung, Brian Strope, and Ray Kurzweil. 2018. Universal sentence encoder.
Pin-Yu Chen, Huan Zhang, Yash Sharma, Jinfeng Yi, and Cho-Jui Hsieh. 2017. Zoo: Zeroth order optimization based black-box attacks to deep neural networks without training substitute models. In Proceedings of the 10th ACM Workshop on AI and Security, pages 15-26. ACM.
Jacob Devlin, M.W Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding.
Yinpeng Dong, Fangzhou Liao, Tianyu Pang, Hang Su, Jun Zhu, Xiaolin Hu, and Jianguo Li. 2018. Boosting adversarial attacks with momentum. In Proceedings of the IEEE CVPR, pages 9185-9193.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and De- jing Dou. 2017. Hotflip: White-box adversarial examples for text classification. arXiv preprint arXiv:1712.06751.
Zhitao Gong, Wenlu Wang, Bo Li, Dawn Song, and Wei-Shinn Ku. 2018. Adversarial texts with gradient methods. arXiv preprint arXiv:1801.07175.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2014. Explaining and harnessing adversarial examples. arXiv preprint arXiv:1412.6572.
Po-Sen Huang, Robert Stanforth, Johannes Welbl, Chris Dyer, Dani Yogatama, Sven Gowal, Krishnamurthy Dvijotham, and Pushmeet Kohli. 2019.
Achieving verified robustness to symbol substitutions via interval bound propagation. arXiv preprint arXiv:1909.01492.
Eric Jang, Shixiang Gu, and Ben Poole. 2016. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144.
Robin Jia, Aditi Raghunathan, Kerem Goksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. arXiv preprint arXiv:1909.00986.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust. A Strong Baseline for Natural Language Attack on Text Classification and Entailment. arXiv e-prints, page.
Yoon Kim. 2014. Convolutional neural networks for sentence classification. arXiv preprint arXiv:1408.5882.
Alexey Kurakin, Ian Goodfellow, and Samy Bengio. 2016. Adversarial examples in the physical world. arXiv preprint arXiv:1607.02533.
Jey Han Lau, Carlos S. Armendariz, Shalom Lappin, Matthew Purver, and Chang Shu. 2020. How furiously can colourless green ideas sleep? sentence acceptability in context. arXiv e-prints, page arXiv:2004.00881.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. Deepfool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE CVPR, pages 2574-2582.
John X Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020a. Reevaluating adversarial examples in natural language. arXiv preprint arXiv:2004.14174.
John X Morris, Eli Lifland, Jin Yong Yoo, and Yanjun Qi. 2020b. Textattack: A framework for adversarial attacks in natural language processing. arXiv preprint arXiv:2005.05909.
Nikola Mrksic, Diarmuid O Seaghda, Blaise Thomson, Milica Gašić, Lina Rojas-Barahona, PeiHao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. arXiv preprint arXiv:1603.00892.
Rafael Müller, Simon Kornblith, and Geoffrey E Hinton. 2019. When does label smoothing help? In NeurIPS, pages 4696-4705.
Nicolas Papernot, Patrick McDaniel, Ian Goodfellow, Somesh Jha, Z Berkay Celik, and Ananthram Swami. 2017. Practical black-box attacks against machine learning. In Proceedings of the 2017 ACM ASI-ACCS, pages 506-519.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th ACL, pages 311-318.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. Glove: Global vectors for word representation. In Proceedings of the EMNLP (2014), pages 1532-1543.
Yi-Ting Tsai, Min-Chu Yang, and Han-Yu Chen. 2019. Adversarial attack on sentiment classification. In Proceedings of the 2019 ACL Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, pages 233-240.
Yequan Wang, Minlie Huang, Li Zhao, et al. 2016. Attention-based LSTM for aspect-level sentiment classification. In Proceedings of the 2016 conference on EMNLP, pages 606-615.
John Wieting and Kevin Gimpel. 2017. *Paranmt-50m: Pushing the limits of paraphrastic sentence embeddings with millions of machine translations.* arXiv preprint arXiv:1711.05732.
Chaowei Xiao, Bo Li, Jun-Yan Zhu, Warren He, Mingyan Liu, and Dawn Song. 2018. Generating adversarial examples with adversarial networks. arXiv preprint arXiv:1801.02610.
Ying Xu, Xu Zhong, Antonio Jimeno Yepes, and Jey Han Lau. 2020. Elephant in the room: An evaluation framework for assessing adversarial examples in nlp.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020. Adversarial attacks on deep-learning models in natural language processing: A survey. ACM TIST, 11(3):1-41.
Yi Zhou, Xiaqing Zheng, Cho-Jui Hsieh, Kai-wei Chang, and Xuanjing Huang. 2020. Defense against adversarial attacks in nlp via dirichlet neighborhood ensemble. arXiv preprint arXiv:2006.11627.