SlowGuess/NAACL_2022_OCR huggingface dataset:
dictated by the PWWS attack algorithm described in Section 3; or a targeted universal concatenation (con) attack (Equation 9), used for the regression task on the L-Bus dataset, seeking to maximise the average score output from the system by appending the same  $N$  words to the end of each input. For classification tasks, the impact of the adversarial attack is measured using the fooling rate, the fraction of originally correctly classified points, misclassified after the attack, whilst for the regression task, the impact is measured as the average increase in the output score.
Dataset	Attack	N	Impact
IMDB	sub	25	Fool: 0.70
Twitter	sub	6	Fool: 0.77
AG News	sub	40	Fool: 0.65
DBpedia	sub	25	Fool: 0.52
L-Bus	con	3	Score: +0.51
Section 4.1 predicts that adversarial attacks in the discrete text space leave residue in a system's encoder embedding space that can be detected using a simple linear classifier. Hence, using the 12-layer Transformer encoder's output CLS token embedding as the encoder embedding space for each dataset's trained system (Table 2), a simple linear classifier, as given in Equation 10, was trained  ${}^{3}$  to detect adversarial examples from the adversarial attacks given for each dataset in Table 3. The training of the detection linear classifier was performed on the training data (Table 1) augmented with an equivalent adversarial example for each original input sample in the dataset. Using the test data samples augmented with adversarial examples (as defined by Table 3), Table 4 compares the efficacy of the linear residue detector to other popular detection strategies  ${}^{4}$  (from Section 4) using the best  ${\mathrm{F}}_{1}$  score. It is evident from the high F-scores,that for most NLP tasks the linear detection approach is better than other state of the art NLP specific and ported image detection approaches.
Table 3: Impact of different  $N$  -word adversarial attacks.
Dataset	Res	Perp	FGWS	MD	Unc
IMDB	0.91	0.68	0.87	0.67	0.75
Twitter	0.84	0.67	0.76	0.67	0.78
AG News	0.95	0.69	0.89	0.68	0.75
DBpedia	0.80	0.67	0.82	0.68	0.90
L-Bus	0.99	0.68	0.91	n/a	0.81
However, an adversary may have knowledge of the detection approach and may attempt to design an attack that directly avoids detection. Hence, for each dataset, the attack approaches were repeated with the added constraint that any attack words that resulted in detection were rejected. The impact of attacks that suppress detection have been presented in Table 5. Generally, it is shown across all NLP tasks that an adversary that attempts to avoid detection of its residue by a previously trained linear classifier, can only generate a significantly less powerful adversarial attack.
Table 4:  ${\mathrm{F}}_{1}$  -score performance of detection approaches.
Dataset	Without	With
IMDB	0.70	0.19
Twitter	0.77	0.23
AG News	0.65	0.16
DBpedia	0.52	0.14
L-Bus	+0.51	+0.23
Table 5: Fooling rate (classification) or score (regression) with and without attack modified to avoid detection.
The aim of this section is to verify that the success of the residue detector can be explained by the two main hypotheses made in Section 4.1. The claim that residue is left by adversarial samples in the central PCA eigenvector components is explored first. For each NLP task a PCA projection matrix is learnt in the encoder embedding space using the original training data samples (Table 1). Using the test data, the residue in the embedding space can be visualized through a plot of the average (across the data) component,  $\rho_{i} = \frac{1}{J}\sum_{j = 1}^{J}\rho_{i,j}$  in each eigenvector direction,  $\mathbf{q}_i$  of the original and attacked data, where
$$
\rho_ {i, j} = \left| \mathcal {F} _ {\mathrm {e n}} \left(\mathbf {x} _ {j}\right) ^ {\mathrm {T}} \mathbf {q} _ {i} \right|, \tag {11}
$$
with  $\mathbf{x}_j$  being the  $j$ th data point. Figure 1 shows an example plot for the Twitter dataset, where  $\rho_i$  is plotted against the eigenvalue rank,  $i$  for the original and attacked data examples. Residue plots for
$^{3}\mathrm{lr} = 0.02$  , epochs  $= 20$  , batch size  $= 200$  #769 parameters Detection Strategies: Mahalanobis Distance (MD) used the same train-test split as the residue approach; Perplexity (Perp) was calculated using the language model from Chen et al. (2016); Uncertainty (Unc) used the best measure out of mutual information, confidence, KL-divergence, expected entropy and entropy of expected and reverse mutual information; and FGWS was implemented using the code given at https://github.com/maximilianmoses/fgws.
other datasets are included in Appendix A. Next, it is necessary to verify that the residue detector specifically uses the residue in the central eigenvector components to distinguish between original and adversarial samples. To establish this, each encoder embedding,  $\mathcal{F}_{\mathrm{en}}(x)$ 's components not within a target subspace of PCA eigenvector directions  $\{\mathbf{q}_i\}_{i=p}^{p+w}$ , are removed, i.e. we have a projected embedding,  $\mathbf{x}^{(\mathrm{p})} = F_{\mathrm{en}}(x) - \sum_{i \notin [p,p+w)} \mathbf{q}_i^T F_{\mathrm{en}}(x) \mathbf{q}_i$ , where  $w$  is a window size to choose. Now, using  $\mathcal{F}_{\mathrm{c1}}(\mathbf{x}^{(\mathrm{p})})$  and a residue detector trained using the modified embeddings,  $\mathbf{x}^{(\mathrm{p})}$ , the classifier's  $(\mathcal{F}_{\mathrm{c1}}(\mathbf{x}^{(\mathrm{p})}))$  accuracy and detector performance (measured using F1 score) can be found. Figure 2 shows the performance of the classifier  $(\mathcal{F}_{\mathrm{c1}}(\mathbf{x}^{(\mathrm{p})}))$  and the detector for different start components  $p$  with the window size,  $w = 5$ . It is clear that the principal components hold the most important information for classifier accuracy, but, as hypothesised in Section 4.1, it is the more central eigenvector components that hold the most information useful for the residue detector, i.e. the subspace defined by  $\{\mathbf{q}_i\}_{i=5}^{10}$  holds the most detectable residue from adversarial examples.
Figure 1: Encoder Embedding Space Residue Plot.
The second hypothesis in Section 4.1 claims that the existence of residue in the central eigenvector components is due to the discrete nature of NLP adversarial attacks. Hence, to analyze the impact of the discrete aspect of the attack, an artificial continuous space attack was constructed for the Twitter NLP system, where the continuous input embedding layer space (Equation 5) of the system is the space in which the attack is performed. Using the Twitter emotion classifier, a PGD (Equation 3) attack was performed on the input embeddings for each token, where the perturbation size was
Figure 2: Performance of classifier and detector with windowed projection of encoder embedding space.
limited to be  $\epsilon = 0.1$  in the  $l_{\infty}$  norm, achieving a fooling rate of 0.73. Note that this form of attack is artificial, as a real adversary can only modify the discrete word sequence (Equation 4). To compare the influence of discrete and continuous attacks on the same system, the average (across dataset)  $l_{2}$  and  $l_{\infty}$  norms of the perturbations in the input layer embedding space were found. Further, a single value summary,  $N_{\sigma}$ , of the residue plot (e.g. Figure 1), was calculated for each attack.  $N_{\sigma}$  is the average difference in standard deviations between the original component mean,  $\rho_{i}^{(\mathrm{orig})}$  and attack mean,  $\rho_{i}^{(\mathrm{attack})}$ ,
$$
N _ {\sigma} = \frac {1}{I} \sum_ {i = 1} ^ {I} \frac {\left| \rho_ {i} ^ {(\text {a t t a c k})} - \rho_ {i} ^ {(\text {o r i g})} \right|}{\sqrt {\operatorname {V a r} _ {j} \left[ \rho_ {i , j} ^ {(\text {o r i g})} \right]}}. \tag {12}
$$
Table 6 reports these metrics for the discrete and artificial continuous NLP adversarial attacks on the Twitter system  ${}^{5}$  . It is apparent that perturbation sizes for the discrete attacks are significantly larger. Moreover,  ${N}_{\sigma }$  is significantly smaller for the continuous space attack, indicating that the residue left by continuous space adversarial attacks is smaller.
Attack	Nσ	l2error	l∞error
Discrete	1.201	50.2±19.2	3.26±0.86
Continuous	0.676	5.35±3.95	0.08±0.03
Table 6: Comparison of token level discrete attack and input embedding layer continuous PGD attack.
To explicitly observe the impact of the nature of data on detectors, adversarial attacks are considered in four domains: the discrete, sequential
Similar trends were found across all datasets (Table A.2)
NLP input space (NLP-disc); the artificial continuous, sequential embedding space of an NLP model (NLP-cont); the continuous, static image input space (Img-cont) and a forced discretised, static image input space (Img-disc). For the NLP-disc and NLP-cont the same attacks as in Table 6 are used. For the continuous image domain (Img-cont), a VGG-16 architecture image classifier trained on CIFAR-100 (Krizhevsky et al., 2009) image data (achieving a top-5 accuracy of  $90.1\%$ ) and attacked using a standard  $\mathbf{l}_{\infty}$  PGD approach (Equation 3) is used. For the discrete image domain (Img-disc), the CIFAR-100 images,  $\mathbf{X} \in \mathbb{Z}_{256}^{R \times R}$  were discretised using function  $\mathcal{Q}_q: \mathbb{Z}_{256}^{R \times R} \to \mathbb{Z}_q^{R \times R}$ , where  $\mathbb{Z}_q = \{0, 1\frac{256}{q - 1}, 2\frac{255}{q - 1}, \ldots, 255\}$ . In this work 2-bit quantization was used, i.e.  $q = 4$ . With this quantization, a VGG-16 architecture was trained to achieve  $78.2\%$  top-5 accuracy. To perform a discrete space attack, a variant of the PWWS synonym substitution attack (Section 3) was implemented, where synonyms were interpreted as closest permitted quantisation values and  $N$  pixel values were substituted. For these different domains, Table 7 compares applicable detection approaches (certain NLP detection approaches are not valid outside the word sequence space) using the best F<sub>1</sub> score, where different attack perturbation sizes are considered ( $N$  substitutions for discrete attacks and for continuous attacks  $|\delta| \leq \epsilon$  for perturbation  $\delta$ ).
Domain	Attack	Res	Unc	MD
NLP-disc	N=3	0.80	0.74	0.67
N=6	0.84	0.78	0.67
NLP-cont	ε=0.1	0.67	0.71	0.68
ε=0.3	0.67	0.80	0.85
Img-disc	N=200	0.78	0.67	0.70
N=400	0.84	0.68	0.72
Img-cont	ε=12	0.68	0.70	0.72
ε=48	0.83	0.81	0.87
Table 7: Portability of detection approaches.
In the discrete domains, the residue detection approach is better than all the other approaches. However, in the continuous data type domains, the Mahalanobis Distance dominates as the detection approach, with the residue detection approach performing the worst. As predicted by the second hypothesis of Section 4.1, the lack of success of the residue detection approach is expected here - the residue detection approach is only successful for discrete space attacks.
To verify that the residue detection approach is agnostic to the type of attack, the residue detector trained on substitution attack examples was evaluated on concatenation attack examples. Using the Twitter dataset, a  $N = 3$  concatenation attack was applied, achieving a fooling rate of 0.59. In this setting, the residue detector (trained on the  $N = 6$  substitution adversarial examples) achieved a  $\mathrm{F_1}$  score of 0.81, which is comparable to the original score of 0.84 (from Table 4). This shows that even with different attack approaches similar forms of residue are produced, meaning a residue detector can be used even without knowledge of the type of adversarial attack.
In recent years, deep learning systems have been deployed for a large number of tasks, ranging from the image to the natural language domain. However, small, imperceptible adversarial perturbations at the input, have been found to easily fool these systems, compromising their validity in high-stakes applications. Defence strategies for deep learning systems have been extensively researched, but this research has been predominantly carried out for systems operating in the image domain. As a result, the adversarial detection strategies developed, are inherently tuned to attacks on the continuous space of images. This work shows that these detection strategies do not necessarily transfer well to attacks on natural language processing systems. Hence, an adversarial attack detection approach is proposed that specifically exploits the discrete nature of perturbations for attacks on discrete sequential inputs.
The proposed approach, termed residue detection, demonstrates that imperceptible attack perturbations on natural language inputs tend to result in large perturbations in word embedding spaces, which result in distinctive residual components. These residual components can be identified using a simple linear classifier. This residue detection approach was found to out-perform both detection approaches ported from the image domain and other state of the art NLP specific detectors.
The key finding in this work is that the nature of the data (e.g. discrete or continuous) strongly influences the success of detection systems and hence it is important to consider the domain when designing defence strategies.
7 Limitations, Risks and Ethics
A limitation of the residue approach proposed in this work is that it requires training on adversarial examples, which is not necessary for other NLP detectors. This means there is a greater computational cost associated with this detector. Moreover, associated with this limitation is a small risk, where in process of generating creative adversarial examples to build a robust residue detector, the attack generation scheme may be so strong that it can more easily evade detection from other existing detectors already deployed in industry. There are no further ethical concerns related to this detector.
This paper reports on research supported by Cambridge Assessment, University of Cambridge. Thanks to Cambridge English Language Assessment for support and access to the Linguuaskill-Business data. The authors would also like to thank members of the ALTA Speech Team.
Siddhant Bhambri, Sumanyu Muku, Avinash Tulasi, and Arun Balaji Buduru. 2019. A study of black box adversarial attacks in computer vision. CoRR, abs/1912.01667.
Battista Biggio and Fabio Roli. 2017. Wild patterns: Ten years after the rise of adversarial machine learning. CoRR, abs/1712.03141.
Matthias Blohm, Gloriana Jagfeld, Ekta Sood, Xiang Yu, and Ngoc Thang Vu. 2018. Comparing attention-based convolutional and recurrent neural networks: Success and limitations in machine reading comprehension. CoRR, abs/1808.08744.
Fabio Carrara, Rudy Becarelli, Roberto Caldelli, Fabrizio Falchi, and Giuseppe Amato. 2019a. Adversarial examples detection in features distance spaces. In Computer Vision - ECCV 2018 Workshops, pages 313-327, Cham. Springer International Publishing.
Fabio Carrara, Rudy Becarelli, Roberto Caldelli, Fabrizio Falchi, and Giuseppe Amato. 2019b. Adversarial Examples Detection in Features Distance Spaces: Subvolume B, pages 313-327. ECCV.
Lucy Chambers and Kate Ingham. 2011. The BULATS online speaking test. Research Notes, 43:21-25.
X. Chen, X. Liu, Y. Qian, M. J. F. Gales, and P. C. Woodland. 2016. Cued-rnnlm — an open-source toolkit for efficient training and evaluation of recurrent neural network language models. In 2016 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 6000-6004.
Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. CoRR, abs/1803.01128.
Kevin Clark, Minh-Thang Luong, Quoc V. Le, and Christopher D. Manning. 2020. ELECTRA: pretraining text encoders as discriminators rather than generators. CoRR, abs/2003.10555.
Gilad Cohen, Guillermo Sapiro, and Raja Giryes. 2019. Detecting adversarial samples using influence functions and nearest neighbors. CoRR, abs/1909.06872.
Council of Europe. 2001. Common European Framework of Reference for Languages: Learning, Teaching, Assessment. Cambridge University Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, abs/1810.04805.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding.
Reuben Feinman, Ryan R. Curtin, Saurabh Shintre, and Andrew B. Gardner. 2017. Detecting adversarial samples from artifacts.
Y. Gal. 2016. Uncertainty in deep learning.
Yarin Gal and Zoubin Ghahramani. 2016. Dropout as a bayesian approximation: Representing model uncertainty in deep learning.
Ian Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In International Conference on Learning Representations.
Kathrin Grosse, Nicolas Papernot, Praveen Manoharan, Michael Backes, and Patrick D. McDaniel. 2016. Adversarial perturbations against deep neural networks for malware classification. CoRR, abs/1606.04435.
Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. 2020. Adversarial attack and defense of structured prediction models.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2015. Deep residual learning for image recognition. CoRR, abs/1512.03385.
Alex Huang, Abdullah Al-Dujaili, Erik Hemberg, and Una-May O'Reilly. 2018. Adversarial deep learning for robust detection of binary encoded malware. CoRR, abs/1801.02950.
Aminul Islam and Diana Inkpen. 2009. Real-word spelling correction using google web it 3-grams. In Proceedings of the 2009 Conference on Empirical Methods in Natural Language Processing: Volume 3 - Volume 3, EMNLP '09, page 1241-1249, USA. Association for Computational Linguistics.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. CoRR, abs/1707.07328.
Alex Krizhevsky, Vinod Nair, and Geoffrey Hinton. 2009. Cifar-100 (canadian institute for advanced research). cifar.
Alexey Kurakin, Ian J. Goodfellow, and Samy Bengio. 2016. Adversarial machine learning at scale. CoRR, abs/1611.01236.
Kimin Lee, Kibok Lee, Honglak Lee, and Jinwoo Shin. 2018. A simple unified framework for detecting out-of-distribution samples and adversarial attacks.
Christian Leibig, Vaneeda Allken, M. Ayhan, Philipp Berens, and S. Wahl. 2017. Leveraging uncertainty information from deep neural networks for disease detection. Scientific Reports, 7.
Jinfeng Li, Shouling Ji, Tianyu Du, Bo Li, and Ting Wang. 2018. Textbugger: Generating adversarial text against real-world applications. CoRR, abs/1812.05271.
Xin Li and Fuxin Li. 2016. Adversarial examples detection in deep networks with convolutional filter statistics. CoRR, abs/1612.07767.
Tsung-Yi Lin, Michael Maire, Serge J. Belongie, Lubomir D. Bourdev, Ross B. Girshick, James Hays, Pietro Perona, Deva Ramanan, Piotr Dólár, and C. Lawrence Zitnick. 2014. Microsoft COCO: common objects in context. CoRR, abs/1405.0312.
Xingjun Ma, Bo Li, Yisen Wang, Sarah M. Erfani, Sudanthi N. R. Wijewickrema, Michael E. Houle, Grant Schoenebeck, Dawn Song, and James Bailey. 2018. Characterizing adversarial subspaces using local intrinsic dimensionality. CoRR, abs/1801.02613.
Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. Learning word vectors for sentiment analysis. In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142-150, Portland, Oregon, USA. Association for Computational Linguistics.
Xiaofeng Mao, Yuefeng Chen, Yuhong Li, Yuan He, and Hui Xue. 2019. Learning to characterize adversarial subspaces.
Eric Mays, Fred J. Damerau, and Robert L. Mercer. 1991. Context based spelling correction. Information Processing & Management, 27(5):517-522.
Pasquale Minervini and Sebastian Riedel. 2018. Adversarily regularising neural NLI models to integrate logical background knowledge. CoRR, abs/1808.08609.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, Omar Fawzi, and Pascal Frossard. 2016. Universal adversarial perturbations. CoRR, abs/1610.08401.
Maximilian Mozes, Pontus Stenetorp, Bennett Kleinberg, and Lewis D. Griffin. 2020. Frequency-guided word substitutions for detecting textual adversarial examples. CoRR, abs/2004.05887.
Paarth Neekhara, Shehzeen Hussain, Shlomo Dubnov, and Farinaz Koushanfar. 2018. Adversarial reprogramming of sequence classification neural networks. CoRR, abs/1809.01829.
Tong Niu and Mohit Bansal. 2018. Adversarial oversensitivity and over-stability strategies for dialogue models. CoRR, abs/1809.02079.
Nicolas Papernot, Patrick D. McDaniel, Ananthram Swami, and Richard E. Harang. 2016. Crafting adversarial input sequences for recurrent neural networks. CoRR, abs/1604.08275.
Vyas Raina, Mark J.F. Gales, and Kate M. Knill. 2020. Universal Adversarial Attacks on Spoken Language Assessment Systems. In Proc. Interspeech 2020, pages 3855-3859.
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In ACL (1), pages 1085-1097.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Ishai Rosenberg, Asaf Shabtai, Lior Rokach, and Yuval Elovici. 2017. Generic black-box end-to-end attack against rnns and other API calls based malware classifiers. CoRR, abs/1707.05970.
Keisuke Sakaguchi, Matt Post, and Benjamin Van Durme. 2017. Grammatical error correction with neural reinforcement learning. CoRR, abs/1707.00299.
Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. CoRR, abs/1707.02812.
Elvis Saravia, Hsien-Chi Toby Liu, Yen-Hao Huang, Junlin Wu, and Yi-Shin Chen. 2018. CARER: Contextualized affect representations for emotion recognition. In Proceedings of the 2018 Conference on
Empirical Methods in Natural Language Processing, pages 3687-3697, Brussels, Belgium. Association for Computational Linguistics.
Alex Serban, Erik Poll, and Joost Visser. 2020. Adversarial examples on object recognition: A comprehensive survey. CoRR, abs/2008.04094.
Lewis Smith and Yarin Gal. 2018. Understanding measures of uncertainty for adversarial example detection.
Mengying Sun, Fengyi Tang, Jinfeng Yi, Fei Wang, and Jiayu Zhou. 2018. Identify susceptible locations in medical records via adversarial attacks on deep predictive models. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery &amp; Data Mining, KDD '18, page 793-801, New York, NY, USA. Association for Computing Machinery.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In International Conference on Learning Representations.
Yicheng Wang and Mohit Bansal. 2018. Robust machine comprehension models via adversarial training. CoRR, abs/1804.06473.
Jin Yong Yoo and Yanjun Qi. 2021. Towards improving adversarial training of nlp models.
Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In Advances in Neural Information Processing Systems, volume 28. Curran Associates, Inc.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2017. Generating natural adversarial examples. CoRR, abs/1710.11342.
Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019. Learning to discriminate perturbations for blocking adversarial attacks in text classification. CoRR, abs/1909.03084.
Training Details