SlowGuess/NAACL_2022_OCR huggingface dataset:
Vyas Raina
ALTA Institute, Cambridge University
vr313@cam.ac.uk
Mark Gales
mjfg@cam.ac.uk
Deep learning based systems are susceptible to adversarial attacks, where a small, imperceptible change at the input alters the model prediction. However, to date the majority of the approaches to detect these attacks have been designed for image processing systems. Many popular image adversarial detection approaches are able to identify adversarial examples from embedding feature spaces, whilst in the NLP domain existing state of the art detection approaches solely focus on input text features, without consideration of model embedding spaces. This work examines what differences result when porting these image designed strategies to Natural Language Processing (NLP) tasks - these detectors are found to not port over well. This is expected as NLP systems have a very different form of input: discrete and sequential in nature, rather than the continuous and fixed size inputs for images. As an equivalent model-focused NLP detection approach, this work proposes a simple sentence-embedding "residue" based detector to identify adversarial examples. On many tasks, it outperforms ported image domain detectors and recent state of the art NLP specific detectors<sup>1</sup>.
In the last decade deep learning based models have demonstrated success in a wide range of application areas, including Natural Language Processing (NLP) (Vaswani et al., 2017) and object recognition (He et al., 2015). These systems may be deployed in mission critical situations, where there is the requirement for a high level of robustness. However, Szegedy et al. (2014) demonstrated that deep models have an inherent weakness: small perturbations in the input can yield significant, undesired, changes in the output from the model. These input perturbations were termed adversarial examples and their generation adversarial attacks.
Adversarial attacks have been developed for systems operating in various domains: image systems (Serban et al., 2020; Biggio and Roli, 2017; Bhambri et al., 2019) and NLP systems (Lin et al., 2014; Samanta and Mehta, 2017; Rosenberg et al., 2017). The characteristics of the input can be very different between these application domains. Broadly, the nature of inputs can be described using two key attributes: static (fixed length) vs sequential and continuous vs discrete. Under this categorisation, image inputs are continuous and static, whilst NLP inputs are discrete and sequential. This work argues that due to the fundamental differences in the input and resulting adversarial perturbations in the different domains, adversarial attack behaviour can vary significantly from one domain to another. Hence, the extensive research on exploring and understanding adversarial perturbation behaviour in the continuous, static world of image systems does not necessarily transfer well to the NLP tasks.
For adversarial attack generation, a number of specific NLP attacks have been proposed that are designed for NLP task inputs (Lin et al., 2014; Samanta and Mehta, 2017; Rosenberg et al., 2017; Huang et al., 2018; Papernot et al., 2016; Grosse et al., 2016; Sun et al., 2018; Cheng et al., 2018; Blohm et al., 2018; Neekhara et al., 2018; Raina et al., 2020; Jia and Liang, 2017; Minervini and Riedel, 2018; Niu and Bansal, 2018; Ribeiro et al., 2018; Iyyer et al., 2018; Zhao et al., 2017). However, there has been less research on developing defence schemes. These defence strategies can be split into two main groups: model modification, where the model or data is altered at training time (e.g. adversarial training (Yoo and Qi, 2021)) and detection, where external systems or algorithms are applied to trained models to identify adversarial attacks. As model modification approaches demand re-training of models, detection approaches are usually considered easier for implementation on de
$^{1}$ Code is available at: https://github.com/rainavyas/NAACL-2022-Residue-Detector
Human Language Technologies, pages 3836 - 3848
ployed systems and thus are often preferred. Hence, this work investigates the portability of popular detection approaches designed for image systems to NLP systems. Furthermore, this work introduces a specific NLP detection approach that exploits the discrete nature of the inputs for NLP systems. This approach out-performs standard schemes designed for image adversarial attack detection, as well as other NLP detection schemes.
The proposed NLP specific detection approach will be referred to as residue detection, as it is shown that adversarial attacks in the discrete, word sequence space result in easily detectable residual components in the sentence embedding space. This residue can be easily detected using a simple linear classifier operating in the encoder embedding space. In addition, this work shows that even when an adversary has knowledge of the linear residual detector, they can only construct attacks at a fraction of the original strength. Hence this work argues that realistic (word level, semantically similar) adversarial perturbations at the natural language input of NLP systems leave behind easily detectable residue in the sentence embedding. Interestingly, the residue detection approach is shown to perform poorly when used to detect attacks in the image domain, supporting the hypothesis that the nature of the input has an important influence on the design of effective defence strategies.
Previous work in the image domain has analysed the output of specific layers in an attempt to identify adversarial examples or adversarial subspaces. First, Feinman et al. (2017) proposed that adversarial subspaces have a lower probability density, motivating the use of the Kernel Density (KD) metric to detect the adversarial examples. Nevertheless, Ma et al. (2018) found Local Intrinsic Dimensionality (LID) was a better metric in defining the subspace for more complex data. In contrast to the local subspace focused approaches of KD and LID, Carrara et al. (2019b) showed that trajectories of hidden layer features can be used to train a LSTM network to accurately discriminate between authentic and adversarial examples. Out performing all previous methods, Lee et al. (2018) introduced an effective detection framework using Mahalanobis Distance Analysis (MDA), where the distance is calculated between a test sample and the closest class-conditional Gaussian distribution in the space
defined by the output of the final layer of the classifier (logit space). Li and Li (2016) also explored using the output of convolutional layers for image classification systems to identify statistics that distinguish adversarial samples from original samples. They find that by performing a PCA decomposition the statistical variation in the least principal directions is the most significant and can be used to separate original and adversarial samples. However, they argue this is ineffective as an adversary can easily suppress the tail distribution. Hence, Li and Li (2016) extract statistics from the convolutional layer output to train a cascade classifier to separate the original and adversarial samples. Most recently, Mao et al. (2019) avoid the use of artificially designed metrics and combine the adversarial subspace identification stage and the detecting adversaries stage into a single framework, where a parametric model adaptively learns the deep features for detecting adversaries.
In contrast to the embedding space detection approaches, Cohen et al. (2019) shows that influence functions combined with Nearest Neighbour distances perform comparably or better than the above standard detection approaches. Other detection approaches have explored the use of uncertainty: Smith and Gal (2018) argues that adversarial examples are out of distribution and do not lie on the manifold of real data. Hence, a discriminative Bayesian model's epistemic (model) uncertainty should be high. Therefore, calculations of the model uncertainty are thought to be useful in detecting adversarial examples, independent of the domain. However, Bayesian approaches aren't always practical in implementation and thus many different approaches to approximate this uncertainty have been suggested in literature (Leibig et al., 2017; Gal, 2016; Gal and Ghahramani, 2016).
There are a number of existing NLP specific detection approaches. For character level attacks, detection approaches have exploited the grammatical (Sakaguchi et al., 2017) and spelling (Mays et al., 1991; Islam and Inkpen, 2009) inconsistencies to identify and detect the adversarial samples. However, these character level attacks are unlikely to be employed in practice due to the simplicity with which they can be detected. Therefore, detection approaches for the more difficult semantically similar attack samples are of greater interest, where the meaning of the textual input is maintained without compromising the spelling or grammatical in-
3837
tegrity. To tackle such word-level, semantically similar examples, Zhou et al. (2019) designed a discriminator to classify each token representation as part of an adversarial perturbation or not, which is then used to 'correct' the perturbation. Other detection approaches (Raina et al., 2020; Han et al., 2020; Minervini and Riedel, 2018) have shown some success in using perplexity to identify adversarial textual examples. Most recently, Mozes et al. (2020) achieved state of the art performance with the Frequency Guided Word Substitution (FGWS) detector, where a change in model prediction after substituting out low frequency words is revealing of adversarial samples.
3 Adversarial Attacks
An adversarial attack is defined as an imperceptible change to the input that causes an undesired change in the output of a system. Often, an attack is found for a specific data point,  $\mathbf{x}$ . Consider a classifier  $\mathcal{F}_{\hat{\theta}}$  with parameters  $\hat{\theta}$ , that predicts a class label for an input data point,  $\mathbf{x}$ , sampled from the input distribution  $\mathcal{X}$ . A successful adversarial attack is where a perturbation  $\delta$  at the input causes the system to miss-classify,
$$
\mathcal {F} _ {\hat {\theta}} (\mathbf {x} + \boldsymbol {\delta}) \neq \mathcal {F} _ {\hat {\theta}} (\mathbf {x}). \tag {1}
$$
When defining adversarial attacks, it is important consider the interpretation of an imperceptible change. Adversarial perturbations are not considered effective if they are easy to detect. Hence, the size of the perturbation must be constrained:
$$
\mathcal {G} (\mathbf {x}, \mathbf {x} + \boldsymbol {\delta}) \leq \epsilon , \tag {2}
$$
where the function  $\mathcal{G}()$  describes the form of constraint and  $\epsilon$  is a selected threshold of imperceptibility. Typically, when considering continuous space inputs (such as images), a popular form of the constraint of Equation 2, is to limit the perturbation in the  $l_{p}$  norm, with  $p\in [1,\infty)$ , e.g.  $||\delta ||_p\leq \epsilon$ .
For whitebox attacks in the image domain, the dominant attack approach has proven to be Projected Gradient Descent (PGD) (Kurakin et al., 2016). The PGD approach, iteratively updates the adversarial perturbation,  $\delta$ , initialised as  $\delta_0 = 0$ . Each iterative step moves the perturbation in the direction that maximises the loss function,  $\mathcal{L}$ , used in the training of the model,
$$
\boldsymbol {\delta} _ {i + 1} = \operatorname {c l i p} _ {\epsilon} \left(\boldsymbol {\delta} _ {i} + \alpha \nabla_ {\boldsymbol {\delta} _ {i}} \mathcal {L} (\mathbf {x} + \boldsymbol {\delta} _ {i}; \hat {\theta})\right), \tag {3}
$$
where  $\alpha$  is an arbitrary step-size parameter and the clipping function,  $\mathrm{clip}_{\epsilon}$ , ensures the imperceptibility constraint of Equation 2 is satisfied.
When considering the NLP domain, a sequential, discrete input of  $L$  words, can be explicitly represented as,
$$
\mathbf {x} = w _ {1: L} = w _ {1}, w _ {2}, \dots , w _ {L - 1}, w _ {L}, \tag {4}
$$
where, the discrete word tokens,  $w_{1:L}$ , are often mapped to a continuous, sequential word embedding (Devlin et al., 2019) space,
$$
\mathbf {h} _ {1: L} = \mathbf {h} _ {1}, \mathbf {h} _ {2}, \dots , \mathbf {h} _ {L - 1}, \mathbf {h} _ {L}. \tag {5}
$$
Attacks must take place in the discrete text space,
$$
\mathbf {x} + \boldsymbol {\delta} = w _ {1: L ^ {\prime}} ^ {\prime} = w _ {1} ^ {\prime}, w _ {2} ^ {\prime}, \dots , w _ {L ^ {\prime} - 1} ^ {\prime}, w _ {L ^ {\prime}}. \tag {6}
$$
This requires a change in the interpretation of the perturbation  $\delta$ . It is not simple to define an appropriate function  $\mathcal{G}()$  in Equation 2 for word sequences. Perturbations can be measured at a character or word level. Alternatively, the perturbation could be measured in the vectorized embedding space (Equation 5), using for example  $l_{p}$ -norm based (Goodfellow et al., 2015) metrics or cosine similarity (Carrara et al., 2019a), which have been used in the image domain. However, constraints in the embedding space do not necessarily achieve imperceptibility in the original word sequence space. The simplest approach is to use a variant of an edit-based measurement (Li et al., 2018),  $\mathcal{L}_{\mathrm{e}}()$ , which counts the number of changes between the original sequence,  $w_{1:L}$  and the adversarial sequence  $w_{1:L'}'$ , where a change is a swap/addition/deletion, and ensures it is smaller than a maximum number of changes,  $N$ ,
$$
\mathcal {L} _ {\mathrm {e}} \left(w _ {1: L}, w _ {1: L ^ {\prime}} ^ {\prime}\right) \leq N. \tag {7}
$$
For the NLP adversarial attacks this work only examines word-level attacks, as these are considered more difficult to detect than character-level attacks. As an example, for an input sequence of  $L$  words, a  $N$ -word substitution adversarial attack,  $w_{1:N}^{\prime}$ , applied at word positions  $n_1, n_2, \ldots, n_N$  gives the adversarial output,  $w_{1:L'}^{\prime}$
$$
\begin{array}{l} w _ {1: L ^ {\prime}} ^ {\prime} = w _ {1}, \dots , w _ {n _ {1} - 1}, w _ {1} ^ {\prime}, w _ {n _ {1} + 1}, \dots , \\ w _ {n N - 1}, w _ {N} ^ {\prime}, w _ {n N + 1}, \dots , w _ {L}. \tag {8} \\ \end{array}
$$
The challenge is to select which words to replace, and what to replace them with. A simple yet effective substitution attack approach that ensures a
small change in the semantic content of a sentence is to use saliency to rank the word positions, and to use word synonyms for the substitutions (Ren et al., 2019). This attack is termed Probability Weight Word Saliency (PWWS). The highest ranking word word can be swapped for a synonym from a pre-selected list of given synonyms. The next most highly ranked word is substituted in the same manner and the process is repeated till the required  $N$  words have been substituted.
The above approach is limited to attacking specific word sequences and so cannot easily be generalised to universal attacks (Moosavi-Dezfooli et al., 2016), where the same perturbation is used for all inputs. For this situation, a simple solution is concatenation (Wang and Bansal, 2018; Blohm et al., 2018), where for example, the same  $N$ -length sequence of words is appended to each input sequence of words, as described in Raina et al. (2020). Here,
$$
w _ {1: L ^ {\prime}} ^ {\prime} = w _ {1}, \dots , w _ {L}, w _ {1} ^ {\prime}, \dots , w _ {N} ^ {\prime}. \tag {9}
$$
In both the substitution attack (Equation 8) and the concatenation attack (Equation 9), the size of the attack can be measured using the number of edits,  $\mathcal{L}_{\mathbf{e}}(w_{1:L}, w_{1:L'}') = N$ .
4 Adversarial Attack Detection
For a deployed system, the easiest approach to defend against adversarial attacks is to use a detection process to identify adversarial examples without having to modify the existing system.
For the image domain Section 2 discusses many of the standard detection approaches. In this work, we select two distinct approaches that have been generally successful: uncertainty (Smith and Gal, 2018), where adversarial samples are thought to result in greater epistemic uncertainty and Mahalanobis Distance (Lee et al., 2018), where the Mahalanobis distance in the logit space is indicative of how out of distribution a sample is (adversarial samples are considered more out of distribution). In the NLP domain, when excluding trivial grammar and spelling based detectors, perplexity based detectors can be used (Raina et al., 2020). Many other NLP specific detectors (Zhou et al., 2019; Han et al., 2020; Minervini and Riedel, 2018) have been proposed, but Mozes et al. (2020)'s FGWS detector is considered the state of art and is thus selected for comparison. Here low frequency words in an input are substituted for higher frequency words
and the change in model prediction is measured - adversarial samples are found to generally have a greater change. This work introduces a further NLP specific detector: residue detection, described in detail in Section 4.1.
When considering any chosen detection measure  $\mathcal{F}_{\mathrm{d}}$ , a threshold  $\beta$  can be selected to decide whether an input,  $w_{1:L}$ , is adversarial or not, where  $\mathcal{F}_{\mathrm{d}}(w_{1:L}) > \beta$ , implies that  $w_{1:L}$  is an adversarial sample. To assess the success of the adversarial attack detection processes, precision-recall curves are used. For the binary classification task of identifying an input as adversially attacked or not, at a given threshold  $\beta$ , the precision and recall values can be computed as  $\mathrm{prec} = \mathrm{TP} / \mathrm{TP} + \mathrm{FP}$  and  $\mathrm{rec} = \mathrm{TP} / \mathrm{TP} + \mathrm{FN}$ , where TP, FP and FN are the standard true-positive, false-positive and false-negative definitions. A single point summary of precision-recall curves is given with the  $\mathrm{F}_1$  score.
4.1 Residue Detection
In this work we introduce a new NLP detection approach, residue detection, that aims to exploit the nature of the NLP input space, discrete and sequential. Here we make two hypotheses:
1. Adversarial samples in an encoder embedding space result in larger components (residue) in central PCA eigenvector components than original examples.
2. The residue is only significant (detectable) for systems operating on discrete data (e.g. NLP systems).
The rationale behind these hypotheses is discussed next.
Deep learning models typically consist of many layers of non-linear activation functions. For example, in the NLP domain systems are usually based on layers of the Transformer architecture (Vaswani et al., 2017). The complete end-to-end model  $\mathcal{F}_{\hat{\theta}}()$  can be treated as a two stage process, with an initial set of layers forming the encoding stage,  $\mathcal{F}_{\mathrm{en}}()$  and the remaining layers forming the output stage,  $\mathcal{F}_{\mathrm{cl}}()$ , i.e.  $F_{\hat{\theta}}(\mathbf{x}) = \mathcal{F}_{\mathrm{cl}}(\mathcal{F}_{\mathrm{en}}(\mathbf{x}))$ .
If the encoding stage of the end-to-end classifier is sufficiently powerful, then the embedding space  $\mathcal{F}_{\mathrm{en}}(\mathbf{x})$  will have compressed the useful information into very few dimensions, allowing the output stage to easily separate the data points into classes (for classification) or map the data points to a continuous value (for regression). A simple Principal
Component Analysis (PCA) decomposition of this embedding space can be used to visualize the level of compression of the useful information. The PCA directions can be found using the eigenvectors of the covariance matrix,  $\mathbf{C}$ , of the data in the encoder embedding space. If  $\{\mathbf{q}_i\}_{i=1}^d$ , where  $d$  is the dimension of the encoder embedding space, represent the eigenvectors of  $\mathbf{C}$  ordered in descending order by the associated eigenvalue in magnitude, then it is expected that almost all useful information is contained within the first few principal directions,  $\{\mathbf{q}_i\}_{i=1}^p$ , where  $p \ll d$ . Hence, the output stage,  $\mathcal{F}_{c1}()$  will implicitly use only these useful components. The impact of a successful adversarial perturbation,  $\mathcal{F}_{\mathrm{en}}(\mathbf{x} + \boldsymbol{\delta})$ , is the significant change in the components in the principal eigenvector directions  $\{\mathbf{q}_i\}_{i=1}^p$ , to allow fooling of the output stage. Due to the complex nature of the encoding stage and the out of distribution nature of the adversarial perturbations, there are likely to be residual components in the non-principal  $\{\mathbf{q}_i\}_{i=p+1}^d$  eigenvector directions. These perturbations in the non-principal directions are likely to be more significant for the central eigenvectors, as the encoding stage is likely to almost entirely compress out components in the least principal eigenvector directions,  $\{\mathbf{q}_i\}_{i=d'+1}^d$ , where  $d' \approx d$ . Hence,  $\{\mathbf{q}_i\}_{i=p+1}^{d'}$  can be viewed as a subspace containing adversarial attack residues that can be used to identify adversarial examples.
The existence of adversarial attack residue in the central PCA eigenvector directions,  $\{\mathbf{q}_i\}_{i = p + 1}^{d'}$  suggests that in the encoder embedding space,  $\mathcal{F}_{\mathrm{en}}(\mathbf{x})$ , adversarial and original examples are linearly separable. This motivates the use of a simple linear classifier as an adversarial attack detector,
$$
P (\operatorname {a d v} | \mathbf {x}) = \sigma \left(\mathbf {W} \mathcal {F} _ {\mathrm {e n}} (\mathbf {x}) + b\right), \tag {10}
$$
where  $\mathbf{W}$  and  $b$  are the parameters of the linear classifier to be learnt and  $\sigma$  is the sigmoid function.
The above argument cannot predict how significant the residue in the central eigenvector space is likely to be. For the discrete space NLP attacks, the input perturbations are semantically small, whilst for continuous space image attacks the perturbations are explicitly small using a standard  $l_{p}$ -norm. Hence, it is hypothesised that NLP perturbations cause larger errors to propagate through the system, resulting in more significant residue in the encoder embedding space than that for image attacks. Thus, the residue technique is only likely to be a feasible detection approach for discrete text attacks.
The hypotheses made in this section are analysed and empirically verified in Section 5.3.
Table 1 describes four NLP classification datasets: IMDB (Maas et al., 2011); Twitter (Saravia et al., 2018); AG News (Zhang et al., 2015) and DBpedia (Zhang et al., 2015). Further, a regression dataset, Linguaskill-Business (L-Bus) (Chambers and Ingham, 2011) is included. The L-Bus data is from a multi-level prompt-response free speaking test i.e. candidates from a range of proficiency levels provide open responses to prompted questions. Based on this audio input a system must predict a score of 0-6 corresponding to the 6 CEFR (Council of Europe, 2001) grades. This audio data was transcribed using an Automatic Speech Recognition system with an average word error rate of  $19.5\%$ .
Dataset	#Train	#Test	#Classes
IMDB	25,000	25,000	2
Twitter	16,000	2000	6
AG News	120,000	7600	4
DBpedia	560,000	70,000	14
L-Bus	900	202	1
All NLP task models were based on the Transformer encoder architecture (Vaswani et al., 2017). Table 2 indicates the specific architecture used for each task and also summarises the classification and regression performance for the different tasks. For classification tasks, the performance is measured by top 1 accuracy, whilst for the regression task (L-Bus), the performance is measured using Pearson Correlation Coefficient (PCC).
Table 1: NLP Datasets.
Dataset	Transformer	Performance
IMDB	BERT	Acc: 93.8%
Twitter	ELECTRA	Acc: 93.3%
AG News	BERT	Acc: 94.5%
DBpedia	ELECTRA	Acc: 99.2%
L-Bus	BERT	PCC: 0.749
Table 2: Performance of models (BERT (Devlin et al., 2018), ELECTRA (Clark et al., 2020)).
Table 3 shows the impact of realistic adversarial attacks on the tasks: substitution (sub) attack (Equation 8), which replaces the  $N$  most salient tokens with a synonym defined by  $\mathrm{WordNet}^2$ , as
$^{2}$ https://wordnet.princeton.edu/