Elephant in the Room: An Evaluation Framework for Assessing Adversarial Examples in NLP

Abstract: An adversarial example is an input transformed by small perturbations that machine learning models consistently misclassify. While there are a number of methods proposed to generate adversarial examples for text data, it is not trivial to assess the quality of these adversarial examples, as minor perturbations (such as changing a word in a sentence) can lead to a significant shift in their meaning, readability and classification label. In this paper, we propose an evaluation framework to assess the quality of adversarial examples based on the aforementioned properties. We experiment with five benchmark attacking methods and an alternative approach based on an auto-encoder, and found that these methods generate adversarial examples with poor readability and content preservation. We also learned that there are multiple factors that can influence the attacking performance, such as the the length of text examples and the input domain.

Adversarial examples, a term introduced in BIBREF0, are inputs transformed by small perturbations that machine learning models consistently misclassify. The experiments are conducted in the context of computer vision (CV), and the core idea is encapsulated by an illustrative example: after imperceptible noises are added to a panda image, an image classifier predicts, with high confidence, that it is a gibbon. Interestingly, these adversarial examples can also be used to improve the classifier — either as additional training data BIBREF0 or as a regularisation objective BIBREF1 — thus providing motivation for generating effective adversarial examples.
The germ of this paper comes from our investigation of adversarial attack methods for natural language processing (NLP) tasks, e.g. sentiment classification, which drives us to quantify what is an “effective” or “good” adversarial example. In the context of images, a good adversarial example is typically defined according two criteria:
it has successfully fooled the classifier;
it is visually similar to the original example.
In NLP, defining a good adversarial example is a little more involving, because while criterion (b) can be measured with a comparable text similarity metric (e.g. BLEU or edit distance), an adversarial example should also:
be fluent or natural;
preserve its original label.
These two additional criteria are generally irrelevant for images, as adding minor perturbations to an image is unlikely to: (1) create an uninterpretable image (where else changing one word in a sentence can render a sentence incoherent), or (2) change how we perceive the image, say from seeing a panda to a gibbon (but a sentence's sentiment can be reversed by simply adding a negative adverb such as not). Without considering criterion (d), generating adversarial examples in NLP would be trivial, as the model can learn to simply replace a positive adjective (amazing) with a negative one (awful) to attack a sentiment classifier.
To the best of our knowledge, most studies on adversarial example generation in NLP have largely ignored these additional criteria BIBREF2, BIBREF3, BIBREF4, BIBREF5. We believe the lack of a rigorous evaluation framework partially explains why adversarial training for NLP models has not seen the same extent of improvement compared to CV models. As our experiments reveal, examples generated from most attacking methods are successful in fooling the classifier, but their language is often unnatural and the original label is not properly preserved.
The core contribution of our paper is to introduce a systematic, rigorous evaluation framework to assess the quality of adversarial examples for NLP. We focus on sentiment classification as the target task, as it is a popular application that highlights the importance of criteria discussed above. We test a number of attacking methods and also propose an alternative approach (based on an auto-encoder) for generating adversarial examples. We learn that a number of factors can influence the performance of adversarial attacks, including architecture of the classifier, sentence length and input domain.
Most existing adversarial attack methods for text inputs are derived from those for image inputs. These methods can be categorised into three types including gradient-based attacks, optimisation-based attacks and model-based attacks.
Gradient-based attacks are mainly white-box attacks that rely on calculating the gradients of the target classifier with respect to the input representation. This class of attacking methods BIBREF6, BIBREF7, BIBREF6 are mainly derived from the fast gradient sign method (FGSM) BIBREF1, and it has been shown to be effective in attacking CV classifiers. However, these gradient-based methods could not be applied to text directly because perturbed word embeddings do not necessarily map to valid words. Other methods such as DeepFool BIBREF8 that rely on perturbing the word embedding space face similar roadblocks. BIBREF5 propose to use nearest neighbour search to find the closest word to the perturbed embedding.
Both optimisation-based and model-based attacks treat adversarial attack as an optimisation problem where the constraints are to maximise the loss of target classifiers and to minimise the difference between original and adversarial examples. Between these two, the former uses optimisation algorithms directly; while the latter trains a seperate model to generate adversarial examples and therefore involves a training process. Some of the most effective attacks for images are achieved by optimisation-based methods, such as the L-BFGS attack BIBREF1 and the C&W attack BIBREF9 in white-box attacks and the ZOO method BIBREF10 in black-box attacks. For texts, the white-box attack HotFlip BIBREF3 and black-box attack DeepWordBug BIBREF11 and TextBugger BIBREF12 are proposed in this category.
In a similar vein, a few model-based attacks have been proposed for images, e.g. BIBREF13 design a generative adversarial network (GAN) to generate the image perturbation from a noise map. The attacking method and target classifier typically form a single large network and the attacking method is trained using the loss from the target classifier. For this reason, it is not very straightforward to use these model-based techniques for text because there is a discontinuity in the network (since words in the adversarial examples are discrete) and so it is not fully differentiable.
There are a number of off-the-shelf neural models for sentiment classification BIBREF14, BIBREF15, most of which are based on long-short term memory networks (LSTM) BIBREF16 or convolutional neural networks (CNN) BIBREF14. In this paper, we pre-train three sentiment classifiers: BiLSTM, BiLSTM$+$A, and CNN. These classifiers are targeted by white-box attacking methods to generate adversarial examples (detailed in Section SECREF9). BiLSTM is composed of an embedding layer that maps individual words to pre-trained word embeddings; a number of bi-directional LSTMs that capture sequential contexts; and an output layer that maps the averaged LSTM hidden states to a binary output. BiLSTM$+$A is similar to BiLSTM except it has an extra self-attention layer which learns to attend to salient words for sentiment classification, and we compute a weighted mean of the LSTM hidden states prior to the output layer. Manual inspection of the attention weights show that polarity words such as awesome and disappointed are assigned with higher weights. Finally, CNN has a number of convolutional filters of varying sizes, and their outputs are concatenated, pooled and fed to a fully-connected layer followed by a binary output layer.
Recent development in transformer-based pre-trained models have produced state-of-the-art performance on a range of NLP tasks BIBREF17, BIBREF18. To validate the transferability of the attacking methods, we also fine-tune a BERT classifier for black-box tests. That is, we use the adversarial examples generated for attacking the three previous classifiers (BiLSTM, BiLSTM$+$A and CNN) as test data for BERT to measure its classification performance to understand whether these adversarial examples can fool BERT.
We experiment with five benchmark attacking methods for texts: FGM, FGVM, DeepFool BIBREF5, HotFlip BIBREF3) and TYC BIBREF4.
To perturb the discrete inputs, both FGM and FGVM introduce noises in the word embedding space via the fast gradient method BIBREF1 and reconstruct the input by mapping perturbed word embeddings to valid words via nearest neighbour search. Between FGM and FGVM, the former introduce noises that is proportional to the sign of the gradients while the latter introduce perturbations proportional to the gradients directly. The proportion is known as the overshoot value and denoted by $\epsilon $. DeepFool uses the same trick to deal with discrete inputs except that, instead of using the fast gradient method, it uses the DeepFool method introduced in BIBREF8 for image to search for an optimal direction to perturb the word embeddings.
Unlike the previous methods, HotFlip and TYC rely on performing one or more atomic flip operations to replace words while monitoring the label change given by the target classifier. In HotFlip, the directional derivatives w.r.t. flip operations are calculated and the flip operation that results in the largest increase in loss is selected. TYC is similar to FGM, FGVM and DeepFool in that it also uses nearest neighbour search to map the perturbed embeddings to valid words, but instead of using the perturbed tokens directly, it uses greedy search or beam search to flip original tokens to perturbed ones one at a time in order of their vulnerability.
The benchmark methods we test (Section SECREF9) are gradient-based and optimisation-based attacks (Section SECREF2). We propose an alternative model-based method to train a seperate generative model to generate text adversarial examples. Denoting the generative model as $\mathcal {G}$, we train it to generate an adversarial example $X^{\prime }$ given an input example $X$ such that $X^{\prime }$ is similar to $X$ but that it changes the prediction of the target classifier $\mathcal {D}$, i.e. $X^{\prime } \sim X$ and $\mathcal {D}(X^{\prime }) \ne \mathcal {D}(X)$.
For images, it is straightforward to combine $\mathcal {G}$ and $\mathcal {D}$ in the same network and use the loss of $\mathcal {D}$ to update $\mathcal {G}$ because $X^{\prime }$ is continuous and can be fed to $\mathcal {D}$ while keeping the whole network differentiable. For texts, $X^{\prime }$ are discrete symbols (typically decoded with argmax or beam-search) and so the network is not fully differentiable. To create a fully differentiable network with $\mathcal {G}+\mathcal {D}$, we propose to use Gumbel-Softmax to imitate the categorical distribution BIBREF19. As the temperature ($\tau $) of Gumbel-Softmax sampling approaches 0, the distribution of the sampled output $X^*$ is identical to $X^{\prime }$. We illustrate this in Figure FIGREF12, where $X^*$ is fed to $\mathcal {D}$ during training while $X^{\prime }$ is generated as the output (i.e. the adversarial example) at test time.
$\mathcal {G}$ is designed as an auto-encoder to reconstruct the input (denoted as AutoEncoder), with the following objectives to capture the evaluation criteria described in Section SECREF1:
$L_{adv}$, the adversarial loss that maximises the cross-entropy of $\mathcal {D}$;
$L_{seq}$, the auto-encoder loss to reconstruct $X^{\prime }$ from $X^{\prime }$;
$L_{sem}$, the cosine distance between the mean embeddings of $X$ and $X^*$.
$L_{adv}$ ensures that the adversarial examples are fooling the classifier (criterion (a)); $L_{seq}$ regulates the adversarial example so that it isn't too different from the original input and has a reasonable language (criteria (b) and (c)); and $L_{sem}$ constrains the underlying semantic content of the original and adversarial sentences to be similar (criterion (b)); so reduces the likelihood of a sentiment flip (indirectly for criteria (d)). Note that criteria (d) is the perhaps the most difficult aspect to handle, as its interpretation is ultimately a human judgement.
We use two scaling hyper-parameters $\lambda _{ae}$ and $\lambda _{seq}$ to weigh the three objectives: $\lambda _{ae}(\lambda _{seq} * L_{seq} + (1-\lambda _{seq})*L_{sem}) + (1-\lambda _{ae})*L_{adv}$. We introduce attention layers to AutoEncoder and test both greedy and beam search decoding to improve the quality of generated adversarial examples. During training we alternate between positive-to-negative and negative-to-positive attack for different mini-batches.
We construct three datasets based on IMDB reviews and Yelp reviews. The IMDB dataset is binarised and split into a training and test set, each with 25K reviews (2K reviews from the training set are reserved for development). We filter out any review that has more than 400 tokens, producing the final dataset (imdb400). For Yelp, we binarise the ratings, and create 2 datasets, where we keep only reviews with $\le $ 50 tokens (yelp50) and $\le $200 tokens (yelp200). We randomly partition both datasets into train/dev/test sets (90/5/5 for yelp50; 99/0.5/0.5 for yelp200). For all datasets, we use spaCy for tokenisation. We train and tune target classifiers (see Section SECREF8) using the training and development sets; and evaluate their performance on the original examples in the test sets as well as the adversarial examples generated by attacking methods for the test sets. Note that AutoEncoder also involves a training process, for which we train and tune AutoEncoder using the training and development sets in yelp50, yelp200 and imdb400. Statistics of the three datasets are presented in Table TABREF22. These datasets present a variation in the text lengths (e.g. the average number of words for yelp50, yelp200 and imdb400 is 34, 82 and 195 words respectively), training data size (e.g. the number of training examples for target classifiers for imdb400, yelp50 and yelp200 are 18K, 407K and 2M, respectively) and input domain (e.g. restaurant vs. movie reviews).
We use the pre-trained glove.840B.300d embeddings BIBREF20 for all 6 attacking methods. For FGM, FGVM and DeepFool, we tune $\epsilon $, the overshoot hyper-parameter (Section SECREF9) and keep the iterative step $n$ static (5). For TYC, besides $\epsilon $ we also tune the upper limit of flipped words, ranging from 10%–100% of the maximum length. For HotFlip, we tune only the upper limit of flipped words, in the range of $[1, 7]$.
We pre-train AutoEncoder to reconstruct sentences in different datasets as we found that this improves the quality of the generated adversarial examples. During pre-training, we tune batch size, number of layers and number of units, and stop the training after the performance on the development sets stops improving for 20K steps. The model is then initialised with the pre-trained weights and trained based on objectives defined in Section SECREF11. During the training process we tune $\lambda _{ae}$ and $\lambda _{seq}$ while keeping the batch size (32) and learning rate ($1e^{-4}$) fixed. As part of our preliminary study, we also tested different values for the Gumbel-softmax temperature $\tau $ and find that $\tau =0.1$ performs the best. Embeddings are fixed throughout all training processes.
For target classifiers, we tune batch size, learning rate, number of layers, number of units, attention size (BiLSTM$+$A), filter sizes and dropout probability (CNN). For BERT, we use the default fine-tuning hyper-parameter values except for batch size, where we adjust based on memory consumption. Note that after the target classifiers are trained their weights are not updated when training or testing the attacking methods.
We propose both automatic metrics and manual evaluation strategies to assess the quality of adversarial examples, based on four criteria defined in Section SECREF1: (a) attacking performance (i.e. how well they fool the classifier); (b) textual similarity between the original input and the adversarial input; (c) fluency of the adversarial example; and (d) label preservation. Note that the automatic metrics only address the first 3 criteria (a, b and c); we contend that criterion (d) requires manual evaluation, as the judgement of whether the original label is preserved is inherently a human decision.
As sentiment classification is our target task, we use the standard classification accuracy (ACC) to evaluate the attacking performance of adversarial examples (criterion (a)).
To assess the similarity between the original and (transformed) adversarial examples (criteria (b)), we compute BLEU scores BIBREF21.
To measure fluency, we first explore a supervised BERT model fine-tuned to predict linguistic acceptability BIBREF17. However, in preliminary experiments we found that BERT performs very poorly at predicting the acceptability of adversarial examples (e.g. it predicts word-salad-like sentences generated by FGVM as very acceptable), revealing the brittleness of these supervised models. We next explore an unsupervised approach BIBREF22, BIBREF23, using normalised sentence probabilities estimated by pre-trained language models for measuring acceptability. In the original papers, the authors tested simple recurrent language models; here we use modern pre-trained language models such as GPT-2 BIBREF24 and XLNet BIBREF18. Our final acceptability metric (ACPT) is based on normalised XLNet sentence probabilities: ${\log P(s)} / ({((5+|s|)/(5+1))^\alpha })$, where $s$ is the sentence, and $\alpha $ is a hyper-parameter (set to 0.8) to dampen the impact of large values BIBREF25.
We only computed BLEU and ACPT scores for adversarial examples that have successfully fooled the classifier. Our rationale is that unsuccessful examples can artificially boost these scores by not making any modifications, and so the better approach is to only consider successful examples.
We present the performance of the attacking methods against 3 target classifiers (Table TABREF23A; top) and on 3 datasets (Table TABREF23B; bottom). We choose 3 ACC thresholds for the attacking performance: T0, T1 and T2, which correspond approximately to accuracy scores of 90%, 80% and 70% for the Yelp datasets (yelp50, yelp200); and 80%, 70% and 50% for the IMDB datasets (imdb400). Each method is tuned accordingly to achieve a particular accuracy. Missing numbers (dashed lines) indicate the method is unable to produce the desired accuracy, e.g. HotFlip with only 1 word flip produces 81.5% accuracy (T1) when attacking CNN on yelp50, and so T0 accuracy is unachievable.
Looking at BLEU and ACPT, HotFlip is the most consistent method over multiple datasets and classifiers. AutoEncoder is also fairly competitive, producing largely comparable performance (except for the yelp200 dataset). Gradient-based methods FGM and FGVM perform very poorly. In general, they tend to produce word salad adversarial examples, as indicated by their poor BLEU scores. DeepFool similarly generates incoherent sentences with low BLEU scores, but occasionally produces good ACPT (BiLSTM at T1 and T2), suggesting potential brittleness of the unsupervised approach for evaluating acceptability.
Comparing the performance across different ACC thresholds, we observe a consistent pattern of decreasing performance over all metrics as the attacking performance increases from T0 to T2. These observations suggest that all methods are trading off fluency and content preservation as they attempt to generate stronger adversarial examples.
We now focus on Table TABREF23A, to understand the impact of model architecture for the target classifier. With 1 word flip as the upper limit for HotFlip, the accuracy of BiLSTM$+$A and BiLSTM drops to T0 (approximately 4% accuracy decrease) while the accuracy of CNN drops to T1 (approximately 13% accuracy decrease), suggesting that convolutional networks are more vulnerable to attacks (noting that it is the predominant architecture for CV). Looking at Table TABREF23B, we also find that the attacking performance is influenced by the input text length and the number of training examples for target classifiers. For HotFlip, we see improvements over BLEU and ACPT as text length increases from yelp50 to yelp200 to imdb400, indicating the performance of HotFlip is more affected by input lengths. We think this is because with more words it is more likely for HotFlip to find a vulnerable spot to target. While for TYC and AutoEncoder, we see improvements over BLEU and ACPT as the number of training examples for target classifier decreases from yelp200 (2M) to yelp50 (407K) to imdb400 (22K), indicating these two methods are less effective for attacking target classifiers that are trained with more data. Therefore, increasing the training data for target classifier may improve their robustness against adversarial examples that are generated by certain methods.
As a black-box test to check how well these adversarial examples generalise to fooling other classifiers, also known as transferability, we feed the adversarial examples from the 3 best methods, i.e. TYC, HotFlip and AutoEncoder, to a pre-trained BERT trained for sentiment classification (Figure FIGREF31) and measure its accuracy. Unsurprisingly, we observe that the attacking performance (i.e. the drop in ACC) is not as good as those reported for the white-box tests. Interestingly, we find that HotFlip, the best method, produces the least effective adversarial examples for BERT. Both TYC and AutoEncoder perform better here, as their generated adversarial examples do well in fooling BERT.
To summarise, our results demonstrate that the best white-box methods (e.g. HotFlip) may not produce adversarial examples that generalise to fooling other classifiers. We also saw that convolutional networks are more vulnerable than recurrent networks, and that dataset features such as text lengths and training data size (for target classifiers) can influence how difficult it is to perform adversarial attack.
Automatic metrics provide a proxy to quantify the quality of the adversarial examples. To validate that these metrics work, we conduct a crowdsourcing experiment on Figure-Eight. Recall that the automatic metrics do not assess sentiment preservation (criterion (d)); we evaluate that aspect here.
We experiment with the 3 best methods (TYC, AutoEncoder and HotFlip) on 2 accuracy thresholds (T0 and T2), using BiLSTM$+$A as the classifier. For each method and threshold, we randomly sample 25 positive-to-negative and 25 negative-to-positive examples. To control for quality, we reserve and annotate 10% of the samples ourselves as control questions. Workers are first presented with 10 control questions as a quiz, and only those who pass the quiz with at least 80% accuracy can continue to work on the task. We display 10 questions per page, where one control question is embedded to continue monitor worker performance. The task is designed such that each control question can only be seen once per worker. We restrict our jobs to workers in United States, United Kingdoms, Australia, and Canada.
To evaluate the criteria discussed in Section SECREF1: (b) textual similarity, (c) fluency, and (d) sentiment preservation, we ask the annotators three questions:
Is snippet B a good paraphrase of snippet A?
$\circledcirc $ Yes $\circledcirc $ Somewhat yes $\circledcirc $ No
How natural does the text read?
$\circledcirc $ Very unnatural $\circledcirc $ Somewhat natural $\circledcirc $ Natural
What is the sentiment of the text?
$\circledcirc $ Positive $\circledcirc $ Negative $\circledcirc $ Cannot tell
For question 1, we display both the adversarial input and the original input, while for question 2 and 3 we present only the adversarial example. As an upper-bound, we also run a survey on question 2 and 3 for 50 random original samples.
We present the percentage of answers to each question in Figure FIGREF38. The green bars illustrate how well the adversarial examples paraphrase the original ones; blue how natural the adversarial examples read; and red whether the sentiment of the adversarial examples is consistent compared to the original.
Looking at the performance of the original sentences (“(a) Original samples”), we see that their language is largely fluent and their sentiment is generally consistent to the original examples', although it's worth noting that the review sentiment of imdb400 can be somewhat ambiguous (63% agreement). We think this is due to movie reviews being more descriptive and therefore creating potential ambiguity in their expression of sentiment.
On content preservation (criterion (b); green bars), all methods produce poor paraphrases on yelp50. For imdb400, however, the results are more promising; adversarial examples generated by HotFlip, in particular, are good, even at T2.
Next we look at fluency (criterion (c); blue bars). We see a similar trend: performance in imdb400 is substantially better than yelp50. In fact we see almost no decrease in fluency in the adversarial examples compared to the original in imdb400. In yelp50, HotFlip and AutoEncoder are fairly competitive, producing adversarial examples that are only marginally less fluent compared to the original at T0. At T2, however, these methods begin to trade off fluency. All in all, the paraphrasability and fluency surveys suggest that imdb400 is an easier dataset for adversarial experiments, and it is the predominant dataset used by most studies.
Lastly, we consider sentiment preservation (criterion (d); red bars). All methods perform poorly at preserving the original sentiment on both yelp50 and imdb400 datasets. The artifact is arguably more profound on shorter inputs, as the original examples in imdb400 have lower agreement in the first place (yelp50 vs. imdb400: 86% vs. 63%). Again both HotFlip and AutoEncoder are the better methods here (interestingly, we observe an increase in agreement as their attacking performance increases from T0 and T2).
Summarising our findings, HotFlip is generally the best method across all criteria, noting that its adversarial examples, however, have poor transferability. TYC generates good black-box adversarial examples but do not do well in terms of content preservation and fluency. AutoEncoder produces comparable results with HotFlip for meeting the four criteria and generates examples that generalise reasonably, but it is very sensitive to the increase of training examples for the target classifier. The ACPT metric appears to be effective in evaluating fluency, as we see good agreement with human evaluation. All said, we found that all methods tend to produce adversarial examples that do not preserve their original sentiments, revealing that these methods in a way “cheat” by simply flipping the sentiments of the original sentences to fool the classifier, and therefore the adversarial examples might be ineffective for adversarial training, as they are not examples that reveal potential vulnerabilities in the classifier.
We propose an evaluation framework for assessing the quality of adversarial examples in NLP, based on four criteria: (a) attacking performance, (b) textual similarity; (c) fluency; (d) label preservation. Our framework involves both automatic and human evaluation, and we test 5 benchmark methods and a novel auto-encoder approach. We found that the architecture of the target classifier is an important factor when it comes to attacking performance, e.g. CNNs are more vulnerable than LSTMs; dataset features such as length of text, training data size (for target classifiers) and input domains are also influencing factors that affect how difficulty it is to perform adversarial attack; and the predominant dataset (IMDB) used by most studies is comparatively easy for adversarial attack. Lastly, we also observe in our human evaluation that on shorter texts (Yelp) these methods produce adversarial examples that tend not to preserve their semantic content and have low readability. More importantly, these methods also “cheat” by simply flipping the sentiment in the adversarial examples, and this behaviour is evident on both datasets, suggesting they could be ineffective for adversarial training.

Figure and table captions:
Figure 1: Framework to generate adversarial examples.
Table 1: Dataset statistics.
Table 2: Results based on automatic metrics. Top half (A) presents 3 different target classifiers evaluated on one dataset (yelp50); bottom half (B) tests 3 datasets using one classifier (BiLSTM+A). For ACPT, less negative values are better. Boldface indicates optimal performance in each column.
Figure 2: Accuracy of BERT on adversarial examples generated from TYC, HOTFLIP and AUTOENCODER for different target classifiers (top row) and for different datasets (bottom row). BERT’s accuracy on original examples are denoted as red line in each plot.
Figure 3: Human evaluation results.

Questions about this paper:
What datasets do they use?

Answers:
1 IMDB dataset and 2 Yelp datasets