SlowGuess/NAACL_2021_OCR huggingface dataset:
Ying Xu* IBM Research Australia
Xu Zhong* IBM Research Australia
Antonio Jimeno Yepes  
IBM Research  
Australia
Jey Han Lau  
University of Melbourne  
Australia
We introduce a grey-box adversarial attack and defence framework for sentiment classification. We address the issues of differentiability, label preservation and input reconstruction for adversarial attack and defence in one unified framework. Our results show that once trained, the attacking model is capable of generating high-quality adversarial examples substantially faster (one order of magnitude less in time) than state-of-the-art attacking methods. These examples also preserve the original sentiment according to human evaluation. Additionally, our framework produces an improved classifier that is robust in defending against multiple adversarial attacking methods. Code is available at: https://github.com/ ibm-aur-nlp/adv-def-text-dist
Recent advances in deep neural networks have created applications for a range of different domains. In spite of the promising performance achieved by neural models, there are concerns around their robustness, as evidence shows that even a slight perturbation to the input data can fool these models into producing wrong predictions (Goodfellow et al., 2014; Kurakin et al., 2016). Research in this area is broadly categorised as adversarial machine learning, and it has two sub-fields: adversarial attack, which seeks to generate adversarial examples that fool target models; and adversarial defence, whose goal is to build models that are less susceptible to adversarial attacks.
A number of adversarial attacking methods have been proposed for image recognition (Goodfellow et al., 2014), NLP (Zhang et al., 2020) and speech recognition (Alzantot et al., 2018a). These methods are generally categorised into three types: white-box, black-box and grey-box attacks. White-box
attacks assume full access to the target models and often use the gradients from the target models to guide the craft of adversarial examples. Black-box attacks, on the other hand, assume no knowledge on the architecture of the target model and perform attacks by repetitively querying the target model. Different from the previous two, grey-box attacks train a generative model to generate adversarial examples and only assume access to the target model during the training phrase. The advantages of grey-box attacking methods include higher time efficiency; no assumption of access to target model during attacking phase; and easier integration into adversarial defending algorithms. However, due to the discrete nature of texts, designing grey-box attacks on text data remains a challenge.
In this paper, we propose a grey-box framework that generates high quality textual adversarial examples while simultaneously trains an improved sentiment classifier for adversarial defending. Our contributions are summarised as follows:
- We propose to use Gumbel-softmax (Jang et al., 2016) to address the differentiability issue to combine the adversarial example generator and target model into one unified trainable network.
- We propose multiple competing objectives for adversarial attack training so that the generated adversarial examples can fool the target classifier while maintaining similarity with the input examples. We considered a number of similarity measures to define a successful attacking example for texts, such as lexical and semantic similarity and label preservation.<sup>1</sup>
- To help the generative model to reconstruct input sentences as faithfully as possible, we introduce a novel but simple copy mechanism to
Without constraint on label preservation, simply flipping the ground-truth sentiment (e.g. the movie is great  $\rightarrow$  the movie is awful) can successfully change the output of a sentiment classifier even though it is not a useful adversarial example.
This work was completed during the employment of the authors in IBM Research Australia.
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4078-4087 June 6-11, 2021. ©2021 Association for Computational Linguistics
the decoder to selectively copy words directly from the input.
- We assess the adversarial examples beyond just attacking performance, but also content similarity, fluency and label preservation using both automatic and human evaluations.
- We simultaneously build an improved sentiment classifier while training the generative (attacking) model. We show that a classifier built this way is more robust than adversarial defending based on adversarial examples augmentation.
Most white-box methods are gradient-based, where some form of the gradients (e.g. the sign) with respect to the target model is calculated and added to the input representation. In image processing, the fast gradient sign method (FGSM; Goodfellow et al. (2014)) is one of the first studies in attacking image classifiers. Some of its variations include Kurakin et al. (2016); Dong et al. (2018). These gradient-based methods could not be applied to texts directly because perturbed word embeddings do not necessarily map to valid words. Methods such as DeepFool (Moosavi-Dezfooli et al., 2016) that rely on perturbing the word embedding space face similar roadblocks.
To address the issue of embedding-to-word mapping, Gong et al. (2018) propose to use nearest-neighbour search to find the closest words to the perturbed embeddings. However, this method treats all tokens as equally vulnerable and replace all tokens with their nearest neighbours, which leads to non-sensical, word-salad outputs. A solution to this is to replace tokens one-by-one in order of their vulnerability while monitoring the change of the output of the target models. The replacement process stops once the target prediction has changed, minimising the number of changes. Examples of white-box attacks that utilise this approach include TYC (Tsai et al., 2019) and HOTFLIP (Ebrahimi et al., 2017).
Different to white-box attacks, black-box attacks do not require full access to the architecture of the target model. Chen et al. (2017) propose to estimate the loss function of the target model by querying its label probability distributions, while Papernot et al. (2017) propose to construct a substitute of the target model by querying its output labels. The latter approach is arguably more realistic because in most
cases attackers only have access to output labels rather than their probability distributions. There is relatively fewer studies on black-box attacks for text. An example is TEXTFOOLER, proposed by Jin et al. (2019), that generates adversarial examples by querying the label probability distribution of the target model. Another is proposed by Alzantot et al. (2018b) where genetic algorithm is used to select the word for substitution.
Grey-box attacks require an additional training process during which full access to the target model is assumed. However, post-training, the model can be used to generate adversarial examples without querying the target model. Xiao et al. (2018) introduce a generative adversarial network to generate the image perturbation from a noise map. It is, however, not trivial to adapt the method for text directly. It is because text generation involves discrete decoding steps and as such the joint generator and target model architecture is non-differentiable.
In terms of adversarial defending, the most straightforward method is to train a robust model on data augmented by adversarial examples. Recently, more methods are proposed for texts, such as those based on interval bound propagation (Jia et al., 2019; Huang et al., 2019), and dirichlet neighborhood ensemble (Zhou et al., 2020).
The purpose of adversarial attack is to slightly perturb an input example  $x$  for a pre-trained target model (e.g. a sentiment classifier)  $f$  so that  $f(x) \neq y$ , where  $y$  is the ground truth of  $x$ . The perturbed example  $x'$  should look similar to  $x$ , which can be measured differently depending on the domain of the input examples.
We propose a grey-box attack and defence framework which consists of a generator  $\mathcal{G}$  (updated), and two copies of a pre-trained target classifier: a static classifier  $\mathcal{C}$  and an updated/augmented classifier  $\mathcal{C}^*$ . During the training phase, the output of  $\mathcal{G}$  is directly fed to  $\mathcal{C}$  and  $\mathcal{C}^*$  to form a joint architecture. Post-training, the generator  $\mathcal{G}$  is used independently to generate adversarial examples (adversarial attack); while the augmented classifier  $\mathcal{C}^*$  is an improved classifier with increased robustness (adversarial defence).
$^{2}\mathcal{C}$  and  $\mathcal{C}^*$  start with the same pre-trained weights, although only  $\mathcal{C}^*$  is updated during training.
Figure 1: Grey-box adversarial attack and defence framework for sentiment classification.
The training phase is divided into attacking steps and defending steps, where the former updates only the generator  $\mathcal{G}$  and learns to introduce slight perturbation to the input by maximising the objective function of the target model  $\mathcal{C}$ . The latter updates  $\mathcal{C}^*$  and  $\mathcal{G}$  by feeding both original examples and adversarial examples generated by  $\mathcal{G}$ . Here, the adversarial examples are assumed to share the same label with their original examples. Effectively, the defending steps are training an improved classifier with data augmented by adversarial examples.
Generating text with discrete decoding steps (e.g. argmax) makes the joint architecture not differentiable. Therefore we propose to use Gumbel-softmax (Jang et al., 2016) to approximate the categorical distribution of the discrete output. For each generation step  $i$ , instead of sampling a word from the vocabulary, we draw a Gumbel-softmax sample  $x_{i}^{*}$  which has the full probability distribution over words in the vocabulary: the probability of the generated word is close to 1.0 and other words close to zero. We obtain the input embedding for  $\mathcal{C}$  and  $\mathcal{C}^*$  by multiplying the sample  $x_{i}^{*}$  with the word embedding matrix,  $M_{\mathcal{C}}$ , of the target model  $\mathcal{C}$ :  $x_{i}^{*} \cdot M_{\mathcal{C}}$ . Figure 1 illustrates our grey-box adversarial attack and defence framework for text.
The generator  $\mathcal{G}$  can be implemented as an autoencoder or a paraphrase generator, essentially differentiated by their data conditions: the former uses the input sentences as the target, while the latter uses paraphrases (e.g. PARANMT-50M (Wieting and Gimpel, 2017)). In this paper, we implement  $\mathcal{G}$  as an auto-encoder, as our preliminary experiments found that a pre-trained paraphrase generator performs poorly when adapted to our test domain, e.g. Yelp reviews.
3.2 Objective Functions
Our auto-encoder  $\mathcal{G}$  generates an adversarial example given an input example. It tries to reconstruct the input example but is also regulated by an adversarial loss term that 'discourages' it from doing so. The objectives for the attacking step are given as follows:
$$
L _ {a d v} = \log p _ {\mathcal {C}} (y | x, \theta_ {\mathcal {C}}, \theta_ {\mathcal {G}}) \tag {1}
$$
$$
L _ {s 2 s} = - \log p _ {\mathcal {G}} (x | x, \theta_ {\mathcal {G}}) \tag {2}
$$
$$
L _ {s e m} = \cos \left(\frac {1}{n} \sum_ {i = 0} ^ {n} \operatorname {e m b} \left(x _ {i}\right), \frac {1}{n} \sum_ {i = 0} ^ {n} \operatorname {e m b} \left(x _ {i} ^ {*}\right)\right) \tag {3}
$$
where  $L_{adv}$  is essentially the negative cross-entropy loss of  $\mathcal{C}$ ;  $L_{s2s}$  is the sequence-to-sequence loss for input reconstruction; and  $L_{sem}$  is the cosine similarity between the averaged embeddings of  $x$  and  $x^{*}$  ( $n =$  number of words). Here,  $L_{s2s}$  encourages  $x'$  (produced at test time) to be lexically similar to  $x$  and helps produce coherent sentences, and  $L_{sem}$  promotes semantic similarity. We weigh the three objective functions with two scaling hyper-parameters and the total loss is:  $L = \lambda_1(\lambda_2L_{s2s} + (1 - \lambda_2)L_{sem}) + (1 - \lambda_1)L_{adv}$ . We denote the auto-encoder based generator trained with these objectives as  $\mathbf{A}\mathbf{E}$ .
An observation from our preliminary experiments is that the generator tends to perform imbalanced attacking among different classes. (e.g. AE learns to completely focus on one direction attacking, e.g. positive-to-negative or negative-to-positive attack). We found a similar issue in white-box attack methods such as FGSM Goodfellow et al. (2014) and DeepFool (Moosavi-Dezfooli et al., 2016). To address this issue, we propose to modify  $L_{adv}$  to be the maximum loss of a particular class in each batch, i.e.
$$
L _ {a d v} = \max  _ {t = 1} ^ {| C |} \left(L _ {a d v} ^ {t}\right) \tag {4}
$$
where  $L_{adv}^{t}$  refers to the adversarial loss of examples in the  $t$ -th class and  $|C|$  the total number of classes. We denote the generator trained with this alternative loss as  $\mathbf{A}\mathbf{E} + \mathbf{B}\mathbf{A}\mathbf{L}$ .
For adversarial defence, we use the same objective functions, with the following exception: we replace  $L_{adv}$  in Equation (1) with the objective function of the classifier  $\mathcal{C}^*$ , i.e.
$$
L _ {d e f} = - \log p _ {\mathcal {C} ^ {*}} ([ y, y ] | [ x, x ^ {*} ], \theta_ {\mathcal {C} ^ {*}}, \theta_ {\mathcal {G}}) \tag {5}
$$
We train the model  $\mathcal{C}^*$  using both original and adversarial examples  $(x$  and  $x^{*})$  with their original label  $(y)$  to prevent  $\mathcal{C}^*$  from overfitting to the adversarial examples.
3.3 Label Preservation
One of the main challenges of generating a textual adversarial example is to preserve its original ground truth label, which we refer to as label preservation. It is less of an issue in computer vision because slight noises added to an image is unlikely to change how we perceive the image. In text, however, slight perturbation to a sentence could completely change its ground truth.
We use sentiment classification as context to explain our approach for label preservation. The goal of adversarial attack is to generate an adversarial sentence whose sentiment is flipped according to the target model prediction but preserves the original ground truth sentiment from the perspective of a human reader. We propose two ways to help label preservation. The first approach is task-agnostic, i.e. it can work for any classification problem, while the second is tailored for sentiment classification.
Label smoothing (+LS). We observe the generator has a tendency to produce adversarial examples with high confidence, opposite sentiment scores from the static classifier  $\mathcal{C}$ . We explore the use of label smoothing (Müller et al., 2019) to force the generator generate examples that are closer to the decision boundary, to discourage the generator from completely changing the sentiment. We incorporate label smoothing in Eq. 1 by redistributing the probability mass of true label uniformly to all other labels. Formally, the smoothed label  $y_{ls} = (1 - \alpha)*y + \alpha /K$  where  $\alpha$  is a hyperparameter and  $K$  is the number of classes. For example, when performing negative-to-positive attack, instead of optimising  $\mathcal{G}$  to produce adversarial examples with label distribution {pos: 1.0, neg: 0.0} (from  $\mathcal{C}$ ), label distribution {pos: 0.6, neg: 0.4} is targeted. Generator trained with this additional constraint is denoted with the +LS suffix.
Counter-fitted embeddings (+CF). Mrkšić et al. (2016) found that unsupervised word embeddings such as GloVe (Pennington et al., 2014) often do not capture synonymy and antonymy relations (e.g. cheap and pricey have high similarity). The authors propose to post-process pre-trained word embeddings with lexical resources (e.g. WordNet) to produce counter-fitted embeddings that better
capture these lexical relations. To discourage the generator  $\mathcal{G}$  from generating words with opposite sentiments, we experiment with training  $\mathcal{G}$  with counter-fitted embeddings. Models using counterfitted embeddings is denoted with  $+\mathbf{CF}$  suffix.
3.4 Generator with Copy Mechanism (+CPY)
White-box or black-box attacking methods are based on adding, removing, or replacing tokens in input examples. Therefore maintaining similarity with original examples is easier than grey-box methods that generate adversarial examples word-by-word from scratch. We introduce a simple copy mechanism that helps grey-box attack to produce faithful reconstruction of the original sentences.
We incorporate a static copy mask to the decoder where it only generates for word positions that have not been masked. E.g., given the input sentence  $x = [w_0, w_1, w_2]$ , target  $x^* = [w_0, w_1, w_2]$ , and mask  $m = [1, 0, 1]$ , at test time the decoder will "copy" from the target for the first input ( $w_0$ ) and third input token ( $w_2$ ) to produce  $w_0$  and  $w_2$ , but for the second input token ( $w_1$ ) it will decode from the vocabulary. During training, we compute cross-entropy only for the unmasked input words.
The static copy mask is obtained from one of the pre-trained target classifiers, C-LSTM (Section 4.2). C-LSTM is a classifier with a bidirectional LSTM followed by a self-attention layer to weigh the LSTM hidden states. We rank the input words based on the self-attention weights and create a copy mask such that only the positions corresponding to the top- $N$  words with the highest weights are generated from the decoder. Generally sentiment-heavy words such as awesome and bad are more likely to have higher weights in the self-attention layer. This self attention layer can be seen as an importance ranking function (Morris et al., 2020b) that determines which tokens should be replaced or replaced first. Models with copy mechanism are denoted with the +CPY suffix.
We conduct our experiments using the Yelp review dataset.3 We binarise the ratings,4 use spaCy for tokenisation,5 and keep only reviews with  $\leq 50$  tokens (hence the dataset is denoted as yelp50).
<sup>3</sup>https://www.yelp.com/dataset
${}^{4}$  Ratings  $\geq  4$  is set as positive and ratings  $\leq  2$  as negative.
$^{5}$ https://spacy.io
We split the data in a  $90/5/5$  ratio and downsample the positive class in each set to be equivalent to the negative class, resulting in 407,298, 22,536 and 22,608 examples in train/dev/test set respectively.
For the target classifiers  $(\mathcal{C}$  and  $\mathcal{C}^*$ ), we pretrain three sentiment classification models using yelp50: C-LSTM (Wang et al., 2016), C-CNN (Kim, 2014) and C-BERT. C-LSTM is composed of an embedding layer, a 2-layer bidirectional LSTMs, a self-attention layer, and an output layer. C-CNN has a number of convolutional filters of varying sizes, and their outputs are concatenated, pooled and fed to a fully-connected layer followed by an output layer. Finally, C-BERT is obtained by fine-tuning the BERT-Base model (Devlin et al., 2018) for sentiment classification. We tune learning rate, batch size, number of layers and number of hidden units for all classifiers; the number of attention units for C-LSTM and convolutional filter sizes and dropout rates for C-CNN specifically.
For the auto-encoder, we pre-train it to reconstruct sentences in yelp50. During pre-training, we tune learning rate, batch size, number of layers and number of hidden units. During the training of adversarial attacking, we tune  $\lambda_{1}$  and  $\lambda_{2}$ , and learning rate  $lr$ . We also test different temperature  $\tau$  for Gumbel-softmax sampling and found that  $\tau = 0.1$  performs the best. All word embeddings are fixed.
More hyper-parameter and training configurations are detailed in the supplementary material.
4.3 Attacking Performance
Most of the existing adversarial attacking methods have been focusing on improving the attack success rate. Recent study shows that with constraints adjusted to better preserve semantics and grammaticality, the attack success rate drops by over 70 percentage points (Morris et al., 2020a). In this paper, we want to understand — given a particular success rate — the quality (e.g. fluency, content/label preservation) of the generated adversarial samples. Therefore, we tuned all attacking methods to achieve the same levels of attack success rates; and compare the quality of generated adversarial examples. Note that results for adver
sarial attack are obtained by using the  $\mathcal{G} + \mathcal{C}$  joint architecture, while results for adversarial defence are achieved by the  $\mathcal{G} + \mathcal{C} + \mathcal{C}^*$  joint architecture.
4.3.1 Evaluation Metrics
In addition to measuring how well the adversarial examples fool the sentiment classifier, we also use a number of automatic metrics to assess other aspects of adversarial examples, following Xu et al. (2020):
Attacking performance. We use the standard classification accuracy (ACC) of the target classifier  $(\mathcal{C})$  to measure the attacking performance of adversarial examples. Lower accuracy means better attacking performance.
Similarity. To assess the textual and semantic similarity between the original and corresponding adversarial examples, we compute BLEU (Papineni et al., 2002) and USE (Cer et al., 2018). For both metrics, higher scores represent better performance.
Fluency. To measure the readability of generated adversarial examples, we use the acceptability score (ACPT) proposed by Lau et al. (2020), which is based on normalised sentence probabilities produced by XLNet (Yang et al., 2019). Higher scores indicate better fluency.
Transferability. To understand the effectiveness of the adversarial examples in attacking another unseen sentiment classifier (TRF), we evaluate the accuracy of C-BERT using adversarial examples that have been generated for attacking classifiers C-LSTM and C-CNN. Lower accuracy indicates better transferability.
Attacking speed. We measure each attacking method on the amount of time it takes on average (in seconds) to generate an adversarial example.
4.3.2 Automatic Evaluation
Comparison between AE variants. We first present results on the development set where we explore different variants of the auto-encoder (generator) in the grey-box model. AE serves as our base model, the suffix  $+\mathrm{BAL}$  denotes the use of an alternative  $L_{adv}$  (Section 3.2), +LS label smoothing (Section 3.3), +CF counter-fitted embeddings (Section 3.3), and +CPY copy mechanism (Section 3.4).
We present the results in Table 1. Attacking performance of all variants are tuned to produce
$^{6}$ Pre-trained BLEU scores are 97.7 and 96.8 on yelp50 using GloVe and counter-fitted embedding, respectively.
7We can in theory tune different methods to achieve higher success rate, but we choose the strategy to use lower success rates so that all methods generate relatively fair quality
examples that annotators can make sense of during human evaluation.
$^{8}$ USE is calculated as the cosine similarity between the original and adversarial sentence embeddings produced by the universal sentence encoder (Cer et al., 2018).
ACC	BLEU	SENT
ALL	POS	NEG	SUC	POS	NEG	AGR	UKN	DAGR
AE	66.0	99.8	28.4	55.3	71.7	58.7	-	-	-
AE+BAL	75.6	72.3	78.8	65.9	73.9	70.9	0.12	0.80	0.08
AE+LS	74.3	77.8	70.4	80.3	84.6	86.3	0.46	0.44	0.10
AE+LS+CF	76.6	66.5	86.7	79.9	82.5	85.0	0.64	0.28	0.08
AE+LS+CF+CPY	77.4	70.9	83.8	85.7	90.6	90.2	0.68	0.30	0.02
Table 1: Performance of adversarial examples generated by five AE variants on yelp50 development set.
approximately  $70\% - 80\%$  accuracy for the target classifier  $\mathcal{C}$  (C-LSTM). For ACC and BLEU, we additionally report the performance for the positive and negative sentiment class separately. To understand how well the adversarial examples preserve the original sentiments, we recruit two annotators internally to annotate a small sample of adversarial examples produced by each of the auto-encoder variants. AGR and DAGR indicate the percentage of adversarial examples where they agree and disagree with the original sentiments, and UKN where the annotators are unable to judge their sentiments.