Year: 2019

Abstract:
While there has been great interest on generating imperceptible adversarial examples in continuous data domain (e.g. image and audio) to explore the model vulnerabilities, generating adversarial text in the discrete domain is still challenging. The main contribution of this paper is to propose a general targeted attack framework AdvCodec for adversarial text generation which addresses the challenge of discrete input space and is easily adapted to general natural language processing (NLP) tasks. In particular, we propose a tree based autoencoder to encode discrete text data into continuous vector space, upon which we optimize the adversarial perturbation. A tree based decoder is then applied to ensure the grammar correctness of the generated text. It also enables the flexibility of making manipulations on different levels of text, such as sentence (AdvCodec(Sent)) and word (AdvCodec(Word)) levels. We consider multiple attacking scenarios, including appending an adversarial sentence or adding unnoticeable words to a given paragraph, to achieve arbitrary targeted attack. To demonstrate the effectiveness of the proposed method, we consider two most representative NLP tasks: sentiment analysis and question answering (QA). Extensive experimental results and human studies show that AdvCodec generated adversarial text can successfully attack the neural models without misleading the human. In particular, our attack causes a BERT-based sentiment classifier accuracy to drop from 0.703 to 0.006, and a BERT-based QA model’s F1 score to drop from 88.62 to 33.21 (with best targeted attack F1 score as 46.54). Furthermore, we show that the white-box generated adversarial texts can transfer across other black-box models, shedding light on an effective way to examine the robustness of existing NLP models.

heading: 1 INTRODUCTION
text: Recent studies have demonstrated that deep neural networks (DNNs) are vulnerable to carefully crafted adversarial examples (Goodfellow et al., 2015; Papernot et al., 2016; Eykholt et al., 2017; Moosavi-Dezfooli et al., 2016). While there are a lot of successful attacks proposed in the continuous data domain including images, audios, and videos, how to effectively generate adversarial examples in the discrete text domain still remains a hard problem. There are several challenges for generating adversarial text: 1) most existing gradient-based adversarial attack approaches are not directly applicable to the discrete structured data; 2) it is less clear how to appropriately measure the naturalness of the generated text compared to the original ones; 3) the manipulation space of text is limited, and it is unclear whether generating a new appended sentence or manipulating individual words will affect human judgements.
So far, existing works on adversarial text generation either leverage heuristic solutions such as genetic algorithms (Jin et al., 2019) to search for potential adversarial sentences, or are limited to attacking specific NLP tasks (Cheng et al., 2018; Lei et al., 2018). In addition, effective targeted attacks have not been achieved by current attacks for any task. In this paper, we aim to provide more insights towards solving these challenges by proposing a unified optimization framework AdvCodec to generate adversarial text against general NLP tasks. In particular, the core component of AdvCodec is a tree based autoencoder which converts discrete text tokens into continuous semantic embedding, upon which the adversarial perturbation will be optimized regarding the chosen adversarial target. Finally, a tree based decoder will decode the generated adversarial continuous embedding vector back to the sentence level based on the tree grammar rules, aiming to both pre-
serve the original semantic meaning and linguistic coherence. An iterative process can be applied here to ensure the attack success rate.
In addition to the general adversarial text generation framework AdvCodec, this paper also aims to explore several scientific questions: 1) Since AdvCodec allows the flexibility of manipulating on different hierarchies of the tree structures, which is more attack effective and which way preserves better grammatical correctness? 2) Is it possible to achieve targeted attack for general NLP tasks such as sentiment classification and QA, given the limited degree of freedom for manipulation? 3) Is it possible to perform blackbox attack in general NLP tasks? 4) Is BERT robust in practice? 5) Do these adversarial examples affect human reader performances?
To address the above questions, we explore two types of tree based autoencoders on the word (AdvCodec(Word)) and sentence level (AdvCodec(Sent)). For each encoding scenario, we generate adversarial text against different sentiment classification and QA models. Compared with the state-of-the-art adversarial text generation methods, our approach achieves significantly higher untargeted and targeted attack success rate. In addition, we perform both whitebox and blackbox settings for each attack to evaluate the model vulnerabilities. Within each attack setting, we evaluate attack strategies as appending an additional adversarial sentence or adding scatter of adversarial words to a paragraph, to evaluate the quantitative attack effectiveness. To provide thorough adversarial text quality assessment, we also perform 7 groups of human studies to evaluate the quality of generated adversarial text compared with the baselines methods, and whether human can still get the ground truth answers for these tasks based on adversarial text. We find that: 1) both word and sentence level attacks can achieve high attack success rate, while the sentence level manipulation can consider the global grammatical constraints and generate high quality adversarial sentences. 2) various targeted attacks on general NLP tasks are possible (e.g. when attacking QA, we can ensure the target to be a specific answer or a specific location within a sentence); 3) the transferability based blackbox attacks are successful in NLP tasks. Transferring adversarial text from stronger models (in terms of performances) to weaker ones is more successful; 4) Although BERT has achieved state-ofthe-art performances, we observe the performance drops are also larger than other standard models when confronted with adversarial examples, which indicates BERT is not robust under the adversarial settings; 5) Most human readers are not sensitive to our adversarial examples and can still answer the right answers when confronted with the adversary-injected paragraphs.
In summary, our main contribution lies on: (1) We propose a general adversarial text generation framework AdvCodec that addresses the challenge of discrete text input to achieve targeted attacks against general NLP tasks (e.g. sentiment classification and QA) while preserving the semantic meaning and linguistic coherence; (2) we propose a novel tree-based text autoencoder that ensures the grammar correctness of generated text; (3) we conduct extensive experiments and successfully attack different sentiment classifiers and QA models with significant higher attack success rate than the state-of-the-art baseline methods; (4) we also perform comprehensive ablation studies including evaluating the attack scenarios of appending an adversarial sentence or adding scatter of adversarial words, as well as appending the adversarial sentence at different positions within a paragraph, and draw several interesting conclusions; (5) we leverage extensive human studies to show that the adversarial text generated by AdvCodec is natural and effective to attack neural models, while barely affecting human’s judgement.

heading: 2 RELATED WORK
text: A large body of works on adversarial examples focus on perturbing the continuous input space. Though some progress has been made on generating adversarial perturbations in the discrete space, several challenges still remain unsolved. For example, Zhao et al. (2017) exploit the generative adversarial network (GAN) to generate natural adversarial text. However, this approach cannot explicitly control the quality of the generated instances. Most existing methods (Liang et al., 2017; Samanta & Mehta, 2017; Jia & Liang, 2017; Li et al., 2018; Jin et al., 2019) apply heuristic strategies to synthesize adversarial text: 1) first identify the features (e.g. characters, words, and sentences) that have the influence on the prediction, 2) follow different search strategies to perturb these features with the constructed perturbation candidates (e.g. typos, synonyms, antonyms, frequent words). For instance, Liang et al. (2017) employ the loss gradient∇L to select important characters and phrases to perturb, while Samanta & Mehta (2017) use typos, synonyms, and important adverbs/adjectives as candidates for insertion and replacement. Once the influential features are obtained, the strategies to

heading: Perturbation
text: apply the perturbation generally include insertion, deletion, and replacement. Such adversarial text generation approaches cannot guarantee the grammar correctness of generated text. For instance, text generated by Liang et al. (2017) are almost random stream of characters. To generate grammarly correct perturbation, Jia & Liang (2017) adopt another heuristic strategy which adds manually constructed legit distracting sentences to the paragraph to introduce fake information. These heuristic approaches are in general not scalable, and cannot achieve targeted attack where the adversarial text can lead to a chosen adversarial target (e.g. adversarial label in classification). Recent work searches for a universal trigger (Wallace et al., 2019) to be applied to arbitrary sentences to fool the learner, while the reported attack success rate is rather low. In contrast, with the tree based autoencoder, the proposed AdvCodec framework is able to generate grammarly correct adversarial text efficiently, achieving high attack success rates on different models.

heading: 3 THE ADVCODEC FRAMEWORK FOR ADVERSARIAL TEXT GENERATION
text: We describe the AdvCodec framework in this section. As illustrated in Figure 1, the key component of the AdvCodec framework is a tree-based autoencoder. The hierarchical and discrete nature of language motivates us to make use of tree-based autoencoder to map discrete text into a high dimensional latent space, which empowers us to leverage the existing optimization based attacking method such as Carlini & Wagner (2016) to both efficiently and effectively generate adversarial text.
Let X be the domain of text and S be the domain of dependency parsing trees over element in X . Formally, a tree-based autoencoder consists of an encoder E : X × S → Z that encodes text x ∈ X along with its dependency parsing tree s ∈ S into a high dimensional latent representation z ∈ Z
and a decoder G : Z × S → X that generates the corresponding text x from the given context vector z and the expected dependency parsing tree s. Given a dependency tree s, E and G form an antoencoder. We thus have the following reconstruction loss to train our tree-based autoencoder:
L = −Ex∼X [log pG(x|s, E(x, s)] (1) As Figure 1 suggests, AdvCodec can operate on different granularity levels to generate either wordlevel or sentence-level contextual representation, and decode it into the adversarial text. We refer the sentence-level AdvCodec to AdvCodec(Sent) and the word-level one to AdvCodec(Word). Both of them will be described in more details in the later part of this section.
3.1 OVERVIEW OF THE ADVCODEC FRAMEWORK Before diving into details, we provide a high level overview of AdvCodec according to the attack scenario and attack capability supported by this framework.
Attack Scenario. Different from the previous adversarial text generation works (Lei et al., 2018; Cheng et al., 2018; Papernot et al., 2016; Miyato et al., 2016; Alzantot et al., 2018) that directly modify critical words in place and might risk changing the semantic meaning or editing the ground truth answers, we are generating the concatenative adversaries. First proposed by Jia & Liang (2017), the concatenative adversary does not change any words in the original paragraph or question, but instead appends a new adversarial sentence to the paragraph to fool the model. However, the concatenative attack also needs to ensure the appended sentence is compatible (Jia & Liang, 2017) with the original paragraph, which in other words means it should not contradict any stated facts in the paragraph, especially the correct answer. In our work, we further push the concept of concatenative adversaries further and propose a more general notion called scatter attack, which means we can inject adversarial words sporadically over the whole paragraph. The concatenative adversarial example falls into our case when those adversarial tokens form a sentence and on the same time the semantic meaning of the sentence does not contradict the original paragraph. Examples of concatenative attack and scatter attack is shown in table 1.
Attack Capability. AdvCodec is essentially an optimization based framework to find the adversarial texts with the optimization goal set to achieve targeted attack. For the sentiment classification task, AdvCodec can perform targeted attack to make the original positive reviews be classified as the most negative one, and vice versa. Particularly in the QA task, we design and implement two kinds of targeted attack: position targeted attack and answer targeted attack. A successful position targeted attack means the model can be fooled to output the answers at specific targeted positions in the paragraph, but the content on the targeted span cannot be guaranteed. In contrast, a successful answer targeted attack is a stronger targeted attack, which refers to the situation when the model always outputs the preset targeted answer pair on the target no matter what the question looks like. An
example of word targeted attack can be found in the table 1. Although our framework is designed as a whitebox attack, our experimental results demonstrate our whitebox generated adversarial words can transfer to other blackbox models with high attack success rate. Finally, because AdvCodec is a unified adversarial text generation framework whose outputs are discrete tokens, it can be applied to different downstream NLP tasks. In this paper, we perform adversarial evaluation on sentiment classification and QA as examples to demonstrate how our framework is adapted to different works.
3.2 ADVCODEC(SENT)
In this subsection, we describe AdvCodec(Sent) and explain how to utilize it to attack sentiment classification models and question answering systems. The main idea comes from the fact that tree structures sometimes have better performances than sequential recurrent models(Li et al., 2015; Iyyer et al., 2014; 2018) and the fact that it is inherently flexible to add perturbations on hierarchical nodes of the tree structures. Motivated by this, we design a novel tree-based autoencoder to simultaneously preserve similar semantic meaning and original syntactic structure.
Encoder. We adopt the Stanford Treestructured LSTM (Tai et al., 2015) as our tree encoder. In the encoding phase, features are extracted and summed from bottom (leaf
node, i.e. word) to top (root node) along the dependency tree, extracted by Stanford CoreNLP Parser (Manning et al., 2014). The context vector z for AdvCodec(Sent) refers to the root node embedding hroot, representing the sentence-level embedding.
Decoder. Following the same dependency tree, we design the text decoder as illustrated in Figure 2. In the decoding phase, we start from the root node and traverse along the dependency tree in level-order. The hidden state hj of the next node j comes from (i) the hidden state hi of the current tree node, (ii) current node predicted word embedding wi, and (iii) the dependency embedding dij between the current node i and the next node j based on the dependency tree. The next node’s corresponding word yj is generated based on the output of the LSTM Cell oj via a linear layer that maps from the hidden presentation oj to the logits that represent the probability distribution of the tree’s vocabulary.
oj , hj = LSTM([hi;wi; dij ]) (2) yj = W · oj + b (3)

heading: 3.2.1 ATTACK SENTIMENT CLASSIFICATION MODEL
text: Initial Seed. Following our pipeline to optimize adversarial sentence AdvSentence appended to the paragraph, we need to first start with an initial seed for optimization. Such initial seed for sentiment classification task can be arbitrary. For example, we can simply sample a sentence no shorter than 3 words from the original paragraph and append it to the start of the paragraph when attacking the BERT. The append position does have a influence on the attack success rate for adversarial attack, and more detailed ablation analysis will be discussed in the next section.
Optimization Procedure. Finding the optimal perturbation z∗ on context vector z, we aim to find z∗ that solves minimize ||z∗||p + cf(z + z∗), (4) where f is the objective function for the targeted attack and c is the constant balancing between the perturbation magnitude and attack target. Specifically, we use the objective function f proposed in Carlini & Wagner (2016) as follows
f(z′) = max(max{Z(G(z′, s))i : i 6= t} − Z(G(z′, s))t,−κ) (5)
where z′ = z + z∗, t is the target class, Z(·) is the logit output of the classification model before softmax and κ is the confidence score to adjust the misclassification rate. The optimal solution is iteratively searched via Adam optimizer (Kingma & Ba, 2014).

heading: 3.2.2 ATTACK QUESTION ANSWERING SYSTEM
text: Initial Seed. Different from attacking sentiment analysis, it is important to choose a good initial seed that is semantically close to the context or the question when attacking QA model. In this way we can reduce the number of iteration steps and attack the QA model more efficiently. Based on the heuristic experiments conducted in the Appendix A.4, we choose to use question words to craft an initial seed. We design a set of coarse grained rules to convert a question sentence to a meaningful declarative statement and assign a target fake answer. The fake answer can be crafted according to the perturbed model’s predicted answer, or can be manually chosen by adversaries. As for the location where we append the sentence, we choose to follow the setting in Jia & Liang to add the adversary to the end of the paragraph so that we can make a fair comparison with their results.
It is worth noting unlike Jia & Liang (2017) that uses complicated rules to ensure the adversarial sentence does not change the ground truth, this heuristic step is the very first step of our framework followed by a series of optimization steps to ensure the ground truth is not changed. In this paper, we ensure our appended adversarial sentences are not contradictory to the ground truth by a) choosing an initial sentence as the initial seed of optimization, b) adding perturbation to the sentence, c) searching for the optimal adversarial sentence, d) ensuring that the adversarial sentence and context sentence are disjoint, otherwise keep the iteration steps. If the maximum steps are reached, the optimization is regarded as a failure.
Optimization Procedure. We follow the same optimization procedure as attacking sentiment classification task except a subtle change of the objective function f due to the difference between QA model and classification model:
f(z′)= 2∑ j=1 max(max{Zj(G(z′, s))i : i 6= t} − Zj(G(z′, s))tj ,−κ) (6)
where Z1(·) is the output logits of answer starting position and Z2(·) is the output logits of answer ending position in the QA system. t1 and t2 are respectively the targeted start position and the targeted end position. For the position targeted attack mentioned in Section 3.1, we expect the model output to be a span in the paragraph from the targeted start position t1 to the targeted end position t2. In contrast, the answer targeted attack requires the model to output the predefined answer spans in the targeted positions and keep them unmodified during the optimization steps by setting gates to the targeted answer span: yj = g1 yj + g2 xj , (j = t1, t1 + 1, ..., t2),where yj refers to the tree decoded adversarial tokens. We set g1 = 1 and g2 = 0 in the position targeted attack, and g1 = 0 and g2 = 1 in the answer targeted attack.

heading: 3.3 ADVCODEC(WORD)
text: Not only we can apply perturbations to the root node of our tree-based autoencoder to generate adversarial sentence, we can also perturb nodes at different hierachical levels of the tree to generate adversarial word. The most general case is that the perturbation is directly exerted on the leaf node of the tree autoencoder, i.e. the word-level perturbation.
AdvCodec(Word) shares the exactly same architectures and optimization steps mentioned above to attack the targeted models. The distinction between AdvCodec(Word) and AdvCodec(Sent) is the context vector z. Formally for the word-level attack, the context vector z are the concatenation of leaf node embedding zi (which corresponds to each word) z = [z1, z2, . . . , zn]. Different from the AdvCodec(Sent) that perturbation is added on the whole sentence, we can control where the perturbations are added by assigning each node a mask as follows: z′i = zi + mask · z∗i (7) When we expect some token zi to be adversarially changed, we can simply assign mask = 1, thus adding the perturbation on the token.
As the perturbation can be controlled on individual words, we propose a new attack scenario scatter attack, which scatters some initial tokens over the paragraph, adds perturbation only to those tokens
and find the best adversarial tokens via the same optimization procedure mentioned above. Moreover, the concatenative adversarial examples (e.g. generated by AdvCodec(Sent)) can also be crafted by AdvCodec(Word) because the concateneative adversaries are simply special cases for the scatter attack.

heading: 4 EXPERIMENTAL RESULTS
text: In this section we will present the experimental evaluation results for AdvCodec. In particular, we target on two popular NLP tasks, sentiment classification and QA. For both models, we perform whitebox and transferability based blackbox attacks. In addition to the model accuracy (untargeted attack evaluation), we also report the targeted attack success rate for AdvCodec. We show that the proposed AdvCodec can outperform other state of the art baseline methods on different models.

heading: 4.1 SENTIMENT ANALYSIS
text: Task and Dataset. In this task, sentiment analysis model takes the user reviews from restaurants and stores as input and is expected to predict the number of stars (from 1 to 5 star) that the user was assigned. We choose the Yelp dataset (Challenge) for sentiment analysis task. It consists of 2.7M yelp reviews, in which we follow the process of Lin et al. (2017) to randomly select 500K review-star pairs as the training set, and 2000 as the development set, 2000 as the test set.
Model. BERT (Devlin et al., 2019) is a transformer (Vaswani et al., 2017) based model, which is unsupervisedly pretrained on a large corpus and is proven to be effective for downstream NLP tasks. Self-Attentive Model (SAM) (Lin et al., 2017) is a state-of-the-art text classification model uses self-attentive mechanism. More detailed model settings are listed in the appendix.
Baseline. Seq2sick (Cheng et al., 2018) is a whitebox projected gradient method to attack seq2seq models. Here, we perform seq2sick attack on sentiment classification models by changing its loss function, which was not evaluated in the original paper.TextFooler (Jin et al., 2019) is a simple yet strong blackbox attack method to generate adversarial text. Following the same setting, Seq2Sick and TextFooler is only allowed to edit the appended sentence or tokens.
Adversarial Evaluation. We perform the baseline attacks and our AdvCodec attack in scatter attack scenario and concat attack scenario under the whitebox settings. Our targeted goal for sentiment classification is the opposite sentiment. Specifically, we set the targeted attack goal as 5-star for reviews originally below 3-star and 1-star for reviews above. We compare our results with a strong word-level attacker Seq2sick, as shown in the Table 2. We can see our AdvCodec(Word) outperforms the baselines and achieves nearly 100% attack success rate on the BERT model. Also, we realize the targeted success rate for AdvCodec(Sent) is lower than the word-level baseline. We assume the reason is that AdvCodec(Sent) has the dependency tree constraints during decoding phase, thus increasing the difficulty to find both grammatical correct and adversarial sentences that can successful attack. On the contrary, the Seq2Sick baseline can edit any words under no semantic or syntactic constraints. Moreover, our following human evaluation exactly confirms AdvCodec(Sent) has better language quality.
Scatter Attack v.s. Concat Attack. In addition, we find scatter attack success rate is slightly lower than concat attack. We think there are two reasons to explain this phenomenon: Firstly, the average number of tokens added in scatter attack is 10, while the average number of tokens added in concat attack is 19. Therefore concat attack has the freedom to manipulate on more words than scatter
attack, thus resulting in higher attack accuracy. Secondly, inserting adversarial tokens to different positions of the passage also affects the success rate, which is shown in Appendix A.5.
Blackbox Attack. We perform transferability based blackbox attacks. We compare our blackbox attack success rate with the blackbox baseline TextFooler and blackbox Seq2Sick based on transferability. Table 3 demonstrates our AdvCodec(Word) model still has the best blackbox targeted and untargeted success rate among all the baseline models.

heading: 4.2 QUESTION ANSWERING (QA)
text: Task and Dataset. In this task, we choose the SQuAD dataset (Rajpurkar et al., 2016) for question answering task. The SQuAD dataset is a reading comprehension dataset consisting of 107,785 questions posed by crowd workers on a set of Wikipedia articles, where the answer to each question must be a segment of text from the corresponding reading passage. To compare our method with other adversarial evaluation works (Jia & Liang, 2017) on the QA task, we evaluate our adversarial attacks on the same test set as Jia & Liang (2017), which consists of 1000 randomly sampled examples from the SQuAD development set. We use the official script of the SQuAD dataset (Rajpurkar et al., 2016) to measure both adversarial exact match rates and F1 scores.
Model. We adapt the BERT model to run on SQuAD v1.1 with the same strategy as that in Devlin et al. (2019), and we reproduce the result on the development set. BiDAF(Seo et al., 2016) is a multi-stage hierarchical process that represents the context at different levels of granularity and uses bidirectional attention flow mechanism to obtain a query-aware context representation.
Baseline. Universal Adversarial Triggers (Wallace et al., 2019) are input-agnostic sequences of tokens that trigger a model to produce a specific prediction when concatenated to any input from a dataset. Here, we compare the targeted attack ability of AdvCodec with it. AddSent (Jia & Liang, 2017) appends a manually constructed legit distracting sentence to the given text so as to introduce fake information, which can only perform untargeted attack.
Adversarial Evaluation. We perform the whitebox attack with different attack methods on our testing models. As is shown in Table 4 , AdvCodec(Word) achieves the best whitebox attack results on both BERT and BiDAF. It is worth noting although BERT has better performances than BiDAF, the performance drop for BERT ∆F1BERT is 55.4 larger than the performance drop for BiDAF ∆F1BiDAF = 53.0, which again proves the BERT is insecure under the adversarial evaluation. We also find the position targeted attack is slightly stronger than the answer targeted attack. We assume it is because the answer targeted attack has fixed targeted answer and limited freedom to alter the appended sentence, but the position targeted attack has more freedom to alter the fake answer from
the targeted position spans. We also tried the scatter attack on QA though the performances are not good. It turns out QA systems highly rely on the relationship between questions and contextual clues, which is hard to break when setting an arbitrary token to a target answer. We discussed in Appendix A.3 the untargeted scatter attack can work well and outperform the baseline methods.
BERT target EM 32.1 43.4 1.4target F1 32.4 46.5 2.1
BiDAF target EM 53.3 71.2 21.2target F1 56.8 75.6 22.6
Then we test the targeted results of whitebox attack methods on QA models. The results are shown in Table 5. It shows that AdvCodec(Word) has the best targeted attack ability on QA. And all our attack methods outperform the baseline(Universal Triggers) when it comes to the targeted results.
Blackbox Attack. We also transfer adversarial texts generated from whitebox attacks to perform blackbox attacks. Table 6 shows the result of the blackbox attack on testing models. All our proposed methods outperform the baseline method(AddSent) when transferring the adversaries among models with same architectures.

heading: 5 HUMAN EVALUATION
text: We conduct a thorough human subject evaluation to assess the human response to different types of generated adversarial text. The main conclusion is that even though these adversarial examples are effective at attacking machine learning models, they are much less noticeable by humans.

heading: 5.1 COMPARISON OF ADVERSARIAL TEXT QUALITY
text: To understand what humans think of our adversarial data quality, we present the adversarial text generated by AdvCodec(Sent) and AdvCodec(Word) based on the same initial seed. Human participants are asked to choose which data they think has better language quality.
In this experiement, we prepare 600 adversarial text pairs from the same paragraphs and initial seeds. We hand out these pairs to 28 Amazon Turks. Each turk is required to annotate at least 20 pairs and at most 140 pairs to ensure the task has been well understood. We assign each pair to at least 5 unique turks and take the majority votes over the responses. Human evaluation results are shown in Table 7, from which we see that the overall vote ratio for AdvCodec(Sent) is 66%, meaning
AdvCodec(Sent) has better language quality than AdvCodec(Word) from a human perspective. This is due to the fact that AdvCodec(Sent) more fully harness the tree-based autoencoder structure compared to AdvCodec(Sent). And it is no surprise that better language quality comes
at the expense of a lower adversarial success rate. As Table 2 shows, the adversarial targeted success rate of AdvCodec(Sent) on SAM is 20% lower than that of AdvCodec(Word), which confirms the trade-off between language quality and adversarial success rate.

heading: 5.2 HUMAN PERFORMANCE ON ADVERSARIAL TEXT
text: Table 8: Human performance on Sentiment Analysis
Method Majority Acc
Origin 0.95 AdvCodec(Word) 0.82 AdvCodec(Sent) 0.82
Table 9: Human performance on QA
Method Majority F1
Origin 90.987 AdvCodec(Word) 82.897 AdvCodec(Sent) 81.784
To ensure that our generated adversarial text are compatible with the original paragraph, we ask human participants to perform the sentiment classification and question answering task both on the original dataset and adversarial dataset. Adversarial dataset on sentiment classification consists of AdvCodec(Sent) concatenative adversarial examples and AdvCodec(Word) scatter attack exmaples. Adversarial dataset on QA consists of concatenative adversarial examples genereated by both AdvCodec(Sent) and AdvCodec(Word). More specifically, we respectively prepare 100 benign and adversarial data pairs for both QA and sentiment classification, and hand out them to 505 Amazon Turks. Each turk is requested to answer at least 5 question and at most 15 questions for the QA task and judge the sentiment for at least 10 paragraphs and at most 20 paragraphs for the sentiment classification task. We also perform a majority vote over Turk’s answers for the same question. The human evaluation results are displayed in Table 8 and Table 9, from which we see that most of our concatenated adversarial text are compatible to the paragraph. While we can spot a drop from the benign to adversarial datasets, we conduct an error analysis in QA and find the error examples are noisy and not necessarily caused by our adversarial text. For adversarial data in the sentiment classification task, we notice that the generated tokens or appended sentences have opposite sentiment from the benign one. However, our evaluation results show human readers can naturally ignore abnormal tokens and make correct judgement according to the context.

heading: 6 DISCUSSION AND FUTURE WORKS
text: Besides the conclusions pointed out in the Introduction section, we also summarize some interesting findings: (1) While AdvCodec(Word) achieves best attack success rate among multiple tasks, we observe a trade-off between the freedom of manipulation and the attack capability. For instance, AdvCodec(Sent) has dependency tree constraints and becomes more natural for human readers than but less effective to attack models than AdvCodec(Word). Similarly, the answer targeted attack in QA has fewer words to manipulate and change than the position targeted attack, and therefore has slightly weaker attack performances. (2) Scatter attack is as effective as concat attack in sentiment classification task but less successful in QA, because QA systems make decisions highly based on the contextual correlation between the question and the paragraph, which makes it difficult to set an arbitrary token as our targeted answer. (3) Transferring adversarial text from models with better performances to weaker ones is more successful. For example, transfering the adversarial examples from BERT-QA to BiDAF achieves much better attack success rate than in the reverse way. (4) We also notice adversarial examples have better transferability among the models with similar architectures than different architectures. (5) BERT models pay more attention to the both ends of the paragraphs and tend to overlook the content in the middle, as shown in Appendix A.5 ablation study that adding adversarial sentences in the middle of the paragraph is less effective than in the front or the end. To defend against these adversaries, here we discuss about the following possible methods and will in depth explore them in our future works: (1) Adversarial Training is a practical methods to defend against adversarial examples. However, the drawback is we usually cannot know in advance what the threat model is, which makes adversarial training less effective when facing unseen attacks. (2) Interval Bound Propagation (IBP) (Dvijotham et al., 2018) is proposed as a new technique to theoretically consider the worst-case perturbation. Recent works (Jia et al., 2019; Huang et al., 2019) have applied IBP in the NLP domain to certify the robustness of models. (3) Language models including GPT2 (Radford et al., 2019) may also function as an anomaly detector to probe the inconsistent and unnatural adversarial sentences.

heading: A ADVCODEC SETTINGS

heading: A.1 AUTOENCODER SELECTION
text: Seq2seq autoencoder. We also tried the tradition sequential architecture (seq2seq) as a different autoencoder in the AdvCodec pipeline. For the seq2seq encoder-decoder, we use a bi-directional LSTM as the encoder (Hochreiter & Schmidhuber, 1997) and a two-layer LSTM plus soft attention mechanism over the encoded states as the decoder (Bahdanau et al., 2015).
During the attack, the LSTM cell sequentially takes the embedding of each word xi as input and output the encoded state hi. The context vector z here refers to the last step’s output hn of the encoder LSTM cell. The perturbation z∗ is added only on the context vector hn without influencing previous encoded states hi (i < n).
As the ablation study, we compare its whitebox attack capability with our AdvCodec on BiDAF on QA task. As table 10 shows, we can see seq2seq based AdvCodec cannot achieve good attack success rate. Moreover, because seq2seq model does not take grammatical constraints into consideration, the quality of generated adversarial text cannot be ensured.
Tree autoencoder. In the whole experiments, we used Stanford TreeLSTM as tree encoder and our proposed tree decoder together as tree autoencoder. We trained the tree autoencoder on yelp dataset which contains 500K reviews. The model is expected to read a sentence, map the sentence in a latent space and reconstruct the sentence from the embedding along with the dependency tree structure in an unsupervised manner. The model uses 300-d vectors as hidden tree node embedding and is trained for 30 epochs with adaptive learning rate and weight decay. After training, the average reconstruction loss on test set is 0.63.

heading: A.2 ATTACK SETTINGS
text: We used the Carlini & Wagner (2016) attack as the optimization procudure to search for the optimal z∗ that can attack the targeted model. We update z∗ iteratively via gradient descent over the optimization function (5) and (6) for different tasks. We use Adam (Kingma & Ba, 2014) as the optimizer, set the learning rate to 0.6 and the optimization steps to 100. We follow the Carlini & Wagner (2016) method to find the suitable parameters in the object function (weight const c and confidence score κ) by binary search.
Algorithm 1 Algorithm of AdvCodec generating adversarial examples 1: procedure ADVCODEC(x, s) . x: initial seed, s: corresponding dependency tree 2: z := E(x, s) . E : encoder of AdvCodec, z: context vector 3: z∗ = 0 . z∗: perturbation on context vector 4: z′ := z + z∗ . z′: perturbed context vector 5: y := G(z′, s) . G: decoder of AdvCodec, y: adversarial sentence 6: f(z′) := the objective function to attack the targeted model 7: while y does not achieve targeted attack do 8: update z∗ by gradient descent over objective function f(z′) 9: end while 10: return y 11: end procedure
We also include our attack algorithm via pseudo-code in Algorithm 1.

heading: A.3 UNTARGETED SCATTER ATTACK ON QA
text: We tried the scatter attack on QA, however, the targeted attack success rate is not satisfactory. It turns out QA systems highly rely on the relationship between questions and contextual clues, which is hard to break when setting an arbitrary token to a target answer. This is also why we use some preliminary approaches to creating a similar fake context when initializing QA appended sentence.
We also performed the untargeted scatter attack on QA. The results are shown in table 11. We insert 30 random tokens (but no more than 1/3 the total words of the paragraph) over the paragraph, optimize and find the adversarial tokens that can cause model output the wrong answers in the untargeted manner. We can see the untargeted scatter attack can also achieve a higher untargeted attack success rate than Jia & Liang (2017).

heading: A.4 HEURISTIC EXPERIMENTS ON CHOOSING THE INITIAL SEED FOR QA
text: We conduct the following heuristic experiments about how to choose a good initialization sentence to more effectively attack QA models. Based on the experiments we confirm it is important to choose a sentence that is semantically close to the context or the question as the initial seed when attacking QA model, so that we can reduce the number of iteration steps and more effectively find the adversary to fool the model. Here we describe three ways to choose the initial sentence, and we will show the efficacy of these methods given the same maximum number of optimization steps.
Random initial sentence. Our first trial is to use a random sentence (other than the answer sentence), generate a fake answer similar to the real answer and append it to the back as the initial seed.
Question-based initial sentence. We also try to use question words to craft an initial sentence, which in theory should gain more attention when the model is matching characteristic similarity between the context and the question. To convert a question sentence to a meaningful declarative statement, we use the following steps:
In step 1, we use the state-of-the-art semantic role labeling (SRL) tools (He et al., 2017) to parse the question into verbs and arguments. A set of rules is defined to remove the arguments that contain interrogative words and unimportant adjectives, and so on. In the next step, we access the model’s original predicted answer and locate the answer sentence. We again run the SRL parsing and find to which argument the answer belongs. The whole answer argument is extracted, but the answer tokens are substituted with the nearest words in the GloVe word vectors (Pennington et al., 2014) that is also used in the QA model. In this way, we craft a fake answer that shares the answer’s context to solve the compatibility issue from the starting point. Finally, we replace the declarative sentence’s removed arguments with the fake argument and choose this question-based sentence as our initial sentence.
Answer-based initial sentence. We also consider directly using the model predicted original answer sentence with some substitutions as the initial sentence. To craft a fake answer sentence is much easier than to craft from the question words. Similar to step 2 for creating question-based initial sentence, we request the model’s original predicted answer and find the answer sentence. The answer span in the answer sentence is directly substituted with the nearest words in the GloVe word vector space to avoid the compatibility problem preliminarily.
Experimental Results. We tried the above initial sentence selection methods on AdvCodec(Word) and perform position targeted attack on BERT-QA given the same maximum optimization steps. The experiments results are shown in table 12. From the table, we find using different initialization methods will greatly affect the attack success rates. Therefore, the initial sentence selection methods are indeed important to help reduce the number of iteration steps and fastly converge to the optimal z∗ that can attack the model.

heading: A.5 ABLATION STUDY ON MODEL ATTENTION
text: To further explore how the location of adversarial sentences affects the attack success rate, we conduct the ablation experiments by varying the position of appended adversarial sentence. We generate the adversarial sentences from the whitebox BERT classification and QA models. Then we inject those adversaries into different positions of the original paragraph and test in another blackbox BERT with the same architecture but different parameters. The results are shown in Table 13 and 14. We see in most time appending the adversarial sentence at the beginning of the paragraph achieves the best attack performance. Also the performance of appending the adversarial sentence at the end of the paragraph is usually slightly weaker than front. This observation suggests that the BERT model might pay more attention to the both ends of the paragraphs and tend to overlook the content in the middle.
Table 14: Blackbox Attack Success Rate after inserting the whitebox generated adversarial sentence to different positions for BERT-QA.
Method Back Mid Front
Adv(Word) EM 32.3 39.1 31.9F1 36.4 43.4 36.3
Adv(Sent) EM 47.0 51.3 42.4F1 52.0 56.7 47.0

heading: B MODEL SETTINGS & HUMAN EVALUATION

heading: B.1 SENTIMENT CLASSIFICATION MODEL
text: BERT. We use the 12-layer BERT-base model 1 with 768 hidden units, 12 self-attention heads and 110M parameters. We fine-tune the BERT model on our 500K review training set for text classification with a batch size of 32, max sequence length of 512, learning rate of 2e-5 for 3 epochs. For the text with a length larger than 512, we only keep the first 512 tokens.
Self-Attentive Model (SAM). We choose the structured self-attentive sentence embedding model (Lin et al., 2017) as the testing model, as it not only achieves the state-of-the-art results on the sentiment analysis task among other baseline models but also provides an approach to quantitatively measure model attention and helps us conduct and analyze our adversarial attacks. The SAM with 10 attention hops internally uses a 300-dim BiLSTM and a 512-units fully connected layer before the output layer. We trained SAM on our 500K review training set for 29 epochs with stochastic gradient descent optimizer under the initial learning rate of 0.1.
1https://github.com/huggingface/pytorch-pretrained-BERT

heading: B.2 SENTIMENT CLASSIFICATION ATTACK BASELINE
text: Seq2sick (Cheng et al., 2018) is a whitebox projected gradient method combined with group lasso and gradient regularization to craft adversarial examples to fool seq2seq models. Here, we define the loss function as Ltarget = max
k∈Y
{ z(k) } − z(t) to perform attack on sentiment classification models
which was not evaluated in the original paper. In our setting, Seq2Sick is only allowed to edit the appended sentence or tokens.
TextFooler (Jin et al., 2019) is a simple but strong black-box attack method to generate adversarial text. Here, TextFooler is also only allowed to edit the appended sentence.

heading: B.3 QA MODEL
text: BiDAF. Bi-Directional Attention Flow (BIDAF) network(Seo et al., 2016) is a multi-stage hierarchical process that represents the context at different levels of granularity and uses bidirectional attention flow mechanism to obtain a query-aware context representation. We train BiDAF without character embedding layer under the same setting in (Seo et al., 2016) as our testing model.

heading: B.4 HUMAN ERROR ANALYSIS IN ADVERSARIAL DATASET
text: We compare the human accuracy on both benign and adversarial texts for both tasks (QA and classification) in revision section 5.2. We spot the human performance drops a bit on adversarial texts. In particular, it drops around 10% for both QA and classification tasks based on AdvCodec as shown in Table 10 and 11. We believe this performance drop is tolerable and the stoa generic based QA attack algorithm experienced around 14% performance drop for human performance (Jia & Liang, 2017).
We also try to analyze the human error cases. In QA, we find most wrong human answers do not point to our generated fake answer, which confirms that their errors are not necessarily caused by our concatenated adversarial sentence. Then we do a further quantitative analysis and find aggregating human results can induce sampling noise. Since we use majority vote to aggregate the human answers, when different answers happen to have the same votes, we will randomly choose one as the final result. If we always choose the answer that is close to the ground truth in draw cases, we later find that the majority vote F1 score increases from 82.897 to 89.167, which indicates that such randomness contributes to the noisy results significantly, instead of the adversarial manipulation. Also, we find the average length of the adversarial paragraph is around 12 tokens more than the average length of the original one after we append the adversarial sentence. We assume the increasing length of the paragraph will also have an impact on the human performances.

heading: C ADVERSARIAL EXAMPLES

heading: C.1 ADVERSARIAL EXAMPLES FOR QA
text: C.1.1 ADVERSARIAL EXAMPLES GENERATED BY ADVCODEC(SENT)

heading: C.1.2 ADVERSARIAL EXAMPLES GENERATED BY ADVCODEC(WORD)

heading: C.2 ADVERSARIAL EXAMPLES FOR CLASSIFICATION

heading: C.2.1 ADVERSARIAL EXAMPLES GENERATED BY ADVCODEC(SENT)

heading: C.2.2 ADVERSARIAL EXAMPLES GENERATED BY ADVCODEC(WORD)
text: Again for both scatter attack and concat attack, the word level manipulation does not take global (sentence-level) grammatical constraints into consideration, it is expected to observe more “free” manipulation than AdvCodec(Sent) and achieves a higher attack success rate at the expense of grammatical correctness.

Review summaries:
This paper proposes a new attack framework AdvCodec for adversarial text generation. The main idea is to use a tree-based autoencoder to embed text data into the continuous vector space and then optimize to find the adversarial perturbation in the vector space. The authors consider two types of attacks: concat attack and scatter attack. Experimental results on sentiment analysis and question answering, together with human evaluation on the generated adversarial text, are provided. 

Motivated by recent development of attack/defense methods addressing the vulnerability of deep CNN classifiers for images, this paper proposes an attack framework for adversarial text generation, in which an autoencoder is employed to map discrete text to a high-dimensional continuous latent space, standard iterative optimization based attack method is performed in the continuous latent space to generate adversarial latent embeddings, and a decoder generates adversarial text from the adversarial embeddings.  Different generation strategies of perturbing latent embeddings at sentence level or masked word level are both explored. Adversarial text generation can take either a form of appending an adversarial sentence or a form of scattering adversarial words into different specified positions. Experiments on both sentiment classification and question answering show that the proposed attack framework outperforms some baselines. Human evaluations are also conducted.