Venue: NAACL 2021

# Contextualized Perturbation for Textual Adversarial Attack

Dianqi Li\({}^{\spadesuit}\) Yizhe Zhang\({}^{\diamondsuit}\) Hao Peng\({}^{\spadesuit}\) Liqun Chen\({}^{\clubsuit}\) Chris Brockett\({}^{\diamondsuit}\) Ming-Ting Sun\({}^{\spadesuit}\) Bill Dolan\({}^{\diamondsuit}\) University of Washington \({}^{\diamondsuit}\)Microsoft Research \({}^{\clubsuit}\)Duke University {dianqili, mts}@uw.edu, hapeng@cs.uw.edu, liqun.chen@duke.edu {Yizhe.Zhang, Chris.Brockett, billdol}@microsoft.com

###### Abstract

Adversarial examples expose the vulnerabilities of natural language processing (NLP) models, and can be used to evaluate and improve their robustness. Existing techniques of generating such examples are typically driven by local heuristic rules that are agnostic to the context, often resulting in unnatural and ungrammatical outputs. This paper presents CLARE, a **C**ontextua**L**ized **A**dversa**R**ial **E**xample generation model that produces fluent and grammatical outputs through a mask-then-infill procedure. CLARE builds on a pre-trained masked language model and modifies the inputs in a context-aware manner. We propose three contextualized perturbations, _Replace_, _Insert_ and _Merge_, that allow for generating outputs of varied lengths. CLARE can flexibly combine these perturbations and apply them at any position in the inputs, and is thus able to attack the victim model more effectively with fewer edits. Extensive experiments and human evaluation demonstrate that CLARE outperforms the baselines in terms of attack success rate, textual similarity, fluency and grammaticality.

+
Footnote †: _Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 5053–5069 June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Adversarial example generation for natural language processing (NLP) tasks aims to perturb input text to trigger errors in machine learning models, while keeping the output close to the original. Besides exposing system vulnerabilities and helping improve their robustness and security Zhao et al. (2018); Wallace et al. (2019); Cheng et al. (2019); Jia et al. (2019), _inter alia_), adversarial examples are also used to analyze and interpret the models' decisions Jia and Liang (2017); Ribeiro et al. (2018).

Generating adversarial examples for NLP tasks can be challenging, in part due to the discrete nature of natural language text. Most recent efforts have explored heuristic rules, such as replacing tokens with their synonyms Samanta and Mehta (2017); Liang et al. (2019); Alzantot et al. (2018); Ren et al. (2019); Jin et al. (2020), _inter alia_). Despite some empirical success, rule-based methods are agnostic to context, limiting their ability to produce natural, fluent, and grammatical outputs Wang et al. (2019); Kurita et al. (2020), _inter alia_).

This work presents CLARE, a **C**ontextua**L**ized **A**dversa**R**ial **E**xample generation model for text. CLARE perturbs the input with a mask-then-infill procedure: it first detects the vulnerabilities of a model and deploys masks to the inputs to indicate missing text, then plugs in an alternative using a pretrained masked language model (e.g., RoBERTa; Liu et al., 2019). CLARE features three contextualized perturbations: _Replace_, _Insert_ and _Merge_, which respectively replace a token, insert a new one, and merge a bigram (Figure 1). As a result, it can generate outputs of varied lengths, in contrast to token replacement based methods that are limited to outputs of the same lengths as the inputs Alzantot et al. (2018); Ren et al. (2019);

Figure 1: Illustration of CLARE. Through a mask-then-infill procedure, the model generates the adversarial text with three contextualized perturbations: _Replace_, _Insert_ and _Merge_. A mask is indicated by “...”. The degree of fade corresponds to the (decreasing) priority of the infill tokens.



Jin et al., 2020). Further, CLARE searches over a wider range of attack strategies, and is thus able to attack the victim model more effectively with fewer edits. Building on a masked language model, CLARE maximally preserves textual similarity, fluency, and grammaticality of the outputs.

We evaluate CLARE on text classification, natural language inference, and sentence paraphrase tasks, by attacking finetuned BERT models Devlin et al. (2019). Extensive experiments and human evaluation results show that CLARE outperforms baselines in terms of attack success rate, textual similarity, fluency, and grammaticality, and strikes a better balance between attack success rate and preserving input-output similarity. Our analysis further suggests that the CLARE can be used to improve the robustness of the downstream models, and improve their accuracy when the available training data is limited. We release our code and models at [https://github.com/cookielee77/CLARE](https://github.com/cookielee77/CLARE).

## 2 Clare

At a high level, CLARE applies a sequence of contextualized perturbation actions to the input. Each can be seen as a _local_ mask-then-infill procedure: it first applies a mask to the input around a given position, and then fills it in using a pretrained masked language model (SS2.1). To produce the output, CLARE scores and descendingly ranks the actions, which are then iteratively applied to the input (SS2.2). We begin with a brief background review and laying out of necessary notation.

Background.Adversarial example generation centers around a **victim** model \(f\), which we assume is a text classifier. We focus on the black-box setting, allowing access to \(f\)'s outputs but _not_ its configurations such as parameters. Given an input sequence \({\bf x}=x_{1}x_{2}\ldots x_{n}\) and its label \(y\) (assume \(f({\bf x})=y\)), an **adversarial example**\({\bf x^{\prime}}\) is supposed to modify \({\bf x}\) to trigger an error in the victim model: \(f({\bf x^{\prime}})\neq f({\bf x})\). At the same time, textual modifications should be minimal, such that \({\bf x^{\prime}}\) is close to \({\bf x}\) and the human predictions on \({\bf x^{\prime}}\) stay the same.1

Footnote 1: In computer vision applications, minor perturbations to continuous pixels can be barely perceptible to humans, thus it can be hard for one to distinguish \({\bf x}\) and \({\bf x^{\prime}}\)Goodfellow et al. (2015). It is not the case for text, however, since changes to the discrete tokens are more likely to be noticed by humans.

This is achieved by requiring the similarity between \({\bf x^{\prime}}\) and \({\bf x}\) to be larger than a threshold: \(\operatorname*{sim}({\bf x^{\prime}},{\bf x})>\ell\). A common choice of \(\operatorname*{sim}(\cdot,\cdot)\) is to encode sentences using neural networks, and calculate their cosine similarity in the embedding space Jin et al. (2020).

### Masking and Contextualized Infilling

At a given position of the input sequence, CLARE can execute three perturbation actions: _Replace_, _Insert_, and _Merge_, which we introduce in this section. These apply masks at the given position with different strategies, and then fill in the missing text based on the unmasked context.

_Replace:_A _Replace_ action substitutes the token at a given position \(i\) with an alternative (e.g., changing "_fundastic_" to "_amazing_" in "The movie is _fantistic_."). It first replaces \(x_{i}\) with a mask, and then selects a token \(z\) from a candidate set \(\mathcal{Z}\) to fill in:

\[\widetilde{{\bf x}} =x_{1}\dots x_{i-1}\text{ [Mask] $x_{i+1}\dots x_{n}$},\] \[\operatorname*{replace}\left({\bf x},i\right) =x_{1}\dots x_{i-1}~{}z~{}x_{i+1}\dots x_{n}.\]

For clarity, we denote \(\operatorname*{replace}\left({\bf x},i\right)\) by \(\widetilde{{\bf x}}_{z}\). To produce an adversarial example,

* \(z\) should fit into the unmasked context;
* \(\widetilde{{\bf x}}_{z}\) should be similar to \({\bf x}\);
* \(\widetilde{{\bf x}}_{z}\) should trigger an error in \(f\).

These can be achieved by selecting a \(z\) such that

* \(z\) receives a high probability from a masked language model: \(p_{\textrm{MLM}}(z\mid\widetilde{{\bf x}})>k\);
* \(\widetilde{{\bf x}}_{z}\) is similar to \({\bf x}\): \(\operatorname*{sim}({\bf x},\widetilde{{\bf x}}_{z})>\ell\);
* \(f\) predicts low probability for the gold label given \(\widetilde{{\bf x}}_{z}\), i.e., \(p_{f}(y\mid\widetilde{{\bf x}}_{z})\) is small.

\(p_{\textrm{MLM}}\) denotes a pretrained masked language model (e.g., RoBERTa; Liu et al., 2019). Using higher \(k\), \(\ell\) thresholds produces outputs that are more fluent and closer to the original. However, this can undermine the success rate of the attack. We choose \(k\), \(\ell\) to trade-off between these two aspects. 2

Footnote 2: \(k\) and \(\ell\) are empirically set as \(5\times 10^{-3}\) and \(0.7\), respectively. This also reduces the computation overhead: in our experiments \(|\mathcal{Z}|\) is \(42\) on average, much smaller than the vocabulary size (\(|\mathcal{V}|=50,265\)).

The first two requirements can be met by the construction of the candidate set: \(\mathcal{Z}=\\ \big{\{}z^{\prime}\in\mathcal{V}\mid p_{\textrm{MLM}}(z^{\prime}\mid \widetilde{{\bf x}})>k,\operatorname*{sim}({\bf x},\widetilde{{\bf x}}_{z^{ \prime}})>\ell\big{\}}\) .

\(\mathcal{V}\) is the vocabulary of the masked language model. To meet the third, we select from \(\mathcal{Z}\) the token that, if filled in, will cause most "confusion" to \(f\):

\[z=\mathop{arg\min}_{z^{\prime}\in\mathcal{Z}}p_{f}(y\mid\widetilde{{\bf x}}_ {z^{\prime}}).\] (1) 

The _Insert_ and _Merge_ actions differ from _Replace_ in terms of masking strategies. The alternative token \(z\) is selected analogously to that in a _Replace_ action.

Invert:This aims to add extra information to the input (e.g., changing "I recommend ..." to "I _highly_ recommend ..."). It inserts a mask after \(x_{i}\) and then fills it. Slightly overloading the notations,

\[\widetilde{\mathbf{x}} =x_{1}\ldots x_{i}\text{ [Mask] $x_{i+1}\ldots x_{n}$},\] \[\operatorname*{insert}\left(\mathbf{x},i\right) =x_{1}\ldots x_{i}~{}z~{}x_{i+1}\ldots x_{n}.\]

This increases the sequence length by 1.

Merge:This masks out a bigram \(x_{i}x_{i+1}\) with _a single_ mask and then fills it, reducing the sequence length by 1:

\[\widetilde{\mathbf{x}} =x_{1}\ldots x_{i-1}\text{ [Mask] $x_{i+2}\ldots x_{n}$},\] \[\operatorname*{merge}\left(\mathbf{x},i\right) =x_{1}\ldots x_{i-1}~{}z~{}x_{i+2}\ldots x_{n}.\]

\(z\) can be the same as one of the masked tokens (e.g., masking out "New York" and then filling in"York"). This can be seen as deleting a token from the input.

For _Insert_ and _Merge_, \(z\) is chosen in the same manner as replace action. 3

Footnote 3: A perturbation will not be considered if its candidate token set is empty.

```
1:Input: Text-label pair \((\mathbf{x},y)\); Victim model \(f\)
2:Output: An adversarial example
3:Initialization:\(\mathbf{x}^{(0)}=\mathbf{x}\)
4:\(\mathcal{A}\leftarrow\varnothing\)
5:for\(1\leq i\leq|\mathbf{x}|\)do
6:\(a\leftarrow\) highest-scoring action from \(\{\\ \operatorname*{replace}(\mathbf{x},i)\), \(\operatorname*{insert}(\mathbf{x},i),\operatorname*{merge}(\mathbf{x},i)\}\)
7:\(\mathcal{A}\leftarrow\mathcal{A}\bigcup\{a\}\)
8:endfor
9:for\(1\leq t\leq T\)do
10:\(a\leftarrow\) highest-scoring action from \(\mathcal{A}\)
11:\(\mathcal{A}\leftarrow\mathcal{A}\setminus\{a\}\)
12:\(\mathbf{x}^{(t)}\leftarrow\) Apply \(a\) on \(\mathbf{x}^{(t-1)}\)
13:if\(f(\mathbf{x}^{(t)}){\neq}y\)thenreturn\(\mathbf{x}^{(t)}\)
14:endif
15:endfor
16:returnNone
```

**Algorithm 1** Adversarial Attack by CLARE

In sum, at each position \(i\) of an input sequence, CLARE first: \((i)\) replaces \(x_{i}\) with a mask; \((ii)\) or inserts a mask after \(x_{i}\); \((iii)\) or merges \(x_{i}x_{i+1}\) into a mask. Then a set of candidate tokens is constructed with a masked language model and a textual similarity function; the token minimizing the gold label's probability is chosen as the alternative token. The combination of these three operations enables conversion between any two sequences.

CLARE first constructs the local actions for all positions in parallel, i.e., the actions at position \(i\) do not affect those at other positions. Then, to produce the adversarial example, CLARE gathers the local actions and selects an order to execute them.

### Sequentially Applying the Perturbations

Given an input pair \((\mathbf{x},y)\), let \(n\) denote the length of \(\mathbf{x}\). CLARE chooses from \(3n\) actions to produce the output: 3 actions for each position, assuming the candidate token sets are not empty. We aim to generate an adversarial example with minimum modifications to the input. To achieve this, we iteratively apply the actions, and first select those minimizing the probability of outputting the gold label \(y\) from \(f\).

Each action is associated with a score, measuring how likely it can "confuse" \(f\): denote by \(a(\mathbf{x})\) the output of applying action \(a\) to \(\mathbf{x}\). The score is then the negative probability of predicting the gold label from \(f\), using \(a(\mathbf{x})\) as the input:

\[s_{(\mathbf{x},y)}(a)=-p_{f}\big{(}y~{}|~{}a(\mathbf{x})\big{)}.\]

Only one of the three actions can be applied at each position, and we select the one with the highest score. This constraint aims to avoid multiple modifications around the same position, e.g., merging "New York" into "Seattle" and then replacing it with "Boston".

Actions are iteratively applied to the input, until an adversarial example is found or a limit of actions \(T\) is reached. Each step selects the highest-scoring action from the remaining ones. Algorithm 1 summarizes the above procedure.4

Footnote 4: Insert and Merge actions change the text length. When any of them is applied, we accordingly change the text indices of affected actions remaining in \(\mathcal{A}\).

Discussion.A key technique of CLARE is the local mask-then-infill perturbation. Compared with existing context-agnostic replacement approaches Alzantot et al. (2018); Jin et al. (2020); Ren et al. (2019), _inter alia_), contextualized infilling produces more fluent and grammatical outputs. Generating adversarial examples with masked language models is also explored by concurrent work 

BERTAttack Li et al. (2020) and BAE Garg and Ramakrishnan (2020).5

Footnote 4: [https://github.com/facebook-challenge-research/](https://github.com/facebook-challenge-research/)

* BERTAttack only replaces tokens and thus can only produce outputs of the same lengths as the inputs. This is analogous with a CLARE model with the _Replace_ action only. BAE entangles replacing and inserting tokens: it inserts _only_ at positions neighboring a replaced token, limiting its attacking capability. Departing from both, CLARE uses three different perturbations (_Replace_, _Insert_ and _Merge_), each allowing efficient attacking against _any_ position of the input, and can produce outputs of varied lengths. As we will show in the experiments (SS3.3), CLARE outperforms both these methods.
* When selecting the attack positions, neither BERAttack or BAE takes into account the tokens to be infilled, whereas CLARE does. This results in better adversarial attack performance according to our ablation study (SS4.1).
* CLARE demonstrates the advantage of using RoBERTa over BERT, which was used in the concurrent works (SS4.1).

## 3 Experiments

We evaluate CLARE on text classification, natural language inference, and sentence paraphrase tasks. We begin by describing the implementation details of CLARE and the baselines (SS3.1). SS3.2 introduces the experimental datasets and the evaluation metrics; the results are summarized in SS3.3.

### Setup

* We experiment with a distilled version of RoBERTa (RoBERTa distill; Sanh et al. (2019) as the masked language model for contextualized infilling. We also compare to base sized RoBERTa (RoBERTabase; Liu et al. (2019) and base sized BERT (BERTbase; Devlin et al. (2019) in the ablation study (SS4.1).
* The similarity function builds on the universal sentence encoder (USE; Cer et al. (2018).
* The victim model is an MLP classifier on top of \({\rm BERT}_{\rm base}\). It takes as input the first token's contextualized representation. We finetune BERT when training the victim model.

Baselines.We compare CLARE with recent state-of-the-art word-level black-box adversarial attack models, including:

* **TextFooler**: a state-of-the-art model by Jin et al. (2020). This replaces tokens with their synonyms derived from counter-fitting word embeddings Mrksic et al. (2016), and uses the same text similarity function as our work.
* **TextFooler+LM**: an improved variant of TextFooler we implemented based on Alzantot et al. (2018) and Cheng et al. (2019). This inherits token replacement from TextFooler, but uses an additional small sized GPT-2 language model Radford et al. (2019) to filter out those candidate tokens that do not fit in the context with calculated perplexity.
* **BERAttack**: a mask-then-infill approach by Li et al. (2020). It greedily replaces tokens with the predictions from BERT. BAE is not listed as it has a similar performance as BERAttack Garg and Ramakrishnan (2020).

We use the open source implementation of the above baselines provided by the authors. More details are included in Appendix SSA.1.

### Datasets and Evaluation

Datasets.We evaluate CLARE with the following datasets:

* **Yelp Reviews**Zhang et al. (2015): a binary sentiment classification dataset based on restaurant reviews.
* **AG News**Zhang et al. (2015): a collection of news articles with four categories: _World_, _Sports_, _Business_ and _Science & Technology_.
* **MNLI**Williams et al. (2018): a natural language inference dataset. Each instance consists of a premise-hypothesis pair, and the model is supposed to determine the relation between them from a label set of _entailment_, _neutral_, and _contradiction_. It covers text from a variety of domains.

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Dataset** & **Avg. Length** & **\# Classes** & **Train** & **Test** & **Acc** \\ \hline Yelp & 130 & 2 & 560K & 38K & 95.9\% \\ AG News & 46 & 4 & 120K & 7.6K & 95.0\% \\ \hline MNLI6  & 23/11 & 3 & 392K & 9.8K & 84.3\% \\ QNLI & 11/31 & 2 & 105K & 5.4K & 91.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 1: Some statistics of datasets. The last column indicates the victim model’s accuracy on the original test set _without_ adversarial attack.

 

* **QNLI**(Wang et al., 2019): a binary classification dataset converted from the Stanford question answering dataset Rajpurkar et al. (2016). The task is to determine whether the context contains the answer to a question. It is mainly based on English Wikipedia articles.

Table 1 summarizes some statistics of the datasets. In addition to the above four datasets, we experiment with DBpedia ontology dataset Zhang et al. (2015), Stanford sentiment treebank Socher et al. (2013), Microsoft Research Paraphrase Corpus Dolan and Brockett (2005), and Quora Question Pairs from the GLUE benchmark. The results on these datasets are summarized in Appendix A.2.

Following previous practice Alzantot et al. (2018), we fine-tune CLARE on training data, and evaluate with 1,000 randomly sampled test instances of lengths \(\leq 100\). In the sentence-pair tasks (e.g., MNLI, QNLI), we attack the longer sentence excluding the tokens that appear in both.

Evaluation metrics.We follow previous works Jin et al. (2020); Morris et al. (2020), and evaluate the models with the following automatic metrics:

* **Attack success rate (A-rate)**: the percentage of adversarial examples that can successfully attack the victim model.
* **Modification rate (Mod)**: the percentage of modified tokens. Each _Replace_ or _Insert_ action accounts for one token modified; a _Merge_ action is considered modifying one token if one of the two merged tokens is kept (e.g., merging bigram \(ab\) into \(a\)), and two otherwise (e.g., merging bigram \(ab\) into \(c\)).
* **Perplexity (PPL)**: a metric used to evaluate the _fluency_ of adversaries Kann et al. (2018); Zang et al. (2020). The perplexity is calculated using small sized GPT-2 with a 50K-sized vocabulary Radford et al. (2019).
* **Grammar error (GErr)**: the absolute number of increased grammatical errors in the successful adversarial example, compared to the original text. Following Zang et al. (2020); Morris et al. (2020), we calculate this by the LanguageTool Naber et al. (2003).7 Footnote 7: [https://www.languagetool.org/](https://www.languagetool.org/)
* **Textual similarity (Sim)**: the cosine similarity between the input and its adversary. Following Jin et al. (2020); Morris et al. (2020), we calculate this using the universal sentence encoder (USE; Cer et al. (2018).

The last four metrics are averaged across those adversarial examples that successfully attack the victim model.

### Results

Table 2 summarizes the results. Overall CLARE achieves the best performance on all metrics consistently across different datasets. Notably, CLARE outperforms BERTattack, the strongest baseline, by a more than 5.4% attack success rate with _fewer_ average modifications to the text. We attribute this to CLARE's flexible attack strategies obtained by combining three different perturbations at any position. Interestingly, using contextualized embeddings does _not_ appear to guarantee better fluency:

\begin{table}
\begin{tabular}{l r r r r r r r r r r} \hline \hline  & \multicolumn{4}{c}{**Yelp** (PPL = 51.5)} & \multicolumn{4}{c}{**AG News (PPL = 62.8)**} \\ \hline
**Model** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **GEr\(\downarrow\)** & **Sim\(\uparrow\)** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **GEr\(\downarrow\)** & **Sim\(\uparrow\)** \\ \hline TextFooler & 77.0 & 16.6 & 163.3 & 1.23 & 0.70 & 56.1 & 23.3 & 331.3 & 1.43 & 0.69 \\ + LM & 34.0 & 17.4 & 90.0 & 1.21 & 0.73 & 23.1 & 21.9 & 144.6 & 1.07 & 0.74 \\ BERAttack & 71.8 & 10.7 & 90.8 & 0.27 & 0.72 & 63.4 & 7.9 & 90.6 & 0.25 & 0.71 \\ \hline Clare & **79.7** & **10.3** & **83.5** & **0.25** & **0.78** & **79.1** & **6.1** & **86.0** & **0.17** & **0.76** \\ \hline \hline \end{tabular} 
\begin{tabular}{l c c c c c c c c c} \hline \hline  & \multicolumn{4}{c}{**MNLI** (PPL = 60.9)} & \multicolumn{4}{c}{**QNLI** (PPL = 46.0)} \\ \hline
**Model** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **GEr\(\downarrow\)** & **Sim\(\uparrow\)** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **GEr\(\downarrow\)** & **Sim\(\uparrow\)** \\ \hline TextFooler & 59.8 & 13.8 & 161.5 & 0.63 & 0.73 & 57.8 & 16.9 & 164.4 & 0.62 & 0.72 \\ + LM & 32.3 & 12.4 & 91.9 & 0.50 & 0.77 & 29.2 & 17.3 & 85.0 & 0.42 & 0.75 \\ BERAttack & 82.7 & 8.4 & 86.7 & 0.04 & 0.77 & 76.7 & 13.3 & 86.5 & 0.03 & 0.73 \\ \hline Clare & **88.1** & **7.5** & **82.7** & **0.02** & **0.82** & **83.8** & **11.8** & **76.7** & **0.01** & **0.78** \\ \hline \hline \end{tabular}
\end{table}
Table 2: Adversarial example generation performance in attack success rate (A-rate), modification rate (Mod), perplexity (PPL), number of increased grammar errors (GFerr), and textual similarity (Sim). The perplexity of the original inputs is indicated in parentheses for each dataset. Bold font indicates the best performance for each metric. All numbers are reported on 1000 test instances. † (\(\downarrow\)) represents that the higher (lower) the better.

 

despite fewer modifications to the text, BERTAttack achieves similar perplexity to language-model-augmented TextFooler on three out of the four datasets, while CLARE consistently outperforms both. In terms of grammatical errors, contextualized models (CLARE and BERAttack) are substantially better than the others, with CLARE performing the best. In terms of similarity, CLARE outperforms all baselines by more than 0.02, a larger gap than BERAttack's improvements over TextFooler variants. We observe similar trends on other datasets in Appendix A.2.

Figure 2 compares trade-off curves between attack success rate and textual similarity. We tune the thresholds for constructing the candidate token sets, and plot textual similarity against the attack success rate. CLARE strikes the best balance, showing a clear advantage in success rate with least similarity drop. We observe similar trends for attack success rate and perplexity trade off.

Human evaluation.We further conduct human evaluation on the AG News dataset. We randomly sample 300 instances which both CLARE and TextFooler successfully attack. For each input, we pair the adversarial examples from the two models, and present them to crowd-sourced judges along with the original input and the gold label. We ask them which they prefer with a neutral option in terms of (1) having a meaning that is closer to the original input (similarity), and (2) being more fluent and grammatical (fluency and grammaticality). Additionally, we ask the judges to annotate adversarial examples, and compare their annotations against the gold labels (label consistency). We collect 5 responses for each pair on every evaluated aspect. Further details are in Appendix A.3.

As shown in Table 3, CLARE has a significant advantage over TextFooler: in terms of similarity 56% responses prefer CLARE, while 16% prefer TextFooler. The trend is similar for fluency & grammaticality (42% vs. 9%). This observation is consistent with results from automatic metrics. On label consistency, CLARE slightly underperforms TextFooler at 68% with a 95% confidence interval (CI) \((66\%,70\%)\), versus 70% with a 95% CI \((68\%,73\%)\). We attribute this to an inherent overlap of some categories in the AG News dataset, e.g., _Science & Technology_ and _Business_, as evidenced by a 71% label consistency for original inputs.

Closing this section, Table 4 compares the adversarial examples generated by TextFooler and CLARE. More samples are listed in Appendix A.4.

## 4 Analysis

This section first conducts an ablation study (SS4.1). We then explore CLARE's potential to be used to improve downstream models' robustness and accuracy in SS4.3. In SS4.2, we empirically observe that CLARE tends to attack noun and noun phrases.

### Ablation Study

We ablate each component of CLARE to study its effectiveness. We evaluate on the 1,000 randomly selected AG news instances (SS3.2). The results are summarized in Table 5.

We first investigate the performance of three perturbations when applied individually. Among three editing strategies, using InsertOnly achieves the best performance, with ReplaceOnly coming a close second. MergeOnly underperforms the other two, partly because the attacks are restricted to bigram noun phrases (SS3.1). Combining all three perturbations, CLARE achieves the best perfor

\begin{table}
\begin{tabular}{l c c c} Metric & CLARE & Neutral & TextFooler \\ \hline Similarity & 56.1\(\pm\)2.5 & 28.1 & 15.8\(\pm\)2.1 \\ Fluency\& Grammaticality & 42.5\(\pm\)2.5 & 48.6 & 8.9\(\pm\)1.5 \\ Label Consistency & 68.0\(\pm\)2.4 & - & 70.1\(\pm\)2.5 \\ \end{tabular}
\end{table}
Table 3: Human evaluation performance in percentage on the AG News dataset. \(\pm\) indicates confidence intervals with a 95% confidence level.

Figure 2: **Left**: Attack success rate and textual similarity trade-off curves (_both higher the better_). **Right**: Attack success rate (_higher the better_) and perplexity (_lower the better_) trade-off curve. The larger area under the two curves indicates the better trade-off between two metrics.

 

To examine the efficiency of attacking order, we compare ReplaceOnly against BERTAttack. Notably, ReplaceOnly outperforms BERTAttack across the board. This is presumably because BERAttack does not take into account the tokens to be infilled when selecting the attack positions.

We now turn to the two constraints imposed when constructing the candidate token set. Perhaps not surprisingly, ablating the textual similarity constraint (_w/o_\({\rm sim}>l\)) decreases textual similarity performance, but increases other aspects. Ablating the masked language model yields a better success rate, but much worse perplexity, grammaticality, and textual similarity.

Finally, we compare CLARE implemented with different masked language models. Table 6 summarizes the results. Overall, distilled RoBERTa achieves the fastest speed without losing performance. Since the victim model is based on BERT, we conjecture that it is less efficient to attack a model using its own information.

### Perturbations by Part-of-speech Tags

In this section, we break down the adversarial attacks by part-of-speech (POS) tags in AG News dataset. We find that most of the adversarial attacks happen to nouns or noun phrases. Presumably, in many topic classification datasets, the prediction heavily relies on some characteristic noun words/phrases. As shown in Table 7, 64% of the _Replace_ actions are applied to nouns. _Insert_ actions tend to insert tokens into noun phrase bigram: two of the most frequent POS bigrams are noun phrases. In fact, around 48% of the _Insert_ actions are applied to noun phrases. This also justifies our choice of only applying _Merge_ to noun phrases.

### Adversarial Training

This section explores CLARE's potential in improving downstream models' accuracy and robustness. Following Tsipras et al. (2018), we use CLARE to generate adversarial examples for AG news training instances, and include them as additional training data. We consider two settings: training with (1) full training data and full adversarial data and (2) 10% randomly-sampled training data and its adversarial data, to simulate the low-resource scenario. For both settings, we compare a BERT-based MLP classifier and a TextCNN (Kim, 2014) classifier without any pretrained embedding.

_Whether adversarial examples, as data augmentation, can help achieve better test accuracy?_ As

\begin{table}
\begin{tabular}{l r r r r r} \hline \hline
**MLM** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **Sim\(\uparrow\)** & **Speed\(\uparrow\)** \\ \hline RoBERTa\({}_{\text{full}}\) & 79.1 & **6.1** & **86.0** & **0.76** & **0.14** \\ RoBERTa\({}_{\text{base}}\) & **79.3** & 6.3 & 88.9 & 0.75 & 0.07 \\ BERTb\({}_{\text{base}}\) & 78.4 & 8.3 & 95.2 & 0.71 & 0.06 \\ \hline \hline \end{tabular}
\end{table}
Table 6: Results of CLARE implemented with different masked language models (MLM). **Speed** is measured by number of processed samples per second.

\begin{table}
\begin{tabular}{p{142.3pt} p{28.5pt}} \hline \hline
**AG** & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc. about using a network the chipmaker is building to deliver live television to Sprint mobile phone customers.} \\ \cline{2-3}  & \multicolumn{1}{c}{Sprint Corps. is in talks with Qualcomm Inc.} \\ TextFooler (Business) & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc.} \\ \cline{2-3}  & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc.} \\ Clare (Business) & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc.} \\ \cline{2-3}  & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc.} \\ Clare (Business) & \multicolumn{1}{c}{Sprint Corp. is in talks with Qualcomm Inc.} \\ \cline{2-3}  & \multicolumn{1}{c}{About using a network Qualcomm is building to deliver _cubic_ television to Sprini mobile phone customers.} \\ \cline{2-3}  & \multicolumn{1}{c}{_Premise_: Let me try it. She began snapping her fingers and saying the word eagerly, but nothing happened.} \\ MNLI (Neutral) & \multicolumn{1}{c}{_Hypothesis_: She became frustrated when the spell didn’t work.} \\ \cline{2-3}  & \multicolumn{1}{c}{_Premise_: _Authorisation_ me _attemping_ it. She _triggered flapping her phirles and said_ the word eagerly, but nothing arisen.} \\ TextFooler (Contra-diction) & \multicolumn{1}{c}{_Hypothesis_: She became frustrated when the spell didn’t work.} \\ \cline{2-3}  & \multicolumn{1}{c}{_Premise_: Let me try it. She began snapping her fingers and saying the word eagerly, but nothing unexpected happened.} \\ Clare (Contaçtion) & \multicolumn{1}{c}{_Hypothesis_: She became frustrated when the spell didn’t work.} \\ \hline \hline \end{tabular}
\end{table}
Table 4: Adversarial examples produced by different models. The gold label of the original is shown below the (bolded) dataset name. _Replace_, Insert and Merge are highlighted in _italic red_, bold blue and sans serif yellow, respectively. (Best viewed in color).

\begin{table}
\begin{tabular}{l c c c c c} \hline \hline
**Module** & **A-rate\(\uparrow\)** & **Mod\(\downarrow\)** & **PPL\(\downarrow\)** & **Germ\(\uparrow\)** & **Sim\(\uparrow\)** \\ \hline Clare & 79.1 & **6.1** & **86.0** & 0.17 & 0.76 \\ \hline MergeOnly* & 47.2 & 6.2 & 95.3 & **0.08** & **0.79** \\ InsertOnly & 68.1 & 7.2 & 93.1 & 0.23 & 0.74 \\ ReplaceOnly & 66.7 & 7.7 & 85.6 & 0.10 & 0.72 \\ BERTAttack & 63.4 & 7.9 & 90.6 & 0.25 & 0.71 \\ \hline _w/o_\({\rm sim}>\ell\) & 82.4 & 6.9 & 86.8 & 0.13 & 0.70 \\ _w/o_\({\rm p_{MLM}}>k\) & **95.7** & 6.8 & 162.8 & 0.71 & 0.61 \\ \hline \hline \end{tabular}
\end{table}
Table 5: Ablation study results. “_w/o_\({\rm sim}>\ell\)” abulates the textual similarity constraint when constructing the candidate sets, while “_w/o_\({\rm p_{MLM}}>k\)" abates the masked language model probability constraint.

 

shown in Table 8, when the full training data is available, adversarial training slightly _decreases_ the test accuracy by 0.2% and 0.5% respectively. This aligns with previous observations Jia et al. (2019). Interestingly, in the low-data scenario with adversarial training, the BERT-based classifier has no accuracy drop, and TextCNN achieves a 2.0% absolute improvement. This suggests that a model with less capacity can benefit more from silver data.

_Does adversarial training help the models defend against adversarial attacks?_ To evaluate this, we use CLARE to attack classifiers trained with and without adversarial examples.9 A higher success rate and fewer modifications indicate a victim classifier is more vulnerable to adversarial attacks. As shown in Table 8, in 3 out of the 4 cases, adversarial training helps to decrease the attack success rate by more than 10.3%, and to increase the number of modifications needed by more than 0.8. The only exception is the TextCNN model trained with 10% data. A possible reason can be that it is trained with little data and thus generalizes less well.

Footnote 9: In preliminary experiments, we found that it is more difficult to use other models to attack a victim model trained with the adversarial examples generated by CLARE, than to use CLARE itself.

These results suggest that CLARE can be used to improve downstream models' robustness, with a negligible accuracy drop.

## 5 Related Work

Textual adversarial attack.An increasing amount of effort is being devoted to generating better textual adversarial examples with various attack models. Character-based models Liang et al. (2019); Ebrahimi et al. (2018); Li et al. (2018); Gao et al. (2018), _inter alia_) use misspellings to attack the victim systems; however, these attacks can often be defended by a spell checker Pruthi et al. (2019); Zhou et al. (2019); Jones et al. (2020). Many sentence-level models Iyyer et al. (2018); Wang et al. (2020); Zou et al. (2020), _inter alia_) have been developed to introduce more sophisticated token/phrase perturbations. These, however, generally have difficulty maintaining semantic similarity with original inputs Zhang et al. (2020a). Recent word-level models explore synonym substitution rules to enhance semantic meaning preservation Alzantot et al. (2018); Jin et al. (2020); Ren et al. (2019); Zhang et al. (2019); Zang et al. (2020), _inter alia_). Our work differs in that CLARE uses three contextualized perturbations that produces more fluent and grammatical outputs.

Text generation with BERT.Generation with masked language models has been widely studied in various natural language tasks, ranging from lexical substitution Wu et al. (2019); Zhou et al. (2019); Qiang et al. (2020); Wu et al. (2019), _inter alia_) to non-autoregressive generation Gu et al. (2018); Lee et al. (2018); Ghazvininejad et al. (2019); Wang and Cho (2019); Ma et al. (2019); Sun et al. (2019); Ren et al. (2020); Zhang et al. (2020b), _inter alia_).

## 6 Conclusion

We have presented CLARE, a contextualized adversarial example generation model for text. It uses contextualized knowledge from pretrained masked language models, and can generate adversarial examples that are natural, fluent and grammatical. With three contextualized perturbation patterns, _Replace_, _Insert_ and _Merge_ in our

\begin{table}
\begin{tabular}{l c r} \hline \hline _**Replace**_ & _Insert_ & _Merge_ \\ \hline _NOUN: 64\%_ & _(NOUN, NOUN): 12\% ADJ-NOUN: 31\%_ \\ _ADJ: 17\%_ & _(ADJ, NOUN): 10\%_ & _NOUN-NOUN: 22\%_ \\ _VERB: 7\%_ & _(NOUN, VERB): 9\%_ & _DT-NOUN: 12\%_ \\ \hline \multicolumn{3}{c}{**Context:** ... Amit Yoran, the government’s cybersecurity chief, abruptly resigned yesterday after a year .... **Replace**: cybersecurity \(\leftarrow\)_[security, surveillance, cryptography, intelligence, encryption ...]_ \\ \multicolumn{3}{c}{**Insert**: cybersecurity \(\leftarrow\)_[technology, defense, intelligentence, program, project ...]_ \\ \multicolumn{3}{c}{**Merge**: cybersecurity chief \(\leftarrow\)_[chief, consultant, administrator, scientist, secretary ...]_ \\ \hline \hline \end{tabular}
\end{table}
Table 7: **Top**: Top-3 POS tags (or POS tag bigrams) and their percentages for each perturbation type. \((a,b)\): insert a token between \(a\) and \(b\). \(a\)-\(b\): merge \(a\) and \(b\) into a token. **Bottom**: An AG news sample, where CLARE perturbs token “cybersecurity.” TextFooler is unable to attack this token since it is out of its vocabularies.

\begin{table}
\begin{tabular}{l c r r} \hline \hline
**Victim Model** & **Acc\(\uparrow\)** & **A-rate\(\downarrow\)** & **Mod\(\uparrow\)** \\ \hline BERT (100\% data) & 95.0 & 79.1 & 6.1 \\  + 100\% adversarial & -0.2 & -18.0 & +5.1 \\ \hline TextCNN (100\% data) & 91.2 & 92.7 & 5.0 \\  + 100\% adversarial & -0.5 & -10.3 & +0.8 \\ \hline \hline BERT (10\% data) & 92.5 & 96.1 & 5.4 \\  + 10\% adversarial & +0.0 & -12.3 & +7.6 \\ \hline TextCNN (10\% data) & 83.6 & 99.0 & 5.6 \\  + 10\% adversarial & +2.0 & -3.5 & +0.3 \\ \hline \hline \end{tabular}
\end{table}
Table 8: Adversarial training results on AG news test set. “Acc” indicates accuracy.

 

arsenal, CLARE can produce outputs of varied lengths and achieves a higher attack success rate than baselines and with fewer edits. Human evaluation shows significant advantages of CLARE in terms of textual similarity, fluency and grammaticality. We release our code and models at [https://github.com/cookielee77/CLARE](https://github.com/cookielee77/CLARE).

## Acknowledgments

We would like to thank the reviewers for their constructive comments. We thank NVIDIA Corporation for the donation of the GPU used for this research. We also thank Tongshuang Wu, Guoyin Wang and Shuhuai Ren for their helpful discussions and feedback.

## References

* M. Alzantot, Y. Sharma, A. Elgohary, B. Ho, M. Srivastava, and K. Chang (2018)Generating natural language adversarial examples. In Proc. of EMNLP, Cited by: SS3.4.
* D. Cer, Y. Yang, S. Kong, N. Hua, N. Limtiaco, R. St John, N. Constant, M. Gualardo-Cespedes, S. Yuan, C. Tar, et al. (2018)Universal sentence encoder. arXiv preprint arXiv:1803.11175. Cited by: SS1.
* Y. Cheng, L. Jiang, and W. Macherey (2019)Robust neural machine translation with doubly adversarial inputs. In Proc. of ACL, Cited by: SS1.
* J. Devlin, M. Chang, K. Lee, and K. Toutanova (2019)BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL, Cited by: SS1.
* W. B. Dolan and C. Brockett (2005)Automatically constructing a corpus of sentential paraphrases. In Proceedings of the Third International Workshop on Paraphrasing, Cited by: SS1.
* J. Ebrahimi, A. Rao, D. Lowd, and D. Dou (2018)Hottlip: white-box adversarial examples for text classification. In Proc. of ACL, Cited by: SS1.
* J. Gao, J. Lauchantin, M. Lou Soffa, and Y. Qi (2018)Black-box generation of adversarial text sequences to evade deep learning classifiers. In IEEE Security and Privacy Workshops (SPW), Cited by: SS1.
* S. Garg and G. Ramakrishnan (2020)Bae: Bert-based adversarial examples for text classification. In Proc. of EMNLP, Cited by: SS1.
* M. Ghazvininejad, O. Levy, Y. Liu, and L. Zettlemoyer (2019)Mask-predict: parallel decoding of conditional masked language models. In Proc. of EMNLP, Cited by: SS1.
* I. J. Goodfellow, J. Shlens, and C. Szegedy (2015)Explaining and harnessing adversarial examples. In Proc. of ICLR, Cited by: SS1.
* J. Gu, J. Bradbury, C. Xiong, V. O. Li, and R. Socher (2018)Non-autoregressive neural machine translation. In Proc. of ICLR, Cited by: SS1.
* M. Iyer, J. Wieting, K. Gimpel, and L. Zettlemoyer (2018)Adversarial example generation with syntactically controlled paraphrase networks. In Proc. of NAACL, Cited by: SS1.
* R. Jia and P. Liang (2017)Adversarial examples for evaluating reading comprehension systems. In Proc. of EMNLP, Cited by: SS1.
* R. Jia, A. Raghunathan, K. Goksel, and P. Liang (2019)Certified robustness to adversarial word substitutions. In Proc. of EMNLP, Cited by: SS1.
* D. Jin, Z. Jin, J. Tianyi Zhou, and P. Szolovits (2020)Is bert really robust? natural language attack on text classification and entailment. In Proc. of AAAI, Cited by: SS1.
* E. Jones, R. Jia, A. Raghunathan, and P. Liang (2020)Robust encodings: a framework for combating adversarial typos. In Proc. of ACL, Cited by: SS1.
* K. Kann, S. Rothe, and K. Filippova (2018)Sentence-level fluency evaluation: references help, but can be spared!. In Proc. of CNLP, pp. 313-323. Cited by: SS1.
* Yoon Kim (2014)Convolutional neural networks for sentence classification. In Proc. of EMNLP, Cited by: SS1.
* Keita Kurita, P. Michel, and G. Neubig (2020)Weight poisoning attacks on pre-trained models. In Proc. of ACL, Cited by: SS1.
* J. Lee, E. Mansimov, and K. Cho (2018)Deterministic non-autoregressive neural sequence modeling by iterative refinement. In Proc. of EMNLP, Cited by: SS1.
* J. Li, S. Ji, T. Du, B. Li, and T. Wang (2018)Textbugger: generating adversarial text against real-world applications. arXiv preprint arXiv:1812.05271. Cited by: SS1.
* L. Li, R. Ma, Q. Guo, X. Xue, and X. Qiu (2020)Bert-attack: Adversarial attack against bert using bert. In Proc. of EMNLP, Cited by: SS1.
* B. Liang, H. Li, M. Su, P. Bian, X. Li, and W. Shi (2019)Deep text classification can be fooled. In Proc. of IJCAI, Cited by: SS1.
* Yinhan Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, and V. Stoyanov (2019)Roberta: a robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.

 

* Ma et al. (2019) Kuezhe Ma, Chunting Zhou, Xian Li, Graham Neubig, and Eduard Hovy. 2019. Flowseq: Non-autoregressive conditional sequence generation with generative flow. In _Proc. of EMNLP_.
* Morris et al. (2020a) John Morris, Eli Lifland, Jin Yong Yoo, Jake Grigsby, Di Jin, and Yanjun Qi. 2020a. Textattack: A framework for adversarial attacks, data augmentation, and adversarial training in nlp. In _Proc. of EMNLP: System Demonstrations_.
* Morris et al. (2020b) John X Morris, Eli Lifland, Jack Lanchantin, Yangfeng Ji, and Yanjun Qi. 2020b. Reevaluating adversarial examples in natural language. In _Proc. of EMNLP Findings_.
* Mrksic et al. (2016) Nikola Mrksic, Diarmuid O Seaghdha, Blaise Thomson, Milica Gasic, Lina M Rojas Barahona, Pei-Hao Su, David Vandyke, Tsung-Hsien Wen, and Steve Young. 2016. Counter-fitting word vectors to linguistic constraints. In _Proc. of NAACL_.
* Naber et al. (2003) Daniel Naber et al. 2003. _A rule-based style and grammar checker_. Citeseer.
* Paszke et al. (2019) Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. 2019. Pytorch: An imperative style, high-performance deep learning library. In _Proc. of NeurIPS_.
* Pruthi et al. (2019) Danish Pruthi, Bhuwan Dhingra, and Zachary C Lipton. 2019. Combating adversarial misspellings with robust word recognition. In _Proc. of ACL_.
* Qiang et al. (2020) Jipeng Qiang, Yun Li, Yi Zhu, and Yunhao Yuan. 2020. A simple bert-based approach for lexical simplification. In _Proc. of AAAI_.
* Radford et al. (2019) Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. _OpenAI Blog_, 1(8):9.
* Rajpurkar et al. (2016) Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In _Proc. of EMNLP_.
* Ren et al. (2019) Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In _Proc. of ACL_.
* Ren et al. (2020) Yi Ren, Jinglin Liu, Xu Tan, Sheng Zhao, Zhou Zhao, and Tie-Yan Liu. 2020. A study of non-autoregressive model for sequence generation. In _Proc. of ACL_.
* Ribeiro et al. (2018) Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging nlp models. In _Proc. of ACL_.
* Samanta and Mehta (2017) Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. _arXiv preprint arXiv:1707.02812_.
* Sanh et al. (2019) Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. DistilBERT, a distilled version of bert: smaller, faster, cheaper and lighter. In _The 5th Workshop on Energy Efficient Machine Learning and Cognitive Computing, NeurIPS_.
* Socher et al. (2013) Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D Manning, Andrew Y Ng, and Christopher Potts. 2013. Recursive deep models for semantic compositionality over a sentiment treebank. In _Proc. of EMNLP_.
* Sun et al. (2019) Zhiqing Sun, Zhuohan Li, Haoqing Wang, Di He, Zi Lin, and Zhihong Deng. 2019. Fast structured decoding for sequence models. In _Proc. of NeurIPS_.
* Tsipras et al. (2018) Dimitris Tsipras, Shibani Santurkar, Logan Engstrom, Alexander Turner, and Aleksander Madry. 2018. Robustness may be at odds with accuracy. In _Proc. of ICLR_.
* Wallace et al. (2019) Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing nlp. In _Proc. of EMNLP_.
* Wang and Cho (2019) Alex Wang and Kyunghyun Cho. 2019. Bert has a mouth, and it must speak: Bert as a markov random field language model. In _Proceedings of the Workshop on Methods for Optimizing and Evaluating Neural Language Generation_.
* Wang et al. (2019a) Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. Glue: A multi-task benchmark and analysis platform for natural language understanding. In _Proc. of ICLR_.
* Wang et al. (2020) Boxin Wang, Hengzhi Pei, Boyuan Pan, Qian Chen, Shuohang Wang, and Bo Li. 2020. T3: Tree-autoencoder regularized adversarial text generation for targeted attack. In _Proc. of EMNLP_.
* Wang et al. (2019b) Xiaosen Wang, Hao Jin, and Kun He. 2019b. Natural language adversarial attacks and defenses in word level. _arXiv preprint arXiv:1909.06723_.
* Williams et al. (2018) Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In _Proc. of NAACL_.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Wu et al. (2019a) Xing Wu, Shangwen Lv, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019a. Conditional bert contextual augmentation. In _Proc. of ICCS_.
* Wu et al. (2019b) Xing Wu, Tao Zhang, Liangjun Zang, Jizhong Han, and Songlin Hu. 2019b. "mask and infill": Applying masked language model to sentiment transfer. In _Proc. of IJCAI_ 

Yuan Zang, Fanchao Qi, Chenghao Yang, Zhiyuan Liu, Meng Zhang, Qun Liu, and Maosong Sun. 2020. Word-level textual adversarial attacking as combinatorial optimization. In _Proc. of ACL_.
* Zhang et al. (2019) Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019. Generating fluent adversarial examples for natural languages. In _Proc. of ACL_.
* Zhang et al. (2020a) Wei Emma Zhang, Quan Z Sheng, Ahoud Alhazmi, and Chenliang Li. 2020a. Adversarial attacks on deep-learning models in natural language processing: A survey. _ACM Transactions on Intelligent Systems and Technology (TIST)_, 11(3):1-41.
* Zhang et al. (2015) Xiang Zhang, Junbo Zhao, and Yann LeCun. 2015. Character-level convolutional networks for text classification. In _Proc. of NeurIPS_.
* Zhang et al. (2020b) Yizhe Zhang, Guoyin Wang, Chunyuan Li, Zhe Gan, Chris Brockett, and Bill Dolan. 2020b. Pointer: Constrained text generation via insertion-based generative pre-training. In _Proc. of ACL_.
* Zhao et al. (2018) Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In _Proc. of ICLR_.
* Zhou et al. (2019a) Wangchunshu Zhou, Tao Ge, Ke Xu, Furu Wei, and Ming Zhou. 2019a. Bert-based lexical substitution. In _Proc. of ACL_.
* Zhou et al. (2019b) Yichao Zhou, Jyun-Yu Jiang, Kai-Wei Chang, and Wei Wang. 2019b. Learning to discriminate perturbations for blocking adversarial attacks in text classification. In _Proc. of EMNLP_.
* Zou et al. (2020) Wei Zou, Shujian Huang, Jun Xie, Xinyu Dai, and Jiajun Chen. 2020. A reinforced generation of adversarial samples for neural machine translation. In _Proc. of ACL_.

 

Appendix

### Additional Experiment Details

**Model Implementation.** All pretrained models and victim models based on RoBERTa and BERTbase are implemented with Hugging Face transformers10(Wolf et al., 2019) based on PyTorch Paszke et al. (2019). RoBERTd distill, RoBERTbase and uncase BERTBase models have 82M, 125M and 110M parameters, respectively. We use RoBERTd distill as our main backbone for fast inference purpose. TextFooler11 and BERTAttack12 are built with their open source implementation provided by the authors. In the implementation of TextFooler+LM, we use small sized GPT-2 language model Radford et al. (2019) to further select those candidate tokens that have top \(20\%\) perplexity in the candidate token set. In the adversarial training (SS4.3), the small TextCNN victim model Kim (2014) has 128 embedding size and \(100\) filters for \(3,4,5\) window size with \(0.5\) dropout, resulting in 7M parameters.

Footnote 10: [https://github.com/huggingface/transformers](https://github.com/huggingface/transformers)

Footnote 11: [https://github.com/jindl1/TextFooler](https://github.com/jindl1/TextFooler)

Footnote 12: [https://github.com/LinyangLee/BERT-Attack](https://github.com/LinyangLee/BERT-Attack)

During the implementation of _w/o_\(p_{\text{MLM}}>k\) in the ablation study (SS4.1), we randomly sample 200 tokens and then apply the similarity constraint to construct candidate set, as exhausting the vocabulary is computationally expensive.

Evaluation Metric.The similarity function sim builds on the universal sentence encoder (USE; Cer et al., 2018) to measure a _local_ similarity at the perturbation position with window size 15 between the original input and its adversary. _All baselines_ are equipped this sim when constructing the candidate vocabulary. The evaluation metric **Sim** uses USE to calculate a _global_ similarity between two texts. These procedures are typically following Jin et al. (2020). We mostly rely on human evaluation (SS3.3) to conclude the significant advantage of preserving textual similarity on CLARE compared with TextFooler.

Data Processing.When processing the data, we keep all punctuation in texts for both victim model training and attacking. This differs the preprocessing setting in TextFooler Jin et al. (2020) as we empirically found that removing punctuation makes the victim model vulnerable. Since GLUE benchmark Wang et al. (2019) does not provide the label for test set, we instead use its dev set as the the test set for the included datasets (MNLI, QNLI, QQP, MRPC, SST-2) in the evaluation. For the sentence-pair tasks (e.g., MNLI, QNLI, QQP, MRPC), we attack the longer one excluding the tokens appearing in both sentences. This is because inference tasks usually require entailed data to have the same keywords, e.g., numbers, name entities, etc. All experiments are conducted on one Nvidia GTX 1080Ti GPU.

### Additional Results

We include the results of DBpedia ontology dataset (**DBpedia**; Zhang et al., 2015, Stanford sentiment treebank (**SST-2**; Socher et al., 2013), Microsoft Research Paraphrase Corpus (**MRPC**; Dolan and Brockett, 2005), and Quora Question Pairs (**QQP**) from the GLUE benchmark in this section. Table 9 summarizes come statistics of these datasets. The results of different models on these datasets are summarized Table 10. Compared with all baselines, CLARE achieves the best performance on attack success rate, perplexity, grammaticality, and similarity. It is consistent with our observation in SS3.3.

### Human Evaluation Details

For each human evaluation on **AG News** dataset, we randomly sampled 300 sentences from the test set combining the corresponding adversarial examples from CLARE and TextFooler (We only consider sentences can be attacked by both models). In order to make the task less abstract, we pair the adversarial examples by the two models, and present them to the participants along with the original input and its gold label. We ask them which one they prefer in terms of (1) having more similar a meaning to the original input (similarity), and (2) being more fluent and grammatical (fluency and grammaticality). We also provide them with a neutral option, when the participants consider the two

\begin{table}
\begin{tabular}{l c r r r r} \hline \hline
**Dataset** & **Avg. Length** & **\# Classes** & **Train** & **Test** & **Acc** \\ \hline SST-2 & 10 & 2 & 67K & 0.9K & 92.3\% \\ DBPedia & 55 & 14 & 560K & 70K & 99.3\% \\ \hline QQP & 13/13 & 2 & 363K & 40K & 91.4\% \\ MRPC & 23/23 & 2 & 3.6K & 1.7K & 81.4\% \\ \hline \hline \end{tabular}
\end{table}
Table 9: Some statistics of datasets. The last column indicates the victim model’s accuracy on the original test set _without_ adversarial attack.

 

indistinguishable. Additionally, we ask the participants to annotate the adversarial examples, and compare their annotations against the gold labels (label consistency). Higher label consistency indicates the model is better at causing the victim model to make errors while preserving human predictions.

Each pair of system outputs was randomly presented to 5 crowd-sourced judges, who indicated their preference for similarity, fluency, and grammaticality using the form shown in Figure 3. The labelling task is illustrated in Figure 4. To minimize the impact of spamming, we employed the top-ranked 30% of U.S. workers provided by the crowdsourcing service. Detailed task descriptions and examples were also provided to guide the judges. We calculate \(p\)-value based on 95% confidence intervals by using 10K paired bootstrap replications, implemented using the R Boot statistical package.

### Qualitative Samples

We include generated adversarial examples by CLARE and TextFooler on **AG News**, **DBpeida**, **Yelp**, **MNLI**, and **QNLI** datasets in Table 11 and Table 12.

 



\begin{table}
\begin{tabular}{l l} \hline
* 0. LOS ANGELES
- Adam Eaton allowed five hits over seven innings for his career
- high 10th victory, Brian Giles homered for the second straight game, and the San Diego Pafres beat the Los Angeles (Sport) & Dodgers 3
- 0 Thursday night. The NL West
- leading Dodgers’ lead was cut to 2 / 2 games over San Francisco
- their smallest since July 31 \\  & Dogger Blank Yanks 3
- 0. Loos ANGELES
- Adams Parades enabling five hits over seven _slugging_ for his career
- high 10th victoria_, Brian Giles homesored for the second straight _matching_, and the _Tome Jose Dodger_ beat the Los (World) & Angeles _Dodger 3_
- 0 Thursday blackness_. The NL _Westerner
- eminent Dodger_’ lead was cut to 2 / 2 games over San San San – their _tiny as juntier_ 31 \\  & Padres Blank Dodgers 3
- 0. Milwaukee NEXT
- Adam Eaton allowed five hits over seven innings for his career
- high 10th victory, Brian Giles homered for the second straight game, and the San Diego Pafres beat the Los Angeles (World) & Dodgers 3
- 0 Thursday night. The NL West
- leading Dodgers’ lead was cut to 2 / 2 games over San Francisco
- their smallest since July 31 ... \\ \hline
**Yelp** & The food at this chain has always been consistently good. Our server in downtown ( where we spent New Year’s ) was new, but that did not impact our service at all. She was prompt and attentive to our needs. \\ TextFooler & The food at this chain has always been _necessarily ok_. Our server in downtown ( where we spent New Year’s ) was new, but that did not impact our service at all. She was _early_ and attentive to our needs. \\ CLARE & The food at this chain has always been looking consistently good. Our server in downtown ( where we spent New (Negative) ) was new, but that did not _enhance_ our service at all. She was prompt and attentive to our needs. \\ \hline
**Velp** & The pho broth is actually flavorful and doesn’t just taste like hot water with beef and nodules. I usually do take out and the order comes out fast during dinner which should be expected with pho, it’s not hard to soak noodles, slice beef and pour broth. \\ (Positive) & The pho broth is actually flavorful and doesn’t just _fasty_ like _torrid waters_ with _slaughter_ and _salads_. I repeatedly_ do take out and the order _poses_ out fast during dinner which should be expected with pho , it’s not _strenuous_ to soak noodles, _severing_ beef and pour broth. \\ CLARE & The pho broth is actually flavorful and doesn’t just taste bland like hot water with beef and noodles. I usually do take out and the order comes out awfully fast during dinner which should be expected with pho, it’s not hard to soak noodles, slice beef and pour broth. \\ \hline \multirow{5}{*}{**MNLI** (Neutral)} & _Premise_: Thebes held onto power until the 12th Dynasty, when its first king, Amenemhet Iwho reigned between 1980 1951 b.c. established a capital near Memphis. \\  & _Hypothesis_: The capital near Memphis lasted only half a century before its inhabitants abandoned it for the next capital. \\ \cline{2-3}  & _Premise_: Thebes _apprehended pour powers_ until the 12th Familial_ , when its _earliest_ king , Amenemhet Iwho reigned between 1980 1951 c.c. established a capital near Memphis . \\ (Contradiction) & _Hypothesis_: The capital near Memphis lasted only half a century before its inhabitants abandoned it for the next capital. \\ \cline{2-3}  & _Premise_: Thebes held onto powerful until the 12th Dynasty, when its first king, Amenemhet Iwho reigned between 1980 1951 bc. thereafter established a capital near Memphis. \\ CLARE & 1980 1951 bc. thereafter established a capital near Memphis. \\ (Contradiction) & _Hypothesis_: The capital near Memphis lasted only half a century before its inhabitants abandoned it for the next capital. \\ \hline
**MNLI** & _Premise_: Hopefully, Wall Street will take voluntary steps to address these issues before it is forced to act. \\ (Entailment



\begin{table}
\begin{tabular}{l l}
**QNLI** & _Premise_: Who overturned the Taft Vale judgement ? \\ (Entailment) & _Hypothesis_: One of the first acts of the new Liberal Government was to reverse the Taff Vale judgement. \\ TextFooler & _Premise_: Who overturned the Taft Vale judgement ? \\ (Not- Entailment) & _Hypothesis_: One of the first acts of the new Liberal Government was to _invest_ the Taff Vale judgement. \\ CLARE & _Premise_: Who overturned the Taft Vale judgement ? \\ (Not- Entailment) & _Hypothesis_: One of the first acts of the new Liberal _Constitution_ was to reverse the Taff Vale judgement. \\ \hline
**QNLI** & _Premise_: What are the software testers aware of ? \\ (Entailment) & _Hypothesis_: Black-box testing treats the software as a black box, examining functionality without any knowledge of internal implementation without seeing the source code. \\ TextFooler & _Premise_: What are the software testers aware of ? \\ (Not- Not- & _Hypothesis_: Black-_boxes testing administers_ the software as a black box, _investigating functions unless_ any \\ Entailment) & knowledge of internal _ful



### Instructions

Below are three short snippets of text. One of them is the REFERENCE TEXT. Determine the extent to which TEXT #1 and TEXT #2 "mean the same thing" as the REFERENCE TEXT.

Then determine which of TEXT #1 or TEXT #2 is better in terms of grammaticality and fluency.

IMPORTANT: The names of entities (e.g., names of people, countries, businesses, and institutions) and numbers may be different between the two sentences. For the purpose of this task you should ignore these differences. Refer to the guidelines for examples.

We would like you to focus on whether the events or state of affairs described in the two texts are the same or different. In other words, are the tests saying similar things about the entities or not? And to what extent are they similar or different?

Please ignore punctuation, spacing, and other minor formatting issues.

#### Reference Text:

N. B. truck driver accused in hoist of 50, 000 cans of Moosehead beer (Canadian Press). Canadian Press-FREDERICTON (CP)-A New Brunswick tuck driver arrested in Ontario this week has been accused by police of stealing 50, 000 cans of Moosehead beer.

**TEXT #1**: N. B. automobiles driver culprit in burglaries of 50, 000 piblchers of Moosehead beer (Canada Pulsar). Toronto Pres-FREDERICTON (CP)-A New Brunswick tuck drivers internet in Simcoe this weeks has been perp by police of burglaries 50, 000 coffers of Moosehead drinkin.

**TEXT #2**: Winding through rural China, train of medicine restores sight. Impatiently, Gopur Samat constituted at the prepared as desemboweded ocular his the and healing from a physicians toward a steered his entitled attention charts 20 legs externally.

#### Meaning Preservation:

IGNORING NMES OF ENTITIES AND NUMBERS, which of the two texts better preserves the meaning of the REFERENCE?

* Definitely Text #1.
* Maybe Text #1.
* Both equally wellbody.
* Maybe Text #2.
* Definitely Text #2.

#### Grammaticality and Fluency:

Which of the two texts is more fluent and grammatical?

* TEXT #1 is clearly better.
* TEXT #1 may be better.
* BOTH are equally head/good.
* TEXT #2 may be better.
* TEXT #2 is clearly better.

#### Comment

You may make a comment in this box if you wish.

Figure 3: Pair-wise comparison in terms of text similarity and fluency & grammaticality on human evaluation.





## 1 Introduction

The purpose of this paper is to develop a new approach for the development of the concept of a novel model. The main idea behind the concept of a novel model is that it can be used as an extension of the framework, and its application in the context of a new model. In particular, we will consider a new model which has been proposed by the authors. We will also discuss how to use the model to describe the model in terms of the model parameters.

In this work, we are interested in the model's behavior of the model under consideration. This is done by considering the model's behavior of the model under consideration. It is shown that the model's behavior is not affected by the model's behavior but rather by the model's behavior. However, the model's behavior is affected by the model's behavior. Therefore, we will focus on the model's behavior of the model under consideration.

We will study the model's behavior of the model under consideration. We will present a new model based on the model's behavior of the model under consideration. We will investigate the model's behavior of the model under consideration. We will show that the model's behavior is affected by the model under consideration. We will analyze the model's behavior of the model under consideration. We will then examine the model's behavior of the model under consideration.

### Model's Behavior

We will study the model's behavior of the model under consideration. We will first introduce the model's behavior of the model under consideration. Then we will consider the model's behavior of the model under consideration. We will assume that the model's behavior is affected by the model under consideration. We will assume that the model's behavior is affected by the model under consideration.