SlowGuess/NAACL_2021_OCR huggingface dataset:
Dianqi Li\* Yizhe Zhang Hao Peng\* Liquin Chen\*
Chris Brockett $^{\diamond}$  Ming-Ting Sun $^{\spadesuit}$  Bill Dolan $^{\diamond}$
$\spadesuit$ University of Washington Microsoft Research Duke University
{dianqili, mts}@uw.edu, hapeng@cs.uw.edu, liqun.chen@duke.edu
{Yizhe.Zhang, Chris.Brockett, billdol}@microsoft.com
Adversarial examples expose the vulnerabilities of natural language processing (NLP) models, and can be used to evaluate and improve their robustness. Existing techniques of generating such examples are typically driven by local heuristic rules that are ag- nostic to the context, often resulting in unnatural and ungrammatical outputs. This paper presents CLARE, a ContextuaLized AdversaRial Example generation model that produces fluent and grammatical outputs through a mask-then-infill procedure. CLARE builds on a pre-trained masked language model and modifies the inputs in a context-aware manner. We propose three contextualized perturbations, Replace, Insert and Merge, that allow for generating outputs of varied lengths. CLARE can flexibly combine these perturbations and apply them at any position in the inputs, and is thus able to attack the victim model more effectively with fewer edits. Extensive experiments and human evaluation demonstrate that CLARE outperforms the baselines in terms of attack success rate, textual similarity, fluency and grammaticality.
Adversarial example generation for natural language processing (NLP) tasks aims to perturb input text to trigger errors in machine learning models, while keeping the output close to the original. Besides exposing system vulnerabilities and helping improve their robustness and security (Zhao et al., 2018; Wallace et al., 2019; Cheng et al., 2019; Jia et al., 2019, inter alia), adversarial examples are also used to analyze and interpret the models' decisions (Jia and Liang, 2017; Ribeiro et al., 2018).
Generating adversarial examples for NLP tasks can be challenging, in part due to the discrete nature of natural language text. Most recent efforts have explored heuristic rules, such as replacing tokens with their synonyms (Samanta and Mehta, 2017;
Original Text
Super ant colony hits
A giant 100km
colony of ants could threaten
local insect species.
Adversarial Text
Australia {Coast, Territory, yesterday...}.
A{gigantic,colossal,dangerous...}100km
{insect, many, numerous...} species.
Figure 1: Illustration of CLARE. Through a mask-then-infill procedure, the model generates the adversarial text with three contextualized perturbations: Replace, Insert and Merge. A mask is indicated by “_”. The degree of fade corresponds to the (decreasing) priority of the infill tokens.
Liang et al., 2019; Alzantot et al., 2018; Ren et al., 2019; Jin et al., 2020, inter alia). Despite some empirical success, rule-based methods are agnostic to context, limiting their ability to produce natural, fluent, and grammatical outputs (Wang et al., 2019b; Kurita et al., 2020, inter alia).
This work presents CLARE, a ContextuaLized AdversaRial Example generation model for text. CLARE perturbs the input with a mask-then-infill procedure: it first detects the vulnerabilities of a model and deploys masks to the inputs to indicate missing text, then plugs in an alternative using a pretrained masked language model (e.g., RoBERTa; Liu et al., 2019). CLARE features three contextualized perturbations: Replace, Insert and Merge, which respectively replace a token, insert a new one, and merge a bigram (Figure 1). As a result, it can generate outputs of varied lengths, in contrast to token replacement based methods that are limited to outputs of the same lengths as the inputs (Alzantot et al., 2018; Ren et al., 2019;
Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 5053-5069 June 6-11, 2021. ©2021 Association for Computational Linguistics
Jin et al., 2020). Further, CLARE searches over a wider range of attack strategies, and is thus able to attack the victim model more effectively with fewer edits. Building on a masked language model, CLARE maximally preserves textual similarity, fluency, and grammaticality of the outputs.
We evaluate CLARE on text classification, natural language inference, and sentence paraphrase tasks, by attacking finetuned BERT models (Devlin et al., 2019). Extensive experiments and human evaluation results show that CLARE outperforms baselines in terms of attack success rate, textual similarity, fluency, and grammaticality, and strikes a better balance between attack success rate and preserving input-output similarity. Our analysis further suggests that the CLARE can be used to improve the robustness of the downstream models, and improve their accuracy when the available training data is limited. We release our code and models at https://github.com/cookielee77/CLARE.
2 CLARE
At a high level, CLARE applies a sequence of contextualized perturbation actions to the input. Each can be seen as a local mask-then-infill procedure: it first applies a mask to the input around a given position, and then fills it in using a pretrained masked language model (§2.1). To produce the output, CLARE scores and descendingly ranks the actions, which are then iteratively applied to the input (§2.2). We begin with a brief background review and laying out of necessary notation.
Background. Adversarial example generation centers around a victim model  $f$ , which we assume is a text classifier. We focus on the black-box setting, allowing access to  $f$ 's outputs but not its configurations such as parameters. Given an input sequence  $\mathbf{x} = x_1x_2\ldots x_n$  and its label  $y$  (assume  $f(\mathbf{x}) = y$ ), an adversarial example  $\mathbf{x}'$  is supposed to modify  $\mathbf{x}$  to trigger an error in the victim model:  $f(\mathbf{x}') \neq f(\mathbf{x})$ . At the same time, textual modifications should be minimal, such that  $\mathbf{x}'$  is close to  $\mathbf{x}$  and the human predictions on  $\mathbf{x}'$  stay the same.
This is achieved by requiring the similarity be
tween  $\mathbf{x}'$  and  $\mathbf{x}$  to be larger than a threshold:  $\mathrm{sim}(\mathbf{x}',\mathbf{x}) > \ell$ . A common choice of  $\mathrm{sim}(\cdot ,\cdot)$  is to encode sentences using neural networks, and calculate their cosine similarity in the embedding space (Jin et al., 2020).
2.1 Masking and Contextualized Infilling
At a given position of the input sequence, CLARE can execute three perturbation actions: Replace, Insert, and Merge, which we introduce in this section. These apply masks at the given position with different strategies, and then fill in the missing text based on the unmasked context.
Replace: A Replace action substitutes the token at a given position  $i$  with an alternative (e.g., changing "fantastic" to "amazing" in "The movie is fantastic"). It first replaces  $x_{i}$  with a mask, and then selects a token  $z$  from a candidate set  $\mathcal{Z}$  to fill in:
$$
\widetilde {\mathbf {x}} = x _ {1} \dots x _ {i - 1} [ \text {M A S K} ] x _ {i + 1} \dots x _ {n},
$$
replace  $(\mathbf{x},i) = x_{1}\dots x_{i - 1}zx_{i + 1}\dots x_{n}$
For clarity, we denote replace  $(\mathbf{x},i)$  by  $\widetilde{\mathbf{x}}_z$  .To produce an adversarial example,
-  $z$  should fit into the unmasked context;
-  $\widetilde{\mathbf{x}}_z$  should be similar to  $\mathbf{x}$ ;
-  $\widetilde{\mathbf{x}}_z$  should trigger an error in  $f$ .
These can be achieved by selecting a  $z$  such that
-  $z$  receives a high probability from a masked language model:  $p_{\mathrm{MLM}}(z \mid \widetilde{\mathbf{x}}) > k$ ;
-  $\widetilde{\mathbf{x}}_z$  is similar to  $\mathbf{x}$ :  $\mathrm{sim}(\mathbf{x},\widetilde{\mathbf{x}}_z) > \ell$ ;
-  $f$  predicts low probability for the gold label given  $\widetilde{\mathbf{x}}_z$ , i.e.,  $p_f(y \mid \widetilde{\mathbf{x}}_z)$  is small.
$p_{\mathrm{MLM}}$  denotes a pretrained masked language model (e.g., RoBERTa; Liu et al., 2019). Using higher  $k$ ,  $\ell$  thresholds produces outputs that are more fluent and closer to the original. However, this can undermine the success rate of the attack. We choose  $k$ ,  $\ell$  to trade-off between these two aspects.
The first two requirements can be met by the construction of the candidate set:  $\mathcal{Z} =$
$$
\left\{z ^ {\prime} \in \mathcal {V} \mid p _ {\mathrm {M L M}} \left(z ^ {\prime} \mid \widetilde {\mathbf {x}}\right) > k, \operatorname {s i m} \left(\mathbf {x}, \widetilde {\mathbf {x}} _ {z ^ {\prime}}\right) > \ell \right\}.
$$
$\mathcal{V}$  is the vocabulary of the masked language model. To meet the third, we select from  $\mathcal{Z}$  the token that, if filled in, will cause most "confusion" to  $f$ :
$$
z = \underset {z ^ {\prime} \in \mathcal {Z}} {\arg \min } p _ {f} (y \mid \widetilde {\mathbf {x}} _ {z ^ {\prime}}). \tag {1}
$$
<sup>1</sup>In computer vision applications, minor perturbations to continuous pixels can be barely perceptible to humans, thus it can be hard for one to distinguish x and x' (Goodfellow et al., 2015). It is not the case for text, however, since changes to the discrete tokens are more likely to be noticed by humans.
$^2 k$  and  $\ell$  are empirically set as  $5\times 10^{-3}$  and 0.7, respectively. This also reduces the computation overhead: in our experiments  $|\mathcal{Z}|$  is 42 on average, much smaller than the vocabulary size  $(|\mathcal{V}| = 50,265)$ .
The Insert and Merge actions differ from Replace in terms of masking strategies. The alternative token  $z$  is selected analogously to that in a Replace action.
Insert: This aims to add extra information to the input (e.g., changing "I recommend ..." to "I highly recommend ..."). It inserts a mask after  $x_{i}$  and then fills it. Slightly overloading the notations,
$$
\widetilde {\mathbf {x}} = x _ {1} \dots x _ {i} [ \text {M A S K} ] x _ {i + 1} \dots x _ {n},
$$
$$
\operatorname {i n s e r t} (\mathbf {x}, i) = x _ {1} \dots x _ {i} z x _ {i + 1} \dots x _ {n}.
$$
This increases the sequence length by 1.
Merge: This masks out a bigram  $x_{i}x_{i + 1}$  with a single mask and then fills it, reducing the sequence length by 1:
$$
\widetilde {\mathbf {x}} = x _ {1} \dots x _ {i - 1} [ \text {M A S K} ] x _ {i + 2} \dots x _ {n},
$$
$$
\operatorname {m e r g e} (\mathbf {x}, i) = x _ {1} \dots x _ {i - 1} z x _ {i + 2} \dots x _ {n}.
$$
$z$  can be the same as one of the masked tokens (e.g., masking out "New York" and then filling in "York"). This can be seen as deleting a token from the input.
For Insert and Merge,  $z$  is chosen in the same manner as replace action.
In sum, at each position  $i$  of an input sequence, CLARE first: (i) replaces  $x_{i}$  with a mask; (ii) or inserts a mask after  $x_{i}$ ; (iii) or merges  $x_{i}x_{i + 1}$  into a mask. Then a set of candidate tokens is constructed with a masked language model and a textual similarity function; the token minimizing the gold label's probability is chosen as the alternative token. The combination of these three operations enables conversion between any two sequences.
CLARE first constructs the local actions for all positions in parallel, i.e., the actions at position  $i$  do not affect those at other positions. Then, to produce the adversarial example, CLARE gathers the local actions and selects an order to execute them.
2.2 Sequentially Applying the Perturbations
Given an input pair  $(\mathbf{x},y)$ , let  $n$  denote the length of  $\mathbf{x}$ . CLARE chooses from  $3n$  actions to produce the output: 3 actions for each position, assuming the candidate token sets are not empty. We aim to generate an adversarial example with minimum modifications to the input. To achieve this, we iteratively apply the actions, and first select those
minimizing the probability of outputting the gold label  $y$  from  $f$ .
Each action is associated with a score, measuring how likely it can "confuse"  $f$ : denote by  $a(\mathbf{x})$  the output of applying action  $a$  to  $\mathbf{x}$ . The score is then the negative probability of predicting the gold label from  $f$ , using  $a(\mathbf{x})$  as the input:
$$
s _ {(\mathbf {x}, y)} (a) = - p _ {f} \left(y \mid a (\mathbf {x})\right).
$$
Only one of the three actions can be applied at each position, and we select the one with the highest score. This constraint aims to avoid multiple modifications around the same position, e.g., merging "New York" into "Seattle" and then replacing it with "Boston".
Actions are iteratively applied to the input, until an adversarial example is found or a limit of actions  $T$  is reached. Each step selects the highest-scoring action from the remaining ones. Algorithm 1 summarizes the above procedure.
Discussion. A key technique of CLARE is the local mask-then-infill perturbation. Compared with existing context-agnostic replacement approaches (Alzantot et al., 2018; Jin et al., 2020; Ren et al., 2019, inter alia), contextualized infilling produces more fluent and grammatical outputs. Generating adversarial examples with masked language models is also explored by concurrent work
4 Insert and Merge actions change the text length. When any of them is applied, we accordingly change the text indices of affected actions remaining in  $\mathcal{A}$ .
${}^{3}$  A perturbation will not be considered if its candidate token set is empty.
BERTAttack (Li et al., 2020) and BAE (Garg and Ramakrishnan, 2020).<sup>5</sup>
- BERTAttack only replaces tokens and thus can only produce outputs of the same lengths as the inputs. This is analogous with a CLARE model with the Replace action only. BAE entangles replacing and inserting tokens: it inserts only at positions neighboring a replaced token, limiting its attacking capability. Departing from both, CLARE uses three different perturbations (Replace, Insert and Merge), each allowing efficient attacking against any position of the input, and can produce outputs of varied lengths. As we will show in the experiments (§3.3), CLARE outperforms both these methods.
- When selecting the attack positions, neither BERTAttack or BAE takes into account the tokens to be inflated, whereas CLARE does. This results in better adversarial attack performance according to our ablation study (§4.1).
- CLARE demonstrates the advantage of using RoBERTa over BERT, which was used in the concurrent works (§4.1).
We evaluate CLARE on text classification, natural language inference, and sentence paraphrase tasks. We begin by describing the implementation details of CLARE and the baselines (§3.1). §3.2 introduces the experimental datasets and the evaluation metrics; the results are summarized in §3.3.
- We experiment with a distilled version of RoBERTa (RoBERTa distill; Sanh et al., 2019) as the masked language model for contextualized infilling. We also compare to base sized RoBERTa (RoBERTa base; Liu et al., 2019) and base sized BERT (BERT base; Devlin et al., 2019) in the ablation study ( $\S 4.1$ ).
- The similarity function builds on the universal sentence encoder (USE; Cer et al., 2018).
- The victim model is an MLP classifier on top of  $\mathrm{BERT}_{\mathrm{base}}$ . It takes as input the first token's contextualized representation. We finetune BERT when training the victim model.
Baselines. We compare CLARE with recent state-of-the-art word-level black-box adversarial
Dataset	Avg. Length	# Classes	Train	Test	Acc
Yelp	130	2	560K	38K	95.9%
AG News	46	4	120K	7.6K	95.0%
\( MNLI^6 \)	23/11	3	392K	9.8K	84.3%
QNLI	11/31	2	105K	5.4K	91.4%
Table 1: Some statistics of datasets. The last column indicates the victim model's accuracy on the original test set without adversarial attack.
attack models, including:
- TextFooler: a state-of-the-art model by Jin et al. (2020). This replaces tokens with their synonyms derived from counter-fitting word embeddings (Mrkšić et al., 2016), and uses the same text similarity function as our work.
- Text,Fooler+LM: an improved variant of TextFooler we implemented based on Alzantot et al. (2018) and Cheng et al. (2019). This inherits token replacement from TextFooler, but uses an additional small sized GPT-2 language model (Radford et al., 2019) to filter out those candidate tokens that do not fit in the context with calculated perplexity.
- BERTAttack: a mask-then-infill approach by Li et al. (2020). It greedily replaces tokens with the predictions from BERT. BAE is not listed as it has a similar performance as BERTAttack (Garg and Ramakrishnan, 2020).
We use the open source implementation of the above baselines provided by the authors. More details are included in Appendix §A.1.
Datasets. We evaluate CLARE with the following datasets:
- Yelp Reviews (Zhang et al., 2015): a binary sentiment classification dataset based on restaurant reviews.
- AG News (Zhang et al., 2015): a collection of news articles with four categories: World, Sports, Business and Science & Technology.
- MNLI (Williams et al., 2018): a natural language inference dataset. Each instance consists of a premise-hypothesis pair, and the model is supposed to determine the relation between them from a label set of entailment, neutral, and contradiction. It covers text from a variety of domains.
6We only examine the performance on the matched set, since the mismatched set is easier to attack.
$^{5}$ Both Li et al. (2020) and Garg and Ramakrishnan (2020) are published concurrently to an initial report of this work.
Yelp (PPL = 51.5)	AG News (PPL = 62.8)
Model	A-rate↑	Mod↓	PPL↓	GErr↓	Sim↑	A-rate↑	Mod↓	PPL↓	GErr↓	Sim↑
TextFooler + LM	77.0	16.6	163.3	1.23	0.70	56.1	23.3	331.3	1.43	0.69
34.0	17.4	90.0	1.21	0.73	23.1	21.9	144.6	1.07	0.74
BERTAttack	71.8	10.7	90.8	0.27	0.72	63.4	7.9	90.6	0.25	0.71
CLARE	79.7	10.3	83.5	0.25	0.78	79.1	6.1	86.0	0.17	0.76
MNLI (PPL = 60.9)	QNLI (PPL = 46.0)
Model	A-rate↑	Mod↓	PPL↓	GErr↓	Sim↑	A-rate↑	Mod↓	PPL↓	GErr↓	Sim↑
TextFooler + LM	59.8	13.8	161.5	0.63	0.73	57.8	16.9	164.4	0.62	0.72
32.3	12.4	91.9	0.50	0.77	29.2	17.3	85.0	0.42	0.75
BERTAttack	82.7	8.4	86.7	0.04	0.77	76.7	13.3	86.5	0.03	0.73
CLARE	88.1	7.5	82.7	0.02	0.82	83.8	11.8	76.7	0.01	0.78
Table 2: Adversarial example generation performance in attack success rate (A-rate), modification rate (Mod), perplexity (PPL), number of increased grammar errors (GErr), and textual similarity (Sim). The perplexity of the original inputs is indicated in parentheses for each dataset. Bold font indicates the best performance for each metric. All numbers are reported on 1000 test instances.  $\uparrow (\downarrow)$  represents that the higher (lower) the better.
- QNLI (Wang et al., 2019a): a binary classification dataset converted from the Stanford question answering dataset (Rajpurkar et al., 2016). The task is to determine whether the context contains the answer to a question. It is mainly based on English Wikipedia articles.
Table 1 summarizes some statistics of the datasets. In addition to the above four datasets, we experiment with DBpedia ontology dataset (Zhang et al., 2015), Stanford sentiment treebank (Socher et al., 2013), Microsoft Research Paraphrase Corpus (Dolan and Brockett, 2005), and Quora Question Pairs from the GLUE benchmark. The results on these datasets are summarized in Appendix A.2.
Following previous practice (Alzantot et al., 2018), we fine-tune CLARE on training data, and evaluate with 1,000 randomly sampled test instances of lengths  $\leq 100$ . In the sentence-pair tasks (e.g., MNLI, QNLI), we attack the longer sentence excluding the tokens that appear in both.
Evaluation metrics. We follow previous works (Jin et al., 2020; Morris et al., 2020a), and evaluate the models with the following automatic metrics:
- Attack success rate (A-rate): the percentage of adversarial examples that can successfully attack the victim model.
- Modification rate (Mod): the percentage of modified tokens. Each Replace or Insert action accounts for one token modified; a Merge action is considered modifying one token if one of the two merged tokens is kept (e.g., merging bigram  $ab$  into  $a$ ), and two otherwise
(e.g., merging bigram  $ab$  into  $c$ ).
- Perplexity (PPL): a metric used to evaluate the fluency of adversaries (Kann et al., 2018; Zang et al., 2020). The perplexity is calculated using small sized GPT-2 with a 50K-sized vocabulary (Radford et al., 2019).
- Grammar error (GErr): the absolute number of increased grammatical errors in the successful adversarial example, compared to the original text. Following (Zang et al., 2020; Morris et al., 2020b), we calculate this by the LanguageTool (Naber et al., 2003).<sup>7</sup>
- Textual similarity (Sim): the cosine similarity between the input and its adversary. Following (Jin et al., 2020; Morris et al., 2020b), we calculate this using the universal sentence encoder (USE; Cer et al., 2018).
The last four metrics are averaged across those adversarial examples that successfully attack the victim model.
Table 2 summarizes the results. Overall CLARE achieves the best performance on all metrics consistently across different datasets. Notably, CLARE outperforms BERTAttack, the strongest baseline, by a more than  $5.4\%$  attack success rate with fewer average modifications to the text. We attribute this to CLARE's flexible attack strategies obtained by combining three different perturbations at any position. Interestingly, using contextualized embeddings does not appear to guarantee better fluency:
<sup>7</sup>https://www.languagetool.org/
Figure 2: Left: Attack success rate and textual similarity trade-off curves (both higher the better). Right: Attack success rate (higher the better) and perplexity (lower the better) trade-off curve. The larger area under the two curves indicates the better trade-off between two metrics.
despite fewer modifications to the text, BERTAttack achieves similar perplexity to language-model-augmented TextFooer on three out of the four datasets, while CLARE consistently outperforms both. In terms of grammatical errors, contextualized models (CLARE and BERTAttack) are substantially better than the others, with CLARE performing the best. In terms of similarity, CLARE outperforms all baselines by more than 0.02, a larger gap than BERTAttack's improvements over TextFooer variants. We observe similar trends on other datasets in Appendix A.2.
Figure 2 compares trade-off curves between attack success rate and textual similarity. We tune the thresholds for constructing the candidate token sets, and plot textual similarity against the attack success rate. CLARE strikes the best balance, showing a clear advantage in success rate with least similarity drop. We observe similar trends for attack success rate and perplexity trade off.
Human evaluation. We further conduct human evaluation on the AG News dataset. We randomly sample 300 instances which both CLARE and TextFooler successfully attack. For each input, we pair the adversarial examples from the two models, and present them to crowd-sourced judges along with the original input and the gold label. We ask them which they prefer with a neutral option in terms of (1) having a meaning that is closer to the original input (similarity), and (2) being more fluent and grammatical (fluency and grammaticality). Additionally, we ask the judges to annotate adversarial examples, and compare their annotations against the gold labels (label consistency). We collect 5 responses for each pair on every eval
Metric	CLARE	Neutral	TextFooler
Similarity	56.1±2.5	28.1	15.8±2.1
Fluency&Grammaticality	42.5±2.5	48.6	8.9±1.5
Label Consistency	68.0±2.4	-	70.1±2.5
Table 3: Human evaluation performance in percentage on the AG News dataset.  $\pm$  indicates confidence intervals with a  $95\%$  confidence level.
uated aspect. Further details are in Appendix A.3.