SlowGuess/NAACL_2022_OCR huggingface dataset:
Chuyun Deng $^{1}$ , Mingxuan Liu $^{1}$ , Yue Qin $^{2}$ , Jia Zhang $^{1,3*}$ , Haixin Duan $^{1,3}$ , Donghong Sun $^{1,3}$
$^{1}$ Tsinghua University,  $^{2}$ Indiana University
$^{3}$ Beijing National Research Center for Information Science and Technology {dengcy20, liumx18}  $@$  mails.tsinghua.edu.cn, qinyue@iu.edu, zhangjia@cernet.edu.cn {duanhx, sundonghong}  $@$  tsinghua.edu.cn
Adversarial texts help explore vulnerabilities in language models, improve model robustness, and explain their working mechanisms. However, existing word-level attack methods trap in a one-to-one attack pattern, i.e., only a single word can be modified in one transformation round, and they ignore the interactions between several consecutive words. In this paper, we propose ValCAT, a black-box attack framework that misleads the language model by applying variable-length contextualized transformations to the original text. Compared to word-level methods, ValCAT expands the basic units of perturbation from single words to spans composed of multiple consecutive words, enhancing the perturbation capability. Experiments show that our method outperforms state-of-the-art methods in terms of attack success rate, perplexity, and semantic similarity on several classification tasks and inference tasks. The comprehensive human evaluation demonstrates that ValCAT has a significant advantage in ensuring the fluency of the adversarial examples and achieves better semantic consistency. We release the code at https://github.com/linerxliner/ValCAT.
Deep learning is successfully applied in a variety of fields, while previous works have found that neural network models are vulnerable to well-constructed adversarial examples (Goodfellow et al., 2015; Kurakin et al., 2017). In general, adversarial examples are constructed by adding imperceptible perturbations to the benign inputs, which can mislead the victim model. And exploring adversarial examples is essential to improving the reliability and robustness of neural network models. Compared to the long-studied image domain, generating adversarial examples on texts is more difficult because texts are discrete, where small changes can alter the original
meaning and make it unnatural (Xu et al., 2020; Zhang et al., 2020).
AG News (Business)	Lucent milestone: A profit Lucent Technologies yesterday posted higher fiscal fourth-quarter earnings, helping lift the telecommunications equipment maker to its first profitable year since 2000.
BERT-Attack (Sci/Tech)	Lucent node: A revenue Lucent tech yesterday reported higher revenue fourth-quarter benefits, which lift the multimedia equipment maker to its first business year year 2000.
ValCAT (Sci/Tech)	Lucent milestone: A profit Lucent [IT] Technologies [recently reported] higher fiscal fourth-quarter earnings, helping lift the telecommunications equipment maker to its first profitable year since 2000.
MNLI (Neutral)	Premise: He caught a grip on himself, fighting the fantasies of his mind, and took another breath of air. Hypothesis: The air tasted like molten metal - the taste of blood.
BERT-Attack (Contradiction)	Hypothesis: The air tasted through boiling armor - the taste of betrayal.
ValCAT (Contradiction)	Hypothesis: The air tasted [nothing like air] - the taste of blood.
Table 1: Instances of adversarial examples. The first is the original text, followed by adversarial examples generated by ValCAT and BERT-Attack. Blue indicates replace operation, and orange indicates insert operation. For ValCAT, words in a bracket are a span perturbed as a whole in one transformation round. The decision results of the victim model are in parentheses.
Most of the sentence-level attack methods (Zhao et al., 2018; Han et al., 2020; Xu et al., 2021) generate adversarial examples by perturbing the latent representation of the text, and the text quality is thus relatively difficult to control. Word-level attack methods (Samanta and Mehta, 2017; Zhang et al., 2019) have received much attention in the recent past. Several previous works explore the perturbation based only on the properties of individual
Human Language Technologies, pages 1735 - 1746
Figure 1: Overview of ValCAT's workflow.
words, with the help of inflectional morphology (Tan et al., 2020), counter-fitting word vectors (Jin et al., 2020), sememe (Zang et al., 2020), etc. The encoder language models, like BERT, give us new insights of incorporating context information into the generation of adversarial candidates in transformation (Li et al., 2020; Garg and Ramakrishnan, 2020; Li et al., 2021). However, the basic units of perturbation in these works are still single words, which limits the scope of the perturbation and overlooks the interactions between words, as examples shown in Table 1. To address this issue, two research questions are raised: 1) How to take into account interactions between words in vulnerable position discovery? 2) How to perturb a vulnerable position with variable-length contextualized transformations?
We propose ValCAT, which generates high-quality adversarial examples by applying variable-length contextualized transformations to the original text. Table 1 demonstrates some sample cases. Specifically, given a benign text, we enumerate all possible spans by traversing the text with sliding windows of different lengths to evaluate the importance of the spans. We utilize two operations, REPLACE and INSERT, to apply adversarial spans generated by the encoder-decoder language model to the text. The encoder can fill a mask token with a single word, while the decoder can predict a sequence after a prompt. Therefore, joint use of encoder and decoder enables ValCAT to generate variable-length contextualized spans at the arbitrary vulnerable position.
Furthermore, we evaluate ValCAT by attacking fine-tuned BERT on several classification tasks and inference tasks. Experiment results show that it outperforms other baseline methods in attack success rate, perplexity, and semantic similarity. In particular, we observe that the variable-length feature can significantly reduce perplexity, which is less than  $50\%$  compared to the best baseline. As for the efficiency, although the multi-word attack
pattern perturbs more words than the one-to-one attack pattern in one transformation, it just requires fewer rounds toward success. In general results, it is in a tolerable perturbation rate, and ValCAT is even lower than baselines on some of the inference datasets. A comprehensive human evaluation further verifies that ValCAT has significant advantages in readability and semantic consistency. The main contributions of this paper are summarized as follows:
- ValCAT is the first variable-length adversarial attack method against language models, i.e., extending the basic units of perturbation from single words to spans.
- Our work proposes the Sliding Window for vulnerable-position discovery and the variable-length contextualized transformation which fully exploits the advantage of encoder-decoder language models.
- Automatic evaluation and human evaluation demonstrate the excellent attack effectiveness of ValCAT and the superior quality of our generated adversarial examples.
2 ValCAT
To further improve the attack effectiveness and simultaneously improve the readability and semantic similarity of the adversarial examples, we propose ValCAT, which can generate high-quality adversarial text by applying variable-length contextualized transformations with the encoder-decoder language model.
Problem Formalization Given a victim model  $F: X \to Y$  and a text  $x = w_{1}w_{2}\dots w_{n - 1}w_{n}$  that can be correctly classified by  $F$ , the attack goal is to generate an adversarial text  $\tilde{x}$ , which can confuse the model prediction, i.e.  $F(\tilde{x}) \neq F(x)$ . A consecutive word sequence with length  $l$  can be denoted as a span  $s_{i:i + l}$ . In the soft-label black-box setting, the attacker only has access to the logit output  $P(y|x)$ . The architecture, parameters, and configurations of  $F$  are unknown to the attacker.
To achieve human imperceptibility, the attacker should minimize textual perturbations and maintain semantic consistency.
ValCAT The attack workflow of ValCAT is illustrated in Figure 1. To locate the most-vulnerable positions for the perturbations in each iteration, ValCAT first rank the spans in  $x$  according to their importance with sliding windows of length 1 to MAX, as lines 2-7 in Algorithm 1. Based on the sorted span ranking  $R$ , ValCAT uses the encoder-decoder language model to generate variable-length contextualized spans, as lines 8-13. Using two transformation operations in line 9, REPLACE and INSERT, we obtain a adversarial text candidate set  $T$ . If some of the candidates can mislead the victim model, ValCAT declares the one with the highest cosine similarity with the original text as the final successful result, as line 11. However, if no candidate successes at this iteration, we select the one with the highest negative impact to the victim model as the basic text for the next iteration, as line 13. Note that, if a span in the ranking  $R$  has been selected as the target span, the subsequent spans which are overlapped with it will be removed from the ranking to avoid multiple modifications on one word. The attack ends when the successful example appears, or when the limit of the perturbation constraints (See Section 3.1) reached. The latter case is considered a final failure. Below we elaborate on the two stages of the attack in Section 2.1 and Section 2.2 in detail.
2.1 Important Span Ranking
Echoing the observation to prior works (Jin et al., 2020; Li et al., 2020), only some key words act as vulnerable positions for the victim model  $F$ . Perturbations over these words can be most beneficial in crafting adversarial examples. Considering the interactions between words, ValCAT performs transformations on several important spans instead of single words.
Given a text  $x$ , we evaluate the importance of a span  $s$  within  $x$  according to how removing the span can impact the model prediction in the black-box setting. Formally, we define the decision difference of  $x$  and  $\tilde{x}$  on the class  $y$  as:
$$
D _ {y} (x, \tilde {x}) = \left\{ \begin{array}{c c c} d _ {y} (x, \tilde {x}), & i f & y = \tilde {y} \\ d _ {y} (x, \tilde {x}) + d _ {\tilde {y}} (\tilde {x}, x), & i f & y \neq \tilde {y} \end{array} \right.
$$
where  $y$  and  $\tilde{y}$  are the predictions of  $x$  and  $\tilde{x}$ , respectively, and  $d_y(x,\tilde{x}) = P(y|x) - P(y|\tilde{x})$  is the
difference of the probabilities that  $x$  and  $\tilde{x}$  are classified as  $y$ . Let  $\tilde{x}_s$  denote the text of replacing span  $s$  by a unknown token [unk]. The importance score of  $s$  with respect to  $x$  is  $IP_x(s) = D_y(x, \tilde{x}_s)$ .
To compare spans of different lengths, we propose the Sliding Windows to measure the importance of variable-length spans. Specifically, we apply multiple sliding windows of the corresponding sizes, which traverse the text from left to right. Each span bounded by a sliding window is sequentially removed from the original text for its importance calculation. Finally, we obtain a set of triples each consisting of the start index, the span length, and the importance score of the span. We rank the triples according to the importance score in descending order.
2.2 Variable-Length Contextualized Transformations
Based on the ranking of triples, ValCAT performs perturbations in a sequential manner, where each step a vulnerable position in the original text is replaced by or inserted with a set of adversarial spans generated by the encoder-decoder language model. The variable-length feature of the adversarial spans renders the language model enough space to produce more contextually appropriate candi
dates to improve fluency. Meanwhile, our variable-length method expands the perturbation units, for it supports multi-word transformations while is compatible with traditional one-to-one transformations. Compared with previous methods, this further improves the attack success rate under the same perturbation constraints. Below we elaborate on the details of the adversarial span generation.
Adversarial Span Generation To generate adversarial candidates of each target span, ValCAT applies the encoder-decoder language model to fill the mask token with a set of spans of varying lengths. First, the encoder language model owns the capability of predicting the masked tokens, which is trained with the masked language modeling (MLM) objective. However, the encoder fills one mask token with only one suitable word rather than multiple words. The ability of the decoder could fill the gap of the single word since the decoder is trained with a causal language modeling (CLM) which can predict the sequence after a prompt. But, the decoder can only generate sequences at the end of the text. Hence, ValCAT combined these two models with their complementary advantages, with the predictive capability at arbitrary positions of encoder and variable-length generation of a decoder. With the candidate adversarial spans for target span, ValCAT performs two kinds of transformation operations, REPLACE and INSERT, to generate the adversarial examples for consideration.
Replace The REPLACE operation substitutes the target span  $s_{i:i + l} = w_i \dots w_{i + l - 1}$  with another span  $s$ . For example, the target span "awesome" in the text "This place is awesome." could be replaced by the adversarial span "pretty good". Specifically, we first replace a mask token [mask] to  $s_{i:i + l}$ :
$$
\tilde {x} ^ {[ i: i + l ]} = w _ {1}... w _ {i - 1} [ \mathrm {m a s k} ] w _ {i + l}... w _ {n},
$$
and generate a set of contextualized spans of varying length,  $S$ , to fill the mask token. The adversarial example is denoted as:
$$
\tilde {x} _ {s} ^ {[ i: i + l ]} = w _ {1} \dots w _ {i - 1} s w _ {i + l} \dots w _ {n},
$$
where  $s\in S$  is a adversarial span.
Since the language model is blind to the information of the target span, some of the generated adversarial spans may deviate from the original meaning to a large extent. To avoid this situation, we only leave the adversarial spans with a high degree of semantic similarity to the original span.
Specifically, we use Universal Sentence Encoder (Cer et al., 2018) to restrict their cosine similarity. We also impose a limit on the perturbation rate (See Section 3.1). To prevent the text from being too long, we constrain the adversarial spans to be at most two words longer than the target span.
Insert The INSERT operation inserts a new span  $s$  in front of the target span  $s_{i:i + l}$ . For example, "I like this quite interesting movie". Similar to the REPLACE operation, it inserts a mask token in front of the target span:
$$
\tilde {x} ^ {i} = w _ {1}... w _ {i - 1} [ \mathrm {m a s k} ] [ w _ {i}... w _ {i + l - 1} ] w _ {i + l}... w _ {n},
$$
and corresponds with the adversarial text  $\tilde{x}_s^i = w_1\ldots w_{i - 1}s w_i\ldots w_n$ . This operation also follows the same perturbation constraints, mentioned in Section 3.1.
In this section, we evaluate ValCAT on two NLP tasks, text classification and natural language inference. To demonstrate the effectiveness of ValCAT in terms of fluency, grammaticality, and semantic consistency, following Li et al. 2021, we design and conduct a comprehensive human evaluation.
Victim Model In this work, we choose fine-tuned BERT as the victim model for the all evaluation tasks. As BERT has achieved good results on a variety of NLU tasks and has been proven to be one of the most representative pre-trained transformers (Devlin et al., 2019).
Span Generation Model To obtain variable-length contextualized spans, we choose T5 (Raffel et al., 2020) as the generation model. T5 is a representative encoder-decoder language model that can predict the missing spans within a corrupted piece of text, benefiting from the fill-in-the-blank pre-training. Also, the large corpus C4 renders T5 rich prior knowledge to enable the diversity and the high quality of the generated spans.
Constraints To achieve human imperceptibility and semantic preservation of the adversarial example, we impose constraints on the word perturbation rate and semantic similarity, as defined in Section 3.3. Following previous practices (Jin et al., 2020; Li et al., 2021), we set the thresholds of these constraints respectively for each dataset, with details shown in Appendix A.
Dataset	Algorithm	Orig Acc	Atk Acc↓	Suc↑	PPL↓	Sim↑	Pert↓	GErr↓
AG News (PPL=98.4)	ValCAT	94.4	35.3	62.6	136.2	0.922	16.1	0.39
BERT-Attack	47.5	49.7	326.9	0.873	17.3	1.12
TextFooler	44.8	52.5	418.6	0.883	15.7	1.37
TextBugger	62.0	34.3	500.7	0.886	19.7	2.78
Yelp Polarity (PPL=71.1)	ValCAT	98.3	6.8	93.1	81.6	0.950	11.6	0.11
BERT-Attack	20.2	79.5	162.9	0.881	14.1	0.15
TextFooler	27.7	71.8	174.5	0.890	11.2	0.33
TextBugger	53.7	45.4	255.4	0.876	16.1	2.17
IMDB (PPL=58.8)	ValCAT	94.6	12.8	86.5	65.6	0.977	6.7	0.09
BERT-Attack	18.3	80.7	98.0	0.950	7.8	0.08
TextFooler	30.2	68.1	95.6	0.956	5.9	0.24
TextBugger	55.9	40.9	123.2	0.952	8.7	1.49
SNLI (PPL=68.0)	ValCAT	89.8	10.6/9.7	88.2/89.2	90.9/78.6	0.854/0.840	21.3/22.9	0.22/0.15
BERT-Attack	34.8/20.5	61.2/77.2	184.5/116.9	0.734/0.741	24.8/19.3	0.31/0.09
TextFooler	41.7/23.3	53.6/74.0	235.9/146.8	0.734/0.745	24.4/19.4	0.64/0.22
TextBugger	55.5/34.9	38.2/61.1	374.0/190.9	0.728/0.754	31.0/25.1	1.80/0.69
MNLI (PPL=79.5)	ValCAT	82.7	7.3/2.5	91.2/97.0	93.5/89.5	0.879/0.869	18.5/20.4	0.19/0.15
BERT-Attack	22.4/17.7	72.9/78.6	172.9/146.9	0.754/0.764	22.6/18.9	0.09/0.12
TextFooler	28.9/21.4	65.1/74.1	228.1/183.4	0.754/0.770	22.1/18.2	0.63/0.38
TextBugger	41.1/34.7	50.3/58.0	317.6/218.2	0.744/0.771	27.3/22.4	1.98/1.05
QNLI (PPL=66.1)	ValCAT	90.0	18.7/6.4	79.2/92.9	70.4/87.5	0.828/0.892	27.8/18.4	0.13/0.18
BERT-Attack	32.5/24.7	63.8/72.6	101.2/215.5	0.734/0.744	24.2/26.5	0.31/0.63
TextFooler	41.0/34.3	54.4/61.9	120.3/252.0	0.757/0.754	20.7/24.0	0.51/1.11
TextBugger	50.4/44.1	43.9/51.0	152.6/388.8	0.753/0.744	27.4/31.8	1.25/3.02
Table 2: Effectiveness of ValCAT in attack success rate (Suc), perplexity (PPL), semantic similarity (Sim), word perturbation rate (Pert), and grammar error (GErr). Bold font indicates the best performance for each metric.  $\uparrow (\downarrow)$  represents that the higher (lower) the better. The original PPL for each dataset is indicated in parentheses under its name. For the natural language inference task, the left side of the slash indicates the result of attacking premise (question) and the right side indicates the result of attacking hypothesis (sentence).
Settings and Computation Cost All results are derived from a single run since there is no randomness in our method. The maximum length of the sliding window is set as 3. In our implementation, we apply SpaCy (Honnibal and Montani, 2017) and NLTK (Loper and Bird, 2002) for text manipulation. We run ValCAT on Intel Xeon E5-2690 2.6GHz Processor with V100 GPU. Averagely it takes 34 secs to generate a successful adversarial example.
3.2 Datasets and Baselines
To investigate the effectiveness of ValCAT on different types of text, we evaluate it on several English datasets. We randomly sample 1000 instances from each of the following datasets: three for text classification, i.e., AG News, Yelp Polarity, and IMDB; and three for natural language inference, i.e., SNLI, MNLI, and QNLI, with detailed informa
tion shown in Appendix B. We compare ValCAT with several state-of-the-art word-level attack methods, i.e., TextBugger, TextFooer, and BERT-Attack. Details of these methods are shown in Appendix C. Note that, all the datasets and baselines are publicly available and are used following their usage specifications.
3.3 Automatic Evaluation
Metrics We evaluate the effectiveness of ValCAT based on the following metrics:
- Attack success rate (Suc): is the percentage of adversarial examples successfully interfered with the victim model's prediction over the text dataset.
- Perplexity (PPL): is an automatic metric to evaluate the probability of a text appearing in a natural corpus. So PPL can reflect the text's natural fluency, the lower the better. We use GPT-2 (Radford et al., 2019) for this calculation.
- Semantic similarity (Sim): is the cosine similarity between the original text and adversarial text calculated by Universal Sentence Encoder (USE) embeddings (Cer et al., 2018).
- Word perturbation rate (Pert): is the proportion of the modified words over the original text. To evaluate the perturbation rate on variable-length perturbations more accurately, we design separate calculations for these two transformation operations. For the REPLACE operation, the number of modifications is  $\max(l_t, l_a) - l_{LCS}$ , where  $l_t$ ,  $l_a$ , and  $l_{LCS}$  are the lengths of the target span, the adversarial span, and their longest-common-subsequence (LCS), respectively. For the INSERT operation, the number of modifications is the length of the inserted span.
- Grammar error (GErr): is the incremental number of grammatical errors of the adversarial example relative to the original text. We use LanguageTool to count the grammatical errors within a text.
Figure 2: Success rate and perplexity of each attack method under the different constraints of word perturbation rate and semantic similarity.
Results The main experimental results are revealed in Table 2. ValCAT outperforms baselines on multiple datasets in both classification tasks and inference tasks. Compared with BERT-Attack, the best baseline method, ValCAT achieves higher attack success rates in all experiments, and the improvement range from  $5.8\%$  even to  $27.0\%$ . This is attributed to ValCAT's ability to apply variable-length contextualized transformations on spans at any position, which largely enriches the pattern of perturbations. In addition, ValCAT achieves the highest semantic similarity with the original text in all tasks. The perplexity of the adversarial example generated by ValCAT is far superior to that of the baselines. It is roughly only  $50\%$  of the perplexity
of BERT-Attack and is almost consistent with the original text. This indicates that the adversarial example generated by ValCAT is with high fluency. The underlying reason is that multi-word transformations render the language model enough space to generate the adversarial spans with approximated distribution with natural language. At the same time, ValCAT reaches the lowest average increase in the number of grammatical errors on five out of nine attacks, with only a slight disadvantage on the natural language inference tasks.
In general, ValCAT generates adversarial examples that achieve more promising attacks and are more imperceptible to humans. Figure 2 shows the performance of ValCAT and baselines on the Yelp dataset under different constraint settings. As we can see, the success rate of ValCAT decays most slowly as the limits of perturbation rate and semantic similarity increase, still achieving a success rate of almost  $70\%$  under the strictest constraint. Moreover, ValCAT can maintain a low perplexity under any constraint, however, other attack methods have to sacrifice attack effect to maintain their fluency.