SlowGuess/NAACL_2022_OCR huggingface dataset:
*Boxin Wang $^{1}$ , *Chejian Xu $^{1}$ , Xiangyu Liu $^{2}$ , Yu Cheng $^{3}$ , Bo Li $^{1}$
2Alibaba Group, 3Microsoft Research
{boxinw2, chejian2, lbo}@illinois.edu
eason.lxy@alibaba-inc.com, yu.cheng@microsoft.com
Recent studies show that pre-trained language models (LMs) are vulnerable to textual adversarial attacks. However, existing attack methods either suffer from low attack success rates or fail to search efficiently in the exponentially large perturbation space. We propose an efficient and effective framework SemAttack to generate natural adversarial text by constructing different semantic perturbation functions. In particular, SemAttack optimizes the generated perturbations constrained on generic semantic spaces, including typo space, knowledge space (e.g., WordNet), contextualized semantic space (e.g., the embedding space of BERT clusterings), or the combination of these spaces. Thus, the generated adversarial texts are more semantically close to the original inputs. Extensive experiments reveal that state-of-the-art (SOTA) large-scale LMs (e.g., DeBERTa-v2) and defense strategies (e.g., FreeLB) are still vulnerable to SemAttack. We further demonstrate that SemAttack is general and able to generate natural adversarial texts for different languages (e.g., English and Chinese) with high attack success rates. Human evaluations also confirm that our generated adversarial texts are natural and barely affect human performance. Our code is publicly available at https://github.com/AI-secure/SemAttack.
Deep neural networks have achieved remarkable success in many machine learning tasks. Particularly, BERT (Devlin et al., 2019) has inspired a suite of large-scale pre-trained language models (Yang et al., 2019; Zhang et al., 2019; Lan et al., 2019), which achieved new SOTA for many NLP tasks. In addition to BERT's dominant performance on English datasets, Tenney et al. (2019) points out that BERT is similarly effective on other languages
Original Input: They need to hire experienced sales rep who are mature enough to handle questions and sales.
Adversarial Input: They need to hire skilled sales rep who are mature enough to handle questions and sales.
Sentiment Prediction: Most Negative  $\rightarrow$  Most Positive
Original Input: 拿什么能吸引你：我们的海外学子？
(Translation: What can attract you: our overseas students?)
Adversarial Input: 拿甚么能吸引你：我们的海外学子？
Topic Prediction: Education News  $\rightarrow$  Entertainment News
Table 1: Adversarial texts generated against English and Chinese BERT classifiers by SemAttack on Yelp and THUCTC datasets. Replacing a word/character with an adversarial one misleads the correct prediction to a wrong class without fooling human.
such as Chinese, whose granularity of words is more complex, given the model's ability to disambiguate information from high-level representations (Ding et al., 2019).
Although effective for many NLP tasks, the robustness of these neural models is often challenged by carefully crafted adversarial examples. Specifically, attackers can add subtle human-imperceptible perturbation to the original input and induce dramatic changes in model output. Current adversarial text generation (Jia and Liang, 2017; Li et al., 2018; Alzantot et al., 2018) is mainly heuristic and only achieves low attack success rates for BERT-based models. Other work (Cheng et al., 2020; Ebrahimi et al., 2018) allows an input word to be substituted by any other word in the vocabulary, which fails to consider the semantic perturbation constraints and is prone to invalid adversarial examples. Recent work (Jin et al., 2020; Zang et al., 2020) relies on external knowledge to constrain the perturbation yet poorly handles large search space that grows exponentially with the input length, as it requires hundreds of queries to generate one adversarial example in practice.
Furthermore, most existing textual adversarial attacks are not generalizable to other languages, due to unique language-dependent characteristics
Findings of the Association for Computational Linguistics: NAACL 2022, pages 176 - 205
Original Input: You don't know what I've experienced here. All I can say is don't go to this place. There's a much better mall in town.
Original Prediction: 1-star (most negative)
Figure 1: Adversarial texts against BERT sentiment classifier generated by SemAttack that formulates two different contextualized semantic perturbation spaces based on BERT embedding clusters (the embedding space is projected by PCA onto 2D space). The word "experienced" reveals different meanings (past tense of the verb "experience" or adjective form) in different contexts (clusters). Our contextualized semantic perturbation chooses "saw" or "encountered" as the perturbation for verb "experienced", while "skilled" or "trained" for the adjective form.
Adversarial Input: You don't know what I've encountered here. All I can say is don't go to this place. There's a much better mall in town.
Adversarial Prediction: 5-star (most positive)
and the lack of universal linguistic resources. Moreover, character-level adversarial attacks designed in English context (Ebrahimi et al., 2017) are often ineffective for Chinese-character-level attacks, as the size of candidate characters increases by two orders of magnitude, resulting in surging computational costs especially for BERT-based models.
We tackle these limitations in textual adversarial attacks by proposing an effective and efficient framework SemAttack, which can be used to further evaluate the robustness of NLP models. We generalize existing word-level attacks and propose generic semantic perturbation functions, which optimize and constrain the perturbations within different semantic spaces, so that the generated adversarial texts retain their semantic meaning. We mainly consider three types of semantic spaces: (1) Tipo Space, using typo words or characters that can fool the models but not human judges; (2) Knowledge Space, utilizing external linguistic knowledge base (e.g., WordNet (Miller, 1995)) as valid perturbation candidates; and (3) Contextualized Semantic Space, exploiting the embedding space of BERT to generate a contextualized perturbation set semantically close to the original word (Figure 1). The contextualized semantic space does not require additional knowledge, and therefore can scale to other languages, especially low-resource languages where a large knowledge base is unavailable.
After the candidate semantic space is determined, SemAttack searches for the optimal perturbation combination. Instead of requiring thousands of queries to generate one adversarial example, opti
mal perturbations can be efficiently found in the embedding space by solving an optimization problem. We also control the magnitude of perturbation to be small as shown in Table 1. Extensive experiments on four datasets demonstrate that SOTA LMs and defense methods are still vulnerable to our adversarial attack, which are natural and barely affects human judgment. For example, the accuracy of BERT sentiment classifier drops from  $70.6\%$  to  $2.4\%$  by simply replacing fewer than  $5\%$  words with our method. Although these adversarial examples are generated in the whitebox setting, they can effectively transfer to two different blackbox attack settings while retaining higher than  $90\%$  attack success rate for BERT and other large-scale LMs such as DeBERTa-XXLarge.
Our contributions are summarized as follows: 1) We propose a unified and effective adversarial attack framework SemAttack by constructing semantic perturbation functions, which constraint perturbations within different semantic spaces and their combinations. 2) SemAttack generates contextualized perturbations that require no external knowledge and thus can easily adapt to different languages. 3) We conducted extensive experiments on different datasets and languages to show that adversarial texts generated by SemAttack are more semantically close to the benign inputs, and achieve much higher attack success rates than existing attack algorithms in different settings. 4) Comprehensive studies demonstrate that SOTA LMs and defenses are still vulnerable to SemAttack, and human evaluation verifies the naturalness and va
lidity of our adversarial examples.
2 SemAttack
Given an input  $\pmb{x} = [x_0, x_1, \dots, x_n]$ , where  $x_i$  is the  $i$ -th input token, the classifier  $f$  maps the input to final logits  $z = f(\pmb{x}) \in \mathbb{R}^C$ , where  $C$  is the number of classes, and outputs a label  $y = \arg \max f(\pmb{x})$ .
During attack, we evaluate the effectiveness of attack algorithms by calculating the targeted attack success rate (TSR):
$$
\operatorname {T S R} = \frac {1}{\left| D _ {\mathrm {a d v}} \right|} \sum_ {\boldsymbol {x} ^ {\prime} \in D _ {\mathrm {a d v}}} \mathbb {1} [ \arg \max  f (\boldsymbol {x} ^ {\prime}) \equiv y ^ {*} ] \tag {1}
$$
and untargeted attack success rate (USR):
$$
\mathrm {U S R} = \frac {1}{| D _ {\mathrm {a d v}} |} \sum_ {\boldsymbol {x} ^ {\prime} \in D _ {\mathrm {a d v}}} \mathbb {1} [ \arg \max  f (\boldsymbol {x} ^ {\prime}) \neq y ] \tag {2}
$$
where the attack algorithm generates one adversarial sentence for each sample to form an adversarial dataset  $D_{\mathrm{adv}}$ ,  $y^{*}$  is the targeted false class,  $y$  is the ground truth label, and  $\mathbb{1}(\cdot)$  is the indicator function.
2.2 Semantic Perturbation Functions
To control adversarial examples to be semantically close to the original input, we design a general form of semantic perturbation function  $\mathcal{F}$ , which takes one token  $x$  as input, and returns its candidate perturbation space  $\mathcal{S} = \{x_0^*, x_1^*, \dots, x_n^*\}$ . We next discuss the types of perturbation function  $\mathcal{F}$ .
Typo-based Perturbation Function  $\mathcal{F}_T$  constrain the search space  $S$  in the typo space, which uses typo words or characters to replace original tokens so that human can still understand the original meaning while models are fooled. In English, we follow the generation process introduced in TextBugger (Li et al., 2018) to generate typos.
In order to illustrate how our proposed method can be easily adapted to multilingual settings, we also generate typo-based semantic space for Chinese. Specifically, for each Chinese token  $x$ , we prepare a set of common Chinese characters  $S$  that look similar ("形近字") or have the same pronunciation ("音近字") as the original token  $x$ . We use the open-source similar Chinese character list that contains more than 9,000 common Chinese characters. To search for the Chinese characters with the same pronunciation (i.e., pinyin), we first query the pronunciation of input  $x$  and then choose the characters returned based on the same pronunciation. If  $x$  is a heteronym that has multiple pronunciations, we only use one pronunciation to do the query. We
also limit the size of Chinese characters of the same pronunciation to be less than 6 so that the search space is not too large. For the Chinese example shown in Table 1, we use “甚” to replace “什” as they share the same pronunciation and are a common typo that will not affect human understanding.
Knowledge-based Perturbation Function  $\mathcal{F}_K$  considers the knowledge space to constrain the perturbation search space  $S$ . Specifically,  $\mathcal{F}_K$  utilizes existing knowledge base to build a candidate perturbation set. In our work, we use WordNet as an example to illustrate how our framework can integrate rule-based knowledge to enhance the quality of adversarial examples. WordNet is a large lexical dataset of more than 200 languages that groups words into sets of cognitive synonyms. With the manually labeled semantic relations among words, synonyms queried from WordNet (i.e., synsets) share the same semantic meaning as the query word  $x$ . Therefore we choose these synonyms returned from WordNet to be the search space  $S$ . We note that WordNet also contains hypernyms and hyponyms information, but including them into the search space may incur some unnatural replacement (e.g., replacing "fifth" with "rank"). Therefore, we only consider synsets as the candidate search space  $S$ . In addition, even for the same token (e.g., "use") in WordNet, it may have different part-of-speech (POS) tags (e.g., "use" as verb or as noun), and thus has different synonyms (e.g., "exploitation" for noun "use" and "practice" as verb "use"), which may result in nonsensical replacement. In order not to include synonyms that have unusual part of speech, we counted the frequency of POS in the synset and only selected the words with the most frequent POS. Using the synonym set  $S$  after filtering, we are able to generate adversarial input texts that mislead models' prediction while barely affect on human understanding.
Contextualized Semantic Perturbation Function  $\mathcal{F}_C$  is a novel perturbation function that explores the BERT embedding space and searches for contextualized perturbation to tackle the issue of most language tokens being polysemous. Previous work (Li et al., 2018; Jin et al., 2020) takes it as a standard practice to use the proximity in embedding space to query the semantic similarity. However, their embedding space is built on a non-contextualized word embedding from GLoVE (Pennington et al., 2014) or Word2Vec (Mikolov et al., 2013), thus failing to consider the polysemy when
generating the perturbation. We propose to explore the BERT embedding space, which is verified by (Hewitt and Manning, 2019; Coenen et al., 2019; Papadimitriou et al., 2021) that BERT embeddings can preserve syntactic and semantic information for word sense disambiguation better than GLoVE or Word2Vec. So the contextualized space from  $\mathcal{F}_C$  is valid semantic perturbations. Similar to our parallel work (Li et al., 2020) of using BERT to generate adversarial perturbations,  $\mathcal{F}_C$  also does not require external linguistic resources such as POS checker. Thus  $\mathcal{F}_C$  can be adapted to other languages, as long as pre-trained BERT of such language models is available.
Specifically, we first choose a set of commonly used tokens  $\mathcal{X}$ . For each word  $x\in \mathcal{X}$ , we select at most 100 example sentences from Wikipedia that contain the word  $x$  so that these sentences represent different meanings of  $x$  in different contexts. We then feed these sentences into a pretrained BERT model to obtain the contextualized embeddings for each word  $x$ . Finally, the contextualized embeddings for all words in  $\mathcal{X}$  formulate a large BERT embedding space. Figure 1 visualizes a BERT embedding space projected into 2D space by PCA.
To query the search space  $S$  for token  $x$ , we first calculate the BERT embedding of token  $x$  given its context sentence. Even for the same token, given different contexts and meanings, BERT will generate distinct representations in the high dimensional embedding space. For the example in Figure 1, the token "experienced" given different contexts have different latent representations and neighbors. Then we use  $k$  nearest neighbors (KNN) algorithm to choose the neighbors of the contextualized embedding of  $x$  as its perturbation search space  $S$ . To ensure high quality of search space  $S$ , we further filter  $S$  and only return the words that appear more frequently than a threshold  $\epsilon$  among  $k$  nearest neighbors. In this way, we remove the noisy tokens that are rarely used and retain the high-quality neighbor tokens whose contextualized semantics are mostly close to the original token  $x$ .
Discussion. The final search space  $S$  can be the union of the search spaces mentioned above. This makes existing defense algorithm (Jones et al., 2020) difficult to apply, as they can only defend against typo-based perturbation but fails to detect other types of perturbation.
$\mathcal{F}$  is a generalization of most existing word-level textual adversarial attacks. Though  $\mathcal{F}_T$  and  $\mathcal{F}_K$
have been discussed in the previous literature (see  $\S$  Related Work), we note that the goal of our paper is not to improve or propose better typo or knowledge perturbation, but to consider multiple semantic spaces at the same time to help generate natural high-quality adversarial examples.
2.3 Attack Algorithm SemAttack
The full pipeline is shown in Appendix Algorithm 1. Essentially, SemAttack searches for the optimal perturbations from different semantic spaces determined by semantic perturbation functions, which is efficiently solved as an optimization problem so that we only perturb as few tokens as possible while achieving the targeted attack.
Unlike generating adversarial examples in the continuous data domain, it is difficult to directly utilize the gradient to guide token substitution due to the discrete nature of text. Thus, we search perturbation in the embedding space and map the perturbed embedding back to tokens. Specifically, the one-hot representation of each discrete token  $\boldsymbol{x}_i \in \mathbb{R}^{|V|}$  ( $V$  is the vocabulary set) is mapped into an embedding space of dimension  $d_c$  via the embedding matrix  $M_e \in \mathbb{R}^{d_c \times |V|}$
$$
\left[ e _ {1}; e _ {2}; \dots ; e _ {n} \right] = M _ {e} \left[ x _ {0}; x _ {1}; \dots ; x _ {n} \right]. \tag {3}
$$
We optimize perturbation  $e^{*}$  added to the original embedding  $e$  for  $m$  iterations. In each iteration, we freeze all the parameters of the classifier  $f$  and optimize variable  $e^{*}$  only. Following Carlini and Wagner (2016), we minimize the loss function as:
$$
\mathcal {L} \left(\boldsymbol {e} ^ {*}\right) = \left\| \boldsymbol {e} ^ {*} \right\| _ {p} + c \cdot g \left(\boldsymbol {x} ^ {\prime}\right), \tag {4}
$$
where the first term controls the magnitude of perturbation, while  $g(\cdot)$  is the attack objective function depending on the attack scenario.  $c$  weighs the attack goal against attack cost.
In targeted attack scenarios, we define  $g(\cdot)$  as:
$$
g \left(\boldsymbol {x} ^ {\prime}\right) = \max  \left[ \max  \left\{f \left(\boldsymbol {x} ^ {\prime}\right) _ {i}: i \neq t \right\} - f \left(\boldsymbol {x} ^ {\prime}\right) _ {t}, - \kappa \right],
$$
where  $t$  is the targeted false class and  $f(\pmb{x}^{\prime})_i$  is the  $i$ -th class logit. A larger  $\kappa$  encourages the classifier to output targeted false class with higher confidence.
In untargeted attack scenarios,  $g(\cdot)$  becomes
$$
g \left(\boldsymbol {x} ^ {\prime}\right) = \max  \left[ f \left(\boldsymbol {x} ^ {\prime}\right) _ {t} - \max  \left\{f \left(\boldsymbol {x} ^ {\prime}\right) _ {i}: i \neq t \right\}, - \kappa \right],
$$
where  $t$  is the ground truth class.
After each iteration of gradient descent, we have an optimized perturbation  $e^{*}$  in the embedding space that tends to fool the classifier  $f$  with small perturbations. We choose the perturbed token  $x_{i}^{\prime} \in S = \mathcal{F}(x_{i})$  that is from the semantic search space  $S$  returned by  $\mathcal{F}(x_i)$  and semantically closest to
the perturbed embedding  $e_i'$ .
$$
\begin{array}{l} \boldsymbol {e} _ {i} ^ {\prime} = \boldsymbol {e} _ {i} + \boldsymbol {e} _ {i} ^ {*}, \\ \boldsymbol {x} _ {i} ^ {\prime} = \underset {\boldsymbol {x} _ {i} ^ {\prime} \in S} {\arg \min } \left(\left\| \boldsymbol {e} _ {i} ^ {\prime} - M _ {e} \boldsymbol {x} _ {i} ^ {\prime} \right\| _ {p}\right). \tag {5} \\ \end{array}
$$
Finally, we obtain an optimal perturbation  $e^{*}$  after repeating the optimization step and token substitution step for  $m$  iterations. Under such settings and constraints, most tokens remain the same and very few are perturbed to their semantically close neighbors. Thus, the adversarial examples still look valid to humans but can fool the models.
In this section, we conduct comprehensive experiments to evaluate our attack method in various settings. We first apply our attack method to two standard NLP models, BERT and SOTA Self-Attention LSTM. We evaluate on two different types of NLP tasks, sentiment analysis and natural language inference (NLI). Secondly, we investigate the effectiveness of SemAttack against SOTA large-scale language models and defense methods. Thirdly, we take Chinese as an example to measure SemAttack's generalization ability across different languages. We evaluate BERT models finetuned on two Chinese datasets. Finally, we conduct extensive human evaluations on both English and Chinese datasets.
We find that: 1) SemAttack can achieve better attack success rates than existing textual adversarial attack methods with better language quality and adversarial transferability. 2) SOTA LMs and defense methods are still vulnerable to our SemAttack. 3) SemAttack is a general textual adversarial attack framework and can be easily adapted to other languages in addition to English with high attack success rates. 4) Adversarial examples generated by SemAttack are natural and barely affect human performance.
3.1 Whitebox and Blackbox Attack
Datasets For sentiment classification task, we choose the standard 5-class sentiment classification dataset, Yelp dataset. Note that unlike previous work (Li et al., 2020; Jin et al., 2020) that uses binary sentiment classification dataset, we focus on the standard 5-class Yelp dataset to further evaluate the targeted attack capability of SemAttack. For NLI task, we choose SNLI dataset. The detailed dataset descriptions are in Appendix §C.
Models We evaluate the robustness of BERT and Self-Attention LSTM (Lin et al., 2017). We present their test accuracy on the benign test sets in Table 2. More hyperparameter settings and training details are discussed in Appendix §B.
Attack Baselines We consider SOTA whitebox and blackbox attack baselines.
- HotFlip (Ebrahimi et al., 2017) is a whitebox attack method for generating adversarial examples on both character-level and word-level. In terms of preserving semantic meaning, we only use word-level attacks in our experiments, which uses gradient-based optimization method to flip words.
- TextFooler (Jin et al., 2020) is a blackbox attack method for generating adversarial text, which uses similarities between pre-calculated word embeddings to find synonyms for each word.
- BERT-Attack (Li et al., 2020) is a strong black-box attack method using pre-trained masked language models such as BERT to replace words in input sentences, where pre-trained masked language models provide candidate words that have high semantic similarity between original texts.
These methods all perform untargeted attacks. We adapt them to both untargeted and targeted attack settings in our experiments.
Attack Goal In the sentiment analysis task, we consider the targeted attack, and choose the most opposite sentiment class as the targeted class, so sentences with original label lower than 2 (negative) are attacked to class 4 (most positive), and others are attacked to class 0 (most negative). In the NLI task, Contradiction and neutral will be attacked to entailment while entailment will be attacked to contradiction.
Adversarial Attack Evaluation We perform SemAttack on BERT and LSTM-based classifiers in both the whitebox and blackbox settings. The whitebox setting approximates the worst-case scenario, where attackers have the access to the model parameters and gradients; while the black-box setting assumes that attackers can only access the model's output confidence.
For the whitebox attack shown in Table 2, SemAttack can outperform all the SOTA baselines and achieve the highest success rates in both untargeted and targeted settings for BERT and LSTM-based models with smaller or comparable perturbation rates. For example, untargeted SemAttack achieves  $97.6\%$  attack success rate
Model	Attack Method	% USR/TSR	% Perturbation
BERT (Acc: 0.706)	HotFlip	71.5/24.0	14.9/44.9
SemAttack (+FT)	42.4/9.3	4.7/9.1
SemAttack (+FK)	84.6/69.3	6.7/13.9
SemAttack (+FC)	91.3/79.7	4.7/11.1
SemAttack (+all)	97.6/93.8	4.3/10.2
Self-Attention LSTM (Acc: 0.705)	HotFlip	16.3/3.2	2.5/17.4
SemAttack (+FT)	67.2/49.4	14.7/21.1
SemAttack (+FK)	47.9/43.6	10.4/18.3
SemAttack (+FC)	67.3/56.5	15.1/23.2
SemAttack (+all)	88.1/84.0	19.2/29.2