SlowGuess/ACL_2020_OCR huggingface dataset:
Yuan Zang $^{1*}$ , Fanchao Qi $^{1*}$ , Chenghao Yang $^{2*†}$ , Zhiyuan Liu $^{1‡}$ , Meng Zhang $^{3}$ , Qun Liu $^{3}$ , Maosong Sun $^{1}$
$^{1}$ Department of Computer Science and Technology, Tsinghua University Institute for Artificial Intelligence, Tsinghua University
$^{2}$ Columbia University  $^{3}$ Huawei Noah's Ark Lab
{zangy17,qfc17}@mails.tsinghua.edu.cn，yangalan1996@gmail.com
{liuzy,sms}@tsinghua.edu.cn,{zhangmeng92,qun.liu}@huawei.com
Adversarial attacks are carried out to reveal the vulnerability of deep neural networks. Textual adversarial attacking is challenging because text is discrete and a small perturbation can bring significant change to the original input. Word-level attacking, which can be regarded as a combinatorial optimization problem, is a well-studied class of textual attack methods. However, existing word-level attack models are far from perfect, largely because unsuitable search space reduction methods and inefficient optimization algorithms are employed. In this paper, we propose a novel attack model, which incorporates the sememe-based word substitution method and particle swarm optimization-based search algorithm to solve the two problems separately. We conduct exhaustive experiments to evaluate our attack model by attacking BiLSTM and BERT on three benchmark datasets. Experimental results demonstrate that our model consistently achieves much higher attack success rates and crafts more high-quality adversarial examples as compared to baseline methods. Also, further experiments show our model has higher transferability and can bring more robustness enhancement to victim models by adversarial training. All the code and data of this paper can be obtained on https://github.com/thunlp/SememePSO-Attack.
Adversarial attacks use adversarial examples (Szegedy et al., 2014; Goodfellow et al., 2015), which are maliciously crafted by perturbing the original input, to fool the deep neural networks
Figure 1: An example showing search space reduction with sememe-based word substitution and adversarial example search in word-level adversarial attacks.
(DNNs). Extensive studies have demonstrated that DNNs are vulnerable to adversarial attacks, e.g., minor modification to highly poisonous phrases can easily deceive Google's toxic comment detection systems (Hosseini et al., 2017). From another perspective, adversarial attacks are also used to improve robustness and interpretability of DNNs (Wallace et al., 2019). In the field of natural language processing (NLP) which widely employs DNNs, practical systems such as spam filtering (Stringhini et al., 2010) and malware detection (Kolter and Maloof, 2006) have been broadly used, but at the same time the concerns about their security are growing. Therefore, the research on textual adversarial attacks becomes increasingly important.
Textual adversarial attacking is challenging. Different from images, a truly imperceptible perturbation on text is almost impossible because of its discrete nature. Even a slightest character-level perturbation can either (1) change the meaning and, worse still, the true label of the original input, or (2) break its grammaticality and naturality. Unfortunately, the change of true label will make the adversarial attack invalid. For example, supposing an adversary changes "she" to "he" in an input
* Indicates equal contribution. Yuan developed the method, designed and conducted most experiments; Fanchao formalized the task, designed some experiments and wrote the paper; Chenghao made the original research proposal, performed human evaluation and conducted some experiments.
† Work done during internship at Tsinghua University
6066
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6066-6080 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
sentence to attack a gender identification model, although the victim model alters its prediction result, this is not a valid attack. And the adversarial examples with broken grammaticality and naturality (i.e., poor quality) can be easily defended (Pruthi et al., 2019).
Various textual adversarial attack models have been proposed (Wang et al., 2019a), ranging from character-level flipping (Ebrahimimi et al., 2018) to sentence-level paraphrasing (Iyyer et al., 2018). Among them, word-level attack models, mostly word substitution-based models, perform comparatively well on both attack efficiency and adversarial example quality (Wang et al., 2019b).
Word-level adversarial attacking is actually a problem of combinatorial optimization (Wolsey and Nemhauser, 1999), as its goal is to craft adversarial examples which can successfully fool the victim model using a limited vocabulary. In this paper, as shown in Figure 1, we break this combinatorial optimization problem down into two steps including (1) reducing search space and (2) searching for adversarial examples.
The first step is aimed at excluding invalid or low-quality potential adversarial examples and retaining the valid ones with good grammaticality and naturality. The most common manner is to pick some candidate substitutes for each word in the original input and use their combinations as the reduced discrete search space. However, existing attack models either disregard this step (Papernot et al., 2016) or adopt unsatisfactory substitution methods that do not perform well in the trade-off between quality and quantity of the retained adversarial examples (Alzantot et al., 2018; Ren et al., 2019). The second step is supposed to find adversarial examples that can successfully fool the victim model in the reduced search space. Previous studies have explored diverse search algorithms including gradient descent (Papernot et al., 2016), genetic algorithm (Alzantot et al., 2018) and greedy algorithm (Ren et al., 2019). Some of them like gradient descent only work in the white-box setting where full knowledge of the victim model is required. In real situations, however, we usually have no access to the internal structures of victim models. As for the other black-box algorithms, they are not efficient and effective enough in searching for adversarial examples.
These problems negatively affect the overall attack performance of existing word-level adversaries
ial attacking. To solve the problems, we propose a novel black-box word-level adversarial attack model, which reforms both the two steps. In the first step, we design a word substitution method based on sememes, the minimum semantic units, which can retain more potential valid adversarial examples with high quality. In the second step, we present a search algorithm based on particle swarm optimization (Eberhart and Kennedy, 1995), which is very efficient and performs better in finding adversarial examples. We conduct exhaustive experiments to evaluate our model. Experimental results show that, compared with baseline models, our model not only achieves the highest attack success rate (e.g.,  $100\%$  when attacking BiLSTM on IMDB) but also possesses the best adversarial example quality and comparable attack validity. We also conduct decomposition analyses to manifest the advantages of the two parts of our model separately. Finally, we demonstrate that our model has the highest transferability and can bring the most robustness improvement to victim models by adversarial training.
In this section, we first briefly introduce sememes, and then we give an overview of the classical particle swarm optimization algorithm.
2.1 Sememes
In linguistics, a sememe is defined as the minimum semantic unit of human languages (Bloomfield, 1926). The meaning of a word can be represented by the composition of its sememes.
In the field of NLP, sememe knowledge bases are built to utilize sememes in practical applications, where sememes are generally regarded as semantic labels of words (as shown in Figure 1). HowNet (Dong and Dong, 2006) is the most well-known one. It annotates over one hundred thousand English and Chinese words with a predefined sets of about 2,000 sememes. Its sememe annotations are sense-level, i.e., each sense of a (polysemous) word is annotated with sememes separately. With the help of HowNet, sememes have been successfully applied to many NLP tasks including word representation learning (Niu et al., 2017), sentiment analysis (Fu et al., 2013), semantic composition (Qi et al., 2019), sequence modeling (Qin et al., 2019), reverse dictionary (Zhang et al., 2019b), etc.
2.2 Particle Swarm Optimization
Inspired by the social behaviors like bird flocking, particle swarm optimization (PSO) is a kind of metaheuristic population-based evolutionary computation paradigms (Eberhart and Kennedy, 1995). It has been proved effective in solving the optimization problems such as image classification (Omran et al., 2004), part-of-speech tagging (Silva et al., 2012) and text clustering (Cagnina et al., 2014). Empirical studies have proven it is more efficient than some other optimization algorithms like the genetic algorithm (Hassan et al., 2005).
PSO exploits a population of interacting individuals to iteratively search for the optimal solution in the specific space. The population is called a swarm and the individuals are called particles. Each particle has a position in the search space and moves with an adaptable velocity.
Formally, when searching in a  $D$ -dimensional continuous space  $\mathcal{S} \subseteq \mathbb{R}^D$  with a swarm containing  $N$  particles, the position and velocity of each particle can be represented by  $\mathbf{x}^n \in \mathcal{S}$  and  $\mathbf{v}^n \in \mathbb{R}^D$  respectively,  $n \in \{1, \dots, N\}$ . Next we describe the PSO algorithm step by step.
(1) Initialize. At the very beginning, each particle is randomly initialized with a position  $\mathbf{x}^n$  in the search space and a velocity  $\mathbf{v}^n$ . Each dimension of the initial velocity  $v_{d}^{n}\in [-V_{max},V_{max}]$ ,  $d\in \{1,\dots ,D\}$ .
(2) Record. Each position in the search space corresponds to an optimization score. The position a particle has reached with the highest optimization score is recorded as its individual best position. The best position among the individual best positions of all the particles is recorded as the global best position.
(3) Terminate. If current global best position has achieved the desired optimization score, the algorithm terminates and outputs the global best position as the search result.
(4) Update. Otherwise, the velocity and position of each particle are updated according to its current position and individual best position together with the global best position. The updating formulae are
$$
\begin{array}{l} v _ {d} ^ {n} = \omega v _ {d} ^ {n} + c _ {1} \times r _ {1} \times \left(p _ {d} ^ {n} - x _ {d} ^ {n}\right) \\ + c _ {2} \times r _ {2} \times \left(p _ {d} ^ {g} - x _ {d} ^ {n}\right), \tag {1} \\ \end{array}
$$
$$
x _ {d} ^ {n} = x _ {d} ^ {n} + v _ {d} ^ {n},
$$
where  $\omega$  is the inertia weight,  $p_d^n$  and  $p_d^g$  are the  $d$ -th dimensions of the  $n$ -th particle's individual best position and the global best position respectively,
$c_{1}$  and  $c_{2}$  are acceleration coefficients which are positive constants and control how fast the particle moves towards its individual best position and the global best position, and  $r_1$  and  $r_2$  are random coefficients. After updating, the algorithm goes back to the Record step.
In this section, we detail our word-level adversarial attack model. It incorporates two parts, namely the sememe-based word substitution method and PSO-based adversarial example search algorithm.
3.1 Sememe-based Word Substitution Method
The sememes of a word are supposed to accurately depict the meaning of the word (Dong and Dong, 2006). Therefore, the words with the same sememe annotations should have the same meanings, and they can serve as the substitutes for each other.
Compared with other word substitution methods, mostly including word embedding-based (Sato et al., 2018), language model-based (Zhang et al., 2019a) and synonym-based methods (Samanta and Mehta, 2017; Ren et al., 2019), the sememe-based word substitution method can achieve a better trade-off between quality and quantity of substitute words.
For one thing, although the word embedding and language model-based substitution methods can find as many substitute words as we want simply by relaxing the restrictions on embedding distance and language model prediction score, they inevitably introduce many inappropriate and low-quality substitutes, such as antonyms and semantically related but not similar words, into adversarial examples which might break the semantics, grammaticality and naturality of original input. In contrast, the sememe-based and, of course, the synonym-based substitution methods does not have this problem.
For another, compared with the synonym-based method, the sememe-based method can find more substitute words and, in turn, retain more potential adversarial examples, because HowNet annotates sememes for all kinds of words. The synonym-based method, however, depends on thesauri like WordNet (Miller, 1995), which provide no synonyms for many words like proper nouns and the number of a word's synonyms is very limited. An empirical comparison of different word substitution methods is given in Section 4.6.
6068
In our sememe-based word substitution method, to preserve grammaticality, we only substitute content words<sup>1</sup> and restrict the substitutes to having the same part-of-speech tags as the original words. Considering polysemy, a word  $w$  can be substituted by another word  $w_{*}$  only if one of  $w$ 's senses has the same sememe annotations as one of  $w_{*}$ 's senses. When making substitutions, we conduct lemmatization to enable more substitutions and delemmatization to avoid introducing grammatical mistakes.
3.2 PSO-based Adversarial Example Search Algorithm
Before presenting our algorithm, we first explain what the concepts in the original PSO algorithm correspond to in the adversarial example search problem.
Different from original PSO, the search space of word-level adversarial example search is discrete. A position in the search space corresponds to a sentence (or an adversarial example), and each dimension of a position corresponds to a word. Formally,  $\mathbf{x}^n = w_1^n\cdots w_d^n\cdots w_D^n$ ,  $w_d^n \in \mathbb{V}(w_d^o)$ , where  $D$  is the length (word number) of the original input,  $w_d^o$  is the  $d$ -th word in the original input, and  $\mathbb{V}(w_d^o)$  is composed of  $w_d^o$  and its substitutes.
The optimization score of a position is the target label's prediction probability given by the victim model, where the target label is the desired classification result for an adversarial attack. Taking a binary classification task as an example, if the true label of the original input is "positive", the target label is "negative", and vice versa. In addition, a particle's velocity now relates to the position change probability, i.e.,  $v_{d}^{n}$  determines how probable  $w_{d}^{n}$  is substituted by another word.
Next we describe our algorithm step by step.
First, for the Initialize step, since we expect the adversarial examples to differ from the original input as little as possible, we do not make random initialization. Instead, we randomly substitute one word of the original input to determine the initial position of a particle. This operation is actually the mutation of genetic algorithm, which has also been employed in some studies on discrete PSO (Higashi and Iba, 2003). We repeat mutation  $N$  times to initialize the positions of  $N$  particles. Each dimension of each particle's velocity is randomly
initialized between  $-V_{max}$  and  $V_{max}$ .
For the Record step, our algorithm keeps the same as the original PSO algorithm. For the Terminate step, the termination condition is the victim model predicts the target label for any of current adversarial examples.
For the Update step, considering the discreteness of search space, we follow Kennedy and Eberhart (1997) to adapt the updating formula of velocity to
$$
v _ {d} ^ {n} = \omega v _ {d} ^ {n} + (1 - \omega) \times \left[ \mathcal {I} \left(p _ {d} ^ {n}, x _ {d} ^ {n}\right) + \mathcal {I} \left(p _ {d} ^ {g}, x _ {d} ^ {n}\right) \right], \tag {2}
$$
where  $\omega$  is still the inertia weight, and  $\mathcal{I}(a,b)$  is defined as
$$
\mathcal {I} (a, b) = \left\{ \begin{array}{c c} 1, & a = b, \\ - 1, & a \neq b. \end{array} \right. \tag {3}
$$
Following Shi and Eberhart (1998), we let the inertia weight decrease with the increase of numbers of iteration times, aiming to make the particles highly dynamic to explore more positions in the early stage and gather around the best positions quickly in the final stage. Specifically,
$$
\omega = \left(\omega_ {m a x} - \omega_ {m i n}\right) \times \frac {T - t}{T} + \omega_ {m i n}, \qquad (4)
$$
where  $0 < \omega_{min} < \omega_{max} < 1$ , and  $T$  and  $t$  are the maximum and current numbers of iteration times.
The updating of positions also needs to be adjusted to the discrete search space. Inspired by Kennedy and Eberhart (1997), instead of making addition, we adopt a probabilistic method to update the position of a particle to the best positions. We design two-step position updating. In the first step, a new movement probability  $P_{i}$  is introduced, with which a particle determines whether it moves to its individual best position as a whole. Once a particle decides to move, the change of each dimension of its position depends on the same dimension of its velocity, specifically with the probability of sigmoid  $(v_{d}^{n})$ . No matter whether a particle has moved towards its individual best position or not, it would be processed in the second step. In the second step, each particle determines whether to move to the global best position with another movement probability  $P_{g}$ . And the change of each position dimension also relies on sigmoid  $(v_{d}^{n})$ .  $P_{i}$  and  $P_{g}$  vary with iteration to enhance search efficiency by adjusting the balance between local and global search, i.e., encouraging particles to explore more
<sup>1</sup>Content words are the words that carry meanings and consist mostly of nouns, verbs, adjectives and adverbs.
6069
Dataset	Task	#Class	Avg. #W	Train	Dev	Test	BiLSTM %ACC	BERT %ACC
IMDB	Sentiment Analysis	2	234	25000	0	25000	89.10	90.76
SST-2	Sentiment Analysis	2	17	6920	872	1821	83.75	90.28
SNLI	NLI	3	8	550152	10000	10000	84.43	89.58
Table 1: Details of datasets and their accuracy results of victim models. "#Class" means the number of classifications. "Avg. #W" signifies the average sentence length (number of words). "Train", "Val" and "Test" denote the instance numbers of the training, validation and test sets respectively. "BiLSTM %ACC" and "BERT %ACC" means the classification accuracy of BiLSTM and BERT.
space around their individual best positions in the early stage and search for better position around the global best position in the final stage. Formally,
$$
P _ {i} = P _ {\max } - \frac {t}{T} \times \left(P _ {\max } - P _ {\min }\right), \tag {5}
$$
$$
P _ {g} = P _ {m i n} + \frac {t}{T} \times (P _ {m a x} - P _ {m i n}),
$$
where  $0 < P_{\min} < P_{\max} < 1$
Besides, to enhance the search in unexplored space, we apply mutation to each particle after the update step. To avoid excessive modification, mutation is conducted with the probability
$$
P _ {m} (\mathbf {x} ^ {n}) = \min  \left(0, 1 - k \frac {\mathcal {E} \left(\mathbf {x} ^ {n} , \mathbf {x} ^ {o}\right)}{D}\right), \tag {6}
$$
where  $k$  is a positive constant,  $\mathbf{x}^o$  represents the original input, and  $\mathcal{E}$  measures the word-level edit distance (number of different words between two sentences).  $\frac{\mathcal{E}(\mathbf{x}^n,\mathbf{x}^o)}{D}$  is defined as the modification rate of an adversarial example. After mutation, the algorithm returns to the Record step.
In this section, we conduct comprehensive experiments to evaluate our attack model on the tasks of sentiment analysis and natural language inference.
4.1 Datasets and Victim Models
For sentiment analysis, we choose two benchmark datasets including IMDB (Maas et al., 2011) and SST-2 (Socher et al., 2013). Both of them are binary sentiment classification datasets. But the average sentence length of SST-2 (17 words) is much shorter than that of IMDB (234 words), which renders attacks on SST-2 more challenging. For natural language inference (NLI), we use the popular Stanford Natural Language Inference (SNLI) dataset (Bowman et al., 2015). Each instance in SNLI comprises a premise-hypothesis sentence pair and is labelled one of three relations including entailment, contradiction and neutral.
As for victim models, we choose two widely used universal sentence encoding models, namely bidirectional LSTM (BiLSTM) with max pooling (Conneau et al., 2017) and  $\mathrm{BERT}_{\mathrm{BASE}}$  (BERT) (Devlin et al., 2019). For BiLSTM, its hidden states are 128-dimensional, and it uses 300-dimensional pre-trained GloVe (Pennington et al., 2014) word embeddings. Details of the datasets and the classification accuracy results of the victim models are listed in Table 1.
We select two recent open-source word-level adversarial attack models as the baselines, which are typical and involve different search space reduction methods (step 1) and search algorithms (step 2).
The first baseline method (Alzantot et al., 2018) uses the combination of restrictions on word embedding distance and language model prediction score to reduce search space. As for search algorithm, it adopts genetic algorithm, another popular metaheuristic population-based evolutionary algorithm. We use "Embedding/LM+Genetic" to denote this baseline method.
The second baseline (Ren et al., 2019) chooses synonyms from WordNet (Miller, 1995) as substitutes and designs a saliency-based greedy algorithm as the search algorithm. We call this method "Synonym+Greedy". This baseline model is very similar to another attack model TextFooer (Jin et al., 2019), which has extra semantic similarity checking when searching adversarial examples. But we find the former performs better in almost all experiments, and thus we only select the former as a baseline for comparison.
In addition, to conduct decomposition analyses of different methods in the two steps separately, we combine different search space reduction methods (Embedding/LM, Synonym and our sememe-based substitution method (Sememe)), and search algorithms (Genetic, Greedy and our PSO).
Metrics	Evaluation Method	Better?
Success Rate	Auto	Higher
Validity	Human (Valid Attack Rate)	Higher
Modification Rate	Auto	Lower
Grammaticality	Auto (Error Increase Rate)	Lower
Fluency	Auto (Perplexity)	Lower
Naturality	Human (Naturality Score)	Higher
Table 2: Details of evaluation metrics. "Auto" and "Human" represent automatic and human evaluations respectively. "Higher" and "Lower" mean the higher/ lower the metric, the better a model performs.
For our PSO,  $V_{max}$  is set to 1,  $\omega_{max}$  and  $\omega_{min}$  are set to 0.8 and 0.2,  $P_{max}$  and  $P_{min}$  are also set to 0.8 and 0.2, and  $k$  in Equation (6) is set to 2. All these hyper-parameters have been tuned on the validation set. For the baselines, we use their recommended hyper-parameter settings. For the two population-based search algorithms Genetic and PSO, we set the maximum number of iteration times ( $T$  in Section 3.2) to 20 and the population size ( $N$  in Section 3.2) to 60, which are the same as Alzantot et al. (2018).
To improve evaluation efficiency, we randomly sample 1,000 correctly classified instances from the test sets of the three datasets as the original input to be perturbed. For SNLI, only the hypotheses are perturbed. Following Alzantot et al. (2018), we restrict the length of the original input to 10-100, exclude the out-of-vocabulary words from the substitute sets, and discard the adversarial examples with modification rates higher than  $25\%$ .
We evaluate the performance of attack models including their attack success rates, attack validity and the quality of adversarial examples. The details of our evaluation metrics are listed in Table 2.