SlowGuess/ACL_2020_OCR huggingface dataset:
Xiaoqing Zheng $^{1,2*}$ , Jiehang Zeng $^{1,2*}$ , Yi Zhou $^{1,2*}$ , Cho-Jui Hsieh $^{3}$ , Minhao Cheng $^{3}$ , Xuanjing Huang $^{1,2}$
$^{2}$ Shanghai Key Laboratory of Intelligent Information Processing
$^{3}$ Department of Computer Science, University of California, Los Angeles, USA
{zhengxq, jhzeng18, yizhou17}@fudan.edu.cn
{chohsieh, mhcheng}@cs.ucla.edu, xjhuang@fudan.edu.cn
Despite achieving prominent performance on many important tasks, it has been reported that neural networks are vulnerable to adversarial examples. Previously studies along this line mainly focused on semantic tasks such as sentiment analysis, question answering and reading comprehension. In this study, we show that adversarial examples also exist in dependency parsing: we propose two approaches to study where and how parsers make mistakes by searching over perturbations to existing texts at sentence and phrase levels, and design algorithms to construct such examples in both of the black-box and white-box settings. Our experiments with one of state-of-the-art parsers on the English Penn Treebank (PTB) show that up to  $77\%$  of input examples admit adversarial perturbations, and we also show that the robustness of parsing models can be improved by crafting high-quality adversaries and including them in the training stage, while suffering little to no performance drop on the clean input data.
Deep neural network-based machine learning (ML) models are powerful but vulnerable to adversarial examples. Adversarial examples also yield broader insights into the targeted models by exposing them to such maliciously crafted examples. The introduction of the adversarial example and training ushered in a new era to understand and improve the ML models, and has received significant attention recently (Szegedy et al., 2013; Goodfellow et al., 2015; Moosavi-Dezfooli et al., 2016; Papernot et al., 2016b; Carlini and Wagner, 2017; Yuan et al., 2019; Eykholt et al., 2018; Xu et al., 2019).
Even though generating adversarial examples for texts has proven to be a more challenging task
Figure 1: Sentence-level attack: An adversarial example (bottom) for the output (top) of a deep neural dependency parser (Dozat and Manning, 2017). Replacing a word "stock" with an adversially-chosen word "exchange" in the sentence causes the parser to make four mistakes (blue, dashed) in arc prediction. The adversarial example preserves the original syntactic structures, and the substitute word is assigned to the same part of speech (POS) as the replaced one. The assigned POS tags (blue) are listed below the words.
than for images and audios due to their discrete nature, a few methods have been proposed to generate adversarial text examples and reveal the vulnerability of deep neural networks in natural language processing (NLP) tasks including reading comprehension (Jia and Liang, 2017), text classification (Samanta and Mehta, 2017; Wong, 2017; Liang et al., 2018; Alzantot et al., 2018), machine translation (Zhao et al., 2018; Ebrahimi et al., 2018; Cheng et al., 2018) and dialogue systems (Cheng et al., 2019). These recent methods attack text examples mainly by replacing, scrambling, and erasing characters or words or other language units under certain semantics-preserving constraints.
Although adversarial examples have been studied recently for NLP tasks, previous work almost exclusively focused on semantic tasks, where the attacks aim to alter the semantic prediction of ML models (e.g., sentiment prediction or question answering) without changing the meaning of original texts. To the best of our knowledge, adversarial
* These authors contributed equally to this work.
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6600-6610 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
examples to syntactic tasks, such as dependency parsing, have not been studied in the literature. Motivated by this, we take the neural network-based dependency parsing algorithms as targeted models and aim to answer the following questions: Can we construct syntactic adversarial examples to fool a dependency parser without changing the original syntactic structure? And can we make dependency parsers robust with respect to these attacks?
To answer these questions, we propose two approaches to study where and how parsers make mistakes by searching over perturbations to existing texts at sentence and phrase (corresponding to subtrees in a parse tree) levels. For the sentence-level attack, we modify an input sentence to fool a dependency parser while such modification should be syntactically imperceptible to humans (see Figure 1). Any new error (excluding the arcs directly connected to the modified parts) made by the parser is accounted as a successful attack.
For the phrase-level (or subtree-level) attack, we choose two phrases from a sentence, which are separated by at least  $k$  words (say  $k \geq 0$ ), and modify one phrase to cause the parser's prediction errors in another target phrase (see Figure 2). Unlike the sentence-level attack, any error occurred outside the target subtree is not considered as a successful attacking trial. It helps us to investigate whether an error in one part of a parse tree may exert long-range influence, and cause cascading errors (Ng and Curran, 2015). We study the sentence-level and subtree-level attacks both in white-box and black-box settings. In the former setting, an attacker can access to the model's architecture and parameters while it is not allowed in the latter one.
Our contributions are summarized as follows: (1) we explore the feasibility of generating syntactic adversarial sentence examples to cause a dependency parser to make mistakes without altering the original syntactic structures; (2) we propose two approaches to construct the syntactic adversarial examples by searching over perturbations to existing texts at sentence and phrase levels in both the black-box and white-box settings; (3) our experiments with a close to state-of-the-art parser on the English Penn Treebank show that up to  $77\%$  of input examples admit adversarial perturbations, and moreover that robustness and generalization of parsing models can be improved by adversarial training with the proposed attacks. The source code is available at (https://github.com/zjiehang/DPAttack).
Figure 2: Phrase-level attack: two separate subtrees in a parse tree are selected, and one of them (left) is deliberately modified to cause a parser to make incorrect arc prediction for another target subtree (right). For example, we can make a neural dependency parser (Dozat and Manning, 2017) to attach the word "difference" in the target subtree to its sibling "in" instead of the correct head "lock" (the subtree's root) by maliciously manipulating the selected leftmost subtree only.
Generating adversarial examples – inputs intentionally crafted to fool a model – has become an important means of exploring model vulnerabilities. Furthermore, adding adversarial examples in the training stage, also known as adversarial training, has become one of the most promising ways to improve model's robustness. Although there is limited literature available for NLP adversarial examples, some studies have been conducted on NLP tasks such as reading comprehension (Jia and Liang, 2017), text classification (Samanta and Mehta, 2017; Wong, 2017; Liang et al., 2018; Alzantot et al., 2018), machine translation (Zhao et al., 2018; Ebrahimi et al., 2018; Cheng et al., 2018), and dialogue systems (Cheng et al., 2019).
Depending on the degree of access to the target model, adversarial examples can be constructed two different settings: white-box and black-box settings (Xu et al., 2019; Wang et al., 2019). In the white-box setting, an adversary can access the model's architecture, parameters and input feature representations while not in the black-box one. The white-box attacks normally yield a higher success rate because the knowledge of target models can be used to guide the generation of adversarial examples. However, the black-box attacks do not require access to target models, making them more practicable for many real-world attacks. Such attacks also can be divided into targeted and non-targeted ones depending on the purpose of adversary. Our phrase-level attack can be viewed as a targeted at
tack towards a specific subtree while the sentence-level attack can be taken as a non-targeted one.
For text data, input sentences can be manipulated at character (Ebrahimi et al., 2018), sememe (the minimum semantic units) (Zang et al., 2019), or word (Samanta and Mehta, 2017; Alzantot et al., 2018) levels by replacement, alteration (e.g. deliberately introducing typos or misspellings), swap, insertion, erasure, or directly making small perturbations to their feature embeddings. Generally, we would like to ensure that the crafted adversarial examples are sufficiently similar to their original ones, and these modifications should be made within semantics-preserving constraints. Such semantic similarity constraints are usually defined based on Cosine similarity (Wong, 2017; Barham and Feizi, 2019; Jin et al., 2019; Ribeiro et al., 2018) or edit distance (Gao et al., 2018).
Text adversarial example generation usually involves two steps: determine an important position (or token) to change; modify it slightly to maximize the model's prediction error. This two-step can be repeated iteratively until the model's prediction changes or certain stopping criteria are reached. Many methods have been proposed to determine the important positions by random selection (Alzantot et al., 2018), trial-and-error testing at each possible point (Kuleshov et al., 2018), analyzing the effects on the model of masking various parts of a input text (Samanta and Mehta, 2017; Gao et al., 2018; Jin et al., 2019; Yang et al., 2018), comparing their attention scores (Hsieh et al., 2019), or gradient-guided optimization methods (Ebrahimi et al., 2018; Lei et al., 2019; Wallace et al., 2019; Barham and Feizi, 2019).
After the important positions are identified, the most popular way to alter text examples is to replace the characters or words at selected positions with similar substitutes. Such substitutes can be chosen from nearest neighbours in an embedding space (Alzantot et al., 2018; Kuleshov et al., 2018; Jin et al., 2019; Barham and Feizi, 2019), synonyms in a prepared dictionary (Samanta and Mehta, 2017; Hsieh et al., 2019), visually similar alternatives like typos (Samanta and Mehta, 2017; Ebrahimi et al., 2018; Liang et al., 2018) or Internet slang and trademark logos (Eger et al., 2019), paraphrases (Lei et al., 2019) or even randomly selected ones (Gao et al., 2018). Given an input instance, Zhao et al. (2018) proposed to search for adversaries in the neighborhood of its corresponding
representation in latent space by sampling within a range that is recursively tightened. Jia and Liang (2017) tried to insert few distraction sentences generated by a simple set of rules into text examples to mislead a reading comprehension system.
Dependency parsing is the task of constructing a parse tree of a sentence that represents its syntactic structure and defines the relationships between "head" words and dependent ones, which modify their heads (see the arcs in Figure 1). In this section, we first describe a graph-based dependency parsing method, and then formally present the adversarial attack problem of dependency parsing.
3.1 Dependency Parsing
Graph-based parsing models learn the parameters to score correct dependency subgraphs over incorrect ones, typically by factoring the graphs directed edges (or arcs), and performs parsing by searching the highest-scoring graph for a given sentence.
Given a sentence  $x$ , we denote the set of all valid parse trees that can be constructed from  $x$  as  $\mathcal{Y}(x)$ . Assume that there exists a graph scoring function  $s$ , the dependency parsing problem can be formulated as finding the highest scoring directed spanning tree for the sentence  $x$ .
$$
y ^ {*} (x) = \underset {\hat {y} \in \mathcal {Y} (x)} {\operatorname {a r g m a x}} s (x, \hat {y}; \theta) \tag {1}
$$
where  $y^{*}(x)$  is the parse tree with the highest score, and  $\theta$  are all the parameters used to calculate the scores. Given a sentence  $x_{[1:n]}$  that is a sequence of  $n$  words  $x_{i}$ ,  $1 \leq i \leq n$ , the score of a graph is usually factorized into the sum of its arc scores to make the search tractable (McDonald et al., 2005).
$$
s (x, \hat {y}; \theta) = \sum_ {\left(x _ {h}, x _ {m}\right) \in A (\hat {y})} s \left(x _ {h}, x _ {m}; \theta\right) \tag {2}
$$
where  $A(\hat{y})$  represents a set of directed edges in the parse tree  $\hat{y}$ . The score of an arc  $(x_h, x_m)$  represents the likelihood of creating a dependency from head  $x_h$  to modifier  $x_m$  in a dependency tree.
A neural network can be considered as a mapping  $f: \mathcal{X} \to \mathcal{Y}$  from an input  $x \in \mathcal{X}$  to a output  $y \in \mathcal{Y}$  with parameters  $\theta$ . For classification problems,  $y$  is a label which lies in some finite set of categories. For the dependency parsing,  $y$  is one of valid parses that can be built from  $x$ . The model  $f$  maps  $x$  to  $y^*$  with the highest score, as defined in Equation (1).
6602
Given the original input  $x$ , adversarial examples are crafted to cause an ML model to misbehave. Following the common definition in previous papers (e.g., Kuleshov et al., (2018)), for a model  $f$  we say  $x'$  is a good adversarial example of  $x$  for untargeted attack if
$$
f \left(x ^ {\prime}\right) \neq y, c \left(x, x ^ {\prime}\right) \leq \epsilon \tag {3}
$$
where  $y$  is the truth output for  $x$ . For targeted attack the goal is to turn  $f(x^{\prime})$  into a particular targeted class, denoted by  $y^\prime$ , under the same constraint in (3). The constraint function  $c:\mathcal{X}\times \mathcal{X}\to \mathbb{R}_+^g$  and a vector of bounds  $\epsilon \in \mathbb{R}^{g}(g\geq 1)$  reflect the notion of the "imperceptibility" of perturbation to ensure that the true label of  $x^{\prime}$  should be the same as  $x$ . In the context of image classification, popular choices of such constraint include  $\ell_0,\ell_2$  and  $\ell_{\infty}$  distances. For natural language tasks,  $x$  and  $x^{\prime}$  are sentences composed with discrete words, and previous methods often define  $c$  to measure the semantic similarity between them, and thus  $x,x^{\prime}$  should have the same semantic meaning while being predicted differently using model  $f$ . In this paper, we consider the syntactic similarity and propose various ways to define such constraint for the dependency parsing task (see Section 4).
Generating adversarial examples can be formulated as an optimization problem of maximizing the probability of  $f(x') \neq y$  by choosing  $x'$  for  $x$  subject to  $c(x, x') \leq \epsilon$ . Algorithms for solving this problem include fast gradient sign method (Goodfellow et al., 2015), iterative methods based on constrained gradient descent (Papernot et al., 2016a), GAN-based strategy (Wong, 2017), genetic algorithms (Alzantot et al., 2018), and submodular set function maximization (Lei et al., 2019).
Adversarial examples are required to maintain the original functionality of the input. In the adversarial NLP literature, previous studies often expect the adversarial examples to retain the same or similar semantic meaning as the original one (Samanta and Mehta, 2017; Wong, 2017; Alzantot et al., 2018; Zhao et al., 2018; Zang et al., 2019). However, in this paper we focus on the dependency parsing task, which focuses on predicting the syntactic structure of input sentences. Therefore, to expose regions of the input space where the dependency parsers perform poorly, we would like the modified examples  $x'$  to preserve the same syntactic structure as the original  $x$ , but slightly relax the constraint
on their similarity in semantic properties. A robust parser should perform consistently well on the sentences that share the same syntactic properties, while differ in their meaning. For example, substituting the word "black" for "white", or "dog" for "cat" are acceptable replacements because they are grammatically imperceptible to humans.
4.1 Adversarial Examples for Parsing
We craft the adversarial examples mainly by replacing few words in an input sentence with carefully selected ones. To preserve the same syntactic structure as the original sentence  $x$ , we impose the following three constraints that should be satisfied by the word replacement when generating the adversarial examples  $x'$ :
(i) The substitute word  $x_{i}^{\prime}$  should fit in well with the context, and can maintain both the semantic and syntactic coherence.
(ii) For any word  $x_{i}$  in an original example, the word  $x_{i}'$  to replace  $x_{i}$  must have the same part-of-speech (POS) as  $x_{i}$ .
(iii) Pronouns, articles, conjunctions, numerals, interjections, interrogative determiners, and punctuations are not allowed to be replaced<sup>1</sup>.
To select a substitute word that agrees well with the context of a sentence, we use the BERT (Devlin et al., 2019) to generate a set of candidate words that are suitable to replace the original word thanks to its bidirectional language model that is capable of capturing the wider context of the entire sentence<sup>2</sup>. Words that are assigned to the same POS generally have similar grammatical properties and display similar syntactic behavior. To enforce the second constraint, we require that the substitute  $x_{i}^{\prime}$  should be assigned to the same part of speech as  $x_{i}$  by a POS tagger like (Samanta and Mehta, 2017; Ebrahimi et al., 2018). We filter out the aforementioned words in the third constraint.
We adopt the following two-step procedure for generating text adversarial examples: choose weak spots (or positions) to change, and then modify them to maximize the model's error. In the black-box setting, we first identify the weak spots of an
<sup>1</sup>We exclude those words from being replaced because either there are very limited number of substitutes available, or such replacements easily lead to syntactic inconsistency.
2We also tried to replace words with their nearest neighbors in the vector space of pre-trained word embeddings such as GloVe (Pennington et al., 2014). However, our preliminary experiments show that these nearest neighbors cannot fit well with the context in many cases since the neighboring words are retrieved without taking the specific context into account.
6603
input sentence with the greedy search strategy by replacing each word, one at a time, with a special "unknown" symbol (<unk>), and examining the changes in unlabeled attachment score (UAS) like (Yang et al., 2018; Gao et al., 2018; Hsieh et al., 2019). For each identified weak spot, we replace it with a word in the candidate set proposed by the BERT to form an attack. We select the substitute word that causes the greatest decrease in UAS while satisfying the aforementioned constraints to construct the adversarial example. This process is repeated until all candidate words are exhausted and every weak spot is tested (see Figure 3).
In the white-box setting, full access to the target model's parameters and features enables us to launch a "surgical" attack by crafting more accurate adversarial examples. We propose a scoring function to determine which parts are more vulnerable to adversarial attacks for an input sentence  $x$  of  $n$  words  $x_{i}$  ( $1 \leq i \leq n$ ) as follows.
$$
\mathcal {F} (x, \theta) = \sum_ {m = 1} ^ {n} \max  [ s (x _ {h}, x _ {m}; \theta) - \max  _ {j \neq h} s (x _ {j}, x _ {m}; \theta), - \varepsilon ]
$$
$$
\mathcal {S} (x _ {i}, \theta) = \left\| \frac {\partial \mathcal {F} (x , \theta)}{\partial e _ {x _ {i}}} \right\| _ {2} \tag {4}
$$
where  $\theta$  are all the parameters of a target dependency parser,  $e_{x_i}$  is the embedding of word  $x_i$ , and  $\varepsilon \geq 0$  denotes a confidence margin. A larger  $\varepsilon$  will lead to a more confident output and a higher success rate, but with the cost of more iterations. The function  $\mathcal{F}(x,\theta)$  sums up all the differences between the score of any ground truth arc  $(x_h,x_m)$  and that of the incorrect, but the highest scoring one with the same dependant  $x_m$ . Generally speaking, the greater the value of this function is, the harder we can find adversarial examples for the input  $x$  because it has a larger margin between the true parse tree and any incorrect one. Minimizing this function maximizes the probability of causing the parser to misbehave.
We determine the importance of words by their values of  $S(x_{i},\theta)$ , namely the norm of the partial derivative of the function  $\mathcal{F}(x,\theta)$  with respect to the word  $x_{i}$ . The key idea is that we use the magnitude of the gradient to decide which words to attack. Assuming we have a set of candidate words  $C_{x_i}$ , we select the optimal one  $x_{i}^{*}$  by:
$$
x _ {i} ^ {*} = \underset {w \in \mathcal {C} _ {x _ {i}}} {\operatorname {a r g m i n}} \left\| e _ {w} - \left(e _ {x _ {i}} - \frac {\alpha}{\mathcal {S} (x _ {i} , \theta)} \frac {\partial \mathcal {F} (x , \theta)}{\partial e _ {x _ {i}}}\right) \right\| _ {2} \tag {5}
$$
where the coefficient  $\alpha$  governs the relative importance of the normalized gradient term. We want
the selected word as close to the replaced one  $x_{i}$  as possible in their embedding space according to the Euclidean distance, where the embedding of  $x_{i}$  is updated in the opposite direction of the gradient at the rate of  $\alpha$ . Such replacement will lead to a decrease in the value of the function  $\mathcal{F}(x,\theta)$ . Our algorithm of generating adversarial examples for dependency parsing in the white-box setting is shown in Figure 4.
$x_{[1:n]}$ : an input sentence of  $n$  words  $x_{i}, 1 \leq i \leq n$ .
$f$  : a target parser.
$\gamma$ : the maximum percentage of words that can be modified.
$\psi$  : the size of the set of candidate words.
Output: an adversarial example  $x'$  of  $x$ .
1:  $\kappa = \gamma \cdot n$  (the maximum number of words to be modified)
2: for each word  $x_{i}$  except those listed in the constraint (iii)
3:  $\hat{x}_i =$  replace  $x_{i}$  with a special symbol  $" <   \mathrm{unk} > "$  in  $x$
4: calculate the unlabeled attachment score of  $f(\hat{x}_i)$ .
5: sort  $\hat{x}_i$  by their UAS, and append the top-  $\kappa$  positions into an ordered index list  $[1:\kappa ]$
6: for each position  $j$  in the list  $[1:\kappa]$
7: generate a set of  $\psi$  candidate words  $\mathcal{C}_j$  by BERT;
8: remove the words from  $\mathcal{C}_j$  if they do not have the same part-of-speech as the  $x_{j}$
9: select the word  $x_{j}^{*} \in \mathcal{C}_{j}$  that causes the greatest decrease in UAS if we replace  $x_{j}$  with  $x_{j}^{*}$  in  $x$ ;
10:  $x' = \text{replace } x_j$  with the word  $x_j^*$  in  $x$ .
11: return  $x'$ .
Figure 3: Adversarial example generation algorithm for dependency parsing in the black-box setting.
4.2 Sentence-level and Phrase-level Attacks
For the sentence-level attack, we simply use the algorithms listed in Figure 3 and 4 to form a attack. For the phrase-level attack, we first choose two phrases (corresponding to two subtrees in a parse) from a sentence, which do not overlap each other and are separated by at least  $k$  words. Then, we try to cause the parser to make mistakes in a target subtree by modifying another one. Unlike the sentence-level attack, any error occurred outside the target subtree will not be counted as a successful trial. Note that even if we can force the parser to change its prediction on the head of the target subtree's root, it is still not considered as a successful attack because the changed edge connects a certain word outside the subtree.
We require that all the subtrees should contain 4 to 12 words $^3$ , and the source subtree to be modified
${}^{3}$  A subtree-level attack can be launched on a sentence if it has at least two subtrees. We ensure that there are enough sen-
6604