SlowGuess/NAACL_2022_OCR huggingface dataset:
Liwen Zhang $^{\ddagger \diamond}$ , Zixia Jia $^{\ddagger \diamond}$ , Wenjuan Han $^{\ddagger \dagger}$ , Zilong Zheng $^{\dagger}$ , Kewei Tu $^{\diamond \star}$
$^{\dagger}$ Beijing Institute for General Artificial Intelligence
{zhanglw1, jiazx, tukw}@shanghaiitech.edu.cn
{hanwenjuan,zlzheng}@bigai.ai
Adversarial attack of structured prediction models faces various challenges such as the difficulty of perturbing discrete words, the sentence quality issue, and the sensitivity of outputs to small perturbations. In this work, we introduce SHARP, a new attack method that formulates the black-box adversarial attack as a search-based optimization problem with a specially designed objective function considering sentence fluency, meaning preservation and attacking effectiveness. Additionally, three different searching strategies are analyzed and compared, i.e., Beam Search, Metropolis-Hastings Sampling, and Hybrid Search. We demonstrate the effectiveness of our attacking strategies on two challenging structured prediction tasks: part-of-speech (POS) tagging and dependency parsing. Through automatic and human evaluations, we show that our method performs a more potent attack compared with pioneer arts. Moreover, the generated adversarial examples can be used to successfully boost the robustness and performance of the victim model via adversarial training.
Adversarial attacking aims to mislead the victim model (e.g., a trained dependency parser) to produce erroneous outputs when feeding adversarial examples. The process can be seen in Figure 1. Adversarial training improves the victim model in terms of performance and robustness by training on adversarial examples. Since structured prediction tasks such as sequence labeling and dependency parsing are critical building blocks of many natural language processing (NLP) systems, it is essential
Structured Prediction	Classification
Ratio of GT Change	0.8	0.2
Meaning Preservation	0.94	0.92
Fluency	142	144
Table 1: Perturbation sensitivity comparisons between structured prediction and classification task.
to study adversarial attacks and defense of structured prediction models (Jia and Liang, 2017; Wang et al., 2019).
However, multiple technical challenges are faced by attackers of structured prediction models in the NLP area. All adversarial attackers for NLP tasks face general challenges related to gradient computation of discrete inputs, grammatical correctness, and meaning preservation (Zhang et al., 2019a; Jia and Liang, 2017; Wang et al., 2019; Cheng et al., 2019b, 2020b). Another potential but important challenge lies in the sensitivity of structured prediction: small perturbations to input sentences may very likely change the target output structures. In contrast, small perturbations typically do not change sentence classification labels. Han et al. (2020) first qualitatively proposed this assumption. We quantitatively investigate this sensitivity in Table 1. Specifically, we attack two typical models (the sentiment classifier (Ren et al., 2019) for the classification task and the dependency parser (Dozat and Manning, 2017) for structured prediction task) using the widely-used word-substitution attackers PWWS (Ren et al., 2019) and FGSM (Goodfellow et al., 2015) respectively. For both tasks, we generate adversarial examples by substituting words with the same proportion. The adversarial examples have similar qualities: fluency with perplexity 142 vs. 144 and meaning-preservation degree with BLEU 0.94 vs. 0.92. However, when asking annotators to label the adversarial examples, we find that around  $80\%$  adversarial examples of
*Kewei Tu is the corresponding author. ‡ These authors contributed equally.
Findings of the Association for Computational Linguistics: NAACL 2022, pages 950 - 961
Figure 1: An illustration of attacking process.
the classification task keep the same target outputs as the original input sentences, while only around  $20\%$  adversarial examples of the structured prediction task have unchanged output target structures. The huge gap shows the sensitivity of structured prediction tasks, verifying the challenge of attacking structured prediction models.
Despite these challenging issues, recently a few researchers are working on attack of structured predictions. Zheng et al. (2020) tries to preserve the original target output structures by replacing words with the same part of speech tags. Wang et al. (2021) follows a similar method to generate adversarial examples. However, these approaches cannot handle the aforementioned sensitivity. Wang et al. (2021) reveals that the syntactic structures of  $25\%$  generated examples of the attacker from Zheng et al. (2020) and  $15\%$  from Wang et al. (2021) are changed, although they both carefully design specific rules based on linguistic prior knowledge to preserve the structures. On the other hand, Han et al. (2020) chooses to generate silver structures instead of assuming unchanged target outputs. However, the proposed method of training a sequence-to-sequence adversarial example generator needs time-consuming training and often leads to ungrammatical and unnatural-looking sentences.
To address the above challenges, in this paper, we propose a novel and efficient attack method: SearcH-based adversarial Attack for stRuctured Prediction (SHARP). We formulate black-box adversarial attack as an optimization problem that seeks to maximize a specially designed objective function for better fluency, contextual consistency, and attacking effectiveness. In addition, we use a pretrained masked language model (PLM) to prune candidate sentences when exploring the search space. While our approach can be applied to any structured prediction tasks, in this paper we evaluate our approach on POS tagging models and de
pendency parsing models. Both automatic and human evaluations show that our method beats previous state-of-the-art (SOTA) approaches by a large margin. We also show that the generated adversarial examples can be used to boost the victim model in terms of accuracy and robustness with adversarial training.
2.1 Structured Prediction
Structured prediction in NLP aims to predict a structured output such as a sequence in the POS tagging task or a tree in the dependency parsing task. Given an input sentence  $\mathbf{x}$ , a structured prediction model predicts the output  $\mathbf{y}$  by maximizing the log conditional probability:
$$
\operatorname *{arg  max}_{\mathbf{y}\in \mathcal{T}}\log P(\mathbf{y}|\mathbf{x})
$$
where  $\mathcal{T}$  is the set of all possible outputs. The prediction model can be trained by maximizing the log probability of the target structure  $\mathbf{y}^*$  given a training set which contains  $(\mathbf{x},\mathbf{y}^{*})$  pairs.
Our purpose is to attack a well-trained structured prediction model through searching adversarial sentences. Besides, by leveraging adversarial sentences and the original training data to retrain the model, we can defend against attacks and enhance the model's robustness.
2.2 Adversarial Attack
Let  $\mathbf{x}^{ori} = \{w_0^{ori}, w_1^{ori}, \dots, w_{N-1}^{ori}\}$  denote an original sentence with  $N$  words. The victim model  $\mathcal{M}_V(\mathbf{x}) : \mathbf{x} \to \mathbf{y}$  has been trained to produce a structured prediction output that is close to the golden structure  $\mathbf{y}^*$ . Then the task for adversarial attack is to fool the model  $\mathcal{M}_V$  by feeding an imperceptible adversarial example  $\mathbf{x}^{adv}$  such that  $\mathcal{M}_V(\mathbf{x}^{adv}) \neq \text{target output of } \mathbf{x}^{adv}$ .
In this work, we focus on the black-box attack setting, where only the outputs of the victim model  $\mathcal{M}_V$  are accessible, while the internal details are
Our implementations are publicly available at https: //github.com/JZXXX/SHARP.
invisible, including the model structure, hyperparameters, training strategy, the training dataset, and gradients over each layer, etc.
3 Search-based Adversarial Attack
3.1 Attacking Objective
Typically, an ideal adversarial natural language example should be: (i) able to fool the victim parser to generate an erroneous output; (ii) fluent and grammatically correct; (iii) semantically consistent with the original sentence  $\mathbf{x}^{ori}$ . We consider the following objectives to address the above requirements, respectively.
Attacking Effectiveness. Ensuring the error of  $\mathcal{M}_V(\mathbf{x}^{adv})$  is non-trivial due to the lack of new ground truth structured outputs. To estimate the new ground truth and further identify if the adversarial sentence can indeed fool the victim model  $\mathcal{M}_V$ , we follow Han et al. (2020) to make use of two external reference models  $\mathcal{M}_A$  and  $\mathcal{M}_B$ . Because we want the victim model to predict wrong outputs of the adversarial examples, a good adversarial example  $\mathbf{x}^{adv}$  should maximize the difference between the predicted structures  $\mathcal{M}_V(\mathbf{x}^{adv})$  and reference outputs  $(\mathcal{M}_A(\mathbf{x}^{adv}), \mathcal{M}_B(\mathbf{x}^{adv}))$ , while minimizing the difference between  $\mathcal{M}_A(\mathbf{x}^{adv})$  and  $\mathcal{M}_B(\mathbf{x}^{adv})$ . Formally, a scoring function can be formulated as
$$
\begin{array}{l} r (\mathbf {x}) = s i m \left(\mathcal {M} _ {A} (\mathbf {x}), \mathcal {M} _ {B} (\mathbf {x})\right) \\ + (1 - \operatorname {s i m} (\mathcal {M} _ {V} (\mathbf {x}), \mathcal {M} _ {A} (\mathbf {x}))) \tag {1} \\ + \left(1 - s i m \left(\mathcal {M} _ {V} (\mathbf {x}), \mathcal {M} _ {B} (\mathbf {x})\right)\right) \\ \end{array}
$$
where  $\operatorname{sim}(\cdot, \cdot) \in [0,1]$  is a similarity function, e.g., Directed Dependency Accuracy (DDA) that evaluates the similarity between two parse trees.
Fluency. We use the perplexity of a PLM to evaluate the grammatical correctness and fluency of the generated sentences following Holtzman et al. (2018); Xu et al. (2018); Pang et al. (2020). For a single sentence  $\mathbf{x}$ , the Perplexity score (PPL) can be computed as
$$
f (\mathbf {x}) = P P L (\mathbf {x}) = P (\mathbf {x}) ^ {- \frac {1}{N}} \tag {2}
$$
where  $N$  denotes the sequence length. A lower perplexity indicates that the sentence is more natural and grammatically correct.
Meaning Preservation. Note that the previous two scores neglect the original sentence  $\mathbf{x}^{ori}$  when attacking, which will commonly result in a "zombie" output, i.e., no matter what the input sentence is, the attack always produces exactly the same adversarial sentence. We maintain the diversity of generated sentences by using a score function to ensure the consistency of meanings between the generated sentences and the original sentences. We use BERTSCORE (Zhang et al., 2019b) to evaluate the similarity, which matches each token in  $\mathbf{x}$  to a token in  $\mathbf{x}^{ori}$  to compute recall, and each token in  $\mathbf{x}^{ori}$  to a token in  $\mathbf{x}$  to compute precision, finally combines precision and recall to compute an F1 measure.
$$
s (\mathbf {x}, \mathbf {x} ^ {\text {o r i}}) = \operatorname {B E R T S C O R E} (\mathbf {x}, \mathbf {x} ^ {\text {o r i}}) \tag {3}
$$
Such metric correlates better with human judgment than traditional measures such as BLEU (Papineni et al., 2002).
Objective Function. Taking together, the objective of our adversarial attack can be defined as a non-negative function:
$$
\mathcal {F} (\mathbf {x}) = \frac {r (\mathbf {x}) \cdot s (\mathbf {x} , \mathbf {x} ^ {\text {o r i}})}{f (\mathbf {x})} \tag {4}
$$
By maximizing  $\mathcal{F}(\mathbf{x})$ , we hope to produce  $\mathbf{x}^{adv}$  that are natural-sounding, human-imperceptible, and effective in attacking the victim model.
3.2 Optimization-based Search
Our optimization problem can be considered as a  $T$ -step sequential decision-making process with its state changes along  $\{\mathbf{x}_0, \mathbf{x}_1, \dots, \mathbf{x}_{T-1}\}$  and  $\mathbf{x}_0 = \mathbf{x}^{ori}$ . At step  $t$ ,  $\mathbf{x}_t$  moves to  $\mathbf{x}_{t+1}$  with respect to
$$
\begin{array}{l} \mathbf {x} _ {t + 1} = a _ {t} ^ {*} \left(w _ {t} ^ {*}, \mathbf {x} _ {t}, w _ {c}\right) \\ \left(a _ {t} ^ {*}, w _ {t} ^ {*}\right) = \arg \max  \mathcal {F} \left(a _ {t} \left(w _ {t}, \mathbf {x} _ {t}, w _ {c}\right)\right) \tag {5} \\ a _ {t} \in \mathcal {A}, w _ {t} \in \mathbf {x} _ {t} \\ \end{array}
$$
where  $w_{t}$  is a selected word in  $\mathbf{x}_t$ , and  $a_{t}$  is a word-level manipulation such that  $\mathbf{x}_{t + 1} = a_t(w_t,\mathbf{x}_t,w_c)$ . In this work, we consider three different manipulations  $\mathcal{A} = \{\text{Replace},\text{Insert},\text{Delete}\}$ :  $\text{Replace}(w^{*},\mathbf{x},w_{c})$  indicates replacing the word  $w^{*}$  in sentence  $\mathbf{x}$  with another word  $w_{c}\in \mathcal{W}$ ;  $\text{Insert}(w^{*},\mathbf{x},w_{c})$  inserts a word  $w_{c}$  after  $w^{*}$ ;  $\text{Delete}(w^{*},\mathbf{x},\text{null})$  simply removes  $w^{*}$  from  $\mathbf{x}$ , null means we do not need another word.
Pruning  $\mathcal{W}$ . Exploring the entire vocabulary set at each step for Replace and Insert will be time-consuming. Therefore, we prune the search space with a pretrained masked language model (PLM), e.g., RoBERTa (Liu et al., 2019). Specifically, the expected position for  $w_{c}$  in sentence  $\mathbf{x}_t$  is replaced with a mask token [MASK] and will be predicted using RoBERTa. Then  $w_{c}$  will be selected from the pruned word set  $\mathcal{W}$  that includes the top  $N_w = 50$  predictions in the masked position according to  $P_{RoBERTa}(w[\text{MASK}]\mid \mathbf{x})$ .
We consider the following three strategies of exploring the search space: Beam Search (BS), Metropolis-Hastings Sampling (MHS) and Hybrid Search (HS).
Beam Search Traditional beam search creates the beam by exhaustively searching all candidates created with one manipulation and one word from  $\mathbf{x}_t$ . Then the top  $k$  (beam size) candidates that maximize  $\mathcal{F}(\mathbf{x})$  are selected and stored in the beam in a greedy manner, each of which will be considered as the input sentence for next step. However, for each sentence, there are a huge number of possible candidates, i.e., different positions for manipulation and different words for replacement or insertion. To reduce the time complexity, at each step, we sample a single type of manipulation  $a \in \mathcal{A}$  and a single word  $w^* \in \mathbf{x}_t$  on which the manipulation is performed. Therefore, the time complexity for a single step is reduced to  $\mathcal{O}(k|\mathcal{W}|)$ . Due to the nature of local optimization of BS, we consider such strategy as exploitation.
Metropolis-Hastings Sampling To get out of the local optima that BS is commonly stuck in, we use a sampling-based approach - Metropolis-Hastings Sampling (Metropolis et al., 1953; Hastings, 1970; Chib and Greenberg, 1995) - to explore the space beyond the starting local optimum and increase the chance of finding a global optima. We consider this strategy as exploration.
Specifically, we can create a proposal  $\mathbf{x}_{t + 1}$  by sampling an action  $a_{t}$ , a selected word  $w\in \mathbf{x}_t$  and a new word  $w_{c}\in \mathcal{W}$ . Given the stationary distribution defined as
$$
\pi (\mathbf {x}) \propto \mathcal {F} (\mathbf {x}), \tag {6}
$$
MHS accepts the proposal with the following rate:
$$
\alpha \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right) = \min  \left\{1, \frac {\pi \left(\mathbf {x} _ {t + 1}\right) T \left(\mathbf {x} _ {t} \mid \mathbf {x} _ {t + 1}\right)}{\pi \left(\mathbf {x} _ {t}\right) T \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right)} \right\}, \tag {7}
$$
$$
\begin{array}{l} T (\mathbf {x} _ {t + 1} | \mathbf {x} _ {t}) = r _ {r} T _ {r} (\mathbf {x} _ {t + 1} | \mathbf {x} _ {t}) + r _ {i} T _ {i} (\mathbf {x} _ {t + 1} | \mathbf {x} _ {t}) \\ + r _ {d} T _ {d} \left(\mathbf {x} _ {t + 1} \mid \mathbf {x} _ {t}\right), \tag {8} \\ \end{array}
$$
where  $T$  defines the overall transition distribution,  $r_r$ ,  $r_i$  and  $r_d$  are transition ratios,  $T_r(\cdot)$ ,  $T_i(\cdot)$  and  $T_d(\cdot)$  are transition likelihoods of Replace, Insert and Delete, respectively. The transition likelihoods can be calculated as:
$$
\begin{array}{l} T _ {r} (\mathbf {x} ^ {\prime} | \mathbf {x}) = \mathbb {1} (w _ {c} \in \mathcal {W}) \cdot \frac {P (R e p l a c e (w ^ {*} , \mathbf {x} , w _ {c}))}{\sum_ {w \in \mathcal {W}} P (R e p l a c e (w ^ {*} , \mathbf {x} , w))} \\ T _ {i} (\mathbf {x} ^ {\prime} | \mathbf {x}) = \mathbb {1} (w _ {c} \in \mathcal {W}) \cdot \frac {P (I n s e r t (w ^ {*} , \mathbf {x} , w _ {c}))}{\sum_ {w \in \mathcal {W}} P (I n s e r t (w ^ {*} , \mathbf {x} , w))} \\ T _ {d} (\mathbf {x} ^ {\prime} | \mathbf {x}) = \left\{ \begin{array}{l l} 1, & \text {i f} \mathbf {x} ^ {\prime} = D e l e t e (w ^ {*}, \mathbf {x}) \\ 0, & \text {o t h e r w i s e} \end{array} \right. \\ \end{array}
$$
where  $P(Replace(w^{*},\mathbf{x},w_{c}))$  and  $P(\text{Insert}(w^{*},\mathbf{x},w_{c}))$  are the probabilities of the sentence  $(w_{0},w_{1},\dots,w_{c},\dots,w_{N - 1})$  and the sentence  $(w_{0},\dots w^{*},w_{c},\dots,w_{N - 1})$  given by a pre-trained language model e.g., RoBERTa, respectively.
Hybrid Search The intuition behind HS stems from the balance between exploration and exploitation. Studies have shown that with a proper balance between exploration and exploitation, the optimization performance can be substantially improved (Črepinšek et al., 2013; Wilson et al., 2021). Specifically, at each step, we randomly select between BS and MHS.
$$
H S = \left\{ \begin{array}{l l} B S & \text {r a n d} <   i / n \\ M H S & \text {r a n d} \geqslant i / n \end{array} , \right. \tag {9}
$$
where  $i$  is the current step number,  $n$  is the maximum number of search steps, and rand is uniformly sampled from  $[0, r]$ ,  $r \in [0, 1]$  controls the exploitation-exploration trade-off. It can be seen that with the increase of  $i$ , HS gradually changes from exploration (MHS) to exploitation (BS). After  $r * n$  step, only BS is used.
3.3 Adversarial Training
Following Goodfellow et al. (2015); Madry et al. (2017), we use adversarial training to resist attacks. More specifically, we defend against attacks and increase model robustness by retraining the model with a mix of adversarial examples and the original data. We choose those adversarial examples where the two reference models  $\mathcal{M}_A$  and  $\mathcal{M}_B$  have the same predictions and take the same predictions as the target outputs.
The PLM used to select candidate word set is RoBERTa (Liu et al., 2019). We search hyperparameters on 300 sentences randomly sampled from the PTB development dataset. The criterion used to select all the hyper-parameters is the token level attacking success rate.
Automatic Evaluation Following Han et al. (2020), we evaluate the adversarial examples on two aspects: generation quality (including fluency and meaning preservation) and attacking efficiency. Specifically, we treat outputs from reference model  $\mathcal{A}$ , reference model  $\mathcal{B}$ , or the agreement part of models  $\mathcal{A} \& \mathcal{B}$  as ground truths and evaluate the following two attacking success rates (ASRs):
- Token-level ASR: the percentage of words in the adversarial examples that are assigned the wrong head without considering the dependence type in dependency parsing or the wrong tag in POS tagging.
- Sentence-level ASR: the percentage of mispredicted examples in the adversarial examples.
Human Evaluation We conduct human evaluation of generation quality (fluency, meaning preservation) and attacking efficiency (token and sentence level). We hire three volunteers with linguistic background to label 50 data (sampled from the PTB test set). For generation quality, we ask the annotators to rate from 1 to 5, the higher, the better. For attacking efficiency, we ask them to manually annotate erroneous outputs in the same way as in automatic evaluation.
4.3 Attack on Dependency Parsing
We apply our approach to the dependency parsing task. We choose the Deep Biaffine parser (Dozat and Manning, 2017) as the victim parser  $\mathcal{P}_V$  and two other SOTA dependency parsers as the reference parsers: StackPTR (Ma et al., 2018) as parser  $\mathcal{P}_A$  and BiST (Kiperwasser and Goldberg, 2016) as parser  $\mathcal{P}_B$ .
The three parsers are all trained with Penn Treebank 3.0 (PTB, Marcus et al. (1994)) dataset following the same hyper-parameters reported in their papers.
Token-level ASR↑
Parser A	Parser B	Parser A & B
MHS	18.7	26.4	10.0
BS	33.6	35.4	31.7
HS	36.6	38.7	32.8
Table 2: Experimental results of three modes: BS, MHS, and HS on PTB development set based on automatic evaluation in the dependency parsing task. ASR: Attacking Success Rate.
Adversarial Attacks To assess our three modes, we random sample 300 samples from the PTB development set. Results are shown in Table 2. Observation shows that HS, taking advantage of both MHS and BS, performs better in attacking effectiveness than single MHS or BS. Therefore, we adopt HS in the following experiments.
Then we summarize automatic evaluation results in Table 3 and human evaluation results in Table 4. Human evaluation is consistent with automatic evaluation: our proposed method significantly outperforms the baseline model at almost all metrics. Particularly, our approach in the "Ours-HS" row demonstrates HS's advantage on the attacking success rate. The attacker from Zheng et al. (2020) uses the black-box setting to attack the same word-based Biaffine model and  $15\%$  words are allowed to be modified. Their method keeps better sentence quality, at the cost of a low ASR. Compared with it, our approach is more than twice better than theirs in ASR and maintains a comparable generation quality. In contrast of the attacker from Zheng et al. (2020) that can only use substitution with the same part of speech tag, our attacker allows more flexible manipulations. A case study is shown in Figure 2, our approach replace But and kept with And and are respectively, and add then. These manipulations lead to a successful attack. Compared with Han et al. (2020), our HS is much more effective in terms of ASR and GQ.
Defense Against Adversarial Attack Attacking the PTB training set, our HS approach can generate about 8000 adversarial examples satisfying  $\mathcal{P}_V(\mathbf{x})\neq \mathcal{P}_A(\mathbf{x}) = \mathcal{P}_B(\mathbf{x})$  . The mixed dataset of adversarial examples and the original training data is used to retrain the victim parser. After adversarial training, the unlabeled attachment score (UAS) of the victim model increases from 95.37 to 95.53. To investigate the significance of the improvement, we perform significance tests on the UAS score.
GQ	Token-level ASR↑	Sentence-level ASR↑
Fluency↓	MP↑	Parser A	Parser B	Parsing A&B	Parser A	Parser B	Parsing A&B
Original	156.02	1	3.9	9.1	1.7	36.7	59.3	21.0
Zheng et al. (2020)	281.99	0.94	7.5	14.5	2.8	52.2	71.4	30.2
Han et al. (2020)	174.16	0.80	14.5	19.9	5.6	78.4	87.7	53.8
Ours-HS	157.41	0.92	35.3	37.8	32.4	98.7	98.8	100.0
Table 3: Experimental results on PTB test set based on automatic evaluation in dependency parsing task. "Original" shows the results of original sentences in the PTB test set. We use the perplexity of GPT-2 (Radford et al., 2019) to evaluate the sentence fluency (Fluency), lower is better. We use the BERTSCORE (Zhang et al., 2019b) to evaluate the meaning preservation (MP). Higher is better. GQ: Generation Quality.
GQ↑	ASR↑
Fluency	MP	Tl.	Sl.
Zheng et al. (2020)	4.57	4.07	15.3	42
Han et al. (2020)	4.07	3.16	23.8	70
Ours-HS	4.21	3.65	38.4	93
Table 4: Experimental results on PTB test set based on human evaluation. TL: Token-level. SL: Sentence-level.
Token-level ASR↓
Parser A	Parser B	Parser A&B
Original	36.6	38.7	32.8
Retrain	32.3	35.0	28.0
Table 5: Token level attacking success rates after adversarial training.
(a) Dependency Parsing
(b) POS Tagging
Figure 2: Adversarial examples of dependency parsing (a) and POS tagging (b) generated by our approach. Red color indicates misprediction.