SlowGuess/ACL_2020_OCR huggingface dataset:
and its target share no word in common. Depending on the purpose of the adversary, adversarial attacks can be divided into two categories: targeted attack and non-targeted attack. The subtree-level attack can be viewed as a targeted attack while the sentence-level attack as a non-targeted one.
A small subtree can be taken as a relatively independent structure. If a parser is robust enough, it should always give the consistent result for a target subtree even when there are some errors in another source subtree that does not overlap with the target one. Therefore, we relax some constraints in the cases of the phrase-level attacks, and allow the words in the source tree to be replaced with any word in the vocabulary if the number of modified words is no more than a given value. With the help of these adversarial examples, we can investigate whether an error in one part of a parse tree may exert long-range influence, and successfully cause cascading errors.
In the black-box setting, we first collect all the subtrees from an input sentence, and then perform trial-and-error testing with every source-target pair. For each pair, we try to modify the source subtree up to  $\kappa$  words (say  $\kappa = 3$ ) by replacing them with other randomly selected words. This process is repeated until a pair is found where the UAS of the target subtree decreases.
In the white-box setting, we can obtain a function as  $\mathcal{F}(x,\theta)$  in Equation (4) for every possible target subtree (excluding its root), and then calculate a score for each source-target pair as follows.
$$
\mathcal {S} \left(x ^ {[ s ]}, x ^ {[ t ]}, \theta\right) = \sum_ {x _ {i} \in x ^ {[ s ]}} \left\| \frac {\partial \mathcal {F} \left(x ^ {[ t ]} , \theta\right)}{\partial e _ {x _ {i}}} \right\| _ {2} \tag {6}
$$
where  $x^{[s]}$  denotes a source subtree, and  $x^{[t]}$  a target one. Such scores can be used to rank the source-target pairs for their potential to deliver a successful attack. Generally, the greater the score is, the more vulnerable the target subtree is to the source one. If we remove the sum from the right hand side of (6), we can obtain the norm of the partial derivative of the function  $\mathcal{F}(x^{[t]},\theta)$  with respect to each word  $x_{i}$  in the source subtree, which helps us to determine which words have higher priority to be changed.
For an input sentence, we successively take one from the list of the source-target pairs in the order of their scores. For each pair, we simultaneously
Figure 4: Adversarial example generation algorithm for dependency parsing in the white-box setting.
replace three words in the source subtree guided by their gradients as Equation (5). More than one word are replaced at each iteration to avoid getting stuck in a local optimum. This two-step procedure is repeated until the parser's prediction changes.
We first describe the target parser as well as its three variants, evaluation dataset, and hyper-parameter settings. We then report the empirical results of the proposed adversarial attacking and training. We also list some adversarial examples generated by our attacking algorithms in Table 5.
5.1 Target Parser and Its Variants
We choose the graph-based dependency parser proposed by Dozat and Manning (2017) as our target model. This well-known parser achieved  $95.7\%$  unlabeled attachment scores (UAS) and  $94.1\%$  labeled attachment scores (LAS) on English PTB dataset and close to state-of-the-art performance on standard treebanks for five other different natural languages (Buchholz and Marsi, 2006).
Specifically, Dozat and Manning (2017) extends bidirectional LSTM-based approach (Kiperwasser and Goldberg, 2016) with biaffine classifiers to predict arcs and labels. They presented two variants of their model: one takes only words as input, and the other takes both the words and their POS tags. Moreover, we use the Stanford POS tagger (Toutanova et al., 2003) to generate the POS tag for each word. In addition to these two, we add a new
tence examples for the experiment. According to our statistics on the English PTB test set,  $35.14\%$  sentences have two such subtrees,  $17.18\%$  have three, and  $8.98\%$  have four or more.
6605
Model	Max%	Word-based	Word + POS	Character-based
UAS	#Word	Succ%	UAS	#Word	Succ%	UAS	#Word	Succ%
Clean	--	95.52	--	--	95.58	--	--	95.73	--	--
Black-box	5%	90.91	0.99	42%	90.87	1.00	41%	91.18	1.09	37%
10%	89.38	1.52	49%	90.20	1.54	43%	88.49	1.99	51%
15%	88.69	2.23	51%	89.86	2.24	44%	85.89	3.08	60%
White-box	5%	87.80	0.60	55%	89.76	0.50	46%	90.37	0.40	37%
10%	83.73	1.50	68%	86.36	1.40	61%	86.58	1.20	54%
15%	80.35	2.40	77%	83.75	2.10	69%	83.25	1.90	64%
Table 1: Results of sentence-level adversarial attacks on a state-of-the-art parser with the English Penn Treebank in both the black-box and white-box settings. "Word-based", "Word + POS", and "Character-based" denote three variants of the model (Dozat and Manning, 2017) with differences in their input forms. "Max%" denotes the maximum percentage of words that are allowed to be modified, "UAS" unlabeled attachment scores, "#Word" the average number of words actually modified, and "Succ%" the success rate in terms of the number of sentences.
Model	Word-based	Word + POS	Character-based
Original	Adv [b]	Adv [w]	Original	Adv [b]	Adv [w]	Original	Adv [b]	Adv [w]
Clean	95.52	95.59	95.16	95.58	95.53	95.05	95.73	95.55	95.34
Attack [b]	88.69	90.03	89.98	89.86	91.86	91.60	85.89	92.93	89.89
Attack [w]	80.35	80.82	88.87	83.75	84.89	90.32	83.25	84.10	86.56
variant that takes characters as inputs, and uses a bidirectional LSTM to generate word representations from the character embeddings.
Table 2: Performance of adversarial training. "Clean" stands for the testing results on the clean data, and "Attack [b]" and "Attack [w]" respectively denote the accuracy under test-time attacks in the black-box ([b]) and white-box ([w]) settings. "Original" and "Adv" denotes the testing and adversarial accuracy of the models without and with the adversarial training.
Model	POS	Word-based	Word + POS
ΔUAS	Succ%	ΔUAS	Succ%
Black-box	JJ	-1.89	23%	-1.13	17%
NN	-2.00	24%	-1.25	20%
RB	-3.13	37%	-2.43	31%
VB	-7.42	48%	-6.17	41%
IN	-11.10	67%	-9.22	62%
White-box	JJ	-4.48	37%	-2.23	25%
NN	-10.53	65%	-8.33	57%
RB	-4.09	40%	-3.14	35%
VB	-13.36	73%	-10.51	63%
IN	-15.58	87%	-13.24	85%
Table 3: The attack success rate and the corresponding changes in UAS by modifying the words with different part-of-speech. "JJ" denotes adjective, "NN" noun, "RB" adverb, "VB" verb, and "IN" preposition.
5.2 Datasets and Hyper-parameter Settings
We evaluate our methods on the English Penn Treebank (PTB), converted into Stanford dependencies using version 3.3.0 of the Stanford dependency converter (de Marneffé et al., 2006). We follow the standard PTB split, using section 2-21 for training, section 22 for development and 23 for testing.
For the target parsing models, we use the same choice of hyperparameters as (Dozat and Manning, 2017): 100-dimensional uncased word embeddings and POS tag vectors; three bi-directional LSTM layers (400 dimensions in each direction); and 500- and 100-dimensional ReLU MLP layers for arc and label predictions respectively. For the character-based variant, we use 100-dimensional character vectors, and 200-dimensional LSTM. The other hyper-parameters were tuned with the PTB 3.3.0 development set by trying only a few different settings. In the following experiments, the maximum size of candidate words  $\psi$  was set to 50, the coefficient  $\alpha$  in Equation (5) to 15, and the maximum number of trials to 40. For each example, we terminate the trials immediately if the drop in UAS is more than  $30\%$  in the white-box setting.
5.3 Results of the Sentence-level Attacks
We now report the empirical studies of adversarial attacks for sentence-level methods. In Table 1, we present both clean accuracy and accuracy under attacks on PTB with the three variants of the parsing model (Dozat and Manning, 2017), where we allow three different,  $5\%$ ,  $10\%$  and  $15\%$  word replacements. A success rate is defined as the number of sentences successfully modified (causing the model to make errors) divided by all the number of sen
4We ask for the copula (linking verbs) to remain the head when its complement is an adjective or noun.
6606
tences to be attempted. The results show that the proposed attacks are effective. With less than two words perturbed on average, our white-box attack can consistently achieve  $>60\%$  success rate.
We also observe that the word-based model is most vulnerable to the adversarial examples among the three variants. Its performance drops  $15.17\%$  in UAS, and  $77\%$  sentence examples admit the adversarial perturbations under the white-box attack with  $15\%$  word replacement. The model taking the words and their POS tags as input ("Word + POS") seems to be more robust against adversarial examples in both settings. One reasonable explanation is that we require the substitute words to have the same part-of-speech as the original ones, and the model can produce more consistent results with the help of the POS tags. The white-box attacks are clearly much more effective than the black-box ones across the three variants of the parsing model and different word replacement rates.
Despite having high success rates, we want to know whether the generated examples are syntactically faithful to and coherent with the original sentences. To evaluate the quality of these adversarial examples, we randomly collect 100 sentences and their adversarial examples each generated in the black-box and white-box settings, and presented them to three human evaluators. The evaluators were asked to examine whether each generated example still preserve the original syntactic structure. We adopted a majority vote for the results, and found that  $80\%$  examples generated in the white-box setting and  $75\%$  in the black-box setting are considered unchanged in their syntactic structures.
The three human evaluators are postgraduate students with at least three years of research experience in syntactic parsing. Those three annotators' pairwise-agreement percentages are  $90\%$ ,  $82\%$ , and  $82\%$  for the adversarial examples generated in the white-box setting, and  $93\%$ ,  $85\%$ ,  $84\%$  for those generated in the black-box setting. Their average Kappa coefficients are  $53.8\%$  (white-box), and  $67.3\%$  (black-box) respectively. In Table 5, we listed five sentences and their adversarial examples generated by our algorithms each in the black-box and white-box settings, which were randomly extracted from the PTB test set.
We would like to know which type of words to modify is most likely to form a successful attack like (Hashemi and Hwa, 2016). In this experiment, we only allowed to replace the words belonging to
one part of speech, and also tried to generate adversarial examples by replacing prepositions, which is forbidden in the above experiments. It can be seen from Table 3 that the following dependencies especially suffer: prepositional, verbal and adverbial phrases. Not surprisingly most of the errors occur with structures which are inherently hard to attach in the dependency parsing.
Model	Word-based	Word + POS
k≥0	k≥1	k≥0	k≥1
Black-box	34.73%	21.72%	19.61%	10.06%
White-box	40.06%	28.66%	25.35%	15.82%
Table 4: The success rate of the phrase-level attacks.
5.4 Results of the Phrase-level Attacks
For the phrase-level attacks, we aim to study whether changes in a source subtree can alter the prediction on another target subtree (see an illustration in Figure 2). We tried two different settings: one asks for the source and target subtrees to be separated by at least one word  $(k \geq 1)$ , and another only requires those two subtrees do not overlap with each other  $(k \geq 0)$ . In the case of  $k \geq 0$ , we can find 1420 sentence examples from the test set, while for  $k \geq 1$ , there are 1340 valid examples that can be used to deliver phrase-level attacks (there are 2416 sentences in total in the PTB test set). Note that all the subtrees should contain 4 to 12 words. For each source-target pair, we allow to modify the source subtree up to 3 words. For some sentences, their adversarial examples can be generated by replacing just one or two words.
The success rate for the phrase-level attacks is defined as the ratio between the number of the sentences where there is at least one source-target subtree pair, such that a modification in the source subtree causes the model to make errors in the target subtree, and the number of the sentences that contain at least one source-target subtree pair, regardless of whether the model is caused to make an error or not. It can be seen from Table 4 that with only three words perturbed, the proposed white-box attack can achieve  $27.47\%$  success rate on average for all the settings. The white-box attacks are again much more effective, and spend less than  $50\%$  of the time to find the most vulnerable pairs than the black-box ones. Like the sentence-level attacks, verbal and prepositional phrases have been shown to be more susceptible to such attacks.
6607
Figure 5: Five adversarial examples each generated by our algorithms in the black-box (top) and white-box (bottom) settings. These adversarial examples were randomly extracted from the test set of the English Penn Treebank (PTB). The original words are highlighted in bold blue font while the substitute words are highlighted in bold green ones. The incorrect arcs (i.e. head-modifier pairs) predicted by the target parser are indicated by dash arrows while the ground truth arcs are indicated by solid arrows.
5.5 Adversarial Training
We also investigated whether our adversarial examples can aid in improving model robustness. We randomly selected  $50\%$  of the training data and generated adversarial examples from them using the algorithms listed in Figure 3 and 4. We merged these adversarial examples with the original training set. Some previous studies show that the models tend to overfit the adversarial examples, and their performance on the clean data will drop if too many adversarial examples are used. Therefore, we used a similar training strategy.
The testing and adversarial performance with and without adversarial training are listed in Table 2. Under all circumstances, adversarial training improved the generalization of the models and made them less vulnerable to the attacks, while suffering little to no loss in on the clean data. For example, 88.69 (column 1, row 2) is the accuracy achieved by the original model on the adversarial examples generated in the black-box setting, 90.03 (column 2, row 2) and 89.98 (column 3, row 2) are the accuracy achieved on the perturbed test data with the test-time adversarial attacks by the models with the adversarial training. It is clear that the robustness of parsing models was improved by the adversarial
training. Furthermore, from the first row of Table 2 these robust models suffer from little to no performance drop on the clean testing data.
In this paper, we study the robustness of neural network-based dependency parsing models. To the best of our knowledge, adversarial examples to syntactic tasks, such as dependency parsing, have not been explored in the literature. We develop the first adversarial attack algorithms for this task to successfully find the blind spots of parsers with high success rates. Furthermore, by applying adversarial training using the proposed attacks, we are able to significantly improve the robustness of dependency parsers without sacrificing their performance on clean data.
The authors would like to thank the anonymous reviewers for their valuable comments. This work was supported by National Key R&D Program of China (No. 2018YFC0830902), Shanghai Municipal Science and Technology Major Project (No. 2018SHZDZX01) and Zhangjiang Lab.
6608
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Samuel Barham and Soheil Feizi. 2019. Interpretable adversarial training for text. Computing Research Repository, arXiv: 1905.12864.
Sabine Buchholz and Erwin Marsi. 2006. CoNLL-X shared task on multilingual dependency parsing. In Proceedings of the International Conference on Computational Natural Language Learning.
Nicholas Carlini and David Wagner. 2017. Towards evaluating the robustness of neural networks. In Proceedings of the IEEE Symposium on Security and Privacy.
Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Minhao Cheng, Jinfeng Yi, Huan Zhang, Pin-Yu Chen, and Cho-Jui Hsieh. 2018. Seq2Sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. Computing Research Repository, arXiv: 1803.01128.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Timothy Dozat and Christopher D. Manning. 2017. Deep biaffine attention for neural dependency parsing. In Proceedings of the International Conference on Learning Representations.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Steffen Eger, Gozde Gul Sahin, Andreas Rucklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding NLP systems. In Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies.
Kevin Eykholt, Ivan Evtimov, Earlence Fernandes, Bo Li, Amir Rahmati, Chaowei Xiao, Atul Prakash, Tadayoshi Kohno, and Dawn Song. 2018. Robust physical-world attacks on deep learning models. In
Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Ji Gao, Jack Lanchantin, Mary Lou Soffa, and Yanjun Qi. 2018. Black-box generation of adversarial text sequences to evade deep learning classifiers. Computing Research Repository, arXiv: 1801.04354.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In Proceedings of the International Conference on Learning Representations.
Homa B. Hashemi and Rebecca Hwa. 2016. An evaluation of parser robustness for ungrammatical sentences. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Yu-Lun Hsieh, Minhao Cheng, Da-Cheng Juan, Wei Wei, Wen-Lian Hsu, and Cho-Jui Hsieh. 2019. On the robustness of self-attentive models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is BERT really robust? a strong baseline for natural language attack on text classification and entailment. Computing Research Repository, arXiv: 1907.11932.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313-327.
Volodymyr Kuleshov, Shantanu Thakoor, Tingfung Lau, and Stefano Ermon. 2018. Adversarial examples for natural language classification problems. OpenReview Submission, id: r1QZ3zbAZ.
Qi Lei, Lingfei Wu, Pin-Yu Chen, Alexandros G. Dimakis, Inderjit S. Dhillon, and Michael Witbrock. 2019. Discrete adversarial attacks and submodular optimization with applications to text classification. In Proceedings of the Conference on Systems and Machine Learning.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of the International Joint Conference on Artificial Intelligence.
Marie-Catherine de Marneffe, Bill MacCartney, and Christopher D. Manning. 2006. Generating typed dependency parses from phrase structure parses. In Proceedings of the International Conference on Language Resources and Evaluation.
Ryan McDonald, Koby Crammer, and Fernando Pereira. 2005. Online large-margin training of dependency parsers. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Seyed-Mohsen Moosavi-Dezfooli, Alhussein Fawzi, and Pascal Frossard. 2016. DeepFool: a simple and accurate method to fool deep neural networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition.
Dominick Ng and James R. Curran. 2015. Identifying cascading errors using constraints in dependency parsing. In Proceedings of the Annual Meeting of the Association for Computational Linguistics and the International Joint Conference on Natural Language Processing.
Nicolas Papernot, Patrick McDaniel, Somesh Jha, Matt Fredrikson, Z. Berkay Celik, and Ananthram Swami. 2016a. The limitations of deep learning in adversarial settings. In Proceedings of the IEEE European Symposium on Security and Privacy.
Nicolas Papernot, Patrick McDaniel, Xi Wu, Somesh Jha, and Ananthram Swami. 2016b. Distillation as a defense to adversarial perturbations against deep neural networks. In Proceedings of the IEEE Symposium on Security and Privacy.
Jeffrey Pennington, Richard Socher, and Christopher Manning. 2014. GloVe: Global vectors for word representation. In Proceedings of the Conference on Empirical Methods in Natural Language Processing.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the Annual Meeting of the Association for Computational Linguistics.
Suranjana Samanta and Sameep Mehta. 2017. Towards crafting text adversarial samples. Computing Research Repository, arXiv: 1707.02812.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian Goodfellow, and Rob Fergus. 2013. Intriguing properties of neural networks. Computing Research Repository, arXiv: 1312.6199.
Kristina Toutanova, Dan Klein, Christopher D. Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the Annual Conference of the North American Chapter of the Association for Computational Linguistics.
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh. 2019. Universal adversarial triggers for attacking and analyzing NLP. In Proceedings of the Conference on Empirical Methods in Natural Language Processing and the International Joint Conference on Natural Language Processing.