SlowGuess/NAACL_2022_OCR huggingface dataset:
We calculate the p-value using the one-tailed sign test with the bootstrap re-sampling from the PTB test set following Chollampatt et al. (2019). We compare the retrained model with the original victim model. The p-values is  $1.61e - 5$  that shows the significance. To test the adversarial robustness, we
use our HS approach to attack the retrained model on 300 randomly sampled data from the PTB development set. As shown in Table 5, adversarial training significantly reduces the token level attacking success rate on all three settings.
4.4 Attack on POS Tagging
We apply our approach to the POS tagging task. We use the tagger from Ma and Hovy (2016) as our victim tagger  $\mathcal{T}_V$ , and we choose two state-of-the-art taggers: Stanford POS tagger (Toutanova et al., 2003) and Senna tagger (Collobert et al., 2011) as our reference tagger  $\mathcal{T}_A$  and tagger  $\mathcal{T}_B$ , respectively. We conduct experiments on the PTB dataset. All the hyper-parameters of the three taggers are the same as reported in their papers.
4.4.1 Main Results
Adversarial Attacks As in the experiments of the dependency parsing task, we first randomly sample 300 samples from the PTB development set to compare our three search methods. Results are shown in Table 7. We can find that the HS still performs the best in the attacking success rate. But it performs a relative more minor advantage in this task than its performance in the dependency parsing task. One possible reason is that, compared to dependency parsing, POS tagging is a simpler task, so BS is effective while MHS (exploration with more randomness) can not bring more benefit. We still adopt HS in the following experiments as the experiments on dependency parsing.
We show the automatic evaluation results on the test set in Table 6 and human evaluation results on the sampled test set in Table 8. Fluency, meaning preservation and attacking success rate of our approach are all above Han et al. (2020). Our approach shows its high efficiency even though in the relatively simple task. Particularly, our approach improves the token level attacking success rate by  $8.4\%$ , and on the sentence level, our ap
GQ	Token-level ASR↑	Sentence-level ASR↑
Fluency↓	MP↑	Tagger A	Tagger B	Tagger A&B	Tagger A	Tagger B	Taggers A&B
Original	156.02	1	2.7	3.4	0.7	38.8	45.8	12.9
Han et al. (2020)	142.59	0.88	13.8	9.5	2.3	76.8	74.6	23.2
Ours-HS	136.16	0.93	12.8	14.2	10.7	96.3	97.0	93.4
Table 6: Experimental results on PTB test set in the POS tagging task based on automatic evaluation.
Token-level ASR↑
Tagger A	Tagger B	Taggers A & B
MHS	5.8	11.2	2.9
BS	14.8	15.9	12.8
HS	15.1	16.3	13.1
Table 7: Experimental results of three modes in the POS tagging task: BS, MHS, and HS on PTB dev set based on automatic evaluation.
GQ↑	ASR↑
Fluency	MP	Tl.	Sl.
Han et al. (2020)	3.88	3.16	8.10	52.0
Ours-HS	4.20	3.59	10.43	88.67
Table 8: Experimental results on PTB test set in the POS tagging task based on human evaluation. TL: Token-level. SL: Sentence-level.
proach shows its powerful attack capability, achieving more than  $70\%$  improvement compared to previous baseline, even though the approach of Han et al. (2020) involves training a generating model and is much more computationally expensive than ours. Case studies are shown in Figure 2(b).
Defense Against Adversarial Attack We also conduct adversarial training in the POS tagging task. To compare with Han et al. (2020), we sample 1000 additional adversarial sentences generated by attacking the PTB training set using our HS approach. We mixed these sentences with the initial training set to retrain the victim tagger. As a result, the accuracy of the tagger improves 0.21 from 97.55 to 97.76 on the PTB test set, while Han et al. (2020) reports a 0.13 improvement in the same setting. That demonstrates the high quality and effectiveness of our generated adversarial sentences.
We conduct our analytical experiments in the dependency parsing task.
Impact of Metric for Reference Models Due to the sensitivity to small perturbations as illustrated in Table 1, the structured-prediction attackers require an automated yet unbiased evaluation scheme that is suitable for assessing the prediction of the adversarial examples. For this purpose, we adopt the agreement of two pre-trained parsers. The criterion
Figure 3: Illustration of  $OIoU$ . Parser A and Parser A' denote the same architectures trained with different random seeds. The numbers in the set denote the mispredicted token numbers.
for selecting these reference parsers is that they should be diverse besides having a high parsing accuracy (Han et al., 2020). Therefore, we propose the metric Opposite Intersection over Union (OIoU) to evaluate the diversity degree of reference parsers. In particular, we propose OIoU as one minus the number of common erroneous predictions from two reference parsers divided by the total number of unique erroneous predictions made by two reference parsers. A quantitative example is demonstrated in Figure 3. The diverse degrees of the parsers on the left and right are 0.631  $(1 - 83 / (26 + 83 + 116))$  and 0.285 respectively. Our experiments find that the parsers on the left with a higher OIoU indeed result in better attacking efficiency than those on the right (12.9% vs. 5.2% token-level ASR). Note that, for fair comparison of SHARP with prior work, we reuse the same reference parsers for all the experiments.
Generalizability We exchange the victim parser and reference parser to show the generalization of our black-box adversarial attack methods. Specifically, we take the Deep Biaffine parser (Dozat and Manning, 2017) as the reference parser  $\mathcal{P}_A$  and the StackPTR (Ma et al., 2018) as the victim parser  $\mathcal{P}_V$ . We repeat the experiments of dependency parsing keeping the same setup of Table 3 except for the parser choice. Experimental results based on automatic evaluation on the test set are shown in Table 9. We can find that our approach still can keep high attacking success rate both on token level and sentence level. Moreover, the fluency of the generated adversarial sentences of Han et al. (2020) becomes worse after changing victim parser, but our approach well maintains the quality of the sentences.
GQ	Token-level ASR↑	Sentence-level ASR↑
Fluency↓	MP↑	Parser A	Parser B	Parsing A&B	Parser A	Parser B	Parsing A&B
Original	156.02	1.00	3.7	8.5	1.5	36.7	57.2	9.2
Han et al. (2020)	244.69	0.92	19.6	23.3	13.4	70.8	77.2	24.3
Ours-HS	142.78	0.94	33.6	36.8	30.8	98.1	98.2	92.4
Table 9: Experimental results on PTB test set in the dependency parsing task based on automatic evaluation with StackPTR as the victim model while the Deep Biaffine parser and BiST as reference models.
GQ	Token-level ASR↑	Sentence-level ASR↑
Fluency↓	MP↑	Parser A	Parser B	Parsing A&B	Parser A	Parser B	Parsing A&B
Ours-HS	188.01	0.92	35.8	39.0	32.8	98.6	98.6	95.6
-s(x, xori)	220.35	0.91	36.2	38.9	33.0	99.3	99.7	97.7
-f(x)	672.54	0.92	44.9	46.9	41.2	99.7	100	100
-r(x)	67.00	0.93	6.72	13.75	3.0	56.8	77.1	24.4
random	919.85	0.88	16.3	25.2	7.9	90.4	93.8	45.5
Table 10: Ablation study on 300 samples of PTB development set in the dependency parsing task based on automatic evaluation.
Ablation Study We show the impact of the three different scores in our objective function. Table 10 shows the automatic evaluation results of generation quality and attacking success rate on 300 samples randomly sampled from the development set. It can be seen that without considering the fluency of generated sentences (Row  $-f(\mathbf{x})$  in Table 10), it is natural that the attacking success rate can be further increased, but the generation quality becomes worse. Without considering meaning reservation between original sentences and generated sentences (Row  $-s(\mathbf{x},\mathbf{x}^{ori})$ ), we can find that  $MP$  drops marginally. To verify the importance of meaning preservation (Row  $s(\mathbf{x},\mathbf{x}^{ori})$ ), we experiment in the same setting as Section 4.3.1 except for using the adversarial examples generated without considering meaning preservation. We find that the unlabeled attachment score (UAS) of the victim model is 94.57 (vs 95.53), which shows that the quality of generated sentences is important to improve the victim model in terms of performance and robustness after retraining on adversarial examples. The importance is also demonstrated by prior work of Wang et al. (2021). Without optimizing  $r(\mathbf{x})$ , we can see that the attack success rate is even lower than random sampling because it needs to promise the quality of sampled sentences. A big gap on both generated quality and attacking success rate between random sampling and our HS approach demonstrates that the strength of our methods.
Impact of Candidate Size and Manipulation
Count Figure 4 and 5 show the result on different candidate sizes and manipulation counts, respectively. We can find that, with the increase of the
candidate size, the attacking success rate also increases, but the growth rate gradually slows down. The manipulation count shows a similar trend.
In our setting, the computing cost has a linear relationship with these two hyper-parameters. Thus we have a trade-off between time and performance: in our experiments, we set the candidate size and manipulation count to 5 and 50, respectively.
Impact of Beam Size We investigate the impact of the beam size under the same computing resources. For example, when the beam size is 1, the candidate number is 10; and when the beam size is 2, the candidate number is 5. The manipulation count keeps the same. The results are shown on Figure 6. It can be seen that 2 is the best.
Causal Analysis of Adversarial Attacks Since SHARP searches the whole sentence space for adversarial attacks without pre-defined templates, the generated adversarial examples have the potential for unseen discoveries. Therefore, we analyze the adversarial examples and conclude some new templates that cause mispredictions of the victim model. We list two observations here. First, uncommon words (e.g., replacing the place name with an adverb or adding a surname) often cause a misprediction. For example, the adversarial example "What is Santa actually worth?" (the original sentence is "What is Santa Fe worth?") makes the victim model mispredict the head of "Santa" being "actually". A probable reason is that interrogative sentences with uncommon words are rare in the training set and the victim model is confused with these out-vocabulary words. The other observation is that symbols such as  $\%$  and  $:$  of
Figure 4: Token-level attacking success rate on different candidate sizes.
Figure 5: Token-level attacking success rate on different manipulation counts.
Figure 6: Token-level attacking success rate on different beam size.
ten mislead the model predictions. Surprisingly, the victim model can predict the correct structure for the sentence "It is widely expected that they will." but fails when the period in this sentence is dropped. We give a simple quantitative analysis of these adversarial types. For the first type (adding a surname to an interrogative sentence), we randomly choose 100 names and write the sentence "What does Name Surname like?" We get  $100\%$  sentence-level attack success. For the second type (adding an adverb after  $\%$  ), we randomly select 100 sentences which contain  $\%$  from the original dataset, and then add adverb "globally" after the  $\%$  to generate new sentences as adversarial samples. We get about  $50\%$  sentence-level attack success in this case.
There is limited literature available for adversarial attacking on structured prediction tasks. Previous adversarial training has been conducted on NLP tasks such as text classification (Liang et al., 2018; Alzantot et al., 2018), machine translation (Zhao et al., 2018; Ebrahimi et al., 2018; Cheng et al., 2020a) and dialogue systems (Cheng et al., 2019a). Recently, adversarial training has also been explored on structured prediction tasks, such as dependency parsing (Zheng et al., 2020; Han et al., 2020). Zheng et al. (2020) replaced some words with adversarily chosen counterparts with the same part of speech tags. They target specific syntactic adversarial sentence examples to attack dependency parser. Han et al. (2020) investigated generation-based attackers for structured prediction tasks.
In this paper, we quantitatively investigate the sensitivity of structured prediction tasks and formulate
the black-box adversarial attack as a search problem that seeks to maximize a specially designed objective function. Both automatic and human evaluations show that our approach beats the previous approaches by a large margin in attacking victim models and simultaneously guarantees better fluency and meaning preservation. Our defense experiments show that the adversarial samples generated by our approach can be used to improve the original model's robustness and performance.
Liwen Zhang, Zixia Jia and Kewei Tu were supported by the National Natural Science Foundation of China (61976139). Wenjuan Han and Zilong Zheng were supported by the National Key R&D Program of China (2021ZD0150200).
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890–2896, Brussels, Belgium. Association for Computational Linguistics.
Minhao Cheng, Wei Wei, and Cho-Jui Hsieh. 2019a. Evaluating and enhancing the robustness of dialogue systems: A case study on a negotiation agent. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3325-3335.
Minhao Cheng, Jinfeng Yi, Pin-Yu Chen, Huan Zhang, and Cho-Jui Hsieh. 2020a. Seq2sick: Evaluating the robustness of sequence-to-sequence models with adversarial examples. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 3601-3608.
Yong Cheng, Lu Jiang, and Wolfgang Macherey. 2019b. Robust neural machine translation with doubly adversarial inputs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4324-4333, Florence, Italy. Association for Computational Linguistics.
Yong Cheng, Lu Jiang, Wolfgang Macherey, and Jacob Eisenstein. 2020b. AdvAug: Robust adversarial augmentation for neural machine translation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5961-5970, Online. Association for Computational Linguistics.
Siddhartha Chib and Edward Greenberg. 1995. Understanding the metropolis-hastings algorithm. The american statistician, 49(4):327-335.
Shamil Chollampatt, Weiqi Wang, and Hwee Tou Ng. 2019. Cross-sentence grammatical error correction. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 435-445, Florence, Italy. Association for Computational Linguistics.
Ronan Collobert, Jason Weston, Léon Bottou, Michael Karlen, Koray Kavukcuoglu, and Pavel Kuksa. 2011. Natural language processing (almost) from scratch. Journal of machine learning research, 12(ArtICLE):2493-2537.
Matej Črepinšek, Shih-Hsi Liu, and Marjan Mernik. 2013. Exploration and exploitation in evolutionary algorithms: A survey. ACM computing surveys (CSUR), 45(3):1-33.
Timothy Dozat and Christopher D Manning. 2017. Deep biaffine attention for neural dependency parsing. 5th International Conference on Learning Representations (ICLR).
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Ian J Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. International Conference on Learning Representations (ICLR).
Wenjuan Han, Liwen Zhang, Yong Jiang, and Kewei Tu. 2020. Adversarial attack and defense of structured prediction models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2327-2338, Online. Association for Computational Linguistics.
W Keith Hastings. 1970. Monte carlo sampling methods using markov chains and their applications.
Ari Holtzman, Jan Buys, Maxwell Forbes, Antoine Bosselut, David Golub, and Yejin Choi. 2018. Learning to write with cooperative discriminators. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1638-1649.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031.
Eliyahu Kiperwasser and Yoav Goldberg. 2016. Simple and accurate dependency parsing using bidirectional LSTM feature representations. Transactions of the Association for Computational Linguistics, 4:313-327.
Bin Liang, Hongcheng Li, Miaoqiang Su, Pan Bian, Xirong Li, and Wenchang Shi. 2018. Deep text classification can be fooled. In Proceedings of the 27th International Joint Conference on Artificial Intelligence, IJCAI'18, page 4208-4215. AAAI Press.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Xuezhe Ma and Eduard Hovy. 2016. End-to-end sequence labeling via bi-directional LSTM-cnns-crf. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1064-1074.
Xuezhe Ma, Zeong Hu, Jingzhou Liu, Nanyun Peng, Graham Neubig, and Eduard Hovy. 2018. Stackpointer networks for dependency parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1403-1414.
Aleksander Madry, Aleksandar Makelov, Ludwig Schmidt, Dimitris Tsipras, and Adrian Vladu. 2017. Towards deep learning models resistant to adversarial attacks. arXiv preprint arXiv:1706.06083.
Mitchell Marcus, Grace Kim, Mary Ann Marcinkiewicz, Robert MacIntyre, Ann Bies, Mark Ferguson, Karen Katz, and Britta Schasberger. 1994. The penn treebank: annotating predicate argument structure. In Proceedings of the workshop on Human Language Technology, pages 114-119. Association for Computational Linguistics.
Nicholas Metropolis, Arianna W Rosenbluth, Marshall N Rosenbluth, Augusta H Teller, and Edward Teller. 1953. Equation of state calculations by fast computing machines. The journal of chemical physics, 21(6):1087-1092.
Bo Pang, Erik Nijkamp, Wenjuan Han, Linqi Zhou, Yixian Liu, and Kewei Tu. 2020. Towards holistic and automatic evaluation of open-domain dialogue
generation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3619-3629.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting on association for computational linguistics, pages 311-318. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Blog, 1(8).
Shuhuai Ren, Yihe Deng, Kun He, and Wanxiang Che. 2019. Generating natural language adversarial examples through probability weighted word saliency. In Proceedings of the 57th annual meeting of the association for computational linguistics, pages 1085-1097.
Kristina Toutanova, Dan Klein, Christopher D Manning, and Yoram Singer. 2003. Feature-rich part-of-speech tagging with a cyclic dependency network. In Proceedings of the 2003 Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics, pages 252-259.
Dilin Wang, Chengyue Gong, and Qiang Liu. 2019. Improving neural language modeling via adversarial training. In International Conference on Machine Learning, pages 6555-6565.
Yuxuan Wang, Wanxiang Che, Ivan Titov, Shay B Cohen, Zhilin Lei, and Ting Liu. 2021. A closer look into the robustness of neural dependency parsers using better adversarial examples. In *Findings of the Association for Computational Linguistics: ACLIJCNLP* 2021, pages 2344-2354.
Robert C Wilson, Elizabeth Bonawitz, Vincent D Costa, and R Becket Ebitz. 2021. Balancing exploration and exploitation with information and randomization. Current Opinion in Behavioral Sciences, 38:49-56.
Jingjing Xu, Xuancheng Ren, Junyang Lin, and Xu Sun. 2018. Dp-gan: diversity-promoting generative adversarial network for generating informative and diversified text. arXiv preprint arXiv:1802.01345.
Huangzhao Zhang, Hao Zhou, Ning Miao, and Lei Li. 2019a. Generating fluent adversarial examples for natural languages. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5564-5569, Florence, Italy. Association for Computational Linguistics.
Tianyi Zhang, Varsha Kishore, Felix Wu, Kilian Q Weinberger, and Yoav Artzi. 2019b. Bertscore: Evaluating text generation with bert. arXiv preprint arXiv:1904.09675.
Zhengli Zhao, Dheeru Dua, and Sameer Singh. 2018. Generating natural adversarial examples. In International Conference on Learning Representations.
Xiaoqing Zheng, Jiehang Zeng, Yi Zhou, Cho-Jui Hsieh, Minhao Cheng, and Xuanjing Huang. 2020. Evaluating and enhancing the robustness of neural network-based dependency parsing models with adversarial examples. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 6600-6610, Online. Association for Computational Linguistics.
A.1 Hyper-Params
The shared hyper-parameters of Beam Search (BS), Metropolis-Hastings Sampling (MHS), and Hybrid Search (HS) are listed in Table 11.
Beam Size	2
Size of Candidate Set	5
Manipulation Count	50
pr	0.5
pi	0.25
pd	0.25
Table 11: Hyper-parameter setting.
In Table 2 and Table 5 of the main body, we use different evaluation metrics to measure results of Parsers A&B compared with Han et al. (2020). Take Token-level ASR as an example, they calculate Parser A&B using  $1 - s / n$ , where  $s$  is the number of tokens of which the three parsers (A, B and C) have the same prediction (i.e.,  $\mathcal{M}_A(x) = \mathcal{M}_B(x) = \mathcal{M}_C(x)$ ), and  $n$  is the total number of tokens. We use  $a / b$ , where  $a$  is the number of tokens that  $\mathcal{M}_A(x) \neq (\mathcal{M}_B(x) = \mathcal{M}_C(x))$ , and  $b$  is the number of tokens that  $\mathcal{M}_B(x) = \mathcal{M}_C(x)$ . Since the assumption behind identifying the ground truth of the structured outputs is that two external reference parsers have the same prediction, it is more reasonable to use  $a / b$  when calculating the attack success rate to identify if the adversarial sentence can indeed fool the victim model. We recalculated all the Parsers A&B results of Han et al. (2020) in the main body of this work. For reference, we give results of our model calculated according to their formula as Table 12. It can be seen that our approach outperforms the approach proposed by Han et al. (2020) even with the original metrics.
Token-level ASR↑
Parser A	Parser B	Parser A & B
Han	13.9	19.2	24.1
our-HS	36.6	38.7	37.6
Table 12: Experimental results on PTB test dataset in dependency parsing task.
Figure 7: Objective score curve on increasing sampling steps.
B Optimizing Process
We show our optimizing curve in the Figure 7. We can see that the objective score is oscillating in the early stages of sampling, because MHS sampling is chosen with high probability in the early stages, which shows the exploration process. After the early stage, we see a clear upward trend of the objective score, where our approach chooses BS with high probability, and BS directly searches the local optimal.