SlowGuess/ACL_2020_OCR huggingface dataset:
We now demonstrate MORPHEUS' ability to craft adversaries for NMT models as well, this time without access to the models' logits. The WMT'14 English-French test set (newstest2014), containing 3,003 sentence pairs, is used for both evaluation and generating adversarial examples. We evaluate our attack on the fairseq implementation of both the Convolutional Seq2Seq (Gehring et al., 2017) and Transformer-big models, and report the BLEU score (Papineni et al., 2002) using fairseq's implementation (Ott et al., 2019).
From our experiments (Table 2), ConvS2S and Transformer-big appear to be extremely brittle even to inflectional perturbations constrained to the same part of speech (56–57% decrease). In addition, some adversarial examples caused the models to regenerate the input verbatim instead of a translation:  $1.4\%$  of the test set for Transformer-big,  $3\%$  for ConvS2S (see Table 9 in the Appendix for some examples). This is likely due to the joint source/target byte-pair encoding (Sennrich et al., 2016) used by both NMT systems to tackle rare word translation.
We experimented with both BLEU and chrF (Popovic, 2015) as our optimizing criterion and achieved comparable results for both, however, MORPHEUS found more adversarial examples that caused the model to output random sentences about Nicolas Sarkozy when optimizing for chrF.
5 Human Evaluation
To test our hypothesis that inflectional perturbations are likely to be relatively natural and semantics preserving, we randomly sample 130 adversar
<sup>6</sup>We use the sacrebleu implementation (Post, 2018).
Plausibility
Native U.S. English Speakers	Unrestricted
SQUAD 2.0	newtest2014	SQUAD 2.0	newtest2014
Native	11.58%	25.64%	22.82%	32.56%
L2 Speaker	42.82%	42.30%	53.58%	52.82%
Beginner	31.79%	23.33%	17.17%	10.25%
Non-human	13.84%	8.71%	6.41%	4.35%
Semantic Equivalence
Native U.S. English Speakers	Unrestricted
SQUAD 2.0	newtest2014	SQUAD 2.0	newtest2014
Highly Likely	52.82%	62.30%	33.84%	40.76%
Likely	20.51%	18.71%	36.15%	33.84%
Somewhat Likely	11.02%	7.94%	22.82%	19.48%
Somewhat Unlikely	6.92%	6.15%	5.38%	4.35%
Unlikely	3.58%	3.07%	1.53%	1.28%
Highly Unlikely	5.12%	1.79%	0.25%	0.25%
Table 4: Human judgements for adversarial examples that caused a significant degradation in performance.
ial examples from each dataset and ask 3 Amazon Mechanical Turk workers to indicate (1) whether the sentences could have been written by a native speaker, L2 speaker, beginner learner, or no human; and (2) the likelihood of the original and adversarial examples sharing the same meaning. To ensure the quality of our results, only Turkers who completed  $>10,000$  HITs with a  $\geq 99\%$  acceptance rate could access our task. For comparison, we also report ratings by native U.S. English speakers, who were selected via a demographic survey and fluency test adapted from Hartshorne et al. (2018). Workers were paid a rate of at least  $\$ 12/\mathrm{hr}$ .
Table 4 shows that Turkers from our unrestricted sample judged  $\sim 95\%$  of our adversaries to be plausibly written by a human and  $92\%$  generally likely to be semantically equivalent to the original examples  $92\%$  of the time, hence validating our hypothesis. Qualitative analysis revealed that "is/are"  $\rightarrow$  "am/been" changes accounted for  $48\%$  of the implausible adversaries.
Discussion. We believe that non-native speakers may tend to rate sentences as more human-like for the following reasons:
Their exposure to another language as a native speaker leads them to accept sentences that mimic errors made by L2 English speakers who share their first language.
Their exposure to the existence of these above-mentioned errors may lead them to be more forgiving of other inflectional errors that are uncommon to them; they may deem these errors as
(a) SQuAD 2.0 dev set
(b) SQuAD 2.0 training set
Figure 2: Comparison of inflectional distributions for SpanBERT $_{\text{SQuAD2}}$ . The adversarial distributions include only examples that degrade model performance. To make the best use of limited space, we omit the RBR, RBS, and NNPS tags since they do not vary much across distributions. Full figures in Appendix D.
plausibly made by an L2 speaker who speaks a different first language from them.
- They do not presume mastery of English, and hence may choose to give the higher score when deciding between 2 choices.
6 Adversarial Fine-tuning
In this section, we extend the standard adversarial training paradigm (Goodfellow et al., 2015) to make the models robust to inflectional perturbations. Since directly running MORPHEUS on the entire training dataset to generate adversaries would be far too time-consuming, we use the findings from our experiments on the respective dev/test sets (Section 4) to create representative samples of good adversaries. This significantly improves robustness to inflectional perturbations while maintaining similar performance on the clean data.
We first present an analysis of the inflectional distributions before elaborating on our method for generating the adversarial training set.
Only adversarial examples that degraded the  $\mathrm{F}_1$  score by  $>50$  and the BLEU score by  $>15$  were considered.
<sup>8</sup>We define a beginner as one who has just started learning the language, and an L2 speaker to be an experienced speaker.
Each task was estimated to take 20-25s to be comfortably completed, but they were routinely completed in under 20s.
SpanBERTSQuAD 2 (F1)
Dataset	Original	Epoch	Adversariably Fine-tuned
Clean	MORPHEUS	Clean	MORPHEUSorig	MORPHEUSadv
SQuAD 2.0 Ans	88.52	69.47 (-21.52%)	1	86.80	85.17 (-1.87%)	82.76 (-4.65%)
4	86.15	84.93 (-1.41%)	82.92 (-3.74%)
SQuAD 2.0 All	87.71	73.26 (-16.47%)	1	86.00	84.72 (-1.48%)	82.41 (-4.17%)
4	87.08	85.93 (-1.32%)	84.71 (-2.72%)
Transformer-big (BLEU)
Dataset	Original	Epoch	Adversariably Fine-tuned
Clean	MORPHEUS	Clean	MORPHEUSorig	MORPHEUSadv
newstest2014	43.16	20.57 (-56.25%)	1	39.84	31.79 (-20.20%)	31.43 (-21.10%)
4	40.60	31.99 (-21.20%)	30.82 (-24.08%)
Table 5: Results from adversarially fine-tuning SpanBERT $_{\text{SQuAD2}}$  and Transformer-big. MORPHEUS $_{\text{orig}}$  refers to the initial adversarial examples, while MORPHEUS $_{\text{adv}}$  refers to the new adversarial examples obtained by running MORPHEUS on the robust model. Relevant results from Table 2 reproduced here for ease of comparison.
6.1 Distributional Analysis
Figure 2a illustrates the overall distributional differences in inflection occurrence between the original and adversarial examples found by MORPHEUS for SQuAD 2.0. Note that these distributions are computed based on the Penn Treebank (PTB) POS tags, which are finer-grained than the universal POS (UPOS) tags used to constrain MORPHEUS' search (Section 4). For example, a UPOS VERB may be actually be a PTB VBD, VBZ, VBG, etc.
We can see obvious differences between the global inflectional distributions of the original datasets and the adversaries found by MORPHEUS. The differences are particularly significant for the NN, NNS, and VBG categories. NNS and VBG also happen to be uncommon in the original distribution. Therefore, we conjecture that the models failed (Section 4) because MORPHEUS is able to find the contexts in the training data where these inflections are uncommon.
6.2 Adversarial Training Set Generation
Since there is an obvious distributional difference between the original and adversarial examples, we hypothesize that bringing the training set's inflectional distribution closer to that of the adversarial examples will improve the models' robustness.
To create the adversarial training set, we first isolate all the adversarial examples (from the dev/test set) that caused any decrease in  $\mathrm{F_1}$ /BLEU score and count the number of times each inflection is used in this adversarial dataset, giving us the inflectional distribution in Figure 2a.
Next, we randomly select an inflection for each
eligible token in each training example, weighting the selection with this inflectional distribution instead of a uniform one. To avoid introducing unnecessary noise into our training data, only inflections from the same UPOS as the original word are chosen. We do this 4 times per training example, resulting in an adversarial training set with a clean-adversarial ratio of  $1:4$ . This can be done in linear time and is highly scalable. Algorithm 2 in Appendix C details our approach and Figure 2b depicts the training set's inflectional distribution before and after this procedure.
Fine-tuning vs. retraining. Existing adversarial training approaches have shown that retraining the model on the augmented training set improves robustness (Belinkov and Bisk, 2018; Eger et al., 2019; Jin et al., 2019). However, this requires substantial compute resources. We show that fine-tuning the pre-trained model for just a single epoch is sufficient to achieve significant robustness to inflectional perturbations yet still maintain good performance on the clean evaluation set (Table 5).
SpanBERT. Following Joshi et al. (2019), we fine-tune  $\mathrm{SpanBERT}_{\mathrm{SQuAD}2}$  for another 4 epochs on our adversarial training set. Table 5 shows the effectiveness of our approach for  $\mathrm{SpanBERT}_{\mathrm{SQuAD}2}$ .
After just a single epoch of fine-tuning,  $\mathrm{SpanBERT}_{\mathrm{SQuAD2}}$  becomes robust to most of the initial adversarial examples with a  $< 2$ -point drop in performance on the clean dev set. More importantly, running MORPHEUS on the robust model fails to significantly degrade its performance.
After 4 epochs, the performance on the clean SQuAD 2.0 dev set is almost equivalent to the original SpanBERT $_{\text{SQuAD}2}$ 's, however this comes at a slight cost: the performance on the answerable questions is slightly lower than before. In fact, if performance on answerable questions is paramount, our results show that fine-tuning on the adversarial training set for 1 epoch would be a better (and more cost effective) decision. Retraining SpanBERT adversarially did not result in better performance.
We also found that weighting the random sampling with the adversarial distribution helped to improve the robust model's performance on the answerable questions (refer to Table 7 in Appendix).
Transformer-big. Similarly, model robustness improves dramatically (56.25% to 20.20% decrease) after fine-tuning for 1 epoch on the adversarial training set with a  $\sim 3$  BLEU point drop in clean data performance (Table 5). Fine-tuning for a further 3 epochs reduced the difference but made the model less robust to new adversarial examples.
We also experimented with using randomly sampled subsets but found that utilizing the entire original training set was necessary for preserving performance on the clean data (see Table 8 in Appendix).
Our anonymous reviewers brought up the possibility of using grammatical error correction (GEC) systems as a defense against inflectional adversaries. Although we agree that adding a GEC model before the actual NLU/translation model would likely help, this would not only require an extra model—often another Transformer (Bryant et al., 2019)—and its training data to be maintained, but would also double the resource usage of the combined system at inference time.
Consequently, institutions with limited resources may choose to sacrifice the experience of minority users rather than incur the extra maintenance costs. Adversarial fine-tuning only requires the NLU/translation model to be fine-tuned once and consumes no extra resources at inference time.
Although we have established our methods' effectiveness at both inducing model failure and robustifying said models, we believe they could be further improved by addressing the following limitations:
1. MORPHEUS finds the distribution of examples that are adversarial for the target model, rather
than that of real L2 speaker errors, which produced some unrealistic adversarial examples.
2. Our method of adversarial fine-tuning is analogous to curing the symptom rather than addressing the root cause since it would have to be performed for each domain-specific dataset the model is trained on.
In future work, we intend to address these limitations by directly modeling the L2 and dialectal distributions and investigating the possibility of robustifying these models further upstream.
Ensuring that NLP technologies are inclusive, in the sense of working for users with diverse linguistic backgrounds (e.g., speakers of World Englishes such as AAVE, as well as L2 speakers), is especially important since natural language user interfaces are becoming increasingly ubiquitous.
We take a step in this direction by revealing the existence of linguistic bias in current English NLP models—e.g., BERT and Transformer—through the use of inflectional adversaries, before using adversarial fine-tuning to significantly reduce it. To find these adversarial examples, we propose MORPHEUS, which crafts plausible and semantically similar adversaries by perturbing an example's inflectional morphology in a constrained fashion, without needing access to the model's gradients. Next, we demonstrate the adversaries' effectiveness using QA and MT, two tasks with direct and wide-ranging applications, before validating their plausibility and semantic content with real humans.
Finally, we show that, instead of retraining the model, fine-tuning it on a representative adversarial training set for a single epoch is sufficient to achieve significant robustness to inflectional adversaries while preserving performance on the clean dataset. We also present a method of generating this adversarial training set in linear time by making use of the adversarial examples' inflectional distribution to perform weighted random sampling.
We would like to express our gratitude to Lav Varshney, Jason Wu, Akhilesh Gotmare, and our anonymous reviewers for their insightful feedback on our paper, and friends who participated in our pilot studies. Samson is supported by Salesforce and the Singapore Economic Development Board under its Industrial Postgraduate Programme.
2928
Eneko Agirre, Daniel Cer, Mona Diab, Aitor Gonzalez-Agirre, and Weiwei Guo. 2013. *SEM 2013 shared task: Semantic textual similarity. In Second Joint Conference on Lexical and Computational Semantics (*SEM), Volume 1: Proceedings of the Main Conference and the Shared Task: Semantic Textual Similarity, pages 32-43, Atlanta, Georgia, USA. Association for Computational Linguistics.
Moustafa Alzantot, Yash Sharma, Ahmed Elgohary, Bo-Jhang Ho, Mani Srivastava, and Kai-Wei Chang. 2018. Generating natural language adversarial examples. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2890-2896, Brussels, Belgium. Association for Computational Linguistics.
Yonatan Belinkov and Yonatan Bisk. 2018. Synthetic and natural noise both break neural machine translation. In 6th International Conference on Learning Representations, Vancouver, BC, Canada.
Steven Bird, Ewan Klein, and Edward Loper. 2009. Natural Language Processing with Python. O'Reilly Media.
Tolga Bolukbasi, Kai-Wei Chang, James Y Zou, Venkatesh Saligrama, and Adam T Kalai. 2016. Man is to computer programmer as woman is to homemaker? debiasing word embeddings. In D. D. Lee, M. Sugiyama, U. V. Luxburg, I. Guyon, and R. Garnett, editors, Advances in Neural Information Processing Systems 29, pages 4349-4357. Curran Associates, Inc.
Shikha Bordia and Samuel R. Bowman. 2019. Identifying and reducing gender bias in word-level language models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Student Research Workshop, pages 7-15, Minneapolis, Minnesota. Association for Computational Linguistics.
Christopher Bryant, Mariano Felice, Øistein E. Andersen, and Ted Briscoe. 2019. The BEA-2019 shared task on grammatical error correction. In Proceedings of the Fourteenth Workshop on Innovative Use of NLP for Building Educational Applications, pages 52–75, Florence, Italy. Association for Computational Linguistics.
David Crystal. 2003. English as a Global Language. Cambridge University Press.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
David M. Eberhard, Gary F. Simons, and Charles D. Fennig, editors. 2019. Ethnologue: Languages of the World, 22 edition. SIL International.
Javid Ebrahimi, Anyi Rao, Daniel Lowd, and Dejing Dou. 2018. HotFlip: White-box adversarial examples for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 31-36, Melbourne, Australia. Association for Computational Linguistics.
Steffen Eger, Gözde Gül Sahin, Andreas Rücklé, Ji-Ung Lee, Claudia Schulz, Mohsen Mesgar, Krishnkant Swarnkar, Edwin Simpson, and Iryna Gurevych. 2019. Text processing like humans do: Visually attacking and shielding nlp systems. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1634-1647, Minneapolis, Minnesota. Association for Computational Linguistics.
Matt Gardner, Joel Grus, Mark Neumann, Oyvind Tafjord, Pradeep Dasigi, Nelson F. Liu, Matthew Peters, Michael Schmitz, and Luke Zettlemoyer. 2018. AllenNLP: A deep semantic natural language processing platform. In Proceedings of Workshop for NLP Open Source Software (NLP-OSS), pages 1-6, Melbourne, Australia. Association for Computational Linguistics.
Jonas Gehring, Michael Auli, David Grangier, Denis Yarats, and Yann N. Dauphin. 2017. Convolutional sequence to sequence learning. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research, pages 1243-1252, International Convention Centre, Sydney, Australia. PMLR.
Ian J. Goodfellow, Jonathon Shlens, and Christian Szegedy. 2015. Explaining and harnessing adversarial examples. In 3rd International Conference on Learning Representations, San Diego, California.
J. Hartshorne, B. Tenenbaum, J., and S. Pinker. 2018. A critical period for second language acquisition: Evidence from 2/3 million english speakers. Cognition, 177:263-277.
Belma Haznedar. 2002. Missing surface inflection in adult and child 12 acquisition. In Proceedings of the 6th Generative Approaches to Second Language Acquisition Conference, pages 140-149, Somerville, Massachusetts. Cascadilla Proceedings Project.
Alex Hern. 2017. Facebook translates 'good morning' into 'attack them', leading to arrest. The Guardian.
Dirk Hovy and Shannon L. Spruit. 2016. The social impact of natural language processing. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 591-598, Berlin, Germany. Association for Computational Linguistics.
Mohit Iyyer, John Wieting, Kevin Gimpel, and Luke Zettlemoyer. 2018. Adversarial example generation with syntactically controlled paraphrase networks. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1875-1885, New Orleans, Louisiana. Association for Computational Linguistics.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Di Jin, Zhijing Jin, Joey Tianyi Zhou, and Peter Szolovits. 2019. Is bert really robust? natural language attack on text classification and entailment. arXiv e-prints, arXiv:1907.11932.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2019. SpanBERT: Improving pre-training by representing and predicting spans. arXiv e-prints, arXiv:1907.10529.
Braj B. Kachru, Yamuna Kachru, and Cecil Nelson, editors. 2009. The Handbook of World Englishes. Wiley-Blackwell.
Nitish Shirish Keskar, Bryan McCann, Lav Varshney, Caiming Xiong, and Richard Socher. 2019. CTRL - A Conditional Transformer Language Model for Controllable Generation. arXiv e-prints, arXiv:1909.05858.
Donna Lardiere. 1998. Case and tense in the 'fossilized' steady state. Second Language Research, 14(1):1-26.
Jacob RE Leimgruber. 2009. Modelling variation in Singapore English. Ph.D. thesis, Oxford University.
Chandler May, Alex Wang, Shikha Bordia, Samuel R. Bowman, and Rachel Rudinger. 2019. On measuring social biases in sentence encoders. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 622-628, Minneapolis, Minnesota. Association for Computational Linguistics.
Paul Michel, Xian Li, Graham Neubig, and Juan Pino. 2019. On evaluation of adversarial perturbations for sequence-to-sequence models. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 3103-3114, Minneapolis, Minnesota. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, Alexei Baevski, Angela Fan, Sam Gross, Nathan Ng, David Grangier, and Michael Auli. 2019. *fairseq: A fast, extensible toolkit for sequence modeling.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics (Demonstrations)*, pages 48-53, Minneapolis, Minnesota. Association for Computational Linguistics.
Myle Ott, Sergey Edunov, David Grangier, and Michael Auli. 2018. Scaling neural machine translation. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 1-9, Brussels, Belgium. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237, New Orleans, Louisiana. Association for Computational Linguistics.
Maja Popovic. 2015. *chrF: character n-gram f-score for automatic MT evaluation*. In *Proceedings of the Tenth Workshop on Statistical Machine Translation*, pages 392–395, Lisbon, Portugal. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting BLEU scores. In Proceedings of the Third Conference on Machine Translation: Research Papers, pages 186-191, Brussels, Belgium. Association for Computational Linguistics.
Philippe Prévost and Lydia White. 2000. Missing surface inflection or impairment in second language acquisition? evidence from tense and agreement. Second Language Research, 16:103-133.
Alec Radford, Jeff Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for