SlowGuess/ACL_2020_OCR huggingface dataset:
machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
John Rickford and Sharese King. 2016. Language and linguistics on trial: Hearing rachel jeantel (and other vernacular speakers) in the courtroom and beyond. Language, 92:948-988.
Rachel Rudinger, Jason Naradowsky, Brian Leonard, and Benjamin Van Durme. 2018. Gender bias in coreference resolution. In Proceedings of the Annual Meeting of the North American Association of Computational Linguistics (NAACL), pages 8-14, New Orleans, Louisiana. Association for Computational Linguistics.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1715-1725, Berlin, Germany. Association for Computational Linguistics.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In 5th International Conference on Learning Representations, Toulon, France.
Harry Seymour. 2004. The challenge of language assessment for african american english-speaking children: A historical perspective. Seminars in Speech and Language, 25:3-12.
Christian Szegedy, Wojciech Zaremba, Ilya Sutskever, Joan Bruna, Dumitru Erhan, Ian J. Goodfellow, and Rob Fergus. 2014. Intriguing properties of neural networks. In 2nd International Conference on Learning Representations, Banff, AB, Canada.
Rachael Tatman. 2017. Gender and dialect bias in YouTube's automatic captions. In Proceedings of the First ACL Workshop on Ethics in Natural Language Processing, pages 53-59, Valencia, Spain. Association for Computational Linguistics.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Lydia White. 2003. Fossilization in steady state 12 grammars: Persistent problems with inflectional morphology. *Bilingualism: Language and Cognition*, 6:129 - 141.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumont, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv e-prints, arXiv:1910.03771.
Walt Wolfram. 2004. The grammar of urban African American Vernacular English. Handbook of varieties of English, 2:111-32.
Wei Emma Zhang, Quan Z. Sheng, Ahoud Alhazmi, and Chenliang Li. 2019a. Adversarial attacks on deep learning models in natural language processing: A survey. arXiv e-prints, arXiv:1901.06796.
Yuan Zhang, Jason Baldridge, and Luheng He. 2019b. PAWS: paraphrase adversaries from word scrambling. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 1298-1308, Minneapolis, Minnesota. Association for Computational Linguistics.
A Examples of Inflectional Variation in English Dialects
African American Vernacular English (Wolfram, 2004)
- They seen it.
- They run there yesterday.
The folks was there.
Colloquial Singapore English (Singlish) (Leimgruber, 2009)
He want to see how we talk.
- It cover up everything in the floss. It's not nice. It look very cheap.
- I want to shopping only.
B More Details on Human Evaluation
Please choose the most suitable option for each question.
Figure 3: Amazon Mechanical Turk UI.
Figure 3 contains a screenshot of the UI we present to crowd workers. We intentionally prime Turkers by asking if the sentence could be written by an L2 speaker instead of directly asking for acceptability/naturalness ratings in order to ensure that they consider these possibilities.
We also do not use the Semantic Textual Similarity evaluation scheme (Agirre et al., 2013); during preliminary pilot studies, we discovered that annotators interpreted certain words in the scheme (e.g., "information", "details", and "topics") considerably differently, introducing substantial noise into an already subjective judgement task.
Possible limitations. It is possible that seeing the original sentence could affect the worker's judgment of the perturbed sentence's plausibility. However, we argue that this is not necessarily negative since seeing the original sentence would make it easier to spot perturbations that are just outright wrong (i.e., a human will not make that error regardless of their level of fluency).
C Adversarial Training Set Generation
D Tables and Figures
SpanBERTSQuAD 2 (F1)
Dataset	Clean	Morpheusseq	Morpheusparallel
SQuAD 2.0 Ans	88.52	69.47 (-21.52%)	74.38 (-15.97%)
SQuAD 2.0 All	87.71	73.26 (-16.47%)	76.64 (-12.62%)
Transformer-big (BLEU)
Dataset	Clean	Morpheusseq	Morpheusparallel
newstest2014	43.16	20.57 (-56.25%)	20.85 (-51.69%)
Table 6: Results of the parallel and sequential approaches to implementing MORPHEUS on SpanBERT $_{\text{SQuAD2}}$  and Transformer-big.
SpanBERTSQuAD 2 (F1)
Weighted	Dataset	Clean	Morpheusorig
Yes	SQuAD 2.0 Ans	86.80	85.17 (-1.87%)
SQuAD 2.0 All	86.00	84.72 (-1.48%)
No	SQuAD 2.0 Ans	84.52	83.15 (-1.62%)
SQuAD 2.0 All	87.12	86.03 (-1.25%)
Table 7: Comparison of results from using weighted vs. uniform random sampling to the create adversarial training set for fine-tuning SpanBERT $_{SQuAD2}$
Transformer-big (BLEU)
Subset	Original	Clean	Morpheusorig
1/20	43.16	30.90	24.95
1/4	43.16	36.59	29.46
Full	43.16	40.60	31.99
Table 8: Results from adversarially fine-tuning Tranformer-big on different subsets of the original training set.
Figure 4: Effect of shuffling the inflection list on the adversarial distribution. We observe that shuffling the inflection list induces a more uniform inflectional distribution by reducing the higher frequency inflections and boosting the lower frequency ones.
Original Source	According to Detroit News, the queen of Soul will be performing at the Sound Board hall of MotorCity Casino Hotel on 21 December.
Adversarial Source	Accorded to Detroit News, the queen of Soul will be performing at the Sound Board hall of MotorCity Casino Hotel on 21 December.
Original Translation	Selon Detroit News, la reine de Soul se produit au Sound Board Hall de l'hôtel MotorCity Casino le 21 décembre.
Original Source	Intersex children pose ethical dilemma.
Adversarial Source	Intersex child posing ethical dilemma.
Original Translation	Les enfants intersexuels posent un dilemme éthique.
Original Source	The Guangzhou-based New Express made a rare public plea for the release of journalist Chen Yongzhou.
Adversarial Source	The Guangzhou-based New Expresses making a rare public plea for the release of journalist Chen Yongzhou.
Original Translation	Le New Express, basé à Guangzhou, a lancé un rare appel public en faveur de la libération dujournaliste Chen Yongzhou.
Original Source	Cue stories about passport controls at Berwick and a barbed wire border along Hadrian's Wall.
Adversarial Source	Cue story about passport controls at Berwick and a barbed wires borders along Hadrian's Walls.
Original Translation	Cue historieires sur le contrôle des passeports à Berwick et une frontière de barbelés le long du mur d'Hadrien.
Table 9: Some of the adversaries that caused Transformer-big to output the source sentence instead of a translation.
(a) SQuAD 2.0 dev set
(b) SQuAD 2.0 training set
Figure 5: Full versions of Figure 2