SlowGuess/NAACL_2022_OCR huggingface dataset:
be leveraged by the model includes both entity knowledge and name clues. Entity knowledge refers to the world knowledge associated with the referred entity, like "Michelle Obama is the wife of Barack Obama," while name clues refer to statistical clues associated with the name's surface form, like "Barack Obama is likely to be a male name", "Barack Obama as an in-distribution name is likely to be the answer for this dataset". While all perturbations break the entity knowledge about the original entity, InDistName doesn't introduce additional domain shift on names and largely preserve the name clues. Going from InDistName to other perturbations, the substitution names become more and more out of the dataset distribution. This performance drop can be attributed to the model's sensitivity to name-related domain shift.
We adopt two measurements to better understand the domain shift on names. As a token-level measurement, we calculate the percentage of test answer entity tokens that are never seen in entities in training answers and entities in training passages, as shown in Table 10. Different datasets have different percentages of unseen tokens in the original test sets, which are mainly affected by the size and diversity of training data. The number goes up with the DBName and RandStr perturbations. As an entity-level measurement, we train an NER model<sup>4</sup> on the training passages, with named entities annotated by spaCy as ground truth. We evaluate the trained model on perturbed test sets and calculate
its accuracy of recognizing the perturbed answer entity. The results are shown in Table 11. As a sign of domain shift, the recognition of answer entities become more difficult when they get perturbed. GPE shows the most significant perturbation drop, which correlate with our observation on the MRC task (§4.2).
SpanBERT's superior robustness over RoBERTa is mainly from handling domain shift. From SearchQA and TriviaQA results in Table 8, we find that RoBERTa and SpanBERT rely similarly on the entity knowledge ( $\sim 13\%$  performance drop from Original to InDistName on SearchQA and  $\sim 11\%$  on TriviaQA). SpanBERT's advantage over RoBERTa is mainly on its good robustness to domain shift on names, shown by the perfomance drop from IndistName to other perturbations. BERT relies slightly more on entity knowledge but much more sensitive to domain shift on names.
4.5 Bias Exhibited by Person Names
National Origins. As the DBName perturbation uses person names in the U.S., it cannot fully reflect the model's robustness behavior when encountering real-world names of different national origins. Therefore, we additional collect names from more countries (India, China) and languages (French, Arabic) to study the potential bias in MRC models. We use the romanized form of names. Table 12 shows the performance comparison of models when evaluated with the person names from different countries and languages on SearchQA and TriviaQA. Names form the U.S. and French-speaking countries generally achieve the highest EM scores. Names from China get the lowest performance for the most of time, with significant EM drop (8.4% on SearchQA and 9.8% on TriviaQA for BERT) from U.S. names. The performance gap between different countries and languages becomes smaller with more robust models.
Other Factors. We also consider other factors of a name that could be related to biased model performance. We limit our scope to the U.S. first names and sample 1500 names from the database. We consider two features for each name. Gender polarity is defined as  $\max \left(\frac{f_m}{f_f},\frac{f_f}{f_m}\right)$ , where  $f_{m},f_{f}$  are the male frequency and female frequency of a name provided by the database. It measures the gender ambiguity of the name. Popularity is defined as  $f_{m} + f_{f}$ . We calculate the EM score for a
4https://spacy.io/usage/training
Country/Language	BERT	RoBERTa	SpanBERT
U.S.	54.6±2.3/54.6±1.6	58.1±0.9/66.1±0.6	63.0±1.1/69.1±0.7
French	55.5±2.2/56.1±1.7	58.2±1.1/66.0±0.5	63.0±1.2/68.8±0.9
India	53.5±2.5/51.9±2.7	56.5±1.9/63.9±0.8	63.0±1.1/68.0±0.4
Arabic	53.3±3.1/48.8±3.2	56.3±2.1/61.8±0.9	62.8±1.0/66.2±0.8
China	46.2±2.5/44.8±3.6	54.0±0.8/63.0±1.4	59.3±2.0/65.2±0.4
RandStr	25.0±1.6/28.9±1.6	22.0±4.7/41.3±0.8	44.6±4.0/57.4±2.4
Table 12: Performance comparison of person names of different national origins. EM scores on the original and perturbed test sets of the PER entity type on SearchQA/TriviaQA.
name by evaluating on a test set where all answer first names get replaced with this name. For what we have tried, we didn't find evidence to support a correlation between each factor and the EM score. For example, with SpanBERT on TriviaQA, names with top  $20\%$  gender polarity gets  $72.7\%$  EM on average; while the bottom  $10\%$  names gets  $72.8\%$  EM. The numbers are  $73.0\%$  vs  $72.7\%$  for popularity. We leave exploring factors that correlate with the difficulty of a name as future work.
4.6 Improving Robustness with Continual Pretraining
SpanBERT's advantage over BERT suggests that some variants of MLM could be helpful for model robustness. To further improve the robustness of SpanBERT, we adopt a training paradigm with an inserted continual pretraining stage and compare MLM with different masking strategies as the objectives.
Training Paradigm. Existing works mainly seek to improve model robustness during finetuning with strategies like data augmentation (Ribeiro et al., 2019; Min et al., 2020), but they usually increase finetuning time and requires additional data. Some recent works (Gururangan et al., 2020; Ye et al., 2021) have explored improving a pretrained language model with "continual pretraining"—continuing to train a pretrained model for more steps with some objective. The generated checkpoint can be used for finetuning on any dataset in the standard way with no additional cost.
Experimental Setup. The masking policy in MLM plays an important role in instructing model learning, which can be potentially used to improve model robustness. Inspired by previous works, we experiment with four heuristic masking policies to implement the MLM objective: MLM (vanilla), MLM (whole word), MLM (span), and
MLM (entity). They perform masking at token, whole-word, span, and entity level respectively. Starting from SpanBERT (-BASE), we run continual pretraining with the above objectives for 8,000 steps. More details are described in Appendix §D.
Results. The results for models finetuned from SpanBERT and different continually pretrained models are shown in Table 13. On SQuAD, all masking policies slightly downgrade the performance. With not much room for robustness improvement, running continual pretraining is probably at the cost of slightly sacrificing the performance due to the inconsistent objective and discontinuous learning rate that are applied when starting the continual pretraining. On SearchQA and TriviaQA, out of the four masking policies, the entity-based masking policy shows consistent improvement over SpanBERT. As analyzed in §4.4, name-related domain shift is a major challenge for the model to handle. By predicting the masked entity, the model is exposed to the diverse entities in the pertraining corpus in a more explicit way, and gain a better sense in recognizing entities. Note that the improvement is not statistically significant in some cases and we leave the exploration of more effective methods to improve model robustness as future work.
Robustness of MRC Models. The robustness of MRC models are usually evaluated against test-time perturbations and out-of-domain data. Research on test-time perturbation proposes perturbation methods at different levels as attacks (Si et al., 2021), such as word replacement with neighbors in the vector space (Rychalska et al., 2018; Jia et al., 2019), question paraphrasing (Gan and Ng, 2019; Ribeiro et al., 2018), sentence distractor injection (Jia and Liang, 2017; Zhou et al., 2020). Another line of research (Fisch et al., 2019; Sen and Saffari, 2020) tests a model on data with out-of-domain passage or question distributions, usually from different datasets. Our work mainly falls into the category of test-time perturbation. We distinguish from previous work by focusing on the effect of entity renaming, with the motivation that entities can have flexible and diverse names in the real life.
Model Robustness to Entity Substitution. It is non-trivial for NLP models to be able to properly handle the large space of named entities. Previ
Model / Perturbation (MIX)	SQuAD	SearchQA	TriviaQA
Original	DBName	RandStr	Original	DBName	RandStr	Original	DBName	RandStr
SpanBERT	86.8±0.5	84.9±0.4	83.0±0.1	73.2±0.7	57.9±1.0	41.5±3.2	79.1±0.1	63.1±0.8	51.9±2.3
SpanBERT w/ continual pretraining
+ MLM (vanilla)	85.6±0.5	83.9±0.2	81.8±0.2	72.3±0.8	57.0±0.3	34.8±2.9	78.9±0.8	64.1±0.2	48.1±2.9
+ MLM (whole word)	86.0±0.7	84.5±0.3	82.7±0.4	72.9±0.4	58.0±0.2	41.6±3.3	79.1±0.5	64.2±0.4	50.1±0.9
+ MLM (span)	85.7±0.2	84.1±0.1	82.6±0.1	73.3±0.5	57.9±0.8	39.5±2.4	79.4±0.8	64.3±0.4*	54.1±1.5
+ MLM (entity)	86.0±0.4	84.3±0.3	82.7±0.1	73.4±0.4	59.3±1.1	48.1±4.6	79.6±0.6	65.9±1.1*	55.5±2.7
Table 13: EM scores of different continually pretrained models on the original and perturbed test sets. Significant improvements  $(p < 0.05)$  over SpanBERT are marked with  $*$ .
ous works use entity substitution to audit or improve model robustness on different tasks like NER (Agarwal et al., 2020; Lin et al., 2021), Natural Language Inference (Mitra et al., 2020), Coreference Resolution (Subramanian and Roth, 2019), and Dialogue State Tracking (Cho et al., 2021). Shwartz et al. (2020) experiment with name swapping to show that a trained MRC model has bias on some U.S. given names due to the grounding effects that associate names with certain entities. Ribeiro et al. (2020) and Balasubramanian et al. (2020) investigate the robustness of models on several tasks with named entity replacement. However, these works didn't systematically test on MRC datasets with different characteristics to unveil the actual robustness challenge. Liu et al. (2021) study the novel entity generalization ability of open-domain QA models by categorizing the test questions based on whether the named entities have been seen during training. Longpre et al. (2021) analyze the memorization behavior of generative open-domain QA models using knowledge conflicts. They use entity substitution to create test passages that contain facts contradicting to what the model has learned during training time. In contrast, we analyze extractive MRC model's robustness when encountering new entities, by evaluating on modified test sets without intentionally introduced knowledge conflicts. The extractive task formulation also makes the model unable to output its memorized knowledge as generative models, leading to different analysis questions and methods.
In this paper, we systematically study the robustness of MRC models to entity name substitution. Specifically, we propose a substitution framework along with candidate names of different implications. We experiment with three pretrained language models on five MRC datasets. We find that
models trained on distantly-supervised datasets are susceptible to entity name substitution, while models trained on human-annotated datasets are relatively robust, with GPE renaming harder than PER and ORG renaming. The lack of robustness can be further attributed to model's overreliance on entity knowledge and name clues. We also find that SpanBERT, which is pretrained using span-level objectives, shows better robustness than BERT and RoBERTa. Leveraging these insights, we study defense approaches based on continual pretraining and demonstrate that entity-based masking policies are beneficial to model's robustness. Future works include systematically studying the effect of background knowledge in MRC, and developing more effective methods to improve the robustness of MRC models.
This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via Contract No. 2019-19051600007, the DARPA MCS program under Contract No. N660011924033, the Defense Advanced Research Projects Agency with award W911NF-19-20271, NSF IIS 2048211, NSF SMA 1829268, and gift awards from Google, Amazon, JP Morgan and Sony. We would like to thank all the collaborators in USC INK research lab for their constructive feedback on the work. We would also like to thank the anonymous reviewers for their valuable comments.
Oshin Agarwal, Yinfei Yang, Byron C. Wallace, and A. Nenkova. 2020. Entity-switched datasets: An approach to auditing the in-domain robustness of named entity recognition models. ArXiv preprint, abs/2004.04123.
Sriram Balasubramanian, Naman Jain, Gaurav Jindal, Abhijeet Awasthi, and Sunita Sarawagi. 2020. What's in a name? are BERT named entity representations just as good for any other name? In Proceedings of the 5th Workshop on Representation Learning for NLP, pages 205-214, Online. Association for Computational Linguistics.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Hyundong Cho, Chinnadhurai Sankar, Christopher Lin, Kaushik Ram Sadagopan, Shahin Shayandeh, Asli Celikyilmaz, Jonathan May, and Ahmad Beirami. 2021. Checkdst: Measuring real-world generalization of dialogue state tracking performance. arXiv preprint arXiv:2112.08321.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. ArXiv preprint, abs/1704.05179.
Adam Fisch, Alon Talmor, Robin Jia, Minjoon Seo, Eunsol Choi, and Danqi Chen. 2019. MRQA 2019 shared task: Evaluating generalization in reading comprehension. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 1-13, Hong Kong, China. Association for Computational Linguistics.
Wee Chung Gan and Hwee Tou Ng. 2019. Improving the robustness of question answering systems to question paraphrasing. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6065-6075, Florence, Italy. Association for Computational Linguistics.
Suchin Gururangan, Ana Marasovic, Swabha Swayamdipta, Kyle Lo, Iz Beltagy, Doug Downey, and Noah A. Smith. 2020. Don't stop pretraining: Adapt language models to domains and tasks. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 8342-8360, Online. Association for Computational Linguistics.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Ming-Wei Chang. 2020. Realm: Retrieval
augmented language model pre-training. ArXiv preprint, abs/2002.08909.
Matthew Honnibal, Ines Montani, Sofie Van Landeghem, and Adriane Boyd. 2020. spaCy: Industrialstrength Natural Language Processing in Python.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 2021-2031, Copenhagen, Denmark. Association for Computational Linguistics.
Robin Jia, Aditi Raghunathan, Kerem Göksel, and Percy Liang. 2019. Certified robustness to adversarial word substitutions. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 4129-4142, Hong Kong, China. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S. Weld, Luke Zettlemoyer, and Omer Levy. 2020. Span-BERT: Improving pre-training by representing and predicting spans. Transactions of the Association for Computational Linguistics, 8:64-77.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, Brussels, Belgium. Association for Computational Linguistics.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. 2020. UNIFIEDQA: Crossing format boundaries with a single QA system. In *Findings of the Association for Computational Linguistics: EMNLP* 2020, pages 1896-1907, Online. Association for Computational Linguistics.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. Transactions of the Association for Computational Linguistics, 7:452-466.
Yuxuan Lai, Chen Zhang, Yansong Feng, Quzhe Huang, and Dongyan Zhao. 2021. Why machine reading
comprehension models learn shortcuts? In Findings of the Association for Computational Linguistics: ACL-IJCNLP 2021, pages 989-1002, Online. Association for Computational Linguistics.
Bill Yuchen Lin, Wenyang Gao, Jun Yan, Ryan Moreno, and Xiang Ren. 2021. Rockner: A simple method to create adversarial examples for evaluating the robustness of named entity recognition models. ArXiv preprint, abs/2109.05620.
Linqing Liu, Patrick Lewis, Sebastian Riedel, and Pontus Stenetorp. 2021. Challenges in generalization in open domain question answering. ArXiv preprint, abs/2109.01156.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. ArXiv preprint, abs/1907.11692.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. *ArXiv* preprint, abs/2109.05052.
Ilya Loshchilov and Frank Hutter. 2019. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, ICLR 2019, New Orleans, LA, USA, May 6-9, 2019. OpenReview.net.
Mitchell P. Marcus, Beatrice Santorini, and Mary Ann Marcinkiewicz. 1993. Building a large annotated corpus of English: The Penn Treebank. Computational Linguistics, 19(2):313-330.
John P Miller, Rohan Taori, Aditi Raghunathan, Shiori Sagawa, Pang Wei Koh, Vaishaal Shankar, Percy Liang, Yair Carmon, and Ludwig Schmidt. 2021. Accuracy on the line: On the strong correlation between out-of-distribution and in-distribution generalization. In International Conference on Machine Learning, pages 7721-7735. PMLR.
Junghyun Min, R. Thomas McCoy, Dipanjan Das, Emily Pitler, and Tal Linzen. 2020. Syntactic data augmentation increases robustness to inference heuristics. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2339-2352, Online. Association for Computational Linguistics.
Arindam Mitra, Ishan Shrivastava, and Chitta Baral. 2020. Enhancing natural language inference using new and expanded training data sets and new learning models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 8504-8511.
Anselmo Peñas, Eduard H Hovy, Pamela Forner, Álvaro Rodrigo, Richard Sutcliffe, Corina Forascu, and Caroline Sporleder. 2011. Overview of qa4mre at clef 2011: Question answering for machine reading evaluation. In CLEF (Notebook Papers/Labs/Workshop), pages 1-20.
Danish Pruthi, Bhuwan Dhingra, and Zachary C. Lipton. 2019. Combating adversarial misspellings with robust word recognition. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 5582-5591, Florence, Italy. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Marco Tulio Ribeiro, Carlos Guestrin, and Sameer Singh. 2019. Are red roses red? evaluating consistency of question-answering models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 6174-6184, Florence, Italy. Association for Computational Linguistics.
Marco Tulio Ribeiro, Sameer Singh, and Carlos Guestrin. 2018. Semantically equivalent adversarial rules for debugging NLP models. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 856-865, Melbourne, Australia. Association for Computational Linguistics.
Marco Tulio Ribeiro, Tongshuang Wu, Carlos Guestrin, and Sameer Singh. 2020. Beyond accuracy: Behavioral testing of NLP models with CheckList. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4902-4912, Online. Association for Computational Linguistics.
Barbara Rychalska, Dominika Basaj, and Przemyslaw Biecek. 2018. Are you tough enough? framework for robustness validation of machine comprehension systems. *ArXiv* preprint, abs/1812.02205.
Priyanka Sen and Amir Saffari. 2020. What do models learn from question answering datasets? In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 2429-2438, Online. Association for Computational Linguistics.
Vered Shwartz, Rachel Rudinger, and Oyvind Tafjord. 2020. "you are grounded!": Latent name artifacts in pre-trained language models. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6850-6861, Online. Association for Computational Linguistics.
Chenglei Si, Ziqing Yang, Yiming Cui, Wentao Ma, Ting Liu, and Shijin Wang. 2021. Benchmarking robustness of machine reading comprehension models. In *Findings of the Association for Computational Linguistics: ACL-IJCNLP* 2021, pages 634-644, Online. Association for Computational Linguistics.