SlowGuess/NAACL_2022_OCR huggingface dataset:
Jun Yan $^{1}$  Yang Xiao $^{2}$  Sagnik Mukherjee $^{3}$  Bill Yuchen Lin $^{1}$  Robin Jia $^{1}$  Xiang Ren $^{1}$
University of Southern California $^{1}$  Fudan University $^{2}$  IIT Kanpur $^{3}$
{yanjun,yuchen.lin,robinjia,xiangren}@usc.edu
17307100059@fudan.edu.cn sagnikm@iitk.ac.in
We study the robustness of machine reading comprehension (MRC) models to entity renaming—do models make more wrong predictions when the same questions are asked about an entity whose name has been changed? Such failures imply that models overly rely on entity information to answer questions, and thus may generalize poorly when facts about the world change or questions are asked about novel entities. To systematically audit this issue, we present a pipeline to automatically generate test examples at scale, by replacing entity names in the original test sample with names from a variety of sources, ranging from names in the same test set, to common names in life, to arbitrary strings. Across five datasets and three pretrained model architectures, MRC models consistently perform worse when entities are renamed, with particularly large accuracy drops on datasets constructed via distant supervision. We also find large differences between models: SpanBERT, which is pretrained with span-level masking, is more robust than RoBERTa, despite having similar accuracy on unperturbed test data. We further experiment with different masking strategies as the continual pretraining objective and find that entity-based masking can improve the robustness of MRC models.<sup>1</sup>
The task of machine reading comprehension (MRC) measures machines' understanding and reasoning abilities. Recent research advances (Devlin et al., 2019; Yang et al., 2019; Khashabi et al., 2020) have driven MRC models to reach or even exceed human performance on several MRC benchmark datasets. However, their actual ability to solve the general MRC task is still questionable (Kaushik and Lipton, 2018; Sen and Saffari, 2020; Sugawara et al., 2020; Lai et al., 2021). While hu
Figure 1: An illustrative example of the robustness to entity renaming and our proposed perturbations for robustness evaluation. "Michael" is from the answer of another test instance. "Ashvith" is a person name from an external database. "Uqlcs" is a random string with the same length as the original name.
mans show robust generalization on reading comprehension, existing works have revealed that MRC models generalize poorly to out-of-domain data distributions (Fisch et al., 2019) and are brittle under test-time perturbations (Pruthi et al., 2019; Jia et al., 2019; Jia and Liang, 2017). All these issues could naturally happen to MRC systems deployed in the wild, hindering them to make reliable predictions on user inputs with great flexibility.
In this work, we focus on an important but understudied type of test-time distribution shift caused by novel entity (e.g., person and company) names. Besides the evidence provided by the surrounding context, an MRC model also has the capacity to leverage the entity information to make predictions (Sugawara et al., 2018; Chen et al., 2016). The information associated with the entity name covers both world knowledge that can change over time and dataset shortcuts that are unlikely to generalize. While contributing to performance on certain benchmarks, the over-reliance on specific entity names leads to an overestimation of model's actual ability to read and comprehend the provided
1Our code and data can be found at https://github. com/INK-UGC-entity-robustness.
Human Language Technologies, pages 508 - 520
passage (Penas et al., 2011). It also hinders model generalization to novel entity names, which itself is challenging due to the large space of valid entity names induced by the flexibility of entity naming. For example, person names can be chosen from a large vocabulary depending on the country, while companies can be named in an even more creative way, not to mention new names that are being invented every day. As illustrated in Figure 1, keeping the reasoning context unchanged, a robust MRC model is supposed to correctly locate the same span of a named entity as the answer, even after it gets renamed.
To audit model robustness, we use entity renaming as test time perturbation to mimic the situation where a deployed MRC model encounters questions asking for novel entity names in the emerging data. We design a general pipeline to generate natural perturbations of MRC instances by swapping the answer entity name with another valid name throughout the passage. We design perturbation rules and collect resources for three types of entities with large name space: Person, Organization, and Geopolitical Entity.
With this proposed analysis framework, we conduct extensive experiments on five datasets and three pretrained language models. Data-wise, we find that distantly supervised MRC datasets lead to less robustness. Entity-wise, we find that geopolitical entities pose a greater challenge than people and organizations when renamed. Model-wise, we find that SpanBERT is more robust than BERT and RoBERTa, mainly due to its lower sensitivity to domain shift on names, which is likely a benefit of its span-focused pretraining objective. Inspired by this, we investigate several continual pretraining objectives and find that an entity-based masking strategy can further improve robustness.
2 Analysis Setup
2.1 Extractive MRC
The task of MRC tests a machine' understanding and reasoning abilities by asking it to answer the question based on the provided passage. We focus on extractive MRC, where the answer is a span in the passage. Formally, given a question  $Q$  and a passage  $P$  of  $n$  tokens  $P = \{x_{1},\ldots ,x_{n}\}$  a model is expected to predict an answer span  $a = \{x_i,\dots ,x_{i + k}\} (1\leq i\leq i + k\leq n)$  in the passage  $P$  as a response to the question  $Q$  .We use exact match (EM) as the metric for MRC evalua
Dataset	# Train	# Dev	# Test	DS?
SQuAD	77,929	8,659	10,507	✘
NQ	84,577	9,367	12,836	✘
HotpotQA	65,636	7,292	5,901	✘
SearchQA	105,646	11,738	16,980	✓
TriviaQA	42,569	4,696	7,785	✓
Table 1: Evaluation datasets. "DS?" indicates whether distant supervision is used for data collection.
tion, which is the percentage of test instances that the model exactly predicts one of the gold answers.
In both real-world scenarios and MRC datasets, a large portion of questions ask about entities like people, organizations and locations. While unmentioned background knowledge about the entities might be helpful for solving the questions, overly relying on it makes the model hard to adapt to updated facts provided by the passage and generalize to novel entities. Especially, we contrast MRC with closed-book QA, which requires a model to directly answer questions without access to any document passage. Closed-book QA tests a model's ability to pack knowledge into its parameters and retrieve knowledge from parameters to answer the question. On the contrary, we expect an MRC model to reason based on the provided passage.
2.2 Evaluation Protocol
We study the robustness of MRC models via test-time perturbation. Given an original test set  $D_{\mathrm{test}}$  and a perturbation function  $f_{\mathrm{perturb}}$  (detailed in §3) as inputs, we construct  $N$  perturbed test sets with  $N$  perturbation seeds. We evaluate the model on the  $N$  perturbed test sets. By averaging the results, we get the average-case EM score as the final metric, which measures the average impact on the model performance caused by the names from a certain perturbation. We set  $N = 5$  in experiments.
2.3 Datasets
We choose five datasets with different characteristics from the MRQA 2019 shared task (Fisch et al., 2019): SQuAD (Rajpurkar et al., 2016), Natural Questions (NQ) (Kwiatkowski et al., 2019), HotpotQA (Yang et al., 2018), SearchQA (Dunn et al., 2017), and TriviaQA (Joshi et al., 2017). Since the official test sets of the MRQA datasets are hidden, we use the development set as the in-house test set, and hold out  $10\%$  of the training data as the in-house development set. Their statistics are shown in Table 1.
As a major difference in data collection, SQuAD, NQ, and HotpotQA employ crowdworkers to annotate the answer span in the passage, while SearchQA and TriviaQA use distant supervision to match the passage with the question. Distant supervision provides no guarantee that the passage contains enough evidence to derive the answer. The context where the entity span shows up may not even be related to the question.
2.4 MRC Models
We experiment with three pretrained language models that have demonstrated strong performance on popular MRC benchmarks. BERT (Devlin et al., 2019) is trained on English Wikipedia plus BookCorpus with masked language modeling (MLM) and next sentence prediction (NSP) as self-supervised objectives. RoBERTa (Liu et al., 2019) improves over BERT mainly by dropping the NSP objective and increasing the pretraining time and the size of pretraining data. SpanBERT (Joshi et al., 2020) masks random contiguous spans to implement MLM and replaces NSP with a span-boundary objective (SBO).
The pretrained language models are finetuned on the MRC dataset to predict the start and end tokens of the answer span based on the concatenated question and passage (Devlin et al., 2019). By default, all pretrained language models in the main experiments are case-sensitive and in their BASE sizes. More details are shown in Appendix §A.
3 Entity Name Substitution
In this section, we introduce our method for perturbing an MRC test set with substitution entity names, i.e., the instantiation of  $f_{\text{perturb}}$ . Generating substitution names is at the core of our evaluation as different kinds of names measure a model's behavior in different situations with different robustness implications. We propose three categories of perturbations on three entity types and collect the corresponding name resources, aiming to audit a model's robustness from different perspectives.
3.1 Perturbation Pipeline
As illustrated in Figure 2, our perturbation pipeline consists of four steps, which are introduced below.
Step 1: Answer Entity Recognition. As we focus on the effect of answer entity renaming, we first identify entities in the answers by performing named entity recognition (NER) with spaCy
Entity Type	Applicable Types of Perturbable Spans
PER(4)	<First Name-Male> (e.g., Richard, Morton)<First Name-Female> (e.g., Lauren, Jennifer)<First Name-Neutral> (e.g., Shine, Frankie)<Last Name> (e.g., Marx, Winfrey)
ORG(5)	<NNP> (e.g., Celtic, Tiffany)<Rare> (e.g., Hufflepuff, Pokemon)
GPE(3)	<GPE-Country> (e.g., Iceland, Algeria)<GPE-State> (e.g., New Brunswick, Ohio)<GPE-City> (e.g., Boston, Sonsonate)
Table 2: Applicable metadata for each entity type in the perturbation pipeline.
Dataset	# PER	# ORG	# GPE	# MIX
SQuAD	1,170	1,095	602	2,613
NQ	3,257	1,207	1,414	5,150
HotpotQA	1,351	824	788	2,614
SearchQA	5,707	2,450	2,248	8,688
TriviaQA	2,747	1,276	1,270	4,351
Table 3: Statistics of the perturbable subsets for different entity types and their union ("MIX").
(Honnibal et al., 2020) on the passage and extract the results on the answer spans. We identify three types of named entities: Person (PER), Organization (ORG), and Geopolitical Entity (GPE). All of them frequently appear as answers and have large space of valid names, making it important and challenging for models to robustly handle.
Step 2: Perturbable Span Identification. To facilitate name substitution, we assign metadata to detected entity names by identifying perturbable spans within the entity name. For each type of entity names, we define the applicable span types in Table 2. The heuristics for identifying each type of perturbable spans are introduced in Appendix §B. Note that given one or more entity types of interest, in this step we filter the test data to only keep a subset of instances with non-empty metadata for the corresponding entity types, which are instances that are ready to be perturbed. Sizes of the perturbable subsets for different entity types and their union (MIX) are shown in Table 3.
Step 3: Candidate Name Sampling. For each perturbable span, we get its substitution name by querying an external dictionary with the span type. The substitution name is randomly sampled from a pool of names in the external dictionary with the same span type. We collect dictionaries with names of different characteristics serving for different analysis purpose, which are detailed in §3.2.
Figure 2: The perturbation pipeline for performing entity name substitution on MRC instances.
Step 4: Name Substitution. Once we have a candidate name for each perturbable span, we perform string mapping on the passage, question, and the gold answer, to finish the entity renaming in MRC instances. The name substitution changes all mentions of the answer entity in the passage while keeping the other reasoning context.
3.2 Candidate Name Collection
We consider three types of candidate names for perturbations in our main experiments to simulate the domain shift of entity names during test time.
In-Distribution Name (InDistName). The set of candidate names with their span types is the same as the perturbable spans with their types identified from the gold answers in the test set. This ensures that no new name is introduced to the test set.
Database Name (DBName). We collect names in the real world by referring to relevant databases. For PER, we collect first names<sup>2</sup> (with gender frequency) and last names<sup>3</sup> from the official statistics of person names in the U.S. (We experiment with names from other countries and languages in §4.5.) We regard a first name as a male/female name if its male/female frequency is two times larger than its frequency of the opposite gender. The remaining names are considered as neutral. Following the practice for identifying perturbable spans, we get the list of country/state/city names using Countries States Cities Database and the NNP list using PTB. Rare words constitute an open vocabulary so they will not be substituted under the DBName perturbation.
Accuracy	PER	ORG	GPE
Perturbable Span Identification	93.3%	86.7%	93.3%
Name Substitution	86.7%	96.7%	96.7%
Table 4: Validity of the two key steps in the perturbation pipeline on 30 randomly sampled TriviaQA instances for each entity type.
Random String (RandStr). The RandStr perturbation is different from the other two as it neglects the query span type when preparing the candidates. We generate a random alphabetical string of the same length and casing as the original perturbable span. Names from low-resource languages can look quite irregular to the pretrained language models. Random string as an extreme case provides an estimation of the performance in this scenario.
3.3 Perturbation Quality
The validity of the perturbed instances depends on the quality of the perturbation pipeline (§3.1). We manually check the accuracy of the perturbation steps on TriviaQA, which demonstrates the largest performance drop as we will show. Out of the four steps in the pipeline (Figure 2), we evaluate the accuracy of step 2 ("Perturbable Span Identification") and step 4 ("Name Substitution") while the accuracy of the other two steps can be inferred. The evaluation details are provided in Appendix §C. As shown in Table 4, our method gets acceptable accuracy on the three entity types, confirming the quality of the perturbed test sets.
The average-case EM scores on the original and perturbed test sets are presented in Figure 3. We report the mean and standard deviation over 3 train
2 https://www.ssa.gov/oact/babynames/limits.html
3 https://www.census.gov/topics/population/  
theology/data/2010_surnames.html
Figure 3: Main results. EM scores for MRC models evaluated on datasets under different perturbations.
BERT@MIX	SQuAD	NQ	HotpotQA	SearchQA	TriviaQA
Original	81.2±0.3	64.4±1.0	60.0±0.2	69.5±1.1	73.4±0.8
InDistName	78.7±0.6	62.0±1.2	56.8±0.4	53.6±1.3	59.0±1.4
DBName	78.8±0.9	62.1±1.3	54.9±0.3	50.2±1.8	50.4±1.6
RandStr	76.9±1.0	59.0±1.7	49.5±0.8	23.6±1.2	25.4±1.4
Table 5: Comparison of different datasets. EM scores of BERT on the original and perturbed test sets of the MIX entity type.
Wrong Entity Error	SQuAD	NQ	HotpotQA	SearchQA	TriviaQA
Original	33.9%	34.3%	27.3%	46.3%	69.0%
InDistName	38.4%	37.5%	32.3%	66.2%	76.6%
DBName	38.0%	37.3%	33.3%	67.2%	77.6%
RandStr	42.7%	41.2%	39.1%	84.7%	86.8%
Table 6: Error analysis. The percentage of wrong entity errors of BERT on the original and perturbed test sets of the MIX entity type.
ing seeds. We analyze the results from several angles by aggregating across certain dimensions.
4.1 Which datasets lead to less robustness?
Training on MRC datasets created with distant supervision leads to less robustness. In Table 5, we show the results of BERT on the original and perturbed test sets, while results of RoBERTa and SpanBERT follow similar patterns. The perturbations on all 3 entity types are combined (shown as "MIX"). We find that models trained on SQuAD, NQ, and HotpotQA (with at most  $6\%$  performance drop under the DBName perturbation)
are significantly more robust than models trained on SearchQA and TriviaQA (with about  $20\%$  performance drop under the DBName perturbation). While the first group of datasets are human-labeled, the later group of datasets are constructed using distant supervision. Such correlation indicates that training noise due to mismatched questions and passages harms model's robustness. We hypothesize the reason to be that, the passage in the human-annotated datasets usually provides enough evidence to derive the answer, so a model is able to learn the actual task of "reading comprehension" from the data. On the contrary, SearchQA and TriviaQA use web snippets as the source of passages. The labeling process of distant supervision assumes that "the presence of the answer string implies the document does answer the question" (Joshi et al., 2017), while the document may or may not contain all facts needed to support the answer. In this case, because the actual reading comprehension task is difficult to learn due to lack of evidence, the model could be prone to use entity-specific background knowledge (e.g. assuming that "Jack Higgins" is a British author regardless of the context) or learn dataset-specific shortcuts associated with certain names via memorization (e.g., choosing "Jack Higgins" whenever it's mentioned in the passage and the question asks for an author), which causes the robustness issue.
To better understand the failure cases, we categor-
BERT	SQuAD	NQ	HotpotQA	SearchQA	TriviaQA
PER-Original	82.8±0.4	69.3±0.9	63.1±0.1	69.7±1.4	73.6±0.7
PER-DBName	81.7±0.8	68.0±1.0	60.8±0.2	54.6±2.3	54.6±1.6
PER-Δ	1.1	1.3	2.3	15.1	19.0
ORG-Original	79.7±0.6	52.1±1.2	58.0±0.3	66.7±1.5	73.8±1.5
ORG-DBName	77.5±1.4	50.8±0.8	55.0±0.6	54.2±1.6	57.0±1.5
ORG-Δ	2.2	1.3	3.0	12.5	16.8
GPE-Original	79.1±1.0	54.5±1.7	55.8±0.6	74.4±0.8	76.4±0.6
GPE-DBName	73.7±1.1	49.5±2.7	43.9±1.3	40.1±0.8	40.1±1.3
GPE-Δ	5.4	5.0	11.9	34.3	36.3
Table 7: Comparison of different entity types. EM scores of BERT on the Original and DBName test sets.
MIX	BERT	RoBERTa	SpanBERT
Original	69.5±1.1/73.4±0.8	74.1±0.2/78.6±0.4	73.2±0.7/79.1±0.1
InDistName	53.6±1.3/59.0±1.4	60.7±0.4/67.8±1.1	60.3±1.4/68.3±0.8
DBName	50.2±1.8/50.4±1.6	54.0±1.0/60.5±0.9	57.9±1.0/63.1±0.8
RandStr	23.6±1.2/25.4±1.4	21.0±4.8/35.6±0.2	41.5±3.2/51.9±2.3
rize the errors made by the model into two classes: wrong entity errors and wrong boundary errors, based on whether the predicted span has any word overlap with the gold answer. We report the percentage of wrong entity errors in Table 6. On all datasets, wrong entity errors make up a larger percentage of all errors when the test sets get perturbed. This suggests that the performance drop is mainly caused by the increasing errors in identifying the correct answer entity rather than accurately predicting the boundary of a correctly-identified answer entity.
4.2 Which entity types are more challenging?
GPE renaming poses the greatest robustness challenge. The renaming of PER and ORG are similarly less challenging. In Table 7, we present the performance drop caused by the DBName perturbation for each entity type. GPE renaming shows the largest performance drop. The comparison of PER and ORG differs across datasets, but their corresponding performance drops are generally similar. The reason is likely to be that the model is only exposed to a small number of distinct GPE names during finetuning compared to PER and ORG. In the training set of TriviaQA, there are 40k ORG names and 54k PER names, but only 12k GPE names. The lack of seen names makes it hard to learn the generalization ability.
Table 8: Comparison of different models. EM scores on the original and perturbed test sets of the MIX entity type on SearchQA/TriviaQA.
MIX	BERT	RoBERTa	SpanBERT
Original	73.4±0.8/76.3±1.1	78.6±0.4/82.3±0.2	79.1±0.1/82.8±0.6
InDistName	59.0±1.4/61.3±0.4	67.8±1.1/70.8±0.6	68.3±0.8/72.1±0.7
DBName	50.4±1.6/52.6±0.7	60.5±0.9/62.6±1.3	63.1±0.8/66.9±0.6
RandStr	25.4±1.4/27.9±2.4	35.6±0.2/35.8±3.1	51.9±2.3/53.6±4.9
Table 9: Comparison of different model sizes. EM scores of the BASE/LARGE variants of models on the original and perturbed test sets of the MIX entity type on TriviaQA.
4.3 Which models are more robust?
On distantly supervised datasets, SpanBERT is more robust than RoBERTa, which is more robust than BERT. In Table 8, we show the performance of the three models under perturbations of the MIX entity type on SearchQA and TriviaQA. While RoBERTa and SpanBERT show comparable performance on the original and InDist-Name test sets, SpanBERT's improvement over RoBERTa becomes larger with more difficult perturbations. Meanwhile, BERT shows even larger performance decreases than RoBERTa. The models' performance differences are mainly attributed to their different pretraining strategies. RoBERTa's improvement over BERT indicates that a better pretraining configuration (as measured by the performance on the in-domain original test set) is also beneficial to the performance on the perturbed test sets, suggesting better generalization ability to the out-of-domain data. This correlation is consistent with the findings in Miller et al. (2021). SpanBERT's particular advantage on the perturbed test sets indicates its span-focused pretraining objective (span-based MLM and span prediction based on boundary tokens) is especially helpful for the span-related robustness, which is desired for the MRC task.
Larger models are not more robust. In Table 9, we compare the performance of the BASE and LARGE variants of pretrained models on TriviaQA. The performance drops from the original test sets to the perturbed test sets are similar for these two variants in most cases, suggesting that simply increasing the model size can not resolve the robustness issue.
4.4 How can we disentangle reasons for performance drop?
Both loss of entity knowledge and domain shift on names happen during renaming. The information associated with the entity name that can
Unseen@MIX	Original/InDistName	DBName	RandStr
SQuAD	33% / 24%	94% / 92%	100% / 100%
NQ	14% / 4%	95% / 89%	100% / 100%
HotpotQA	19% / 3%	96% / 89%	100% / 100%
SearchQA	7% / 0%	92% / 78%	100% / 100%
TriviaQA	21% / 2%	93% / 83%	100% / 100%
Table 10: The percentage of test answer entity tokens that are never seen in the training answer/passage entities.
Accuracy	Original	InDistName	DBName	RandStr
PER	75.6%	73.4%	58.0%	60.0%
ORG	60.1%	50.6%	49.2%	48.0%
GPE	84.4%	83.4%	43.2%	27.6%
Table 11: Accuracy of the trained NER model to recognize the original and perturbed answer entities on SQuAD.