Experiments and Further Analyses ::: Results of Reading and Reasoning Skills	We report the results for the skills in Table TABREF10. In the following, % indicates a relative change from the original F1/accuracy unless specified otherwise. In this section, we describe the notable findings for several skills. The observations for all other skills are explained in Appendix F. $s_2$ and $s_3$: recognizing content words and function words. On all datasets, the relative changes for $s_2$ were greater than those for $s_3$. However, it is remarkable that even with function words alone, the model could achieve 53.0% and 17.4% F1 on CoQA and SQuAD v1.1, respectively. On ARC, RACE, and SWAG, the model showed more than 40% accuracy ($>$25% of random choice). As for content words only, on all answer extraction datasets, the performance was greater than 78.7% that of the original. On all multiple-choice datasets, it was more than 90.2%.

Experiments and Further Analyses ::: Further Analyses	To complement the observations in Section SECREF11, we performed further experiments as follows. The whole question and/or context ablation. To correctly interpret the result for $s_1$, we should know the performance on the empty questions. Likewise, for multiple-choice questions, the performance on the empty context should be investigated to reveal biases contained in the answer options. Therefore, we report the baseline results on the whole question and/or context ablations. Our results are reported in Table TABREF17. Although the performance on SQuAD v2.0 was relatively high, we found that the model predicted no answer for all of the questions (in this dataset, almost half of the questions are no answer). The other answer extraction datasets showed a relative drop of 80–90%. This result is not surprising since this setting forces the model to choose an answer span arbitrarily. On the multiple-choice datasets, on the other hand, the accuracies were higher than those of random choice (50% for MultiRC and 25% for the others), which implies that some bias exists in the context and/or the options. Training and evaluating on the modified context.

Assessing the Benchmarking Capacity of Machine Reading Comprehension Datasets	Existing analysis work in machine reading comprehension (MRC) is largely concerned with evaluating the capabilities of systems. However, the capabilities of datasets are not assessed for benchmarking language understanding precisely. We propose a semi-automated, ablation-based methodology for this challenge; By checking whether questions can be solved even after removing features associated with a skill requisite for language understanding, we evaluate to what degree the questions do not require the skill. Experiments on 10 datasets (e.g., CoQA, SQuAD v2.0, and RACE) with a strong baseline model show that, for example, the relative scores of a baseline model provided with content words only and with shuffled sentence words in the context are on average 89.2% and 78.5% of the original score, respectively. These results suggest that most of the questions already answered correctly by the model do not necessarily require grammatical and complex reasoning. For precise benchmarking, MRC datasets will need to take extra care in their design to ensure that questions can correctly evaluate the intended skills.

A question that was raised during the main analysis is what would happen if the model was trained on the modified input. For example, given that the performance with the content words only is high, we would like to know the upper bound performance when the model is forced to ignore function words also during training. Hence we trained the model with the ablations for the following skills: $s_3$ content words only; $s_6$ context word shuffle; and $s_7$ sentence word shuffle. The results are reported in the bottom rows of Table TABREF17. On almost all datasets, the baseline model trained on the ablation training set ($s_3^{\prime }$, $s_6^{\prime }$, and $s_7^{\prime }$) displayed higher scores than that on the original training set ($s_3$, $s_6$, and $s_7$). On CoQA, for instance, the relative change from the original score was only $-$8.3% when the model was trained on $s_3$ content words only.

Dataset Diagnosis by Input Ablation ::: Formulation	Our methodology uses a set of requisite skills and corresponding ablation methods. By checking the solvability of questions after applying the ablation methods, we can quantify to what degree the questions allow unintended solutions that do not require the requisite skills. Users can define an arbitrary set of skills to suit their purposes. We develop a method $\sigma _i$ that ablates features necessary for the corresponding skill $s_i$ in a set of requisite skills $S$. For $(x, y) \in X \times Y$, whenever $f(x) = y$, if $f(\sigma _i(x)) = y$, we recognize that $x$ is solvable without $s_i$. Here, $X$ is the input, $Y$ is the gold labels, $(x, y)$ is a pair consisting of an input instance and its gold-standard answer, and $f$ is a model. When the performance gap between the original and the modified dataset is small, we can infer that most of the questions already solved are solvable without $s_i$.

Experiments and Further Analyses ::: Experimental Settings	Datasets. We use 10 datasets. For answer extraction datasets in which a reader chooses a text span in a given context, we use (1) CoQA , (2) DuoRC , (3) HotpotQA (distractor) , (4) SQuAD v1.1 , and (5) SQuAD v2.0 . For multiple choice datasets in which a reader chooses a correct option from multiple options, we use (6) ARC (Challenge) , (7) MCTest , (8) MultiRC , (9) RACE , and (10) SWAG . For the main analysis, we applied our ablation methods to development sets. We included SWAG because its formulation can be viewed as a multiple-choice MRC task and we would like to analyze the reasons for the high performance reported for the baseline model on this dataset . For preprocessing the datasets, we use CoreNLP . We specify further details in Appendix B. Models. As the baseline model, we used BERT-large . We fine-tuned it on the original training set of each dataset and evaluated it on a modified development set.

Introduction	Machine reading comprehension (MRC) is a testbed for evaluating natural language understanding (NLU), by letting machines answer questions about given texts . Although MRC could be the most suitable task for evaluating NLU  and the performance of systems is comparable to humans on some existing datasets , it has been found that the quality of existing datasets might be insufficient for requiring precise understanding . Whereas these analyses are useful to investigate the performance of systems, however, it is still necessary to verify the fine-grained capabilities of datasets for benchmarking NLU. In the design of MRC datasets, it is implicitly assumed that questions test a cognitive process of language understanding . As various aspects of such a process, we can use requisite skills for answering questions such as coreference resolution and commonsense reasoning . Considering skills as metrics would be useful for analyzing datasets. However, for most datasets, the skills required to answer existing questions are not identified, or significant human annotation is needed. In this study, we propose a semi-automated, ablation-based methodology to analyze the capabilities of MRC datasets to benchmark NLU. Our motivation is to investigate to what extent a dataset allows unintended solutions that do not need requisite skills.

The annotation of the necessity also shows that, however, reconstructable features (function words in this case) for all the questions are not necessary to answer them (i.e., Case $\gamma $). Therefore, we could not find any question in Case $\delta $. We report the annotation results in Appendix H. It is not easy for the annotator to completely ignore the information of reconstructed features. We leave designing a solid, scalable annotation scheme for future work. In summary, we found that almost all ablated features are unreconstructable. Although for some questions ablated features are reconstructable for the content words only, these words are not necessarily required for answering the questions. Overall, this result supports our observations in Section SECREF4, i.e., questions already solved in existing datasets do not necessarily require complex language understanding.

Experimental Details	In this section, we provide details of the specifications used in our experiments. Datasets. For CoQA, since this dataset allows for yes/no/unknown questions, we appended these words to the end of the context. These special words were not allowed to be dropped. Additionally, we appended the previous question-answer pair prior to the current question so that the model can consider the history of the QA conversation. To compute the performance on SQuAD v2.0, we used the best F1 value that was derived from the predictions with a no-answer threshold of $0.0$. For DuoRC, we used the ParaRC dataset (the official preprocessed version provided by the authors). When training a model on DuoRC and HotpotQA, we used the first answer span; i.e., the document spans that have no answer span were not used in training. For MCTest and RACE, we computed accuracy by combining MC160 with MC500 and Middle with High, respectively.

Related Work	We briefly survey existing interpretation methods and skill-based analyses for NLU tasks. Interpretation methods. A challenge with the MRC task is that we do not know the extent to which a successful model precisely understands natural language. To analyze a model's behavior, existing studies mainly proposed modification of the input. For example,  showed that the performance of existing models on SQuAD  significantly degrades when manually verified distracting sentences are added to the given context. In addition,  demonstrated that MRC models do not necessarily change their predictions even when most question tokens are dropped. Likewise, for the natural language inference task,  proposed to hide the premise and to evaluate a model using only the hypothesis. These kinds of analyses are helpful for detecting biases that are unintentionally included in datasets. Nonetheless, to assure that a dataset can evaluate various aspects of NLU, more fine-grained detail is needed than what is allowed by inspection using existing methods. Skills as units of interpretation. In the topic of interpretable machine learning,  defined the concept of cognitive chunks as the basic units of explanation. In the MRC task, we consider that requisite skills to answer questions are appropriate as such units.

Our proposed approach can be expanded in two further directions: (i) inner-sentence and (ii) multiple-sentence levels. For (i), we can focus on understanding of specific linguistic phenomena. This includes logical and semantic understanding such as in FraCaS  and SuperGLUE . To investigate particular syntactic phenomena, we might be able to use existing analysis methods . For (ii), our skills can include complex/implicit reasoning, e.g., spatial reasoning  and lexically dependent causal reasoning . Although we do not need to include all of these skills in a single dataset, we need to consider the generalization of models across them.

For MultiRC, which is allowed to have multiple correct options for a question, we cast a pair consisting of a question and one option as a two-option multiple choice (i.e., whether its option is true or false) and computed the micro-averaged accuracy for the evaluation. The SWAG dataset is a multiple-choice task of predicting which event is most likely to occur next to a given sentence and the subject (noun phrase) of a subsequent event. We cast the first sentence as the context and the subject of the second sentence as the question. To compute F1 scores for the answer extraction datasets, we used the official evaluation scripts provided for the answer extraction datasets. Ablation methods. For $\sigma _4$ vocabulary anonymization, we used the tags as shown in Table TABREF23 and @other tags for the other POS tags. For $\sigma _{10}$ logical words dropped, as logic-related terms, we used the following: all, any, each, every, few, if, more, most, no, nor, not, other, same, some, and than.

We therefore perform the annotation with the following queries: (i) are ablated features reconstructable? and (ii) are reconstructable features really necessary for answering? When the answers for both queries are yes, a question is in Case $\delta $. In the annotation, we define that features in a question are reconstructable if the features existing around the rationale for answering the question are guessable. We also require that these features are necessary to decide the answer if the correct answer becomes undecidable without them. Results. For both datasets, the annotation shows that, not surprisingly, almost all features are unreconstructable in the shuffled sentence/context words and the vocabulary anonymization (except for one example in RACE). When these questions are solvable / unsolvable by humans, we can say that features are unnecessary (Case $\alpha $) / necessary (Case $\beta $) for answering the questions. In contrast, the annotators could guess function words for some questions even if these words are dropped (SQuAD: 55.0% and RACE: 15.0%).

This result means that most of the solved questions are solvable even with the sentence words shuffled. However, we should not say that all questions must require this skill; a question can require the performance of some complex reasoning (e.g., logical and multi-hop reasoning) and merely need to identify the sentence that gives the correct answer without precisely understanding that sentence. Nevertheless, if the question is not intended to require such reasoning, we should care whether it can be solved with only a (sentence-level) bag of words. In order to ensure that a model can understand the precise meaning of a described event, we may need to include questions to evaluate the grammatical and syntactic understanding into a dataset. $s_8$: discourse relation understanding (sentence order shuffle). The smallest drop, excluding SWAG, which has one context sentence, was $-$1.3%, on SQuAD v1.1. Except for HotpotQA, the datasets show small drops (less than 10%), which indicates that most solved questions do not require understanding of adjacent discourse relations and are solvable even if the sentences appear in an unnatural order. For SQuAD v2.0, we observed that the model recall increases for the no-answer questions.

Qualitative Evaluation ::: Reconstructability of Ablated Features	Motivation. We also seek to investigate the reconstructability of ablated features. Even if a question falls under Case A in the previous section, it might require the skill intended by the ablation; If a reader is able to guess the dropped information and uses it to solve the question, we cannot say that the question does not require the corresponding skill. For example, even after dropping function words ($\sigma _3$), we might be able to guess which function word to fill in a cloze based on grammaticality and lexical knowledge. Such reconstructable features possibly exist for some ablation methods. However, they are not critical if they are unnecessary for answering questions. We can list the following cases: ablated features are ($\alpha $) unreconstructable and unnecessary, ($\beta $) unreconstructable and necessary, ($\gamma $) reconstructable and unnecessary, and ($\delta $) reconstructable and necessary. To verify that ablation methods work, we need to confirm that there are few questions of Case $\delta $. The other cases are not critical to our observations in the main experiment.

Although $s_3^{\prime }$ and $s_7^{\prime }$ with RACE were exceptions, their learning did not converge within the specified number of epochs. We observed that for all datasets the relative upper bounds of performance were on average 92.5%, 80.1%, and 91.8% for $s_3$, $s_6$, and $s_7$, respectively. These results support our observations in Section SECREF11, that is, the questions allow solutions that do not necessarily require these skills, and thus fall short of testing precise NLU. Even without tuning on the ablation training set, however, our methods can make an optimistic estimation of questions that are possibly dubious for evaluating intended skills. Data leakage in BERT for SWAG. BERT's performance on SWAG is close to the performance by humans (88.0%). However, the questions and corresponding options for SWAG are generated by a language model trained on the BookCorpus , on which BERT's language model is also pretrained. We therefore suspect that there is severe data leakage in BERT's language model as reported in .

For $\sigma _4$ vocabulary anonymization, we train the model after the anonymization. For ARC, MCTest, and MultiRC, we fine-tuned a model that had already been trained on RACE to see the performance gained by transfer learning . We report the hyperparameters of our models in Appendix C. Although we trained the baseline model on the original training set, it is assumed that the upper-bound performance can be achieved by a model trained on the modified training set. Therefore, in Section SECREF16, we also see the extent to which the performance improves when the model is trained on the modified training set. Ablation methods. $\sigma _2$ and $\sigma _3$: we use a set of stopwords from NLTK  as function words. All other words are regarded as content words. We do not drop punctuation. When a token is dropped, it is replaced with an [UNK] token to preserve the correct answer span. $\sigma _4$: we use the same ID for the same word in a single given context but different IDs for different contexts. For inflectional words, we anonymize them using their lemma.

Statistics of the Examined MRC Datasets	Table TABREF27 shows the statistics for the examined MRC datasets.

On the other hand, the performance on the multiple choice datasets was significantly worse; if multiple choices do not have sufficient word overlap with the given context, there is no way to infer the correct answer option. Therefore, this result shows that multiple choice datasets might have a capacity for requiring more complex understanding beyond matching patterns between the question and the context than the answer extraction datasets. $s_6$: recognizing the context word order (context words shuffle). We found that for the answer extraction datasets, the relative performance decreased by 55.6% on average. A moderate number of questions are solvable even with the context words shuffled. We also found that, surprisingly, the average decrease was 21.3% for the multiple choice datasets. The drop on MCTest is more prominent than that on the others. We posit that this is because its limited vocabulary makes questions more context dependent. ARC, in contrast, uses factoid texts, and appears less context dependent. $s_7$: grasping sentence-level compositionality (sentence words shuffle). The performance with sentence words shuffled was greater than 60% and 80% those of the original dataset on the answer extraction and multiple-choice datasets, respectively.

Conclusion	Existing analysis work in MRC is largely concerned with evaluating the capabilities of systems. By contrast, in this work, we proposed an analysis methodology for the benchmarking capacity of datasets. Our methodology consists of input-ablation tests, in which each ablation method is associated with a skill requisite for MRC. We exemplified 12 skills and analyzed 10 datasets. The experimental results suggest that for benchmarking sophisticated NLU, datasets should be more carefully designed to ensure that questions correctly evaluate the intended skills. In future work, we will develop a skill-oriented method for crowdsourcing questions.

Full Observations of the Main Results	In this appendix, we describe the results for the reading and reasoning skills not mentioned in Section 4.2. $s_1$: recognizing question words. For the first four answer-extraction datasets, the performance decreased by more than 70%. For the multiple-choice datasets, the performance decreased by an average of 23.9%. $s_5$: attending to the whole context other than similar sentences. Even with only the most similar sentences, the baseline models achieved a performance level greater than half their original performances in 8 out of 10 datasets. In contrast, HotpotQA showed the largest decrease in performance. This result reflects the fact that this dataset contains questions requiring multi-hop reasoning across multiple sentences. $s_9$–$s_{12}$: various types of reasoning. For these skills, we can see that the performance drops were small; given that the drop for $s_3$ recognizing content words alone was under 20%, we can infer that specific types of reasoning might not be critical for answering the questions.

A limitation of our method is that it can not draw conclusions regarding questions that remain unsolved, and thus we need to assume a reasonable level of performance for existing models on the dataset to be analysed. Given our findings, we posit that MRC datasets should be carefully designed, e.g., by filtering questions using methods such as the ones we propose, so that their questions correctly benchmark the intended NLU skills.

Qualitative Evaluation	In this section, we qualitatively investigate our ablation methods in terms of the human solvability of questions and the reconstructability of ablated features. We analyze questions of SQuAD v1.1 and RACE which cover both answering styles and are influential in the community. We randomly sampled 20 questions from each dataset that are correctly solved (100% F1 and accuracy) by the baseline model on the original datasets. Our analysis covers four ablation methods ($\sigma _3$ content words only (involving $\sigma _{10,11,12}$), $\sigma _4$ vocabulary anonymization, $\sigma _6$ context word shuffle, and $\sigma _7$ sentence word shuffle) which provided specific insights in Section SECREF4.

Some types of reasoning, however, might play an essential role for some datasets: $s_9$ numerical reasoning in HotpotQA (whose questions sometimes require answers with numbers) and $s_{11}$ pronoun coreference resolution in DuoRC (consisting of movie scripts).

In Section SECREF5, we perform a complementary inspection of questions with our ablation methods in terms of the solvability of questions and the reconstructability of ablated features. Finally we discuss, in Section SECREF6, two requirements for developing MRC to benchmark NLU: the control of question solvability and the comprehensiveness of requisite skills. Our contributions are as follows: We propose a semi-automated methodology to analyze the benchmarking capacity of MRC datasets in terms of requisite skills for answering questions. With an example set of 12 skills and corresponding input-ablation methods, we use our methodology and examine 10 existing datasets with two answering styles. Our analysis shows that the relative performance on questions with content words only, shuffled sentence words, and shuffled sentence order averaged 89.2%, 78.5%, and 95.4% of the original performance, indicating that the questions might be inadequate for evaluating grammatical and complex reasoning. These results suggest that most of the questions currently solved in MRC may be insufficient for evaluating various skills.

These results imply that most of the questions already solved do not necessarily require grammatical and syntactic reasoning, in which function words are used. $s_4$: recognizing vocabulary beyond POS tags. Surprisingly, for SQuAD v1.1, the baseline model achieved 61.2% F1. It only uses 248 tokens as the vocabulary with the anonymization tags and no other actual tokens. For the other answer extraction datasets, the largest drop (73.6% relative) is by HotpotQA; it has longer context documents than the other datasets, which seemingly makes its questions more difficult. To verify the effect of its longer documents, we also evaluated the baseline model on HotpotQA without distracting paragraphs. We found that the model's performance was 56.4% F1 (the original performance was 76.3% F1 and its relative drop was 26.1%) which is much higher than that on the context with distracting paragraphs (16.8% F1). This indicates that adding longer distracting documents contributes to encouraging machines to understand a given context beyond matching word patterns.

We therefore perform the annotation of questions with human solvability; We define that a question is solvable if a reasonable rationale for answering the question can be found in the context. Results. Table TABREF20 shows the human solvability along with the baseline model's performance on the sampled questions. The model's performance is taken from the model trained on the original datasets except for the vocabulary anonymization method. For the content words only on both datasets, the human solvability is higher than the baseline performance. Although these gaps are not significant, we might be able to infer that the baseline model relies on content words more than humans (Case B). Given that the high performance of both humans and the baseline model, most of the questions fall into Case A, i.e., they are easy and do not necessarily require complex reasoning involving the understanding of function words. For the other three methods, the human solvability is lower than the baseline performance. This result indicates that the questions correctly solved only by the baseline model may contain unintended biases (Case C). For example, the gap in the context word shuffle of RACE is significant (30.0% vs. 75.0%).

This leads to the following intuition: if a question is correctly answered (or solvable) even after removing features associated with a given skill, the question does not require the skill. We show an example of our ablation method in Figure FIGREF1. Suppose we wish to analyze a dataset's capacity to evaluate understanding of texts beyond the information of part-of-speech (POS) tags. To this end, we replace context and question words with POS tags and ID numbers. If a model can still correctly answer this modified question, the question does not necessarily require deep understanding of texts but matching word patterns only. Questions of this kind might be insufficient for developing a model that understands texts deeply as they may reduce models to recognizing superficial word overlaps. Our methodology uses a set of requisite skills and corresponding ablation methods. Inspired by the computational model of reading comprehension , we exemplify 12 skills on two classes: reading and reasoning (Section SECREF3). Then, we present a large-scale analysis over 10 existing datasets using a strong baseline model (Section SECREF4).

For example, are would be replaced with @verb2 (= is) if it appeared in Figure FIGREF1. In addition, to retain the information of the POS tags, we append its POS tag after each inflectional anonymized word (e.g., is is replaced with @verb{ID} [VBZ]). $\sigma _6$: because it is necessary to maintain the correct answer span in the answer extraction datasets, we split the context into segments that have the same length as the gold answer span and shuffle them. $\sigma _7$: as with $\sigma _6$, we split each sentence into segments and shuffle them within each sentence. For $\sigma _6$ to $\sigma _8$, we averaged the scores over five runs with different seeds and report their variances in Appendix D.

Qualitative Evaluation ::: Human Solvability after the Ablation	Motivation. In Section SECREF4, we observed that the baseline model exhibits remarkably high performance on some ablation tests. To interpret this result, we investigate if a question is solvable by humans and the model. Concretely, the question after the ablation can be (A) solvable by both humans and the model, (B) solvable by humans but unsolvable by the model, (C) unsolvable by humans but solvable by the model, or (D) unsolvable by both humans and the model. For Case A, the question is easy and does not require complex language understanding. For Cases B and C, the model may use unintended solutions because (B) it does not use the same solution as humans or (C) it cleverly uses biases that humans cannot recognize. For Case D, the question may require the skill intended by the ablation method. Although Cases A to C are undesirable for evaluating the systems' skills, it seems to be useful to distinguish them for further improvement of the dataset creation.

Dataset Diagnosis by Input Ablation ::: Example Set of Requisite Skills	In this section, we exemplify a skill set that consists of 12 skills along with two classes; reading and reasoning (Table TABREF5). In psychology, there is a tradition of theoretical research on human text comprehension. The construction–integration model  is one of the most acknowledged theories. This model assumes that human text comprehension consists of two processes: (i) construction, in which a reader elaborates concepts and propositions in the text and (ii) integration, in which the reader associates the propositions to understand them consistently. We associate this two-step process with our two classes. Reading skills. This class deals with six skills of observing and recognizing word appearances, which are performed before reasoning. In MRC, it has been shown that some existing questions can be solved by reading a limited number of words in the question and the context (e.g., by simply attending to context tokens that are similar to those of the questions ). Our goal of this class is, therefore, to ensure that the questions require the reading of the whole question and context uniformly. Reasoning skills.

Discussion	In this section, we discuss two requirements for developing the MRC task as an NLU benchmark. The control of question solvability. Not to allow the model to focus on unintended objectives, we need to ensure that each question is unsolvable without its intended requisite skill. Therefore, when benchmarking, we first need to identify necessary features whose presence determines the question's solvability. To identify them, we might need to perform ablation testing with humans. Further, we need to evaluate a model in both regular and ablation settings. This is because a model may detect some biases that enable it to solve the question; such biases can actually be false for humans and may be acquired by the model through overfitting to datasets. Nonetheless, there is a case in which, even if we can identify necessary features, the model can have prior, true knowledge (e.g., world knowledge) of the correct answer. In this case, the model can answer the question without the context. To avoid this circumvention, we may need to evaluate the model on fictional texts. Comprehensiveness of requisite skills. Another aspect of NLU benchmarking is the comprehensiveness of skills.

On the other hand, if the gap is large, a sizable proportion of the solved questions may require $s_i$. We note that we cannot draw general conclusions for instances given by conditions other than the abovementioned one. Consider the case where $f(x) = y$ and $f(\sigma _i(x)) \ne y$, for example. This only means that $f$ cannot solve $x$ without the features ablated by $\sigma _i$. We cannot conclude that $x$ requires $s_i$ in every model because there might exist a model that can solve $x$ without $s_i$. However, if there is at least one model $f$ that solves $x$ without $s_i$, this may indicate an unintended way to solve $x$ while ignoring $s_i$. Therefore our methodology only requires a single baseline model. Users can choose an arbitrary model for their purposes.

This class comprises six skills of relational reasoning among described entities and events such as pronoun coreference resolution and logical reasoning. Although these skills are essential for sophisticated NLU, it is difficult to precisely determine whether these types of reasoning are genuinely required in answering a question. Therefore, in this class, we define reasoning-related skills that are performed using the explicit information contained in the context (e.g., $s_9$ explicit logical reasoning and $s_{12}$ reasoning about explicit causality). In the following, we highlight some of the defined skills. Skill $s_1$ is inspired by  and . Although their studies proposed dropping question tokens based on their model-based importance or the question length, we instead drop tokens other than interrogatives as interpretable features. Our vocabulary anonymization ($s_4$) is mainly inspired by  where they anonymized named entities to make their MRC task independent of prior knowledge. Our shuffle-based methods ($s_6$ to $s_8$) are inspired by existing analyses for other tasks , , . Among them, our purpose for $s_7$ is to analyze whether a question requires precise reasoning performed over syntactic and grammatical aspects in each sentence.

The Annotation Results	Table TABREF29 shows the frequency of questions for Cases $\alpha $ to $\delta $ for SQuAD v1.1 and RACE. See Section 5.2 for details.

The remaining skills are described in Appendix A. Although our proposed definitions can be extended, they are sufficient for the purpose of demonstrating and evaluating our approach. In Section SECREF6, we discuss further directions to develop purpose-oriented skill sets.

Because F1 score is computed between the has- and no-answer question subsets, the scores tend to be higher than those for SQuAD v1.1.

Detailed Results of SQuAD v2.0	We report the ablation results for has-answer and no-answer questions in SQuAD v2.0 in Table TABREF28.

To confirm this issue, we trained a model without the context (i.e., the first given sentence). The accuracy on the development set, which was also without the context, was 74.9% (a relative decrease of 12.2%). This result suggests that we need to pay more attention to the relations of corpora on which a model is trained and evaluated, but leave further analysis for future work.

Figure FIGREF21 shows a question that is unsolvable for humans but can be solved by the baseline model. We conjecture that while humans cannot detect biases easily, the model can exploit biases contained in the answer options and their relations to the given context.

Acknowledgments	We would like to thank Max Bartolo, Pasquale Minervini, and the anonymous reviewers for their insightful comments. This work was supported by JSPS KAKENHI Grant Numbers 18H03297 and 18J12960 and JST ACT-X Grant Number JPMJAX190G.

Our Defined Requisite Skills	Reading skills. As $s_2$ and $s_3$, we propose limiting the information available in the context by dropping content and function words respectively, which is intended to ascertain the extent to which a question depends on the given word type (e.g., a preposition in before a time-related expression for a when question). Skill $s_5$ provides a heuristic of the relative levels of attention between a question and the context. Skill $s_6$ is used to ensure that a model can extract the information conditioned on the word order. Reasoning skills. Skill $s_8$ is for the understanding of discourse relations between adjacent sentences, which relies on information given by the sentence order in the context. When we shuffle the sentence order, various relations, such as causality and temporality, are expected to be broken. Skills $s_9$ to $s_{12}$ are defined more specifically; we drop tokens that explicitly emphasize important roles in specific skills such as if and not in logical reasoning.

For $\sigma _{12}$ causal words dropped, as causality-related terms, we used the following: as, because, cause, since, therefore, and why. For $\sigma _3^{\prime }$ training with content words only, we dropped function words as well as punctuation marks so that the model would see only content words. We show examples of questions for the ablation method $\sigma _{4}$ in Figure FIGREF24.

Hyperparameters of the Baseline Model	Hyperparameters used in the baseline model are shown in Table TABREF25.

Performance Variances in Shuffle Methods	We report the variance for shuffling methods $s_6$ context words shuffle, $s_7$ sentence words shuffle, and $s_8$ sentence order shuffle in Table TABREF26.

A skill-based analysis was conducted by , who proposed classifications of knowledge and reasoning. Prior to this,  also defined a set of 13 requisite skills. However, there are two main issues with these approaches: (i) the human annotation does not necessarily reveal unintended biases that machines can make use of, and (ii) it requires costly annotation efforts. Therefore, we posit that a machine-based analysis is needed and that it should be performed in an automated manner.

Question: what are all the datasets they experiment with?
Answer: CoQA , DuoRC , HotpotQA , SQuAD v1.1 , SQuAD v2.0, ARC (Challenge), MCTest , MultiRC , RACE , SWAG