Saku Sugawara*, Yusuke Kido*, Hikaru Yokono*,  and  Akiko Aizawa◇◆ ↑The University of Tokyo, 7-3-1 Hongo, Bunkyo-ku, Tokyo, Japan ↑Fujitsu Laboratories Ltd., 4-1-1 Kamikodanaka, Nakahara-ku, Kawasaki, Japan ◇National Institute of Informatics, 2-1-2 Hitotsubashi, Chiyoda-ku, Tokyo, Japan sakus@is.s.u-tokyo.ac.jp mail@yusuk.eki.do aizawa@nii.ac.jp yokono.hikaru@jp.fujitsu.com
Abstract
Knowing the quality of reading compre- hension (RC) datasets is important for the development of natural-language un- derstanding systems.  In this study, two classes of metrics were adopted for eval- uating RC datasets: prerequisite skills and readability. We applied these classes to six existing datasets, including MCTest and SQuAD, and highlighted the characteris- tics of the datasets according to each met- ric and the correlation between the two classes. Our dataset analysis suggests that the readability of RC datasets does not di- rectly affect the question difficulty and that it is possible to create an RC dataset that is easy to read but difficult to answer.
Introduction
1
A major goal of natural language processing (NLP) is to develop agents that can understand natural language.  Such an ability can be tested with a reading comprehension (RC) task that re- quires the agent to read open-domain documents and answer questions about them.  Constructing systems with RC competence is challenging be- cause RC comprises multiple processes includ- ing parsing, understanding cohesion, and infer- ence with linguistic and general knowledge. Clarifying what a system achieves is important in the development of RC systems.  To achieve robust improvement, systems should be measured according to a variety of metrics beyond simple accuracy.  However, a current problem is that most RC datasets are presented only with superfi- cial categories, such as question types (e.g., what, where, and who) and answer types (e.g., numeric, location, and person).  In addition, Chen et al. (2016) noted that some questions in datasets may not be suited to the testing of RC systems. In such Figure 1: Examples of RC questions from SQuAD (Rajpurkar et al., 2016) and MCTest (Richardson et al., 2013) (the Contexts are excerpts). situations, it is difficult to obtain an accurate as- sessment of the RC system. Norvig (1989) argued that questions that are easy for humans to answer often turn out to be difficult for machines.  For example, consider the two RC questions in Figure 1.  The first example is from SQuAD (Rajpurkar et al., 2016), although the document is taken from a Wikipedia article and was therefore written for adults.  The question is answerable simply by noticing one sentence, with- out needing to fully understand the content of the text.  On the other hand, consider the second exam- ple from MCTest (Richardson et al., 2013 ), which was written for children and is easy to read. Here, answering the question involves gathering infor- mation from multiple sentences and utilizing a combination of several skills, such as understand- ing causal relations (Sara wanted… → they went to…), coreference resolution ( Sara and Her Dad = they), and complementing ellipsis ( baseball team = team). These two examples show that the read- ability of the text does not necessarily correlate with the difficulty of answering questions about it. Furthermore, the accompanying categories of ex- isting RC datasets cannot help with the analysis of this issue. In this study, our goal is to investigate how these two types of difficulty, namely "answering ques- tions" and "reading text," are correlated in RC. Corresponding to each type, we formalize two classes of evaluation metrics, prerequisite skills and readability , and analyze existing RC datasets. Our intention is to provide the basis of an eval- uation methodology of RC systems to help their robust development. Our two classes of metrics are inspired by the analysis in McNamara and Magliano (2009) of human text comprehension in psychology. They considered two aspects of text comprehen- sion, namely "strategic/skilled comprehension" and “text ease of processing.” Our first class defines metrics for "strate- gic/skilled comprehension," namely the difficulty of comprehending the context when answering questions.  We adopted the set of prerequisite skills that Sugawara et al. (2017) proposed for the fine- grained analysis of RC capability. Their study also presented an important observation of the relation between the difficulty of an RC task and prereq- uisite skills: the more skills that are required to answer a question, the more difficult is the ques- tion.  Based on this observation, in this work, we assume that the number of skills required to an- swer a question is a reasonable indication of the difficulty of the question.  This is because each skill corresponds to one of the functions of an NLP system, which has to be capable of that function- ality. Our second class defines metrics for "text ease of processing," namely the difficulty of reading the text.  We regard it as readability of the text in terms of syntactic and lexical complexity.  From among readability studies in NLP, we adopt a wide range of linguistic features proposed by Vajjala and Meurers (2012), which can be used for texts with no available annotations. The contributions of this paper are as follows. 1. We adopt two classes of evaluation metrics to show the qualitative features of RC datasets. Through analyses of RC datasets, we demon- strate that there is only a weak correlation be- tween the difficulty of questions and the read- ability of context texts in RC datasets. 2. We revise a previous classification of pre- requisite skills for RC. Specifically, skills of knowledge reasoning are organized by us- ing insights of entailment phenomena in NLP and human text comprehension in psychol- ogy. 3. We annotate six existing RC datasets, com- pared to the two datasets considered in Sug- awara and Aizawa (2016), with our organized metrics being used in the comparison.  We have made the results publicly available¹ and report on the characteristics of the datasets and the differences between them. We should note that, in this study, RC datasets with different task formulations were annotated with prerequisite skills under the same conditions. Annotators first saw a context, a question, and its answer.  They selected the sentences required to provide the answer, and then annotated them with appropriate prerequisite skills.  That is, the datasets were annotated from the point of view of whether the context entailed the hypothesis con¬ structed from the pair of the question and answer. This means that our methodology cannot quantify the systems' competence in searching the context for necessary sentences and answer candidates. In other words, our methodology can be only used to evaluate the competence of understanding RC questions as contextual entailments. The remainder of this paper is divided into the following sections. First, we discuss related work in Section 2. Next, we specify our two classes of metrics in Section 3.  In Section 4, we annotate existing RC datasets with the prerequisite skills. Section 5 gives the results of our dataset analysis and Section 6 discusses their implications.  Section 7 presents our conclusions.
Related Work
2
2.1
Reading Comprehension Datasets
In this section, we present a short history of RC datasets.  To our knowledge, Hirschman et al. (1999) were the first to use NLP methods for RC. Their dataset comprised reading materials for grades 3–6 with simple 5W ( wh-) questions. Sub- sequent investigations into questions of natural language understanding focused on other formu- lations, such as question answering (Yang et al., 2015; Wang et al., 2007; Voorhees et al., 1999) and textual entailment (Bentivogli et al., 2010; Sam- mons et al., 2010; Dagan et al., 2006).  One of the RC tasks of the time was QA4MRE (Sutcliffe et al., 2013).  The highest accuracy achieved for this task was 59% and the size of the dataset was very limited: there were only 224 gold-standard questions, which is insufficient for machine learn- ing methods. This means that an important issue for design- ing RC datasets is their scalability.  Richardson et al. (2013) presented MCTest, which is an open- domain narrative dataset for gauging comprehen- sion at a child's level.  This dataset was created by crowdsourcing and was based on a scalable methodology.  Since then, additional large-scale datasets have been proposed with the development of machine learning methods in NLP. For exam- ple, the CNN/Daily Mail dataset (Hermann et al., 2015) and CBTest (Hill et al., 2016) have approx- imately 1.4M and 688K passages, respectively. These context texts and questions were automat- ically curated and generated from large corpora. However, Chen et al. (2016) indicated that approx- imately 25% of the questions in the CNN/Daily Mail dataset are either unsolvable or nonsensical. This dataset-quality issue highlights the demand for more stable and robust sourcing methods. Several additional RC datasets were presented in the last half of 2016, involving large docu- ments and sensible queries that were guaranteed by crowdsourcing or other human testing.  They were intended to provide large and high-quality content for machine learning models.  Nonethe- less, as shown in the examples of Figure 1, they were not offered with metrics that could evaluate NLP systems adequately with respect to the diffi- culty of questions and the surface features of texts.
Reading Comprehension in Psychology
2.2
In psychology, there is a rich tradition of research on human text comprehension. The construction– integration (C–I) model (Kintsch, 1988) is one of the most basic and influential theories. This model assumes a connectional and computational archi- tecture for text comprehension.  It assumes that comprehension is the processing of information based on the following two steps. 2 1. Construction: read sentences or clauses as in- puts; form and elaborate concepts and propo-
2. Integration:  associate the contents to un- derstand them consistently (e.g., coreference, discourse, and coherence). During these steps, three levels of representa- tion are constructed (van Dijk and Kintsch, 1983): the surface code (i.e., wording and syntax), the textbase (i.e., text propositions with cohesion), and the situation model (i.e., mental represen- tation). Based on these assumptions, McNa- mara and Magliano (2009) proposed two aspects of text comprehension, namely "strategic/skilled comprehension” and “text ease of processing.” We adopted these assumptions as the basis of our two classes of evaluation metrics (Section 3 ). In an alternative approach, Kintsch (1993) pro- posed two dichotomies for the classification of human inferences, including the knowledge-based inference assumed in the C–I model. The first di- chotomy is between inferences that are automatic and those that are controlled. However, Graesser et al. (1994) indicated that this distinction is am- biguous, because there is a continuum between the two states that depends on individuals. Therefore, this dichotomy is unsuited to empirical evaluation, which is our focus. The second dichotomy is be- tween inferences that are retrieved and those that are generated. Retrieved means that the informa- tion used for inference is retrieved entirely from the context. In contrast, when inferences are gen- erated, the reader uses external knowledge that goes beyond the context. A similar distinction was proposed by McNa- mara and Magliano (2009), namely that between bridging and elaboration.  A bridging inference connects current information to other information that has been encountered previously. Elaboration connects current information to external knowl- edge that is not included in the context.  We use these two types of inference in the classification of knowledge reasoning.
3
Evaluation Metrics for Datasets
Following the depiction of text comprehension by McNamara and Magliano (2009), we adopted two classes for the evaluation of RC datasets: prereq- uisite skills and readability. For the prerequisite skills class (Section 3.1 ), we refined RC skills that were proposed by Sug- awara et al. (2017) and Sugawara and Aizawa (2016).  However, a problem in these studies is that their categorization of knowledge reasoning was provisional and with a weak theoretical back- ground. Therefore, in this study, we reorganized the cat- egory of knowledge reasoning in terms of textual entailment in NLP and human text comprehension in psychology.  In research on textual entailment, several methodologies have been proposed for the precise analysis of entailment phenomena (Dagan et al., 2013 ; LoBue and Yates, 2011 ). In psychol- ogy research, as described in Section 2.2, McNa- mara and Magliano (2009) proposed a similar dis- tinction for inferences: bridging versus elabora- tion.  We utilized these insights in developing a comprehensive but not overly specific classifica- tion of knowledge reasoning. Our prerequisite skills class includes the textbase and situation model (van Dijk and Kintsch, 1983).  In our terminology, this means understanding each fact and associating multiple facts in a text, such as the relations of events, char- acters, or the topic of a story. The skills also in- volve knowledge reasoning, which is divided into several metrics according to the distinctions of hu- man inferences. This point is discussed by Kintsch (1993) and McNamara and Magliano (2009).  It also accords with the classification of entailment phenomena by Dagan et al. (2013) and LoBue and Yates (2011). Readability metrics (Section 3.2) are quantita- tive measures used to assess the difficulty of read- ing, with respect to vocabulary and the complexity of texts.  In this study, they measure the compe- tence in understanding the first basic representa- tion of a text, called the surface code (van Dijk and Kintsch, 1983).
3.1
Prerequisite Skills
Based on the 10 RC skills in Sugawara et al. (2017), we identified 13 prerequisite skills, which are presented below.  (We use * and † to indicate skills that have been modified/elaborated from the original definition or have been newly introduced in this study, respectively.) 1. Object tracking* : jointly tracking or grasp- ing of multiple objects, including sets or member- ships (Clark, 1975).  This skill is a version of the list/enumeration used in the original classification, renamed to emphasize its scope with respect to multiple objects. 2. Mathematical reasoning*: we merged sta- tistical and quantitative reasoning with mathemat- ical reasoning.  This skill is a renamed version of mathematical operations. 3.  Coreference resolution*:  this skill has a small modification to include an anaphora (Dagan et al., 2013). It is similar to direct reference (Clark, 1975). 4. Logical reasoning*: we identified this skill as the understanding of predicate logic, e.g., con- ditionals, quantifiers, negation, and transitivity. Note that this skill, together with mathematical reasoning, is intended to align with the offline skills described by Graesser et al. (1994). 5. Analogy*: understanding of metaphors in- cluding metonymy and synecdoche (see LoBue and Yates (2011) for examples of synecdoche.) 6.  Causal relation: understanding of causal- ity that is represented by explicit expressions such as “why,” “because,” and “the reason for” (only if they exist). 7. Spatiotemporal relation: understanding of spatial and/or temporal relationships between mul- tiple entities, events, and states. In addition, we propose the following four cate- gories by refining the "commonsense reasoning" category proposed originally in Sugawara et al. (2017). 8. Ellipsis†: recognizing implicit/omitted infor- mation (argument, predicate, quantifier, time, or place). This skill is inspired by Dagan et al. (2013) and the discussion in Sugawara et al. (2017). 9. Bridging † : inference supported by grammat- ical and lexical knowledge (e.g., synonymy, hy- pernymy, thematic role, part of events, idioms, and apposition). This skill is inspired by the concept of indirect reference in the literature (Clark, 1975). Note that we exclude direct reference because it is covered by coreference resolution (pronominal- ization) and elaboration (epithets). 10. Elaboration† : inference using known facts, general knowledge (e.g., kinship, exchange, typi- cal event sequence, and naming), and implicit re- lations (e.g., noun compounds and possessives) (see Dagan et al. (2013) for details).  Bridging and elaboration are distinguished by the knowl- edge used in inferences being grammatical/lexical or general/commonsense, respectively. 11.  Meta-knowledge † :  using  knowledge  that includes a reader, writer, or text genre (e.g., nar- ratives and expository documents) from meta- viewpoints (e.g., Who are the principal charac- ters of the story?  or What is the main subject of this article?). Although this skill can be regarded as part of elaboration , we defined it as an inde- pendent skill because this knowledge is specific to RC. We were motivated by the discussion in Smith et al. (2015). Whereas the above 11 skills involve multiple items, the final pair of skills involve only a single sentence. 12. Schematic clause relation : understand- ing of complex sentences that have coordination or subordination, including relative clauses. 13.  Punctuation*:  understanding  of  punctu- ation marks (e.g., parenthesis, dash, quotation, colon, or semicolon). This skill is a renamed ver- sion of special sentence structure . Concerning the original definition, we regarded "scheme" in fig- ures of speech as ambiguous and excluded it. We defined ellipsis as a independent skill, and appo- sition was merged into bridging.  Similarly, un- derstanding of constructions was merged into the idioms in bridging. Note that we did not construct this classification to be dependent on particular RC systems in NLP. This was because our methodology is intended to be general and applicable to many kinds of archi- tectures. For example, we did not consider the di- chotomy between automatic and controlled infer- ences because the usage of knowledge is not nec- essarily the same for all RC systems.
Readability Metrics
3.2
In this study, we evaluated the readability of texts based on metrics in NLP. Several studies have ex- amined readability in various applications, such as second-language learning (Razon and Barnden, 2015) and text simplification (Aluisio et al., 2010), and from various aspects, such as development measures in second-language acquisition (Vajjala and Meurers, 2012) and discourse relations (Pitler and Nenkova, 2008). Of these, we adopted the classification of lin- guistic features proposed by Vajjala and Meurers (2012). This was because they presented a com- parison of a wide range of linguistic features fo- cusing on second-language acquisition and their method can be applied to plain text. 3 We list the readability metrics in Table 1, which were reported by Vajjala and Meurers (2012) as - Ave. no. of characters per word ( NumChar ) - Ave. no. of syllables per word ( NumSyll ) - Ave. sentence length in words (MLS) -  Proportion of words in AWL (AWL) -  Modifier variation (ModVar) - No. of coordinate phrases per sentence ( CoOrd ) -  Coleman–Liau index (Coleman) - Dependent clause-to-clause ratio ( DC/C ) -  Complex nominals per clause ( CN/C ) -  Adverb variation (AdvVar) Table 1: Readability metrics. AWL refers to the Academic Word List. 4 the top 10 features that affect human readabil- To classify these metrics, we can identify ity. three classes:  lexical  features ( NumChar ,  Num- Syll , AWL , AdvVar , and ModVar ), syntactic fea- tures ( MLS , CoOrd , DC/C , and CN/C ), and tradi- tional features ( Coleman ). We applied these met- rics only to sentences that needed to be read in an- swering questions. However, because these metrics were proposed for human readability, they do not necessarily cor- relate with those used in RC systems.  Therefore, in any system analysis, ideally we would have to consult a variety of features.
4
Annotation of Reading Comprehension
Datasets
We annotated six existing RC datasets with the prerequisite skills.  We explain the annotation procedure in Section 4.1 and the annotated RC datasets in Section 4.2.
4.1
Annotation Procedure
We prepared annotation guidelines according to Sugawara et al. (2017). The guidelines include the definitions and examples of the skills and annota- tion instructions. Four annotators were asked to simulate the pro- cess of answering questions in RC datasets, using only the prerequisite skills, and to annotate ques- tions with one or more skills required in answer- ing.  For each task in the datasets, the annotators saw simultaneously the context, question, and its answer. When a dataset contained multiple-choice questions, we showed all candidate answers and labeled the correct one with an asterisk.  The an-
notators then selected the sentences that needed to be read to be able to answer the question and de- cided on the set of prerequisite skills required. The annotators were allowed to select nonsense for unsolvable or unanswerable questions (e.g., the "coreference error" and "ambiguous" ques- tions described in Chen et al. (2016)) to distin- guish them from any solvable questions that re- quired no skills.
4.2
Datasets
As summarized in Table 2, the annotation was per- formed on six existing RC datasets:  QA4MRE (Sutcliffe et al., 2013 ), MCTest (Richardson et al., 2013), SQuAD (Rajpurkar et al., 2016), Who-did- What (Onishi et al., 2016), MS MARCO (Nguyen et al., 2016), and NewsQA (Trischler et al., 2016). We selected these datasets to enable coverage of a variety of genres, query sourcing methods, and task formulations.  From each dataset, we ran- domly selected 100 questions.  This number was considered sufficient for the degree of analysis of RC datasets performed by Chen et al. (2016). The questions were sampled from the gold-standard dataset of QA4MRE and the development sets of the other RC datasets. (We explain the method of choosing questions for the annotation in Appendix A.) For a variety of reasons, there were other datasets we did not annotate in this study. CNN/Daily Mail (Hermann et al., 2015) is anonymized and contains errors, according to Chen et al. (2016), making it unsuitable for anno- tation. We considered CBTest (Hill et al., 2016) to be devised as language-modeling tasks rather than RC-related tasks.  LAMBADA (Paperno et al.,
QA4MRE MCTest SQuAD WDW MARCO NewsQA
2016) texts are formatted for machine reading, with all tokens in lower case, which would seem to disallow inferences based on proper nouns and render them unsuitable for human reading and an- notation.
5
Results of the Dataset Analysis
We now present the results of evaluating the RC datasets according to the two classes of metrics. In the annotation of prerequisite skills, the inter- annotator agreement was 90.1% for 62 randomly sampled questions. The evaluation was performed with respect to the following four aspects:  (i) fre- quencies of prerequisite skills required for each RC dataset; (ii) number of prerequisite skills re- quired per question; (iii) readability metrics for each RC dataset; and (iv) correlation between readability metrics and the number of required prerequisite skills. (i) Frequencies of prerequisite skills (see Ta- ble 3):  QA4MRE had the highest scores for fre- quencies among the datasets. This seems to reflect
the fact that QA4MRE involves technical docu- ments that contain a wide range of knowledge, multiple clauses, and punctuation.  Moreover, the questions are devised by experts. MCTest achieved a high score for several skills (best for causal relation and meta-knowledge and second-best for coreference resolution and spa- tiotemporal relation), but a low score for punctua- tion. These scores seem to be because the MCTest dataset consists of narratives. Another dataset that achieved notable scores is Who-did-What.  This dataset achieved the highest score for ellipsis . This is because the questions of Who-did-What are automatically generated from articles not used as context.  This methodology tends to avoid textual overlap between a question and its context, thereby requiring frequently the skills of ellipsis, bridging, and elaboration. With regard to nonsense , MS MARCO and Who-did-What received relatively high scores. This appears to have been caused by the automated sourcing methods, which may generate a separa- tion between the contents of the context and ques- tion (i.e., web segments and a search query in MS MARCO, and a context article and question article in Who-did-What).  In contrast, NewsQA had no nonsense questions.  Although this result was af- fected by our filtering (described in Appendix A), it is important to note that the NewsQA dataset in- cludes annotations of meta-information whether or not a question makes sense ( is_question_bad ). (ii) Number of required prerequisite skills (see Table 4):  QA4MRE had the highest score. On average, each question required 3.25 skills. There were few questions in QA4MRE that re- quired zero or one skill, whereas such questions were contained more frequently in other datasets. Table 4 also indicates that more than 90% of the MS MARCO questions required fewer than three skills according to the annotation. (iii) Readability metrics for each dataset (see Table 5):  SQuAD and QA4MRE achieved the highest scores for most metrics.  This reflects the fact that Wikipedia articles and technical docu- ments usually require a high-grade level of un- derstanding.  In contrast, MCTest had the lowest scores, with its dataset consisting of narratives for children. (iv) Correlation between numbers of re- quired prerequisite skills and readability met- rics (see Figures 2 and 3, and Table 6): our main interest was in the correlation between prerequi- site skills and readability.  To investigate this, we examined the relation between the number of re- quired prerequisite skills and readability metrics.
We used the Flesch–Kincaid grade level (Kincaid et al., 1975) as an intuitive reference for read- ability.  This value represents the typical num- ber of years of education required to understand texts based on counts of syllables, words, and sen- tences. Figures 2 and 3 show the relation between two values for each dataset and for each question, respectively.  Figure 2 shows the trends of the datasets.  QA4MRE was relatively difficult both to read and to answer, whereas SQuAD was dif- ficult to read but easy to answer.  For further in- vestigation, we selected three datasets (QA4MRE, MCTest, and SQuAD) and plotted all of their questions in Figure 3. Three separate domains can be seen. Table 6 presents Pearson's correlation coeffi- cients between the number of required prerequisite skills and each readability metric for all questions in the RC datasets. Although there are weak corre- lations, from 0.025 to 0.416, these results demon- strate that there is not necessarily a strong correla- tion between the two values. This leads to the fol- lowing two insights.  First, the readability of RC datasets does not directly affect the difficulty of their questions. That is, RC datasets that are diffi- cult to read are not necessarily difficult to answer. Second, it is possible to create difficult questions from the context that are easy to read. MCTest is a good example.  The context texts in the MCTest dataset are easy to read, but the difficulty of its questions compares to that for the other datasets. To summarize our results in terms of each RC dataset, we can make the following observations. -  QA4MRE is difficult both to read and to answer among the datasets analyzed. This would seem to follow its questions being devised by experts. -  MCTest is a good example of an RC dataset that is easy to read but difficult to answer.  We pre- sume that this is because the corpus genre (i.e., narrative) reflects the trend in required skills for the questions. - SQuAD is difficult to read, along with QA4MRE, but relatively easy to answer com- pared with the other datasets. - Who-did-What performs well in terms of its query-sourcing method. Although its questions are created automatically, they are sophisticated in terms of knowledge reasoning. However, the automated sourcing method must be improved to exclude nonsense questions. - MS MARCO is a relatively easy dataset in terms of prerequisite skills. However, one prob- lem is that the dataset contained nonsense ques- tions. - NewsQA is advantageous in that it provides meta-information on the reliability of the ques- tions. Such information enabled us to avoid us- ing nonsense questions, as for the training of machine learning models. 6 Discussion In this section, we discuss several issues regarding the construction of RC datasets and the develop- ment of RC systems using our methodology. How to utilize the two classes of metrics for system development : one possible scenario for developing an RC system is that it is first built to solve an easy-to-read and easy-to-answer dataset. The next step would be to improve the system so that it can solve an easy-to-read but difficult-to- answer dataset (or its converse).  Finally, only af- ter it can solve such datasets should the system be applied to difficult-to-read and difficult-to-answer datasets.  The metrics of this study may be useful in preparing appropriate datasets for each step by measuring their properties. The datasets can then be ordered according to the grades of the metrics and applied to each step of the development, as in curriculum learning (Bengio et al., 2009) and transfer learning (Pan and Yang, 2010). Corpus genre: attention should be paid to the genre of the corpus used to construct a dataset. Ex- pository documents such as news articles tend to require factorial understanding. Most existing RC datasets use such texts because of their availabil- ity.  On the other hand, narrative texts may have a closer correspondence to our everyday experience, involving the emotions and intentions of charac- ters (Graesser et al., 1994).  To build agents that work in the real world, RC datasets may have to be constructed from narratives. Question type: in contrast to factorial under- standing, comprehensive understanding of natural language texts needs a better grasp of global co- herence (e.g., the main point or moral of the text, the goal of a story, or the intention of characters) from the broad context (Graesser et al., 1994). Most questions in current use require only local coherence (e.g., referential relations and thematic roles) within a narrow context.  An example of a question based on global coherence would be to give a summary of the text, as used in Hermann et al. (2015).  It could be generated automatically by techniques of abstractive text summarization (Rush et al., 2015; Ganesan et al., 2010). Annotation issues :  we  found  questions  for which there were disagreements regarding non- sense decisions.  For example, some questions can be solved by external knowledge without even seeing their context.  Therefore, we should clar- ify what constitutes a "solvable" or "reasonable" question for RC. In addition, annotators reported that the prerequisite skills did not easily treat questions whose answer was "none of the above" in QA4MRE. We considered these "no answer" questions difficult, in that systems have to decide not to select any of the candidate answers, and our methodology failed to specify them. Competence in selecting necessary sentences: as mentioned in Section 1, our methodology can- not evaluate competence in selecting sentences that need to be read to answer questions. In a brief analysis, we further investigated sentences in the context of the datasets that were selected in the an- notation.  Analyses were performed in two ways. For each question, we counted the number of re- quired sentences and their distance apart. 4  The first row of Table 7 gives the average number of re- quired sentences per question for each RC dataset. Although the scores are reasonably close, MCTest required multiple sentences to be read most fre- quently.  The second row gives the average dis-
Table 7:  Average number and distance apart of sentences that need to be read to answer a ques- tion in the RC datasets. tance apart of the required sentences.  QA4MRE required the longest distance because readers had to look for clues in the long context texts. In con- trast, SQuAD and MS MARCO had lower scores. Most of their questions seemed to be answered by reading only a single sentence.  Of course, the scores for distances will depend on the length of the context texts. Metrics of RC for machines:  our underlying assumption in this study is that, in the develop- ment of interactive agents such as dialogue sys- tems, it is important to make the systems behave in a human-like way.  This has also become a distinguishing feature of recent RC task design, and one that has never been explicitly considered in conventional NLP tasks.  To date, the differ- ence between human and machine RC has not at- tracted much research attention.  We believe that our human-based evaluation metrics and analy- sis will help researchers to develop a method for the step-by-step construction of better RC datasets and improved RC systems.
7
Conclusion
In this study, we adopted evaluation metrics that comprise two classes, namely refined prerequisite skills and readability , for analyzing the quality of RC datasets.  We applied these classes to six ex- isting datasets and highlighted their characteris- tics according to each metric. Our dataset analysis suggests that the readability of RC datasets does not directly affect the difficulty of the questions and that it is possible to create an RC dataset that is easy to read but difficult to answer.  In future work, we plan to use the analysis from the present study in constructing a system that can be applied to multiple datasets.
Acknowledgments
We would like to thank anonymous reviewers This work was for their insightful comments. supported by JSPS KAKENHI Grant Numbers 15H02754 and 16K16120. Sandra Aluisio, Lucia Specia, Caroline Gasperin, and Carolina Scarton. 2010.  Readability assessment for text simplification.  In Proceedings of the NAACL HLT 2010 Fifth Workshop on Innovative Use of NLP for Building Educational Applications . Asso- ciation for Computational Linguistics, pages 1–9. https://aclweb.org/anthology/W10-1001. Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. 2009.  Curriculum learning.  In Proceedings of the 26th annual international con- ference on machine learning. ACM, pages 41–48. https://doi.org/10.1145/1553374.1553380. Luisa Bentivogli, Elena Cabrio, Ido Dagan, Danilo Giampiccolo, Medea Lo Leggio, and Bernardo Magnini. 2010.  Building textual entailment special- ized data sets: a methodology for isolating linguistic phenomena relevant to inference. In Proceedings of the 7th International Conference on Language Re- sources and Evaluation . Citeseer. Danqi Chen, Jason Bolton, and Christopher D. Man- A thorough examination of the ning. 2016. CNN/Daily Mail reading comprehension task.  In Proceedings of the 54th Annual Meeting of the Asso- ciation for Computational Linguistics. Association for Computational Linguistics, pages 2358–2367. https://aclweb.org/anthology/P16-1223. Herbert  H  Clark.  1975. Bridging. In  Proceed- ings of the 1975 workshop on Theoretical is- sues in natural language processing. Association for Computational Linguistics, pages 169–174. https://doi.org/10.3115/980190.980237. Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual en- tailment challenge. In Machine learning predictive challenges. uncertainty, evaluating object classification, visual and recognising Springer, tectual entailment, pages 177–190. https://doi.org/10.1007/11736790_9. Ido Dagan, Dan Roth, Mark Sammons, and Fabio Mas- simo Zanzotto. 2013.  Recognizing textual entail- ment: Models and applications.  Synthesis Lectures on Human Language Technologies 6(4):1–220. Kavita Ganesan, ChengXiang Zhai, and Jiawei Han. 2010. Opinosis: a graph-based approach to abstrac- tive summarization of highly redundant opinions.  In Proceedings of the 23rd international conference on computational linguistics . Association for Compu- tational Linguistics, pages 340–348. Arthur C Graesser, Murray Singer, and Tom Trabasso. 1994. Constructing inferences during narrative text comprehension.  Psychological review 101(3):371. https://doi.org/10.1037/0033-295X.101.3.371. Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Su- leyman, and Phil Blunsom. 2015.  Teaching ma- chines to read and comprehend.  In Advances in Neu- ral Information Processing Systems . pages 1693– 1701. Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016.  The goldilocks principle:  Reading children's books with explicit memory representa- tions.  In Proceedings of the International Confer- ence on Learning Representations. Lynette Hirschman, Marc Light, Eric Breck, and John D Burger. 1999. Deep read: A read- ing comprehension system. In   Proceedings of the 37th Annual Meeting of the Associa- tion for Computational Linguistics. Association for Computational Linguistics, pages 325–332. https://doi.org/10.3115/1034678.1034731. J Peter Kincaid, Robert P Fishburne Jr, Richard L Rogers, and Brad S Chissom. 1975.  Derivation of new readability formulas (automated readability in- dex, fog count and flesch reading ease formula) for navy enlisted personnel.  Chief of Naval Technical Training, Research Branch Report 8-75. Walter Kintsch. 1988. The role of knowledge in discourse comprehension: A construction- integration model.  Psychological review 95(2):163. https://doi.org/10.1037/0033-295X.95.2.163. Walter Kintsch. 1993.  Information accretion and re- duction in text processing:  Inferences.  Discourse processes 16(1-2):193–202. Peter LoBue and Alexander Yates. 2011.  Types of common-sense knowledge needed for recognizing textual entailment.  In Proceedings of the 49th An- nual Meeting of the Association for Computational Linguistics:  Human Language Technologies. Asso- ciation for Computational Linguistics, pages 329– 334. https://aclweb.org/anthology/P11-2057. Danielle S McNamara and Joe Magliano. 2009.  To- ward a comprehensive model of comprehension. Psychology of learning and motivation 51:297–384. https://doi.org/10.1016/S0079-7421(09)51009-2. Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016.  MS MARCO: A human gener- ated machine reading comprehension dataset.  CoRR abs/1611.09268. Peter Norvig. 1989. Marker passing as a weak method for text inferencing.  Cognitive Science 13(4):569– 620. https://doi.org/10.1207/s15516709cog1304_4. Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gim- pel, and David McAllester. 2016.  Who did What: A large-scale person-centered cloze dataset. In Pro- ceedings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, pages 2230–2235. https://aclweb.org/anthology/D16-1241. A sur- Sinno Jialin Pan and Qiang Yang. 2010. vey on transfer learning.  IEEE Transactions on knowledge and data engineering 22(10):1345–1359. https://doi.org/10.1109/TKDE.2009.191. Denis Paperno, Germán Kruszewski, Angeliki Lazari- dou, Ngoc Quan Pham, Raffaella Bernardi, San- dro Pezzelle, Marco Baroni, Gemma Boleda, and Raquel Fernandez. 2016. The LAMBADA dataset: Word prediction requiring a broad discourse context. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics. Associ- ation for Computational Linguistics, pages 1525– 1534. https://aclweb.org/anthology/P16-1144. Emily Pitler and Ani Nenkova. 2008. Revisiting read- ability:  A unified framework for predicting text quality.  In Proceedings of the 2008 Conference on Empirical Methods in Natural Language Pro- cessing. Association for Computational Linguistics, pages 186–195.  https://aclweb.org/anthology/D08- 1020. Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016.  SQuAD: 100,000+ questions for machine comprehension of text.  In Proceed- ings of the 2016 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, pages 2383–2392. https://aclweb.org/anthology/D16-1264. Abigail Razon and John Barnden. 2015. A new approach to automated text readability classifica- tion based on concept indexing with integrated part-of-speech n-gram features. In  Proceedings of the International Conference Recent Advances in Natural Language Processing. pages 521–528. https://aclweb.org/anthology/R15-1068. Matthew Richardson, J.C. Christopher Burges, and Erin Renshaw. 2013.  MCTest:  A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Em- pirical Methods in Natural Language Processing. pages 193–203.  http://aclweb.org/anthology/D13- 1020. Alexander M. Rush, Sumit Chopra, and Jason We- A neural attention model for ab- ston.  2015. stractive sentence summarization. In Proceed- ings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing. Associa- tion for Computational Linguistics, pages 379–389. https://aclweb.org/anthology/D15-1044. Mark Sammons, V.G.Vinod Vydiswaran, and Dan Roth. 2010. "ask not what textual entail- In   Proceedings ment can do for you…”. of the 48th Annual Meeting of the Associa- tion for Computational Linguistics. Association for Computational Linguistics, pages 1199–1208. https://aclweb.org/anthology/P10-1122. Ellery Smith, Nicola Greco, Matko Bosnjak, and An- dreas Vlachos. 2015.  A strong lexical matching method for the machine comprehension test.  In Pro- ceedings of the 2015 Conference on Empirical Meth- ods in Natural Language Processing. Association for Computational Linguistics, pages 1693–1698. Saku Sugawara and Akiko Aizawa. 2016.  An anal- ysis of prerequisite skills for reading compre- In Proceedings of the Workshop on hension. Uphill Battles in Language Processing:  Scaling Early Achievements to Robust Methods. Associ- ation for Computational Linguistics, pages 1–5. https://aclweb.org/anthology/W16-6001. Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. 2017.  Prerequisite skills for reading comprehen- sion:  Multi-perspective analysis of mctest datasets and systems.  In AAAI Conference on Artificial In- telligence. pages 3089–3096. Richard Sutcliffe, Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Corina Forascu, Yassine Benajiba, and Petya Osenova. 2013. Overview of QA4MRE main task at CLEF 2013. Working Notes, CLEF . Adam Trischler, Tong Wang, Xingdi Yuan, Justin Har- ris, Alessandro Sordoni, Philip Bachman, and Ka- heer Suleman. 2016. NewsQA: A machine compre- hension dataset. CoRR abs/1611.09830. Sowmya Vajjala and Detmar Meurers. 2012.  On im- proving the accuracy of readability classification us- ing insights from second language acquisition.  In Proceedings of the Seventh Workshop on Build- ing Educational Applications Using NLP. Associa- tion for Computational Linguistics, pages 163–173. https://aclweb.org/anthology/W12-2019. Teun Adrianus van Dijk and Walter Kintsch. 1983. Strategies of discourse comprehension . Citeseer. Ellen M Voorhees et al. 1999.  The TREC-8 question answering track report. In TREC . volume 99, pages 77–82. Mengqiu Wang, Noah A. Smith, and Teruko Mita- What is the Jeopardy model? a mura.  2007. quasi-synchronous grammar for QA.  In Proceed- ings of the 2007 Joint Conference on Empirical Methods in Natural Language Processing and Com- putational Natural Language Learning. Associa- tion for Computational Linguistics, pages 22–32. https://aclweb.org/anthology/D/D07/D07-1003. Wen-tau Yih, and Christopher Meek. Yi Yang, WikiQA: A challenge dataset for open- 2015. In   Proceedings question answering. domain 2015 Conference on Empirical Meth- of the ods in Natural Language Processing. Association for Computational Linguistics, pages 2013–2018. https://aclweb.org/anthology/D15-1237. In this appendix, we explain the method of choos- ing questions for annotation. QA4MRE (Sutcliffe et al., 2013):  the gold- standard dataset comprised four different topics and four documents for each topic.  We randomly selected 100 main and auxiliary questions so that at least one question for each document was in- cluded. MCTest (Richardson et al., 2013): this dataset comprised two sets:  MC160 and MC500.  Their development sets had 80 tasks in total, with each containing context texts and four questions.  We randomly chose 25 tasks (100 questions) from the development sets. SQuAD (Rajpurkar et al., 2016):  this dataset included Wikipedia articles involving various top- ics, with the articles being divided into paragraphs. We randomly chose 100 paragraphs from 15 arti- cles and used only one question from each para- graph for the annotation. Who-did-What (WDW) (Onishi et al., 2016): this dataset was constructed from the English Gi- gaword newswire corpus (v5).  Its questions were automatically created using a different article from that used for context.  In addition, questions that could be solved by a simple baseline method were excluded from the dataset. MS MARCO (MARCO) (Nguyen et al., 2016): each task in this dataset comprised several seg- ments, one question, and its answer. We randomly chose 100 tasks (100 questions) and only used seg- ments whose attribute was is_selected = 1 as con- text. NewsQA (Trischler et al., 2016):  we ran- domly chose questions that satisfied the fol- lowing conditions:   is answer absent = 0, is_question_bad = 0, and validated_answers do not include bad_question or none.