Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability Saku Sugawara♠, Yusuke Kido♠, Hikaru Yokono✿, Akiko Aizawa�♠ ♠University of Tokyo *Fujitsu Laboratories Ltd. ◇National Institute of Informatics August 1, ACL 2017
1/40

ID: MCTest MC160.dev.29 (1) multiple: Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escap- ing? Answer: A) Mountain   *B) Forest   C) Cave   D) Castle
2 / 40

ID:  MCTest MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A:  A) Mountain   *B) Forest   C) Cave   D) Castle Coreference resolution (she = princess) Commonsense reasoning (escaping = climbed down) Temporal relation (climbed → wandered)
3 / 40

ゃDeepRead (1999)ゃQA4MRE (2013) + MCTest (2013)ゃCNN/Daily Mail (2015) ✿ SQuAD (2016)ゃLAMBADA (2016) ✿ Who-did-What (2016)ゃMS MARCO (2016)ゃNewsQA (2016) ✿ WikiSuggest (ACL 2017)ゃTriviaQA (ACL 2017) ✿ RACE (EMNLP 2017)
4 / 40

5 / 40

6 / 40

→  Only with accuracy, we cannot tell what the systems understand and what they don't. ← Chen+ (2016) shows: CNN/Daily Mail datasets contain unanswerable or ambiguous questions
7 / 40

ゃHow can we evaluate and analyze our RC systems? →  Propose evaluation metrics for RC →  Focus on prerequisite skills and readability
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question: What are members who have been baptized as an infant or child but who have not subsequently professed their own faith? Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wan- dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
9 / 40

ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have not subsequently professed their own faith? Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping.  She wan¬ dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
10 / 40

ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping.  She wan¬ dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain   *B) Forest   C) Cave   D) Castle
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r. [ → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where d'A → Regure reading multile sentence with skills Answer: A) Mount:
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have not subseque   Difficult-to-read & Easy-to-answer Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princes a Answer: A) Mountain   *B) Fores | Easy-to-read & Difficult-to-answer
14 / 40

1. Define prerequisite skills and readability metrics ✈ 13 prerequisite skills and 10 readability measures
15 / 40

1. Define prerequisite skills and readability metrics :- 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills ✈ 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA
1. Define prerequisite skills and readability metrics :- 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills ✈ 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA 3.  Calculate readability of the datasets ♪ Readability of "context sentences necessary for answering" (selected in the annotation) (≠ whole context)
17 / 40

1. Define prerequisite skills and readability metrics :- 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills ✈ 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA 3.  Calculate readability of the datasets ♪ Readability of “context sentences necessary for answering” (selected in the annotation) (≠ whole context) 4.  Analyze the datasets on two types of difficulties → See the relation between skills and readability
18 / 40

19 / 40

This study shows statistics of datasets & an observation on the relation between skills (difficulty in answering) and readability (difficulty in reading)
20 / 40

ゃNew knowledge reasoning skills in this study “Commonsense reasoning” is updated to new 4 skills for more detailed analysis
21 / 40

← Previous study (Sugawara+, AAAI 2017): skills are based on existing NLU tasks in NLP →  Analyzed MCTest dataset and three systems, and showed that "the more skills are required, the more difficult to answer (lower accuracy)."ゃWe regard this as the difficulty of answering
22 / 40

ゃ13 skillls: mult-label annotationゃ6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA � 100 questions for each datasetゃ4 annotators: graduate NLP studentsゃFor 62 randomly sampled questions, 90.1% agreement ← Annotation: choose skill labels and “necessary context sentences” for answering ➪ Sentences are used to calculate readability measures
24 / 40

➪ Ave. Num. of characters per word (NumChar) ← Ave. Num. of syllables per word (NumSyll) ✿ Ave. sentence length in words (MLS) ✿ Proportion of words in AWL (AWL) ➪ Modifier variation (ModVar) ➪ Num. of coordinate phrases per sentence (CoOrd) ✿ Coleman-Liau index (Coleman) ← Dependent clause to clause ratio (DC/C) ← Complex nominals per clause (CN/C) ✿ Adverb variation (AdvVar) Figure: 10 readability measure from Vajjala and Meurers (2012).
31 / 40

*F-K = Flesch-Kincaid grade level = education level required to understand the text.
● There is only a weak correlation between readability metrics and numbers of required skills →  Difficult to read ⇒ difficult to answer (and vice versa) →  It is possible to create a dataset that consists of an easy-to-read context and difficult-to-answer questions.
34 / 40

For development of RC systems, select datasets in the following steps:
Answerability of questions ☞ We cannot evaluate the difference among “truely difficult,” "non sense," and "no answer" when we encounter systems' incorrect answers (and even in human performance!) + It is not easy to maintain the quality of questions especially in crowd-based sourcing.
Answerability of questions ➪ We cannot evaluate the difference among "truely difficult," "non sense," and "no answer" when we encounter systems' incorrect answers (and even in human performance!) + It is not easy to maintain the quality of questions especially in crowd-based sourcing. Reading comprehension seen as textual entailment Multiple premises (from context) ⇒ hypothesis (from Q+A) ● Our methodology cannot deal with the competence of: 1. Collecting evidences from context 2. Generating hypothesis from question and answer candidates
37 / 40

Corpus genreゃAre Wikipedia or news articles enough? Example: narratives are close to our everyday experience (characters' emotions, intentions, and actions)
38 / 40

Corpus genreゃAre Wikipedia or news articles enough? Example: narratives are close to our everyday experience (characters' emotions, intentions, and actions) Relation between RC and other NLP fields : How can RC systems contribute to other fields in which natural language understanding is required? ↑ RC may be able to lay a foundation of “contextual” NLU. > Example: dialogue systems dialogue history = context, utterance = question
39 / 40

Proposed evaluation metrics for RC 1.  Defined two classes of metrics: prerequisite skills and readability 2.  Annotated RC datasets with the skills 3.  Calculated readability of datasets 4.  Analyzed datasets →  Results can be used for evaluation of systems Observation ● There is only a weak correlation between readability metrics and numbers of required skills
40 / 40

Table: Pearson's correlation coefficients ( r ) with the p-values ( p ) in all RC datasets
42 / 40

ゃTextual entailment → Recognizing and testing: premise → hypothesis ✿ Reading comprehension as textual entailment → Recognizing and testing: Multiple premises → hypothesis Issue: Our metrics cannot evaluate the following processes in RC: ✿ Multiple premises ← gathered from context sentencesゃhypothesis ← generated from answer candidates ⇒  depending on context length and question formulation
43 / 40

4 Ellipsis ✈ Recognizing implicit/omitted information ♯ e.g. She is a smart student → She is a student
44 / 40

φ Ellipsis → Recognizing implicit/omitted information ✈ e.g. She is a smart student → She is a student ✦ Bridging ♯ Inferences supported by grammatical and lexical information F e.g. She loves sushi. → She likes sushi.
45 / 40

φ Ellipsis
→ Recognizing implicit/omitted information ✈ e.g. She is a smart student → She is a student ✦ Bridging ✈ Inferences supported by grammatical and lexical information F e.g. She loves sushi. → She likes sushi. ✿ Elaboration ✈ Inference using known facts and general knowledge → e.g. The writer of Hamlet was Shakespeare → Shakespeare wrote Hamlet
46 / 40

φ Ellipsis
→ Recognizing implicit/omitted information ✈ e.g. She is a smart student → She is a student ✦ Bridging ✈ Inferences supported by grammatical and lexical information F e.g. She loves sushi. → She likes sushi. ✿ Elaboration ✈ Inference using known facts and general knowledge → e.g. The writer of Hamlet was Shakespeare → Shakespeare wrote HamletゃMeta-knowledge } Inference using external knowledge including a reader, writer, and text genre ♯ e.g. Who is the main character in this story?
47 / 40