Evaluation Metrics for the Machine Reading Comprehension Task Saku Sugawara University of Tokyo November 7, 2017 @ Tohoku University http://penzant.net
1/69

1.  Overview of the machine reading comprehension (RC) task “What is the RC task?” 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" Sugawara+ (2017a, AAAI) & Sugawara+ (2017b, ACL) 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?"
2/69

1.  Overview of the machine reading comprehension (RC) task “What is the RC task?” 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" Sugawara+ (2017a, AAAI) & Sugawara+ (2017b, ACL) 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?"
3/69

ID: MCTest MC160.dev.29 (1) multiple: Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escap- ing? Answer: A) Mountain   *B) Forest   C) Cave   D) Castle Task: context, question, and answer
4/69

ID:  MCTest MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A:  A) Mountain   *B) Forest   C) Cave   D) Castle Coreference resolution (she = princess) Commonsense reasoning (escaping = climbed down) Temporal relation (climbed → wandered)
5/69

Towards the Machine Comprehension of Text: An Essay (Burges 2013) “a machine comprehends a passage of text if, for any question regarding that text that can be answered correctly by a majority of native speakers, that machine can provide a string which those speakers would agree both answers that question" Overview of QA4MRE Main Task at CLEF 2013 (Sutcliffe+ 2013) "RC tests do not require only semantic understanding but they assume a cognitive process which involves using implications and presuppositions, retrieving the stored information, performing inferences to make implicit information explicit." → the RC task is an evaluation method for language understanding systems in terms of their behavior, and tests a cognitive process that involves several skills, such as performing inferences using background knowledge, by letting the system answer questions about a given text. 6/69 Representative Datasets & MCTest (2013) ❁ CNN/Daily Mail (2015) # SQuAD (2016)
7/69

❀ Richardson et al. ù Children stories and questions written by crowdworkers ฿ Multiple choice & Pros: X Story-based RC = characters' intentions, relations of events, commonsense... 88 Limited vocabulary ❀ Cons 8 Not large: 660 stories with 4 questions each ometimes he'd reach into all the food. Other times he'd sled on the deck and get see what kind of trouble ne could get the grocery store and pulled all f n ¥ pay, and instead bead. hat is the name of the trouble making turtle? at did James pull off of the shelves in the gro-
❀ Hermann et al. ฿ News articles ù Cloze (fiil-in-blank) # Answer extraction & Pros & 140M Qs, various topics ✈ Cons ® Contains errors in coreference (Chen + 2016)
🌸 Rajpurkar et al. ❀ Wikipedia articles 🌸 Answer extraction & Pros & 100K Qs, various topics Super_Bowl_50 The Stanford Question Answering Dataset Super Rowl 50 was an American football game to determine the champion of the National Football League (NFL) for the 2015 season. The American Football Conference (AFC) chargeson Derwir Brencm defeated the National Footbull Conference (NFC) chartoion Carolina Parthers 24–10 to cam their third Super Bowl tide. The game was played on February 7, 2016, at Levi's Stadium in the San Francisco Bay Area at Santa Clara, California. As this was the 50th Super Bowl, the Isague emphasized the "solden applyersary" with xarious rold-sherned initiatives, as well as temporarily suspending the tradition of naming each Super Bowl garse with Roman numerals (under which the game would have been known as "Super Bowl L"), so that the logo could prominently feature the Arabic numerals 50.
& Cons
® Goldberg (2017) says “pattern matching" & Easily fooled by Jia+ (2017)’s adversarial examples .501 oncos | Denver Bronc Which NFI team represented the NFC at Super Bowl 502 Ground Truth Answers: Carolina Panthers: [Carolina] Panthers | Carolina Panthers Where did Super Bowl 50 take place Ground Truth Answers:  Santa Clara, California  Levi's Stadium] Levi's Stadium in the San Francisco Bay Area at Santa Clara, California.
Question answering ❀ Q + A without the explicit context e.g., Q: 'What is solid CO2 commonly called?' A: dry ice ❀ Include “searching” from the resources
19 / 69

Question answering ❀ Q + A without the explicit context e.g., Q: 'What is solid CO2 commonly called?' A: dry ice ❀ Include “searching” from the resources Reading comprehension as question answering ❀ Q + A with the explicit context ❀ Inquire not only general knowledge but the context-dependent information about temporal situations/stories
20/69

Textual entailment ❁ Recognizing and testing: premise → hypothesis e.g., Premise: 'alcohol reduces blood pressure' → Hypothesis: 'alcohol affects blood pressure'
21/69

Textual entailment ❀ Recognizing and testing: premise → hypothesis e.g., Premise: 'alcohol reduces blood pressure' → Hypothesis: 'alcohol affects blood pressure' Reading comprehension as textual entailment ❁ Recognizing and testing: multiple premises (from context) → hyposhesis (from Q+A) ﴾ Cf. FraCaS (Cooper, 1996) (issues: simple & small) ❀ Etzioni+ (2006): “Machine reading […] combines multiple textual entailment steps to form a coherent set of beliefs based on the text." (see also Manning (2006))
22 / 69

1.  Overview of the machine reading comprehension (RC) task “What is the RC task?” 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?"
23 / 69

→  Only with accuracy, we cannot tell what the systems understand and what they don't. ❀ Chen+ (2016) shows: CNN/Daily Mail datasets contain unanswerable or ambiguous questions
24/69

❀ How can we evaluate and analyze our RC systems? →  Propose evaluation metrics for RC →  Focus on prerequisite skills and readability
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question: What are members who have been baptized as an infant or child but who have not subsequently professed their own faith? Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wan- dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
26/69

ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have not subsequently professed their own faith? Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping.  She wan¬ dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping.  She wan¬ dered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain  *B) Forest   C) Cave   D) Castle
28/69

ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways.  Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princess wander to after escaping? Answer: A) Mountain   *B) Forest   C) Cave   D) Castle
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have r. [ → Answerble simply by noticing one sentence Answer: Baptized N ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where d r → Regure reading multile sentence with skills Answer: A) Mount:
ID: SQuAD (2016), United_Methodist_Church Context: The United Methodist Church (UMC) practices infant and adult baptism.  Baptized Members are those who have been baptized as an infant or child, but who have not subsequently professed their own faith. Question:  What are members who have been baptized as an infant or child but who have not subseque   Difficult-to-read & Easy-to-answer Answer: Baptized Members ID: MCTest (2013), mc160.dev.29 Context: The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there are no electric poles but where there are some caves. Question: Where did the princes a Answer: A) Mountain   *B) Fores | Easy-to-read & Difficult-to-answer
31/69

1. Define prerequisite skills and readability metrics 88 13 prerequisite skills and 10 readability measures
1. Define prerequisite skills and readability metrics 88 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills & 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA
33/69

1. Define prerequisite skills and readability metrics 88 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills & 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA 3.  Calculate readability of the datasets & Readability of "context sentences necessary for answering" (selected in the annotation) (≠ whole context)
34/69

1. Define prerequisite skills and readability metrics 88 13 prerequisite skills and 10 readability measures 2.  Annotate RC datasets with skills & 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA 3.  Calculate readability of the datasets & Readability of "context sentences necessary for answering" (selected in the annotation) (≠ whole context) 4.  Analyze the datasets on two types of difficulties & See the relation between skills and readability
35/69

36/69

This study shows statistics of datasets & an observation on the relation between skills (difficulty in answering) and readability (difficulty in reading)
37 / 69

﴾ New knowledge reasoning skills in this study “Commonsense reasoning” is updated to new 4 skills for more detailed analysis
38/69

❀ Previous study (Sugawara+, AAAI 2017): skills are based on existing NLU tasks in NLP →  Analyzed MCTest dataset and three systems, and showed that "the more skills are required, the more difficult to answer (lower accuracy)." ❀ We regard this as the difficulty of answering
39 / 69

❀ 13 skillls: mult-label annotation ❀ 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MS MARCO, NewsQA ❀ 100 questions for each dataset & 4 annotators: graduate NLP students ❀ For 62 randomly sampled questions, 90.1% agreement ❀ Annotation: choose skill labels and “necessary context sentences” for answering ❀ Sentences are used to calculate readability measures
41/69

❀ Ave. Num. of characters per word (NumChar) ❀ Ave. Num. of syllables per word (NumSyll) ❀ Ave. sentence length in words (MLS) ❀ Proportion of words in AWL (AWL) ❀ Modifier variation (ModVar) ❀ Num. of coordinate phrases per sentence (CoOrd) ❀ Coleman-Liau index (Coleman) ঞ Dependent clause to clause ratio (DC/C) ❀ Complex nominals per clause (CN/C) ❁ Adverb variation (AdvVar) Figure: 10 readability measure from Vajjala and Meurers (2012).
48/69

*F-K = Flesch-Kincaid grade level = education level required to understand the text.
❀ There is only a weak correlation between readability metrics and numbers of required skills →  Difficult to read ⇒ difficult to answer (and vice versa) →  It is possible to create a dataset that consists of an easy-to-read context and difficult-to-answer questions.
51/69

steps: easy-to-read & difficult-to-answer dataset
Answerability of questions ❀ We cannot evaluate the difference among “truely difficult,” "non sense," and "no answer" when we encounter systems' incorrect answers (and even in human performance!) ❀ It is not easy to maintain the quality of questions especially in crowd-based sourcing.
Answerability of questions ❀ We cannot evaluate the difference among “truely difficult,” “non sense,” and “no answer” when we encounter systems’ incorrect answers (and even in human performance!) ❀ It is not easy to maintain the quality of questions especially in crowd-based sourcing. Corpus genre ❀ Are Wikipedia or news articles enough? Example: narratives are close to our everyday experience (characters' emotions, intentions, and actions)
54/69

Proposed evaluation methodology for RC 1.  Defined two classes of metrics: prerequisite skills and readability 2.  Annotated RC datasets with the skills 3.  Calculated readability of datasets 4.  Analyzed datasets →  Results can be used for evaluation of systems Observation ❀ There is only a weak correlation between readability metrics and numbers of required skills
55/69

1.  Overview of the machine reading comprehension (RC) task “What is the RC task?” 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?"
56 / 69

1.  Overview of the machine reading comprehension (RC) task “What is the RC task?” 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?" →  What factors affect the difficulty of RC questions?
57 / 69

1. Read a given text and understand the meaning of a question about it & e.g. which type of entitiy is required.
58/69

1. Read a given text and understand the meaning of a question about it & e.g. which type of entitiy is required. 2.  If needed (e.g., in answer extraction questions), find answer candidates from the context
59 / 69

1. Read a given text and understand the meaning of a question about it & e.g. which type of entitiy is required. 2.  If needed (e.g., in answer extraction questions), find answer candidates from the context 3.  Make hypotheses using the question and each answer candidate
60/69

1. Read a given text and understand the meaning of a question about it & e.g. which type of entitiy is required. 2.  If needed (e.g., in answer extraction questions), find answer candidates from the context 3.  Make hypotheses using the question and each answer candidate 4. Test each hypothesis whether it can be entailed from a given text or its sub-sentences
61/69

1. Read a given text and understand the meaning of a question about it & e.g. which type of entitiy is required. 2.  If needed (e.g., in answer extraction questions), find answer candidates from the context 3.  Make hypotheses using the question and each answer candidate 4. Test each hypothesis whether it can be entailed from a given text or its sub-sentences 5.  Choose a hypothesis that is most likely to be entailed, and return its corresponding candidate
A.  Understanding a question itself & Involving some skills such as coreference resolution → Easier if a question does not require background knowledge (difficult example: "what is the main theme of this passage?")

A.  Understanding a question itself
& Involving some skills such as coreference resolution → Easier if a question does not require background knowledge (difficult example: "what is the main theme of this passage?") B.  Identifying answer candidates & Recognizing entity types (wh-) and clause (why/how) →  Easier if there are few answer candidates (e.g., there is only one temporal expression in the context for a when question) Multiple choice seems better (cf. Levesque (2013))
64/69

C.  Choosing evidential sentence(s) from a given text 88 Which sentence is informative for examining a hypothesis →  Easier if the number of sentences is small (like TE)
C.  Choosing evidential sentence(s) from a given text 88 Which sentence is informative for examining a hypothesis →  Easier if the number of sentences is small (like TE) D.  Testing entailment between evidnece and hypothesis & How many prerequisite skills are required (Partly revealed by our evaluation methodology) →  Easier if no skill is required (e.g., the hypothesis is almost similar to the evidence)
66 / 69

1.  Overview of the machine reading comprehension task "What is the reading comprehension (RC) task?" 2.  Evaluation methodology for RC datasets/systems "How can we evaluate our systems/datasets?" 3.  Discussion for constructing RC datasets: "How can we create difficult but not too difficult questions?"
67 / 69

Burges (2013) Towards the Machine Comprehension of Text: An Essay, technical report. 88  Chen et al. (2016) A Thorough Examination of the CNN/Daily Mail Reading Comprehension Task, in ACL. 88  Cooper et al. (1996) Using the Framework, by the FraCaS Consortium. 88  Etzioni et al. (2006) Machine Reading, in AAAI. 8駅 Goldberg (2017) Near human performance in question answering? (http://u.cs.biu.ac.il/~yogo/on-squad.pdf). 88  Hermann et al. (2015) Teaching Machines to Read and Comprehend, in NIPS. Jia et al. (2017) Adversarial Examples for Evaluating Reading Comprehension Systems, in EMNLP.耶 Lai et al. (2017) RACE: Large-scale ReAding Comprehension Dataset From Examinations, in EMNLP. Levesque (2013) On our best behaviour, in IJCAI. Joshi et al. (2017) TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension, in ACL.
86  Manning (2006) Local textual inference: Its hard to circumscribe, but you know it when you see it-and NLP needs it, unpublished manuscript. 88  Nguyen et al. (2016) MS MARCO: A Human Generated MAchine Reading COmprehension Dataset. ￼  Onishi et al. (2016) Who did What: A Large-Scale Person-Centered Cloze Dataset, in EMNLP. 88  Rajpurkar et al. (2016) SQuAD: 100,000+ Questions for Machine Comprehension of Text, in EMNLP. 49 Richardson et al. (2013) MCTest: A Challenge Dataset for the Open-Domain Machine Comprehension of Text, in EMNLP. 88  Sugawara et al. (2017a) Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems, in AAAI. 88  Sugawara et al. (2017b) Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills and Readability, in ACL. 88  Sutcliffe et al. (2013) Overview of QA4MRE Main Task at CLEF 2013, in CLEF. 88  Trischler et al. (2016) NewsQA: A Machine Comprehension Dataset, in REP4NLP.
Table: Pearson's correlation coefficients ( r ) with the p-values ( p ) in all RC datasets
71/69

# Ellipsis ® Recognizing implicit/omitted information & e.g. She is a smart student → She is a student

# Ellipsis
& Recognizing implicit/omitted information & e.g. She is a smart student → She is a student & Bridging 88 Inferences supported by grammatical and lexical information & e.g. She loves sushi. → She likes sushi.
73/69

# Ellipsis
& Recognizing implicit/omitted information & e.g. She is a smart student → She is a student & Bridging 88 Inferences supported by grammatical and lexical information & e.g. She loves sushi. → She likes sushi.
# Elaboration
88 Inference using known facts and general knowledge 88 e.g. The writer of Hamlet was Shakespeare → Shakespeare wrote Hamlet
74/69

# Ellipsis
& Recognizing implicit/omitted information & e.g. She is a smart student → She is a student & Bridging 88 Inferences supported by grammatical and lexical information & e.g. She loves sushi. → She likes sushi. # Elaboration 88 Inference using known facts and general knowledge 88 e.g. The writer of Hamlet was Shakespeare → Shakespeare wrote Hamlet 🌸 Meta-knowledge 88 Inference using external knowledge including a reader, writer, and text genre 88 e.g. Who is the main character in this story?
75/69