Prerequisite Skills for Reading Comprehension: Multi-perspective Analysis of MCTest Datasets and Systems *Saku Sugawara (The University of Tokyo) Hikaru Yokono (Fujitsu Laboratories Ltd.) Akiko Aizawa (National Institute of Informatics, Japan) AAAI-17 February 8, 2017
1/36

What is reading comprehension?
ID:  MCTest MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A:  A) Mountain   *B) Forest   C) Cave   D) Castle
ID:  MCTest MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A:  A) Mountain   *B) Forest   C) Cave   D) Castle Coreference resolution (she = princess) Commonsense reasoning (escaping = climbed down) Temporal relation (climbed → wandered)
3/36

→ Only with accuracy, we cannot tell what the systems understand and what they don't.
4/36

ゃHow can we evaluate and analyze our RC systems? →  Simgle accuracy-based analysis is insufficient!!! →  We need a methodology for more detailed analysis
5/36
Our study: A Methodology for Evaluation of Reading Comprehension Consists of three steps: 1. Define prerequisite skills 2.  Annotate existing RC tasks with the skills 3. Analyze datasets and systems

7/36

1. Define prerequisite skills
2. Annotation RC questions with defined skills

9/36

3. Analyze datasets and systems

10 / 36

1. Define prerequisite skills
3. Analyze datasets and systems
11/36

The asterisks ( " ) with items represent "understanding of." / CoNLLst2015 = Shallow Discourse Parsing for PDTB
The asterisks ( " ) with items represent "understanding of." / CoNLLst2015 = Shallow Discourse Parsing for PDTB
Context: The name of John's sister is Sylvia. John was annoyed because his sister ate his cake. Q1: Why was John annoyed? A1: Because his sister ate his cake.
14 / 36

Context: The name of John's sister is Sylvia. John was annoyed because his sister ate his cake. Q1: Why was John annoyed? A1: Because his sister ate his cake. Required skill: } Causal relation Why → Because …
15/36

Context: The name of John's sister is Sylvia. John was annoyed because his sister ate his cake. Q2: Why was John irritated? A2: Because Sylvia ate his cake.
16 / 36

Context: The name of John's sister is Sylvia. John was annoyed because his sister ate his cake. Q2: Why was John irritated? A2: Because Sylvia ate his cake. Required skills: ÷ Causal relation Why → Because … → Commonsense reasoning annoyed = irritated The name of John's sister is Sylvia.  = Sylvia is John's sister. ÷ Coreference resolution his sister = John's sister
17 / 36

1. Define prerequisite skills 2.  Annotate existing RC tasks with the skills 3. Analyze datasets and systems
18 / 36

ゃMCTest development sets [Richardson+ 2013] Corpus: narratives written for young children Task formulation: multiple choice 320 questions from the development sets from MC160+MC500 Agreement: 85% (two annotators; for sampled questions) Additional:ゃSQuAD development set [Rajpurkar+ 2016] 4 Corpus: expository texts from Wikipedia ✿ Task: selecting text span in context 80 questions from the development set (v1.1) Details are reported in Sugawara+ (2016) EMNLP workshop
19 / 36

ID:  MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A: Forest ➪ Coreference resolution: · She in C2 = the princess in C1 · She in C3 = the princess in C1やTemporal relation: · the actions in C1 → wandered out … in C2 → went into … in C3
20 / 36

ID:  MC160.dev.29 (1) multiple: C1:  The princess climbed out the window of the high tower and climbed down the south wall when her mother was sleeping. C2:  She wandered out a good ways. C3:  Finally she went into the forest where there are no electric poles but where there are some caves. Q:  Where did the princess wander to after escaping? A: Forest  Commonsense reasoning: · escaping in Q ⇒ the actions in C1 · wandered out in C2 and went into the forest in C3 ⇒ wander to the forest in Q and A ➥ S/R clause (=complex) sentence and Special sentence structure: · C1 = the princess climbed out ... and [the princess] climbed down ...  (ellipsis)
21/36

1. Define prerequisite skills 2.  Annotate existing RC tasks with the skills 3. Analyze datasets and systems
22/36

1.  Baseline SW+D [Richardson+ 2013] → Sliding window + word distance algorithm 2.  Smith LexMatch [Smith+ 2015] → Baseline + lexical matching method ✈ (stemming + question type + coreference) 3.  Yin ABCNN [Yin+ 2016] → Attention-based CNN without any linguistic features ✈ Answers a question as textual entailment
23/36

Table: Frequencies and accuracies (%) for each prereq. skill.
❖ The three systems are not good at these two skills? → At least we can say that "need more development for them" 27 / 36 Annotation Results - Numbers of required skills
Table: Frequencies and accuracies (%) for required numbers of prereq. skills for each question. Annotation Results - Numbers of required skills
↑ Observation: the more required skills, the more difficult?
Table: Frequencies and accuracies (%) for each prereq. skill.
MCTest: narratives with characters and events SQuAD: expository articles
SQuAD: written for adults (Wikipedia) and questions are paraphrased by crowdworkers who wrote them
32/36

Table: Frequencies and accuracies (%) for required numbers of prereq. skills for each question.
& Mathematical ops.  and Logical rsng.  were few required →  We will try to annotate other (specialized) datasets. (e.g., Aristo and FraCaS) to check if the skill definitions are valid.ゃAnalogy was also few required → Was the definition ambiguous? : Commonsense reasoning needs fine-grained subclasses →  We will take entailment phenomena into account. →  There are many classifications of knowledge and inference in AI/NLP and psychology.
35/36
Summary
We proposed an evaluation methodology for RC: 1. Defined ten prerequisite skills in terms of NLU tasks 2.  Annotated existing RC tasks with the skills 3.  Analyzed datasets and systems Observation: : There may be the coreelation of number of required skills in each question and the difficulty of the question. Thank you!!ゃAnnotation data is now available!! http://www-al.nii.ac.jp/mctest-rcskills-annot/ ✿ Questions, comments: sakus(at)is.s.u-tokyo.ac.jp
36/36

ゃTextual entailment → Recognizing and testing: premise → hypothesis ✿ Reading comprehension as textual entailment → Recognizing and testing: Multiple premises → hypothesis Issue: Our methodology cannot evaluate the following processes:ゃMultiple premises ← gathered from context sentencesゃhypothesis ← generated from answer candidates ⇒  depending on context length and question formulation
38 / 36

We can see the weakness of a system from the difference of combinations of skills: ExampleゃAccuracy for (skill 1, skill 4, skill 5): 60% Accuracy for (skill 1, skill 4, skill 7): 80% Accuracy for (skill 1, skill 4): 80% →  not good at skill 5?
39 / 36