An Overview and Analysis ofReading Comprehension Tasks
Saku Sugawara @ Nagoya NLP seminar
2017-06-19

Introduction
Saku Sugawara / 菅原 朔
D1 (2017 Apr. -) @ Dept. of Computer Science, Univ. Tokyo
Aizawa lab. (National Institute of Informatics)
Interests:
natural language understanding, knowledge, inference, ...
saku (at) nii.ac.jp
http://penzant.net
2Today●Overview of reading comprehension (RC) tasks●Evaluation methodology for reading comprehension○ Prerequisite skills and MCTest analysis (AAAI 2017) [paper] [slide]○ Prerequisite skills and readability metrics (ACL 2017) [paper] [slide]●Observations○ The more skills required to answer, the more difficult for systems○ No correlation between “numbers of required skills” and “readability”

3Task Example - MCTest (2013)

Context + Question + Answer
4

Task Example

5Short history
Deep Read (Hirschman 1999) [paper]: Textbooks and WH questions↓2000s: Question answering / Information retrieval / Textual entailment○ QA tutorial @ NAACL2016 [slide] / TE @ ACLwiki [web] RTE [paper]↓2011-2013: Some reading comprehension tasks: QA4MRE, MCTest

●
●

●
↓

2015-2016: Large-scale & Automated○ bAbI (not RC?), CNN/Daily Mail, Children’s Book Test, Who-did-What●↓●

2016-2017: Large-scale & Crowdsourced○ SQuAD, LAMBADA, MS MARCO, NewsQA
6

Deep Read (1999)●●

Elementary level textbook (60docs w/ 300Qs)
Wh- questions / bag-of-words system
7

Recent Trends
QA4MRE (2011-2013) [paper1] [paper2] [web]MCTest (2013) [paper] [web] (not found now)bAbI (2015) [paper] [web] [slide]
CNN/Daily Mail (2015) [paper] [web]Children’s Book Test (2015) [paper] [web]SQuAD (2016) [paper] [web]
LAMBADA (2016) [paper] [web]Who-did-What (2016) [paper] [web]MS MARCO (2016) [paper] [web]NewsQA (2016) [paper] [web]8CLEF QA4MRE (2011-2013)
CLEF 2011-2013: Question Answering for
Machine Reading Evaluation [paper1] [paper2] [web]♠ Source: Technical documents○ Four topics: Alzheimer, AIDS, Climate Change, Music & Society (2013)♣ Formulation: Multiple choice (5 options including “none of the above”)♥ Pros○ Hard questions (several question types)○ Detailed analysis / Various languages○ Offering background knowledge○ Auxiliary (relaxed) questions♦ Cons○ Small questions / Limited topics
9

CLEF QA4MRE (2011-2013)10

MCTest (2013)
By Microsoft Research / EMNLP2013 [paper] [web] (not found now)
♠ Source: Stories written by crowdworkers♣ Formulation○ Multiple choice (4 options)○ Questions are also written by crowdworkers♥ Pros○ Story-based RC■ Characters’ intentions, relations of events, commonsense...
○ Limited vocabulary (written for children)♦ Cons○ Not large: 660 stories with 4 questions each11

MCTest (2013)

12

bAbI tasks (2015)
By Facebook AI Research / ICLR2016 [paper] [web] [slide]♠ Source: Automatically generated sentences♣ Formulation○ Context sentences + questions○ 20 tasks♥ Pros: Analysis is easy
○ Many systems are
not good at Path Finding♦ Cons○ Very small vocabulary○ No scalability for further RC
13

CNN/Daily Mail (2015)
By DeepMind / NIPS2015 [paper] [web]♠ Source: News articles (CNN/Daily Mail)♣ Formulation○ Cloze in article titles○ Title is regarded as a summaryof its content○ Entities are anonymized♥ Pros: Large (1.4M) / Multiple topics♦ Cons: Contain errors (cf. Chen+ (2016) [paper])○ Coreference errors and ambiguous/hard questions
14

CNN/Daily Mail - Example

15

CNN/Daily Mail - Attention

16

Children’s Book Test (2015)
By Facebook AI Research / ICLR2016 [paper] [web]♠ Source: Children’s books (from Project Gutenberg)♣ Formulation: Cloze for named entities, nouns, verbs, prepositions♥ Pros○ Large (688k) / Story-based reading♦ Cons○ Language modeling task (rather than RC)

from wikipedia17

Children’s Book Test (2015)18

SQuAD (2016)
By Stanford / EMNLP2016 [paper] [web]♠ Source: Wikipedia articles♣ Formulation: Text span selection♥ Pros: Large (100k) / Various topics♦ Cons: Not many questions that require multiple sentence reasoning

19

SQuAD (2016) - Example

20

LAMBADA (2016)
By Univ. Trento and Univ. Amsterdam / ACL2016 [paper] [web]♠ Source: Narratives (from Book Corpus by Zhu+ (2015) [paper])♣ Formulation: Cloze in the last sentence of a context○ More than 80% of passages include target words in the context♥ Pros: Low baseline (SOTA as of 2017S1: 0.49 by Chu+ (2017) [paper])♦ Cons○ Too difficult…? / Contextual word prediction task21

Who-did-What (2016)
By Toyota Technological Institute at Chicago / EMNLP2016 [paper] [web]♠ Source: News articles (Gigaword corpus v5)♣ Formulation: Cloze using two articles regarding the same topic and persons♥ Pros: Large (200K) / Automatically generated queries♦ Cons: Including nonsense questions

22

Who-did-What - Example

23

MS MARCO (2016)
By Microsoft Research [paper] [web]♠ Source○ Search query and web pages○ Human generated answers♣ Formulation○ Question + some passages○ Five question types: description, numeric, entity, person, and location♥ Pros○ Large (100K) / Including candidate (not directly related) passages♦ Cons○ Difficult to evaluate (not exact match but ROUGE/BLEU)
24

MS MARCO - Answer generation25

NewsQA (2016)
By Maluuba (MS) [paper] [web]♠ Source○ CNN articles♣ Formulation○ Crowdsourced QA○ Text span selection♥ Pros○ Large (120K) / “More challenging than existing datasets, e.g., SQuAD”♦ Cons○ Issue remains in human performance (See openreview of the paper)○ (However, what is “human performance”?)
26NewsQA - Example

27

Dataset at a Glance
DatasetYear

SourceFormulation

Query

Size (Q)QA4MRE

2013

Technical doc

Multiple choice (5 ops)

Experts240

MCTest2013

Narrative

Multiple choice (4 ops)

Crowdworkers

2640

bAbI

2015

Simple text

Word selection (context)

Automated

10K (*20)

CNN/Daily Mail

2015

News article

Cloze (word from context)

Automated

1.4M

Children’s Book Test

2015

Narrative

Cloze (word from candidates)

Automated

688K

SQuAD

2016

Wikipedia

Text span selection (context)

Crowdworkers

100K

Who-did-What

2016

News article

Cloze (word from context)

Automated

200K

LAMBADA

2016

Narrative

Cloze (word from context*0.8)

Crowdworkers

10K

MS MARCO2016

Web page

Text span selection* (context)

Search engine

100K

NewsQA

2016

News article

Text span selection (context)

Crowdworkers

120K
28

Dataset at a Glance
DatasetYear

SourceFormulation

Query

Size (Q)QA4MRE

2013

Technical doc

Multiple choice (5 ops)

Experts240

MCTest2013

Narrative

Multiple choice (4 ops)

Crowdworkers2640

bAbI

2015

Simple text

Word selection (context)

Automated

10K (*20)

CNN/Daily Mail

2015

News article

Cloze (word from context)

Automated

1.4M

Children’s Book Test

2015

Narrative

Cloze (word from candidates)

Automated

688K

SQuAD

2016

Wikipedia

Text span selection (context)

Crowdworkers

100K

Who-did-What

2016

News article

Cloze (word from context)

Automated

200K

LAMBADA

2016

Narrative

Cloze (word from context*0.8)

Crowdworkers

10K

MS MARCO2016

Web page

Text span selection* (context)

Search engine

100K

NewsQA

2016

News article

Text span selection (context)

Crowdworkers120K
29

Dataset at a Glance
DatasetYear

SourceFormulation

QuerySize (Q)QA4MRE

2013

Technical doc

Multiple choice (5 ops)

Experts240

MCTest2013

Narrative

Multiple choice (4 ops)

Crowdworkers

2640

bAbI

2015

Simple text

Word selection (context)

Automated

10K (*20)

CNN/Daily Mail

2015

News article

Cloze (word from context)

Automated

1.4M

Children’s Book Test

2015

Narrative

Cloze (word from candidates)

Automated

688K

SQuAD

2016

Wikipedia

Text span selection (context)

Crowdworkers

100K

Who-did-What

2016

News article

Cloze (word from context)

Automated200KLAMBADA

2016

Narrative

Cloze (word from context*0.8)

Crowdworkers

10K

MS MARCO2016

Web page

Text span selection* (context)

Search engine

100K

NewsQA

2016

News article

Text span selection (context)

Crowdworkers

120K
30

Dataset at a Glance
DatasetYear

SourceFormulation

Query

Size (Q)QA4MRE

2013

Technical doc

Multiple choice (5 ops)

Experts240

MCTest2013

Narrative

Multiple choice (4 ops)

Crowdworkers

2640

bAbI

2015

Simple text

Word selection (context)

Automated

10K (*20)

CNN/Daily Mail

2015

News article

Cloze (word from context)

Automated

1.4M

Children’s Book Test2015

Narrative

Cloze (word from candidates)

Automated

688K

SQuAD

2016

Wikipedia

Text span selection (context)

Crowdworkers

100K

Who-did-What

2016

News article

Cloze (word from context)

Automated

200K

LAMBADA

2016

Narrative

Cloze (word from context*0.8)

Crowdworkers

10K

MS MARCO2016

Web page

Text span selection* (context)

Search engine

100K

NewsQA

2016

News article

Text span selection (context)

Crowdworkers

120K
31

Dataset at a Glance
DatasetYear

SourceFormulation

Query

Size (Q)QA4MRE

2013

Technical doc

Multiple choice (5 ops)

Experts240

MCTest2013

Narrative

Multiple choice (4 ops)

Crowdworkers

2640

bAbI

2015

Simple text

Word selection (context)

Automated

10K (*20)

CNN/Daily Mail

2015

News article

Cloze (word from context)

Automated

1.4M

Children’s Book Test

2015

Narrative

Cloze (word from candidates)

Automated

688K

SQuAD

2016

Wikipedia

Text span selection (context)

Crowdworkers

100K

Who-did-What

2016

News article

Cloze (word from context)

Automated

200K

LAMBADA

2016

Narrative

Cloze (word from context*0.8)

Crowdworkers

10K

MS MARCO2016

Web page

Text span selection* (context)

Search engine

100K

NewsQA

2016

News article

Text span selection (context)

Crowdworkers

120K
32

Others...
By Allen Institute for AI (AI2) [web]● Elementary level science examquestions (Aristo) [web] [paper]● Textbook Question Answering[web] [paper]○ Diagrams and images
By Stanford● Stanford Natural Language
Inference Corpus [web] [paper]○ Automatically generated
textual entailment corpus○ 570k human-written pairs
33

Others...
●

FraCaS (1996) [web]●Winograd Schema Challenge(2011) [web]●Winograd Schema Challenge(2011) [web] [paper]

34

Issue●●Large dataset○ How can we analyze our systems?
Only with accuracy, we cannot tell what the systems understand and what
they don’t.

35

Research Question

Q: How can we evaluate and analyze our RC systems?
→ We propose an evaluation methodology for RC36

Research Question
Sub-questions:
Q1. How can we measure the difficulty of RC questions?
→ Propose “prerequisite skills”→ Analyze one RC dataset and systems

Q2. What is the relation between the readability of texts and the difficultyof questions?
→ Use “readability metrics” for dataset analysis→ Refine prereq. skills and analyze multiple RC datasets
37

Our Methodology

38

Our Methodology

39

Our Methodology

40Our Methodology

41

Our Methodology

42Our Methodology

43

1. Prerequisite Skills

44

1. Prerequisite Skills

45

Our Methodology

46

2. Annotation: MCTest (2013)

●
●
●

●
●

By Microsoft Research / EMNLP2013
Source: Stories written by crowdworkers
Formulation○ Multiple choice (4 options)○ Questions are also written by crowdworkers
Annotation: 320 questions (MC160+MC500)
Agreement: 85% (two annotators; for sampled questions)47

2. Annotation: MCTest (2013)48

2. Annotation: MCTest (2013)49

Our Methodology

50

3. Three Analyzed Systems

51

3. Annotation Result / Skills

52

3. Annotation Result / Skills

53

3. Annotation Result / Skills

54

3. Annotation Result / Skills

55

3. Annotation Result / Skills→The three systems are not good at these two skills?

56

3. Number of Required Skills

57

3. Number of Required Skills→ The more skills, the more difficult?
583. Annotation Result / Skills→ The more skills, the more difficult?
59Research Question - Next
Sub-questions:
Q1. How can we measure the difficulty of RC questions?
A1. Required numbers of prerequisite skills
Next:
Q2. What is the relation between the readability of texts and the difficultyof questions?
→ Use “readability metrics” for dataset analysis→ Refine prereq. skills and analyze multiple RC datasets
60

Why Readability?

61

Why Readability?

62

Why Readability?

63Prereq. Skills and Readability

64

Readability Metrics
10 readability metrics from Vajjala and Meurers (2012) [paper]
●
●
●
●
●
●
●
●
●
●

Ave. Num. of characters per word (NumChar)
Ave. Num. of syllables per word (NumSyll)
Ave. sentence length in words (MLS)
Proportion of words in AWL (AWL)
Modifier variation (ModVar)
Num. of coordinate phrases per sentence (CoOrd)
Coleman-Liau index (Coleman)
Dependent clause to clause ratio (DC/C)
Complex nominals per clause (CN/C)
Adverb variation (AdvVar)
65

Readability Metrics

66

Refined Prerequisite Skills

67

Four Reasoning Skills●Ellipsis○ Recognizing implicit/omitted information (argument, predicate,quantifier, time, or place)○ e.g. She is a smart student → She is a student●Bridging○ Inferences between two facts supported by grammatical and conceptual
knowledge (synonymy, hypernymy, thematic role, part of events, idioms,and apposition)○ e.g. She loves sushi. → She likes sushi.
68Four Reasoning Skills●Elaboration○ Inference using known facts, general knowledge (e.g., kinship, exchange,typical event sequence, and naming), and implicit relations (e.g., noun
compounds and possessives)○●e.g. The writer of Hamlet was Shakespeare → Shakespeare wrote Hamlet

Meta-knowledge○ Inference using external knowledge including a reader, writer, character,and genre○ e.g. Who is the main character in this story?
69

Analyzed Six Datasets

Annotation:
6*100 questions
13 skills
4 annotators
90% agreement70

Skill Frequencies (%)

71

Skill Frequencies (%)

MCTest:Narratives with characters

72

Skill Frequencies (%)

QA4MRE:
Questions are written by experts73

Skill Frequencies (%)

Who did What (WDW):
Automatically generated queries
MS MARCO:
Search queries and web pages

74

Number of Required Skills

75Number of Required Skills

76Readability Metrics

77

Readability Metrics

Flesch= Flesch-Kincaid grade level= the number of years required to understand the text

78

Skills and Readability

79

Skills and Readability

80

Obsearvation
There is only a weak correlation between readability metrics and numbers ofrequired skills● “Difficult to read” does not mean “difficult to answer”● It is possible to create a dataset that consists of an easy-to-read context
and difficult-to-answer questions

Q2. What is the relation between the readability of texts and the difficultyof questions?
A2. There is only a weak correlation between them81Textual Entailment and RC
Textual entailment (cf. RTE, FraCaS, SNLI)
Premise(s) → hypothesis
Reading comprehension
Multiple premises in context → hypothesis as Q+A
Our methodology cannot evaluate the following processes in RC:● Multiple premises○ Skill for gathering premises from context sentences● Hypothesis○ Skill for choosing and generating hypothesis from answer candidates
82

Research Question Summary
Sub-questions:
Q1. How can we measure the difficulty of RC questions?
A1. Required numbers of prerequisite skills

Q2. What is the relation between the readability of texts and the difficultyof questions?
A2. There is only a weak correlation between them83

Summary●Overview of reading comprehension (RC) tasks○ (Deep Read,) QA4MRE, MCTest, bAbI, CNN/Daily Mail, CBT, SQuAD,LAMBADA, Who-did-What, MS MARCO, NewsQA ----- 10 datasets!

●Evaluation methodology for reading comprehension○ Prerequisite skills and system/dataset analysis (AAAI 2017)○ Prerequisite skills and readability metrics (ACL 2017)●Observations○ The more skills required to answer, the more difficult for systems○ No correlation between “numbers of required skills” and “readability”
84