Evaluation Metrics for
the Machine Reading Comprehension Task
Saku Sugawara
University of Tokyo
October,@ Google Tokyo

http://penzant.net/

Agenda. Overview of the machine reading comprehension (RC) task
“What is the RC task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”
Sugawara+ (a, AAAI) & Sugawara+ (b, ACL). Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”/

Agenda. Overview of the machine reading comprehension (RC) task
“What is the RC task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”
Sugawara+ (a, AAAI) & Sugawara+ (b, ACL). Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”/

Reading Comprehension (RC) Task
ID: MCTest MC.dev. ( ) multiple:
Context: The princess climbed out the window of the hightower and climbed down the south wall when her motherwas sleeping. She wandered out a good ways. Finally she
went into the forest where there are no electric poles butwhere there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle

Task: context, question, and answer/

Reading Comprehension (RC) Task
ID: MCTest MC.dev.

( ) multiple:

C : The princess climbed out the window of the high tower andclimbed down the south wall when her mother was
sleeping.
C : She wandered out a good ways.
C : Finally she went into the forest where there are no electricpoles but where there are some caves.
Q: Where did the princess wander to after escaping?
A: A) Mountain *B) Forest C) Cave D) Castle
Coreference resolution (she = princess)Commonsense reasoning (escaping = climbed down)
Temporal relation (climbed → wandered)/Definitions
Towards the Machine Comprehension of Text: An Essay (Burges)“a machine comprehends a passage of text if, for any question
regarding that text that can be answered correctly by a majority ofnative speakers, that machine can provide a string which those
speakers would agree both answers that question”
Overview of QA MRE Main Task at CLEF

(Sutcliffe+)“RC tests do not require only semantic understanding but they assume
a cognitive process which involves using implications andpresuppositions, retrieving the stored information, performing
inferences to make implicit information explicit.”→ the RC task is an evaluation method for language understanding systems
in terms of their behavior, and tests a cognitive process that involves several
skills, such as performing inferences using background knowledge, by letting
the system answer questions about a given text.
/

Representative Datasets

MCTest ()
CNN/Daily Mail (SQuAD ())/

MCTest ()

Richardson et al.
Children stories andquestions written by
crowdworkers
Multiple choice
Pros:
Story-based RC =characters’ intentions,relations of events,
commonsense...
Limited vocabularyCons
Not large:
stories withquestions each/

CNN/Daily Mail (Hermann et al.
News articles
Cloze (fiil-in-blank)
Answer extraction)Pros
M Qs, various topics

Cons
Contains errors incoreference (Chen +)/SQuAD ()

Rajpurkar et al.
Wikipedia articles
Answer extraction
Pros
K Qs, various topics

Cons
Goldberg () says “patternmatching”
Easily fooled by Jia+ ()’sadversarial examples/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/

Datasets
DatasetQA MRE
MCTestCNN/
Daily Mail
SQuAD
Who-didWhat
MS
MARCO
NewsQA
TriviaQA
RACE

Year

Question/
AnswerGiven
text

Genre or
SourceComplete/
Technical
Multichoice Document documents
Complete/
CraftedMultichoice Paragraphs
story
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraph Wikipedia
News
Cloze/
Extraction Document
articleComplete/
Extraction Paragraphs Web page
News
Complete/
Extraction Document
articleWikipedia/
Complete/
Extraction Paragraph Web page
Complete/
Exam
Multichoice Document

Question Question
sourcing
size

ReaderExpertStudent

Crowdworker
Automated
Crowdworker
Automated
SearchengineCrowdworker
QuizwebsiteExpert

Child
. M

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Adult

K

Student/RC and Question Answering
Question answering
Q + A without the explicit context
e.g., Q: ’What is solid CO commonly called?’ A: dry ice
Include “searching” from the resources/RC and Question Answering
Question answering
Q + A without the explicit context
e.g., Q: ’What is solid CO commonly called?’ A: dry ice
Include “searching” from the resources
Reading comprehension as question answering
Q + A with the explicit context
Inquire not only general knowledge but thecontext-dependent information about temporalsituations/stories/

RC and Textual Entailment
Textual entailment
Recognizing and testing:premise → hypothesise.g., Premise: ’alcohol reduces blood pressure’→ Hypothesis: ’alcohol affects blood pressure’/RC and Textual Entailment
Textual entailment
Recognizing and testing:premise → hypothesise.g., Premise: ’alcohol reduces blood pressure’→ Hypothesis: ’alcohol affects blood pressure’
Reading comprehension as textual entailment
Recognizing and testing:multiple premises (from context) → hyposhesis (from Q+A)
Cf. FraCaS (Cooper,) (issues: simple & small)+Etzioni (): “Machine reading [...] combines multiple
textual entailment steps to form a coherent set of beliefs
based on the text.” (see also Manning ())/

Agenda. Overview of the machine reading comprehension (RC) task
“What is the RC task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”. Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”/

Motivation: Accuracy is not Enough
Dataset A

System X

Q
Q
Q
..
.

x
o
x
..
.

Q

o

Accuracy. %→ Only with accuracy, we cannot tell what the systems
understand and what they don’t.
Chen+ () shows: CNN/Daily Mail datasets containunanswerable or ambiguous questions/

Our Research Question

How can we evaluate and analyze our RC systems?
→ Propose evaluation metrics for RC→ Focus on prerequisite skills and readability/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not subsequently professed their own faith?
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower andclimbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not subsequently professed their own faith?
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower andclimbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not subsequently professed their own faith?
→ Answerble simply by noticing one sentence
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower andclimbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not subsequently professed their own faith?
→ Answerble simply by noticing one sentence
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower
and climbed down the south wall when her mother was sleeping.
She wandered out a good ways. Finally she went into the forest where
there are no electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not→subsequently
professed
their own
Answerble simply
by noticing
onefaith?
sentence
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower
and climbed down the south wall when her mother was sleeping.
She wandered out a good ways. Finally she went into the forest where
there are no electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
→ Requrereading
multileAnswer: A) Mountain*B) ForestC) CaveD)sentenceCastle with skills/

Motivation: Two Types of Difficulties
ID: SQuAD (), United_Methodist_Church
Context: The United Methodist Church (UMC) practices infant andadult baptism. Baptized Members are those who have been baptized
as an infant or child, but who have not subsequently professed theirown faith.
Question: What are members who have been baptized as an infant orchild but who have not subsequently professed their own faith?
Difficult-to-read & Easy-to-answer
Answer: Baptized Members
ID: MCTest (
), mc
.dev.
Context: The princess climbed out the window of the high tower
and climbed down the south wall when her mother was sleeping.
She wandered out a good ways. Finally she went into the forest where
there are no electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
& Difficult-to-answer
Answer: A) Mountain *B) Forest Easy-to-readC) Cave D) Castle/

Our study: Evaluation Metrics for RC. Define prerequisite skills and readability metrics
prerequisite skills andreadability measures/

Our study: Evaluation Metrics for RC. Define prerequisite skills and readability metrics
prerequisite skills andreadability measures. Annotate RC datasets with skills
datasets: QA MRE, MCTest, SQuAD, Who-did-What, MS
MARCO, NewsQA/

Our study: Evaluation Metrics for RC. Define prerequisite skills and readability metrics
prerequisite skills andreadability measures. Annotate RC datasets with skills
datasets: QA MRE, MCTest, SQuAD, Who-did-What, MS
MARCO, NewsQA. Calculate readability of the datasets
Readability of “context sentences necessary for answering”(selected in the annotation) (6= whole context)/

Our study: Evaluation Metrics for RC. Define prerequisite skills and readability metrics
prerequisite skills andreadability measures. Annotate RC datasets with skills
datasets: QA MRE, MCTest, SQuAD, Who-did-What, MS
MARCO, NewsQA. Calculate readability of the datasets
Readability of “context sentences necessary for answering”(selected in the annotation) (6= whole context). Analyze the datasets on two types of difficulties
See the relation between skills and readability/

System Analysis by Accuracy

Dataset A

System X

Q
Q
Q
..
.

x
o
x
..
.

Q

o

Accuracy. %/System Analysis by the Skills and Readability
Dataset A
Prerequisite Skills
Question Skill Skill · · · Skill

Readability Metrics
RM RM · · · RM

System X

Q
Q
Q
..
.

x
x
..
.

o
x
..
.

···
···
···
..
.

x
..
.

.
.
.
..
.

.
.
.
..
.

···
···
···
..
.

.
.
.
..
.

x
o
x
..
.

Q

o

o

···

o

.

.

···

.

o

. %

. % ···

. %

-

-

···

-

. %

AccuracyThis study shows statistics of datasets & an observation on the relation
between skills (difficulty in answering) and readability (difficulty in reading)/Prerequisite Skills. Object tracking. Mathematical reasoning. Coreference resolution. Logical reasoning. Analogy. Causal relation. Spatiotemporal relation. Ellipsis. Bridging. Elaboration. Meta-knowledge. Schematics clause relation. Punctuation

New knowledge reasoning skills in this study“Commonsense reasoning” is updated to new
more detailed analysis

skills for/Prerequisite Skills. Object tracking. Mathematical reasoning. Coreference resolution. Logical reasoning. Analogy. Causal relation. Spatiotemporal relation. Ellipsis. Bridging. Elaboration. Meta-knowledge. Schematics clause relation. Punctuation

Previous study (Sugawara+ , AAAI): skills are based on
existing NLU tasks in NLP→ Analyzed MCTest dataset and three systems, and showedthat “the more skills are required, the more difficult toanswer (lower accuracy).”
We regard this as the difficulty of answering/Annotated RC Datasets:

Qs for each

RC dataset

Genre

Query
sourcing

Task
formulation

QA MRE()
MCTest
(
)
SQuAD
(
)
Who-did-What()
MS MARCO()
NewsQA()

Technical
documents
Narratives bycrowd workers
Wikipedia
articlesNews articles(Gigaward v )Segmented
web pages
News articles

Handcraftedby experts
Crowd sourced

MultiplechoiceMultiplechoiceText span
selection
Cloze

Crowd sourced
Automated fromother articles
Search engine
queries
Crowd sourced

Description
Text span
selection/

Annotation with the Prerequisite Skills
skillls: mult-label annotation
datasets: QA MRE, MCTest, SQuAD, Who-did-What, MS
MARCO, NewsQA
questions for each dataset
annotators: graduate NLP studentsFor
randomly sampled questions, . % agreement
Annotation: choose skill labels and “necessary context
sentences” for answering
Sentences are used to calculate readability measures/

Result: Frequencies (%) of Prerequisite Skills
Skills. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation
Nonsense

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

10.0

1.0

3.0

27.0

14.0

1.0
/

Result: Frequencies (%) of Prerequisite Skills
Skills. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation
Nonsense

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

10.0

1.0

3.0

27.0

14.0

1.0
/

Result: Frequencies (%) of Prerequisite Skills
Skills. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation
Nonsense

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

10.0

1.0

3.0

27.0

14.0

1.0
/

Result: Frequencies (%) of Prerequisite Skills
Skills. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation
Nonsense

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

10.0

.

.

.

14.0

.
/

Result: Numbers of Required Skills#Skills QA MRE MCTest SQuAD WDW MARCO NewsQA

Ave.

2.0
13.0
13.0
20.0
14.0
13.0
10.0
1.0
1.0
0.0
3.0

18.0
36.0
24.0
15.0
4.0
1.0
1.0
0.0
0.0
0.0
0.0

27.0
33.0
24.0
6.0
6.0
1.0
0.0
0.0
0.0
0.0
0.0

2.0
5.0
14.0
22.0
16.0
6.0
6.0
2.0
0.0
0.0
0.0

15.0
35.0
29.0
6.0
2.0
0.0
0.0
0.0
0.0
0.0
0.0

13.0
26.0
23.0
25.0
9.0
2.0
1.0
0.0
0.0
0.0
0.0

3.25

1.56

1.28

2.43

1.19

1.99

/

Result: Numbers of Required Skills#Skills QA MRE MCTest SQuAD WDW MARCO NewsQA

Ave.

2.0
13.0
13.0
20.0
14.0
13.0
10.0
1.0
1.0
0.0
3.0

18.0
36.0
24.0
15.0
4.0
1.0
1.0
0.0
0.0
0.0
0.0

27.0
33.0
24.0
6.0
6.0
1.0
0.0
0.0
0.0
0.0
0.0

2.0
5.0
14.0
22.0
16.0
6.0
6.0
2.0
0.0
0.0
0.0

15.0
35.0
29.0
6.0
2.0
0.0
0.0
0.0
0.0
0.0
0.0

13.0
26.0
23.0
25.0
9.0
2.0
1.0
0.0
0.0
0.0
0.0

3.25

1.56

1.28

2.43

1.19

1.99

/

Calculation of Readability
Ave. Num. of characters per word (NumChar)
Ave. Num. of syllables per word (NumSyll)
Ave. sentence length in words (MLS)
Proportion of words in AWL (AWL)
Modifier variation (ModVar)
Num. of coordinate phrases per sentence (CoOrd)
Coleman-Liau index (Coleman)
Dependent clause to clause ratio (DC/C)
Complex nominals per clause (CN/C)
Adverb variation (AdvVar)
Figure:readability measure from Vajjala and Meurers ().

/

Result: Readability Metrics
Mesures

QA MRE

NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar

.
.

F–K
Words
∗

.
.
.
.
.
.
.
.
.
.

MCTest SQuAD

WDW

.8
.
.8 8
.
.
.
.
.
.6
.

.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.6

.
.

.
.
.
.
.
.
.
.
.
.

.
.

MARCO NewsQA
.
.
.
.
.
.
.
.
.
.

.
.

.
.

.

F–K = Flesch-Kincaid grade level= education level required to understand the text.
/

Relation btwn Skills and Readability
16

F-K grade level

14
12

WDW

SQuAD
NewsQAMARCOQA4MRE

108
6

MCTest

4
21.0

1.5

2.0

2.5

3.0

Average number of required skills

3.5/

Observation

There is only a weak correlation between readability
metrics and numbers of required skills→ Difficult to read ; difficult to answer (and vice versa)→ It is possible to create a dataset that consists of aneasy-to-read context and difficult-to-answer questions.

/How to Use This Observation?
For development of RC systems, select datasets in the following
steps:
easy-to-read & easy-to-answer
dataseteasy-to-read & difficult-to-answer
datasetdifficult-to-read & easy-to-answer
datasetdifficult-to-read & difficult-to-answer
dataset/Short Discussion
Answerability of questions
We cannot evaluate the difference among “truely difficult,”“non sense,” and “no answer” when we encounter systems’
incorrect answers (and even in human performance!)
It is not easy to maintain the quality of questions especiallyin crowd-based sourcing.

/Short Discussion
Answerability of questions
We cannot evaluate the difference among “truely difficult,”“non sense,” and “no answer” when we encounter systems’
incorrect answers (and even in human performance!)
It is not easy to maintain the quality of questions especiallyin crowd-based sourcing.
Corpus genre
Are Wikipedia or news articles enough?
Example: narratives are close to our everyday experience(characters’ emotions, intentions, and actions)/Summary of the Second Part
Proposed evaluation methodology for RC. Defined two classes of metrics:
prerequisite skills and readability. Annotated RC datasets with the skills. Calculated readability of datasets. Analyzed datasets→ Results can be used for evaluation of systems
Observation
There is only a weak correlation between readability
metrics and numbers of required skills/

Agenda. Overview of the machine reading comprehension (RC) task
“What is the RC task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”. Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”/

Agenda. Overview of the machine reading comprehension (RC) task
“What is the RC task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”. Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”→ What factors affect the difficulty of RC questions?

/How do we solve RC questions?
. Read a given text and understand the meaning of aquestion about it
e.g. which type of entitiy is required.

/How do we solve RC questions?
. Read a given text and understand the meaning of aquestion about it
e.g. which type of entitiy is required.

. If needed (e.g., in answer extraction questions), find answer
candidates from the context/

How do we solve RC questions?
. Read a given text and understand the meaning of aquestion about it
e.g. which type of entitiy is required.

. If needed (e.g., in answer extraction questions), find answer
candidates from the context. Make hypotheses using the question and each answer
candidate/

How do we solve RC questions?
. Read a given text and understand the meaning of aquestion about it
e.g. which type of entitiy is required.

. If needed (e.g., in answer extraction questions), find answer
candidates from the context. Make hypotheses using the question and each answer
candidate. Test each hypothesis whether it can be entailed from agiven text or its sub-sentences/

How do we solve RC questions?
. Read a given text and understand the meaning of aquestion about it
e.g. which type of entitiy is required.

. If needed (e.g., in answer extraction questions), find answer
candidates from the context. Make hypotheses using the question and each answer
candidate. Test each hypothesis whether it can be entailed from agiven text or its sub-sentences. Choose a hypothesis that is most likely to be entailed, andreturn its corresponding candidate/

Parameters of the Difficulty

A. Understanding a question itself
Involving some skills such as coreference resolution→ Easier if a question does not require background knowledge(difficult example: “what is the main theme of this passage?”)/Parameters of the Difficulty

A. Understanding a question itself
Involving some skills such as coreference resolution→ Easier if a question does not require background knowledge(difficult example: “what is the main theme of this passage?”)

B. Identifying answer candidates
Recognizing entity types (wh-) and clause (why/how)→ Easier if there are few answer candidates (e.g., there is only
one temporal expression in the context for a when question)
Multiple choice seems better (cf. Levesque ())/

Parameters of the Difficulty

C. Choosing evidential sentence(s) from a given text
Which sentence is informative for examining a hypothesis→ Easier if the number of sentences is small (like TE)/

Parameters of the Difficulty

C. Choosing evidential sentence(s) from a given text
Which sentence is informative for examining a hypothesis→ Easier if the number of sentences is small (like TE)

D. Testing entailment between evidnece and hypothesis
How many prerequisite skills are required(Partly revealed by our evaluation methodology)→ Easier if no skill is required (e.g., the hypothesis is almost
similar to the evidence)/Summary. Overview of the machine reading comprehension task
“What is the reading comprehension (RC) task?”. Evaluation methodology for RC datasets/systems“How can we evaluate our systems/datasets?”. Discussion for constructing RC datasets:“How can we create difficult but not too difficult questions?”/Reference I
Burges () Towards the Machine Comprehension of Text: An Essay, technical report.

Chen et al. () A Thorough Examination of the CNN/Daily Mail Reading
Comprehension Task, in ACL.
Cooper et al. () Using the Framework, by the FraCaS Consortium.

Etzioni et al. () Machine Reading, in AAAI.

Goldberg () Near human performance in question answering?
(http://u.cs.biu.ac.il/~yogo/on-squad.pdf).
Hermann et al. () Teaching Machines to Read and Comprehend, in NIPS.

Jia et al. (EMNLP.

) Adversarial Examples for Evaluating Reading Comprehension Systems, in

Lai et al. (in EMNLP.

) RACE: Large-scale ReAding Comprehension Dataset From Examinations,

Levesque () On our best behaviour, in IJCAI.

Joshi et al. () TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for
Reading Comprehension, in ACL.
Manning () Local textual inference: Its hard to circumscribe, but you know it when
you see it–and NLP needs it, unpublished manuscript.
/

Reference II
Nguyen et al. (Dataset.

) MS MARCO: A Human Generated MAchine Reading COmprehension

Onishi et al. (EMNLP.

) Who did What: A Large-Scale Person-Centered Cloze Dataset, in

Rajpurkar et al. (in EMNLP.

) SQuAD:,+ Questions for Machine Comprehension of Text,

Richardson et al. () MCTest: A Challenge Dataset for the Open-Domain Machine
Comprehension of Text, in EMNLP.
Sugawara et al. (a) Prerequisite Skills for Reading Comprehension: Multi-perspective
Analysis of MCTest Datasets and Systems, in AAAI.
Sugawara et al. (b) Evaluation Metrics for Machine Reading Comprehension:
Prerequisite Skills and Readability, in ACL.
Sutcliffe et al. () Overview of QA MRE Main Task at CLEF

Trischler et al. () NewsQA: A Machine Comprehension Dataset, in REP NLP.

, in CLEF.

/

/Appendix: Correlation btwn Readability andthe Skills
Metricsrp

NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

F–K

.

.

Table: Pearson’s correlation coefficients (r) with the p-values (p) in all
RC datasets/

Appendix: Knowledge reasoning
Ellipsis
Recognizing implicit/omitted information
e.g. She is a smart student → She is a student/

Appendix: Knowledge reasoning
Ellipsis
Recognizing implicit/omitted information
e.g. She is a smart student → She is a student

Bridging
Inferences supported by grammatical and lexical information
e.g. She loves sushi. → She likes sushi.

/

Appendix: Knowledge reasoning
Ellipsis
Recognizing implicit/omitted information
e.g. She is a smart student → She is a student

Bridging
Inferences supported by grammatical and lexical information
e.g. She loves sushi. → She likes sushi.

Elaboration
Inference using known facts and general knowledge
e.g. The writer of Hamlet was Shakespeare → Shakespeare
wrote Hamlet/

Appendix: Knowledge reasoning
Ellipsis
Recognizing implicit/omitted information
e.g. She is a smart student → She is a student

Bridging
Inferences supported by grammatical and lexical information
e.g. She loves sushi. → She likes sushi.

Elaboration
Inference using known facts and general knowledge
e.g. The writer of Hamlet was Shakespeare → Shakespeare
wrote Hamlet

Meta-knowledge
Inference using external knowledge including a reader, writer,
and text genre
e.g. Who is the main character in this story?
/