Evaluating Natural Language Understandingin Machine Reading Comprehension

Saku Sugawara
Department of Computer Science, University of Tokyo
PhD defense, January,/Evaluation of Natural Language Understanding
Goal
Developing a system that understand human languages= Computationally modeling language understanding→ Studying human language understanding/Evaluation of Natural Language Understanding
Goal
Developing a system that understand human languages= Computationally modeling language understanding→ Studying human language understanding
Tasks
Turing Test ()
Question answering (s-)
Recognizing textual entailment (Machine reading comprehension (

-)
-)
/

Machine Reading Comprehension (MRC) Task

ID: MCTest MC.dev. ( ) multiple:
Context: The princess climbed out the window of the high tower andclimbed down the south wall when her mother was sleeping. She wandered out a good ways. Finally she went into the forest where there
are no electric poles but where there are some caves.
Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Machine Reading Comprehension (MRC) Task
ID: MCTest MC.dev. ( ) multiple:
C : The princess climbed out the window of the high tower and climbed
down the south wall when her mother was sleeping.
C : She wandered out a good ways.
C : Finally she went into the forest where there are no electric poles butwhere there are some caves.
Q: Where did the princess wander to after escaping?
A: A) Mountain *B) Forest C) Cave D) Castle
Coreference resolution (she = princess)Commonsense reasoning (escaping = climbed down)
Temporal relation (climbed → wandered)/

MRC Datasets and Systems
Datasets
MCTest ( K) QA MRE (bAbI (

Systems
Feature-based models
LSTM-based models)

K) CNN/Daily Mail ( . M) CBT (K)(BiDAF: . M)
SQuAD (K) WikiReading ( M) LAMBADA (did-What (K) NewsQA (K) MS MARCO (K) WhoK)

TriviaQA (K) RACE (K) QAngaroo (tiveQA ( K) MCScript ( K) ...

K) Narra-

ARC ( K) CliCR (K) MultiRC ( K) SQuAD . (DuoRC (K) HotpotQA ( K) QuAC (K) CoQA (...
DROP (K) ReCoRD (K) MCScript . (K)
K)

Transformer-based models(GPT- /BERT/XLNet:
M?)K) ...

/

MRC Datasets and Systems
Datasets
MCTest ( K) QA MRE (bAbI (

Systems
Feature-based models
LSTM-based models)

K) CNN/Daily Mail ( . M) CBT (K)(BiDAF: . M)
SQuAD (K) WikiReading ( M) LAMBADA (did-What (K) NewsQA (K) MS MARCO (K) WhoK)

TriviaQA (K) RACE (K) QAngaroo (tiveQA ( K) MCScript ( K) ...

K) Narra-

ARC ( K) CliCR (K) MultiRC ( K) SQuAD . (DuoRC (K) HotpotQA ( K) QuAC (K) CoQA (...
DROP (K) ReCoRD (More thanK) MCScript . (datasets

K)
K)

Transformer-based models(GPT- /BERT/XLNet:
M?)K) ...

Huge models/

Major Datasets
Dataset
MCTest
CNN/Daily
Mail
SQuAD
RACE
HotpotQA
DROP

Year

Domainchildrenstories
news
articlesWikipedia
articlesEnglishexams
open domain
in Wikipedia
Wikipedia
articles

Sourcing

Ans Style

Focus

automated

multiplechoiceentitycloze

crowdsourced

extraction

experts

multiplechoicefirst largescale dataset
large scale,written by humans
written by expert,
various domains

crowdsourced

extraction

multihop reasoning

crowdsourced

generation

discrete reasoning

crowdsourced

first dataset/

Systems achieved human-level performance...

/Issue : Evaluation Metrics
Systems achieved human-level performance. But the simple metric tells us...
Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %/Issue : Evaluation Metrics
Systems achieved human-level performance. But the simple metric tells us...
Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %→What skills the system has???
Corefernece resolution?
Commonsense reasoning?
Logical reasoning?
Understanding discourse
relations?

/Issue : Evaluation Metrics
Systems achieved human-level performance. But the simple metric tells us...
Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %→What skills the system has???
Corefernece resolution?
Commonsense reasoning?
Logical reasoning?
Understanding discourse
relations?

No interpretability and explainability/

Issue : Question Quality in Recent Datasets
Adversarial examples: [Jia and Liang,]
MRC models are fooled by manually injected distracting sentences
Context:

Peyton Manning is the oldest quarterback ever to play in a Super Bowl
at age . The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age
and is currently Denver’s
Executive Vice President of Football Operations.
Question: What is the name of the quarterback who wasin Super Bowl
XXXIII?
Predictions: John Elway/

Issue : Question Quality in Recent Datasets
Adversarial examples: [Jia and Liang,]
MRC models are fooled by manually injected distracting sentences
Context:

Peyton Manning is the oldest quarterback ever to play in a Super Bowl
at age . The past record was held by John Elway, who led the Broncos to victory in Super Bowl XXXIII at age
and is currently Denver’s Executive Vice President of Football Operations. Quarterback
Jeff Dean had jersey number
in Champ Bowl XXXIV.
Question: What is the name of the quarterback who wasin Super Bowl
XXXIII?
Predictions: John Elway → Jeff Dean

No validity and generalizability/

Two Issues and Research Questions. Evaluation metrics
No explainability and interpretability→ How to evaluate reading comprehension? (§ )/

Two Issues and Research Questions. Evaluation metrics
No explainability and interpretability→ How to evaluate reading comprehension? (§ ). Question quality
No validity and generalizability→ How to ensure questions require precise NLU? (§ )/

Two Issues and Research Questions. Evaluation metrics
No explainability and interpretability→ How to evaluate reading comprehension? (§ ). Question quality
No validity and generalizability→ How to ensure questions require precise NLU? (§ )& Benchmarking capability of MRC datasets→ How to specify high-quality questions with organized metrics? (§ )Important for the explainability in NLU study and practical use/

Issues and Motivation (in a Broad Sense)
Issues in Current NLP
Reproducibility of findings [Bouthillier et al.,]= Findings in a dataset/task are transferable to other datasets/tasks?
→ How can we accumulate findings in the study?
Leaderboards tell us nothing about the task when both datasets andmodels are black-box.

/

Issues and Motivation (in a Broad Sense)
Issues in Current NLP
Reproducibility of findings [Bouthillier et al.,]= Findings in a dataset/task are transferable to other datasets/tasks?
→ How can we accumulate findings in the study?
Leaderboards tell us nothing about the task when both datasets andmodels are black-box.
Underlying Motivation & Goal
Create a theoretical foundation for the evaluation of NLU.
Contribute to make the NLU study (NLP in general?) more meaningful.
/Overview§ ,§§Evaluating Natural Language Understandingin Machine Reading Comprehension. Evaluation
Metrics

How to evaluate reading
comprehension?
→ Requisite skills andtext readability
(AAAI
, ACL
)

§

. Question
Quality

How to ensure questions
require precise NLU?
→ Heuristics for identifying
easy & hard questions(EMNLP)§ . Benchmark

Capacity
How to specify high-quality
questions with organized
metrics?
→ Automated analysis
methods for requisite skill
(AAAI
)
/

Background: System Analysis by Accuracy (Issue )Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %→What skills the system has???
Corefernece resolution?
Commonsense reasoning?
Logical reasoning?
Understanding discourse
relations?

No interpretability and explainability/

Motivation: System Analysis by Skills as Metrics

Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %/Motivation: System Analysis by Skills as Metrics

Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %Dataset A
Prerequisite Skills
Question Skill Skill · · · Skill
Q
Q
Q
..
.

System X

x
x
..
.

X
x
..
.

···
···
···
..
.

x
..
.

x
X
x
..
.

Q

X

X

···

X

X

Accuracy. %. % ···

. %

. %

→

/

Motivation: System Analysis by Skills as Metrics

Dataset A

System X

Q
Q
Q
..
.

x
X
x
..
.

Q

X

Accuracy. %Dataset A
Prerequisite Skills
Question Skill Skill · · · Skill
Q
Q
Q
..
.

System X

x
x
..
.

X
x
..
.

···
···
···
..
.

x
..
.

x
X
x
..
.

Q

X

X

···

X

X

Accuracy. %. % ···

. %

. %

→

Decompose the performance into skills → Detailed analysis/

Chapter : Evaluation Metrics for MRC (ACL)

SQuAD ()
Context: The United Methodist Church (UMC) practices infant and adult baptism. Baptized Members
are those who have been baptized as an infant orchild, but who have not professed their own faith.
Question: What are members who have been
baptized as an infant or child but who have notprofessed their own faith?
Answer: Baptized Members
MCTest ()
Context: The princess climbed out the window of thehigh tower and climbed down the south wall when
her mother was sleeping. She wandered out a good
ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after
escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Chapter : Evaluation Metrics for MRC (ACL)

Difficult-to-read & Easy-to-answer

SQuAD ()
Context: The United Methodist Church (UMC) practices infant and adult baptism. Baptized Members
are those who have been baptized as an infant orchild, but who have not professed their own faith.
Question: What are members who have been
baptized as an infant or child but who have notprofessed their own faith?
Answer: Baptized Members

MCTest () Easy-to-read & Difficult-to-answer
Context: The princess climbed out the window of thehigh tower and climbed down the south wall when
her mother was sleeping. She wandered out a good
ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after
escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle/

Chapter : Evaluation Metrics for MRC (ACL)

Difficult-to-read & Easy-to-answer

SQuAD ()
Context: The United Methodist Church (UMC) practices infant and adult baptism. Baptized Members
are those who have been baptized as an infant orchild, but who have not professed their own faith.
Question: What are members who have been
baptized as an infant or child but who have notprofessed their own faith?
Answer: Baptized Members

MCTest () Easy-to-read & Difficult-to-answer
Context: The princess climbed out the window of thehigh tower and climbed down the south wall when
her mother was sleeping. She wandered out a good
ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after
escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle

What we did in this chapter:. Defined two classes of metrics:
requisite skills and readability. Annotated questions with theskills (multi labeling). Analyzed & compared datasets/

Chapter : Evaluation Metrics for MRC (ACL)

Difficult-to-read & Easy-to-answer

SQuAD ()
Context: The United Methodist Church (UMC) practices infant and adult baptism. Baptized Members
are those who have been baptized as an infant orchild, but who have not professed their own faith.
Question: What are members who have been
baptized as an infant or child but who have notprofessed their own faith?
Answer: Baptized Members

MCTest () Easy-to-read & Difficult-to-answer
Context: The princess climbed out the window of thehigh tower and climbed down the south wall when
her mother was sleeping. She wandered out a good
ways. Finally she went into the forest where there areno electric poles but where there are some caves.
Question: Where did the princess wander to after
escaping?
Answer: A) Mountain *B) Forest C) Cave D) Castle

What we did in this chapter:. Defined two classes of metrics:
requisite skills and readability. Annotated questions with theskills (multi labeling). Analyzed & compared datasets
Provide fine-grained evaluation metrics/

Requisite Skills. Object tracking. Mathematical reasoning. Coreference resolution. Logical reasoning. Analogy. Causal relation. Spatiotemporal relation. Ellipsis. Bridging. Elaboration. Meta-knowledge. Schematics clause relation. Punctuation

Skills are newly defined for MRC, based on existing NLP tasks.
Related works in NLU tasks don’t cover discourse level skills.
- Knowledge types in RTE [LoBue and Yates,- Reasoning types in science QA [Jansen et al.,]]/

Numbers of Required Skills (AAAI#Skills

MCTest
Freq.
.
.
.
.
.
.

)

Accuracy
Baseline
Smith+
SW+D
LexMatch
.
.
.
.
.
.

.
.
.
.
.
.

Yin+
ABCNN
.
.
.
.
.
.

/

Numbers of Required Skills (AAAI#Skills

MCTest
Freq.
.
.
.
.
.
.

)

Accuracy
Baseline
Smith+
SW+D
LexMatch
.
.
.
.
.
.

.
.
.
.
.
.

Yin+
ABCNN
.
.
.
.
.
.

Previous study (Sugawara+a) observed that “the more skills are
required, the more difficult to answer (lower accuracy).”→ # requisite skills in a question = the difficulty of answering it/Result: Frequencies (%) of Requisite Skills
Skill \ Dataset. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

Qs * datasets
across several
answering styles
Asked to annotatewith skills needed foranswering
Agreement >%/Result: Frequencies (%) of Requisite Skills
Skill \ Dataset. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

Qs * datasets
across several
answering styles
Asked to annotatewith skills needed foranswering
Agreement >%MCTest= narrative→ coreference?

/Result: Frequencies (%) of Requisite Skills
Skill \ Dataset. Tracking. Math.
. Coref. resol.
. Logical rsng.
. Analogy. Causal rel.
. Sptemp rel.
. Ellipsis. Bridging. Elaboration. Meta. Clause rel.
. Punctuation

QA MRE MCTest SQuAD WDW MARCO NewsQA
11.0
4.0
32.0
15.0
7.0
1.0
26.0
13.0
69.0
60.0
1.0
52.0
34.0

6.0
4.0
49.0
2.0
0.0
6.0
9.0
4.0
26.0
8.0
1.0
40.0
1.0

3.0
0.0
13.0
0.0
0.0
0.0
2.0
3.0
42.0
13.0
0.0
28.0
24.0

8.0
3.0
19.0
8.0
7.0
2.0
2.0
16.0
59.0
57.0
0.0
42.0
20.0

6.0
0.0
15.0
1.0
0.0
0.0
0.0
2.0
36.0
18.0
0.0
27.0
14.0

2.0
1.0
24.0
2.0
3.0
4.0
3.0
15.0
50.0
36.0
0.0
34.0
25.0

Qs * datasets
across several
answering styles
Asked to annotatewith skills needed foranswering
Agreement >%MCTest= narrative→ coreference?
QA MRE= written by experts→ reasoning?

/Calculation of Readability
Avg. Num. of characters per word (NumChar)
Avg. Num. of syllables per word (NumSyll)
Avg. sentence length in words (MLS)
Proportion of words in Academic Word List (AWL)
Modifier variation (ModVar)
Num. of coordinate phrases per sentence (CoOrd)
Coleman-Liau index (computed by #letters and #sentences) (Coleman)
Dependent clause to clause ratio (DC/C)
Complex nominals per clause (CN/C)
Adverb variation (AdvVar)
Figure:readability measure from Vajjala and Meurers [].
/Result: Readability Metrics
Measures QA MRE
NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar
F–K
Words
∗

.
.
.
.
.
.
.
.
.
.
.
.

MCTest SQuAD

WDW

MARCO

NewsQA

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.8
.
.8 8
.
.
.
.
.
.6
.

.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.6

.
.

.
.

.
.

.
.

.

F–K = Flesch-Kincaid grade level = education level required to understand the text.

/

Result: Readability Metrics
Measures QA MRE
NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar
F–K
Words
∗

.
.
.
.
.
.
.
.
.
.
.
.

MCTest SQuAD

WDW

MARCO

NewsQA

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.8
.
.8 8
.
.
.
.
.
.6
.

.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.6

.
.

.
.

.
.

QA MRE, SQuAD, WDW= e.g., news & Wikipedia
articles.
.

.

F–K = Flesch-Kincaid grade level = education level required to understand the text.

/

Result: Readability Metrics
Measures QA MRE
NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar
F–K
Words
∗

.
.
.
.
.
.
.
.
.
.
.
.

MCTest SQuAD

WDW

MARCO

NewsQA

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.8
.
.8 8
.
.
.
.
.
.6
.

.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.6

.
.

.
.

.
.

QA MRE, SQuAD, WDW= e.g., news & Wikipedia
articles
MCTest= stories for children.
.

.

F–K = Flesch-Kincaid grade level = education level required to understand the text.

/

Result: Readability Metrics
Measures QA MRE
NumChar
NumSyll
MLS
AWL
ModVarCoOrd
Coleman
DC/C
CN/C
AdvVar
F–K
Words
∗

.
.
.
.
.
.
.
.
.
.
.
.

MCTest SQuAD

WDW

MARCO

NewsQA

.
.
.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.
.
.

.8
.
.8 8
.
.
.
.
.
.6
.

.
.

.
.

.
.
.
.
.
.
.
.

.
.
.
.
.
.
.
.

.6

.
.

.
.

.
.

QA MRE, SQuAD, WDW= e.g., news & Wikipedia
articles
MCTest= stories for children.
.

.

F–K = Flesch-Kincaid grade level = education level required to understand the text.

/

Relation between Skills and Readability
16

F-K grade level

14
12

WDW

SQuAD
NewsQAMARCOQA4MRE

108
6

MCTest

4
21.0

1.5

2.0

2.5

3.0

Average number of required skills

3.5/

Relation between Skills and Readability
16

F-K grade level

14
12

WDW

SQuAD
NewsQAMARCOQA4MRE

108
6

MCTest

4
21.0

1.5

2.0

2.5

3.0

Average number of required skills

3.5

There is only a weak correlation → readability 6= difficulty/Summary of Chapter
Observations
Difficult texts do not necessarily make difficult questions→ Text readability 6= question difficulty
When controlling the question difficulty, we can focus on easy texts(e.g., story for children) rather than difficult texts (e.g., news articles)/Summary of Chapter
Observations
Difficult texts do not necessarily make difficult questions→ Text readability 6= question difficulty
When controlling the question difficulty, we can focus on easy texts(e.g., story for children) rather than difficult texts (e.g., news articles)
Research Question and Contribution
Q: How to evaluate reading comprehension beyond simple accuracy?
A: Define a comprehensive set of requisite skills and readability measures→ Provide fine-grained and human-based evaluation metrics for MRC/

Overview§ ,§§Evaluating Natural Language Understandingin Machine Reading Comprehension. Evaluation
Metrics

How to evaluate reading
comprehension?
→ Requisite skills andtext readability
(AAAI
, ACL
)

§

. Question
Quality

How to ensure questions
require precise NLU?
→ Heuristics for identifying
easy & hard questions(EMNLP)§ . Benchmark

Capacity
How to specify high-quality
questions with organized
metrics?
→ Automated analysis
methods for requisite skill
(AAAI
)
/

Background: Question Quality in NLU Tasks (Issue )Annotation artifacts
NLU tasks contain unintended patterns specific to certain answer classes
PremiseA woman selling bamboo sticks talking to two men on a loading dock.

Entailment
NeutralContradiction

There are at least three people on a loading dock.
A woman is selling bamboo sticks to help provide for her family.
A woman is not taking money for any of her sticks.

SNLI/MultiNLI [Gururangan et al.,], StoryClozeTest [Schwartz et al.,]/

Background: Question Quality in NLU Tasks (Issue )Annotation artifacts
NLU tasks contain unintended patterns specific to certain answer classes
PremiseA woman selling bamboo sticks talking to two men on a loading dock.

Entailment
NeutralContradiction

There are at least three people on a loading dock.
A woman is selling bamboo sticks to help provide for her family.
A woman is not taking money for any of her sticks.

SNLI/MultiNLI [Gururangan et al.,], StoryClozeTest [Schwartz et al.,]Adversarial examples
SQuAD [Jia and Liang,], SST/SNLI/SQuAD [Wallace et al.,]/

Background: Question Quality in NLU Tasks (Issue )Annotation artifacts
NLU tasks contain unintended patterns specific to certain answer classes
PremiseA woman selling bamboo sticks talking to two men on a loading dock.

Entailment
NeutralContradiction

There are at least three people on a loading dock.
A woman is selling bamboo sticks to help provide for her family.
A woman is not taking money for any of her sticks.

SNLI/MultiNLI [Gururangan et al.,], StoryClozeTest [Schwartz et al.,]Adversarial examples
SQuAD [Jia and Liang,], SST/SNLI/SQuAD [Wallace et al.,]Questions may fail to require precise understanding?
/Motivation: What Makes Questions Easier?
What kind of understanding actually happens?
Context:

Question:
Answer:In November, Sony Pictures Entertainment was targeted by hackers
who released details of confidential e-mails between Sony executives regarding several high-profile film projects. Included within these were several memos relating to the production of Spectre , claiming that [...]. Eon
Productions issued a statement [...].
When did hackers get into the Sony Pictures e-mail system?
November/Motivation: What Makes Questions Easier?
What kind of understanding actually happens?
Context:

Question:
Answer:In November, Sony Pictures Entertainment was targeted by hackers
who released details of confidential e-mails between Sony executives regarding several high-profile film projects. Included within these were several memos relating to the production of Spectre , claiming that [...]. Eon
Productions issued a statement [...].
When did hackers get into the Sony Pictures e-mail system?
November. Recognizing entity type
Single candidate answer November

for when/Motivation: What Makes Questions Easier?
What kind of understanding actually happens?
Context:

Question:
Answer:In November, Sony Pictures Entertainment was targeted by hackers
who released details of confidential e-mails between Sony executives regarding several high-profile film projects. Included within these were several memos relating to the production of Spectre (), claiming that [...].
In February, Eon Productions issued a statement [...].
When did hackers get into the Sony Pictures e-mail system?
November. Recognizing entity type
Single candidate answer November

for when/Motivation: What Makes Questions Easier?
What kind of understanding actually happens?
Context:

Question:
Answer:In November, Sony Pictures Entertainment was targeted by hackers
who released details of confidential e-mails between Sony executives regarding several high-profile film projects. Included within these were several memos relating to the production of Spectre (), claiming that [...].
In February, Eon Productions issued a statement [...].
When did hackers get into the Sony Pictures e-mail system?
November. Recognizing entity type
Single candidate answer November

for when. Attending words between Context and Question
Sony, Pictures, hackers, emails, Sony...
/

Motivation: What Makes Questions Easier?
What kind of understanding actually happens?
Context:

Question:
Answer:In November, Sony Pictures Entertainment was targeted by hackers
who released details of confidential e-mails between Sony executives regarding several high-profile film projects. Included within these were several memos relating to the production of Spectre (), claiming that [...].
In February, Eon Productions issued a statement [...].
When did hackers get into the Sony Pictures e-mail system?
November. Recognizing entity type
Single candidate answer November

for when. Attending words between Context and Question
Sony, Pictures, hackers, emails, Sony...

→ Use these information to classify Easy & Hard questions/

Chapter : What Makes Questions Easier? (EMNLP)What we did in this chapter:/

Chapter : What Makes Questions Easier? (EMNLP)What we did in this chapter:. Propose two heuristics to identify Easy & Hard questions ofdatasets with regard to the baseline performance
Entity type-based heuristic
Attention-based heuristic/

Chapter : What Makes Questions Easier? (EMNLP)What we did in this chapter:. Propose two heuristics to identify Easy & Hard questions ofdatasets with regard to the baseline performance
Entity type-based heuristic
Attention-based heuristic. Analyze these subsets by annotating with validity and requisite skills
Validity: solvability, multiple candidates answers, unambiguity
Skills: word matching, knowledge reasoning, etc. (single labeling)/Chapter : What Makes Questions Easier? (EMNLP)What we did in this chapter:. Propose two heuristics to identify Easy & Hard questions ofdatasets with regard to the baseline performance
Entity type-based heuristic
Attention-based heuristic. Analyze these subsets by annotating with validity and requisite skills
Validity: solvability, multiple candidates answers, unambiguity
Skills: word matching, knowledge reasoning, etc. (single labeling)

Enable to collect questions that require a deeper understanding of texts/

Two Heuristics → Easy and Hard Subsets
A. Entity type-based heuristic
Q: How many questions are solved only with the first k tokens? (for simplicity)/

Two Heuristics → Easy and Hard Subsets
A. Entity type-based heuristic
Q: How many questions are solved only with the first k tokens? (for simplicity)

B. Attention-based heuristic
Q: How many questions have their answers in the most similar sentence?
We compute unigram overlap to get intuitive results/Two Heuristics → Easy and Hard Subsets
A. Entity type-based heuristic
Q: How many questions are solved only with the first k tokens? (for simplicity)

B. Attention-based heuristic
Q: How many questions have their answers in the most similar sentence?
We compute unigram overlap to get intuitive results

Heuristics
Answer inmost sim sentence

Yes
No

Score on the first twoquestion tokens (k = 2)
>0
0
Easy
Easy
Easy
Hard
/

Easy and Hard Subsets
Answer extraction Description Multiple choice

Easy

Hard

RACE MCScript ARC-E

ARC-C

Full

F /Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
.

.

.

.
.

.

.

.

.

MARCO NarraQA MCTest
.

.

.

.

.
.

.

.
.

.
.

.
.

.
.

/

Easy and Hard Subsets
Answer extraction Description Multiple choice

Easy

Hard

RACE MCScript ARC-E

ARC-C

Full

F /Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
.

.

.

.
.

.

.

.

.

MARCO NarraQA MCTest
.

.

.

.

.
.

.

.
.

.
.

.
.

.
.

The baseline performances: Easy >>> Hard
→ We overestimate the performance?
/

Annotation Results: Requisite Skills
Word match

Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest/Annotation Results: Requisite Skills
Word match

Skill proportion (%)

100

E H

E H

E H

E H

Paraphrase

E H

E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

Word matching: Easy > Hard/

Annotation Results: Requisite Skills
Word match

Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

Word matching: Easy > Hard
Knowledge reasoning: Hard > Easy/

Summary of Chapter
Observations
The baseline performances: Easy >>> Hard
→ Overestimate the current performance?
Knowledge reasoning & multi sentence reasoning: Hard > Easy
Multiple choice datasets are better in validity and reasoning types/

Summary of Chapter
Observations
The baseline performances: Easy >>> Hard
→ Overestimate the current performance?
Knowledge reasoning & multi sentence reasoning: Hard > Easy
Multiple choice datasets are better in validity and reasoning types
Research Question and Contribution
Q: How to ensure questions require precise NLU?
A: Develop a filtering method using two simple heuristics→ Enable to collect questions that require a deeper understanding of texts/

Overview§ ,§§Evaluating Natural Language Understandingin Machine Reading Comprehension. Evaluation
Metrics

How to evaluate reading
comprehension?
→ Requisite skills andtext readability
(AAAI
, ACL
)

§

. Question
Quality

How to ensure questions
require precise NLU?
→ Heuristics for identifying
easy & hard questions(EMNLP)§ . Benchmark

Capacity
How to specify high-quality
questions with organized
metrics?
→ Automated analysis
methods for requisite skill
(AAAI
)
/

Motivation: Skill-based & Automated Analysis§ Requisite skills for MRC
Manual annotation of requisite skills in the MRC task
Enable to detailed evaluation but necessitate much annotation cost/Motivation: Skill-based & Automated Analysis§ Requisite skills for MRC
Manual annotation of requisite skills in the MRC task
Enable to detailed evaluation but necessitate much annotation cost
§ Heuristics for MRC
Automated analysis of question difficulty in the MRC task
Easy to use, but still don’t provide detailed information/

Motivation: Skill-based & Automated Analysis§ Requisite skills for MRC
Manual annotation of requisite skills in the MRC task
Enable to detailed evaluation but necessitate much annotation cost
§ Heuristics for MRC
Automated analysis of question difficulty in the MRC task
Easy to use, but still don’t provide detailed information→ § Skill-based & automated analysis for MRC
Machines may find bypass solutions.
→ Simple human annotation 6= true requisite skills? ⇒ low explainability/

Analysis Methods in MRC/

Intuition: Ablation of Features as Dataset Analysis
Previous Work: analyzing model behavior by input modification
Drop tokens [Kaushik and Lipton,
Replace tokens [Cirik et al.,]Shuffle tokens [Nie et al.,]]/Intuition: Ablation of Features as Dataset Analysis
Previous Work: analyzing model behavior by input modification
Drop tokens [Kaushik and Lipton,
Replace tokens [Cirik et al.,]Shuffle tokens [Nie et al.,]]↓Intuition: ablation of features
If a solved question can be still solved even after removing featuresassociated with a skill, the question do not require that skill.
/Ablation of Features: Shuffling Sentence Words
Context
What colour is your name ? One person says her name is the colour red . Synesthesia
is not a common condition . For these people , the everyday world can be a colourful
and interesting place .
Context with shuffled sentence words
is colour your What name ? One is person colour her says red the name . Synesthesia
a common is not condition . world the colourful , can For be people place everyday
and a interesting these .
Question
What is this passage mainly about?
Options(A) An unusual condition. (B) People who like colour. (C) The colour of pain. (D) Music and art.
Prediction before and after shuffling(A) An unusual condition. → (A) An unusual condition.

/

Ablation of Features: Shuffling Sentence Words
Context
What colour is your name ? One person says her name is the colour red . Synesthesia
is not a common condition . For these people , the everyday world can be a colourful
and interesting place .
Context with shuffled sentence words
is colour your What name ? One is person colour her says red the name . Synesthesia
a common is not condition . world the colourful , can For be people place everyday
and a interesting these .
Question
What is this passage mainly about?
Options(A) An unusual condition. (B) People who like colour. (C) The colour of pain. (D) Music and art.
Prediction before and after shuffling(A) An unusual condition. → (A) An unusual condition.

Does this question require the syntax-level information?
/

Chapter : Assessing the Benchmarking Capacity of Datasets
(AAAI

)

What we did in this chapter:/

Chapter : Assessing the Benchmarking Capacity of Datasets
(AAAI

)

What we did in this chapter:. Propose a semi-automated, ablation-based methodology for analyzing
the benchmarking capacity of MRC datasets.
skills and corresponding ablation methods/

Chapter : Assessing the Benchmarking Capacity of Datasets
(AAAI

)

What we did in this chapter:. Propose a semi-automated, ablation-based methodology for analyzing
the benchmarking capacity of MRC datasets.
skills and corresponding ablation methods. Evaluate to what degree the questions do not require the skills
existing MRC datasets from the answer extraction and multiple choice/Chapter : Assessing the Benchmarking Capacity of Datasets
(AAAI

)

What we did in this chapter:. Propose a semi-automated, ablation-based methodology for analyzing
the benchmarking capacity of MRC datasets.
skills and corresponding ablation methods. Evaluate to what degree the questions do not require the skills
existing MRC datasets from the answer extraction and multiple choice

Enable to precisely evaluate the benchmarking capacity of datasets.
/

Skills and Ablation Methods: Reading Level ( – ). Recognizing the whole question except for interrogatives
Drop all words except interrogatives (wh- words and how) in a question.

. Recognizing content words
Drop content words in the context.

. Recognizing function words
Drop function words in the context.

. Recognizing vocabulary
Anonymize context and questions words with their part-of-speech tag.

. Attending the whole context other than similar sentences
Keep the sentences that are the most similar to the question.

. Recognizing the word order
Randomly shuffle all words in the context.
/

Skills and Methods: Reasoning Level ( – ). Grasping sentence-level compositionality
Randomly shuffle the words in all the sentences except the last token.

. Bridging reasoning
Randomly shuffle the order of the sentences in the context.

. Performing basic arithmetic operations
Replace numerical expressions with random numbers.

. Explicit logical reasoning
Drop logical terms such as not, every, and if.

. Resolving pronoun coreferences
Drop personal and possessive pronouns.

. Reasoning about explicit causality
Drop causal terms/clauses such as because and therefore.
/

Other Examples/

Results: Trained & Evaluated on the Ablated Dataoriginal

content words only

vocab anonymize.

context word shuff

sentence word shuff

100

F /Acc

80
60
40
20
0
CoQA

DuoRC

Hotpot

SQuAD

Answer extraction

SQuAD

ARC

MCTestMultiRC

RACE

SWAG

Multiple choice/Results: Trained & Evaluated on the Ablated Dataoriginal

content words only

vocab anonymize.

context word shuff

sentence word shuff

100

F /Acc

80
60
40
20
0
CoQA

DuoRC

Hotpot

SQuAD

Answer extraction

SQuAD

ARC

MCTestMultiRC

RACE

SWAG

Multiple choiceContent words only & Sentence word shuffle : small drop from the original/Results: Trained & Evaluated on the Ablated Dataoriginal

content words only

vocab anonymize.

context word shuff

sentence word shuff

100

F /Acc

80
60
40
20
0
CoQA

DuoRC

Hotpot

SQuAD

Answer extraction

SQuAD

ARC

MCTestMultiRC

RACE

SWAG

Multiple choiceContent words only & Sentence word shuffle : small drop from the original
CoQA, SQuAD, SQuAD : relatively high performance on Vocabulary anonymization/

Results: Trained & Evaluated on the Ablated Dataoriginal

content words only

vocab anonymize.

context word shuff

sentence word shuff

100

F /Acc

80
60
40
20
0
CoQA

DuoRC

Hotpot

SQuAD

Answer extraction

SQuAD

ARC

MCTestMultiRC

RACE

SWAG

Multiple choiceContent words only & Sentence word shuffle : small drop from the original
CoQA, SQuAD, SQuAD : relatively high performance on Vocabulary anonymization
Multiple choice datasets: high performance on Context word shuffle/Summary of Chapter
Observations
Most of the questions already answered correctly by the baseline model
do not necessarily require lexical, grammatical and complex reasoning.
→ Existing questions may fail to require complex understanding of texts
For precise benchmarking, MRC datasets will need to take extra care in
their design to ensure that questions require the intended skills.

/

Summary of Chapter
Observations
Most of the questions already answered correctly by the baseline model
do not necessarily require lexical, grammatical and complex reasoning.
→ Existing questions may fail to require complex understanding of texts
For precise benchmarking, MRC datasets will need to take extra care in
their design to ensure that questions require the intended skills.
Research Question and Contribution
Q: How to specify high-quality questions with organized metrics?
A: Proposed analysis methods for datasets using feature ablation→ Enable to precisely evaluate the benchmarking capacity of datasets.
/

Overview§ ,§§Evaluating Natural Language Understandingin Machine Reading Comprehension. Evaluation
Metrics

How to evaluate reading
comprehension?
→ Requisite skills andtext readability
(AAAI
, ACL
)

§

. Question
Quality

How to ensure questions
require precise NLU?
→ Heuristics for identifying
easy & hard questions(EMNLP)§ . Benchmark

Capacity
How to specify high-quality
questions with organized
metrics?
→ Automated analysis
methods for requisite skill
(AAAI
)
/

Contributions§ . How to evaluate reading comprehension?
Defined a comprehensive set of requisite skills and readability measures→ Provide fine-grained evaluation metrics for NLU beyond simple accuracy.

/

Contributions§ . How to evaluate reading comprehension?
Defined a comprehensive set of requisite skills and readability measures→ Provide fine-grained evaluation metrics for NLU beyond simple accuracy.

§ . How to ensure questions require precise NLU?
Developed a question-filtering method using two simple heuristics→ Enable to automatically collect hard questions that require a deeper
understanding of texts beyond using superficial cues.

/

Contributions§ . How to evaluate reading comprehension?
Defined a comprehensive set of requisite skills and readability measures→ Provide fine-grained evaluation metrics for NLU beyond simple accuracy.

§ . How to ensure questions require precise NLU?
Developed a question-filtering method using two simple heuristics→ Enable to automatically collect hard questions that require a deeper
understanding of texts beyond using superficial cues.

§ . How to specify high-quality questions with organized metrics?
Proposed analysis methods of datasets using feature ablation→ Enable to assess the capabilities of datasets for benchmarking NLU.
/

Summary of the Thesis
Requirements of MRC Datasets
Explainability (cf. psychological study of human text understanding)
Evaluation metrics reflect the question intention in human terms.
→ Explain what is evaluated and what successful models can do.

Validiation (cf. validity in psychometrics)
Questions reliably evaluate the intended skills without bypass solutions.
→ Ensure that the intended skills are evaluated.

/Summary of the Thesis
Requirements of MRC Datasets
Explainability (cf. psychological study of human text understanding)
Evaluation metrics reflect the question intention in human terms.
→ Explain what is evaluated and what successful models can do.

Validiation (cf. validity in psychometrics)
Questions reliably evaluate the intended skills without bypass solutions.
→ Ensure that the intended skills are evaluated.

Why Important?
Hypothesis verification for the scientific study of NLU
Accountability in practical applications such as assisting human
intelligent activities/

Publications. Saku Sugawara, Stenetorp Pontus, Kentaro Inui, and Akiko Aizawa.
. Assessing
the benchmarking capacity of machine reading comprehension datasets. In
Proceedings of AAAI Conference on Artificial Intelligence (AAAI), to appear.
. Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa.
. What makes
reading comprehension questions easier?. In Proceedings of theConference on
Empirical Methods in Natural Language Processing (EMNLP), pages.
. Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa.
a. Evaluation
metrics for machine reading comprehension: Prerequisite skills and readability. In
Proceedings of the th Annual Meeting of the Association for Computational
Linguistics (Volume : Long Papers) (ACL), pages.
. Saku Sugawara, Hikaru Yokono, and Akiko Aizawa.
b. Prerequisite skills forreading comprehension: Multi-perspective analysis of mctest datasets and systems. In
AAAI Conference on Artificial Intelligence (AAAI), pages.
/References I
Xavier Bouthillier, César Laurent, and Pascal Vincent.
. Unreproducible research is reproducible. In
Proceedings of the th International Conference on Machine Learning, volumeof Proceedings of Machine
Learning Research, pages–, Long Beach, California, USA. PMLR.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes.
. Reading wikipedia to answer open-domainquestions. In Proceedings of the th Annual Meeting of the Association for Computational Linguistics (Volume: Long Papers), pages–. Association for Computational Linguistics.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wen-tau Yih, Yejin Choi, Percy Liang, and Luke Zettlemoyer.
.
QuAC: Question answering in context. In Proceedings of theConference on Empirical Methods in Natural
Language Processing, pages–, Brussels, Belgium. Association for Computational Linguistics.
Volkan Cirik, Louis-Philippe Morency, and Taylor Berg-Kirkpatrick.
. Visual referring expression recognition:
What do systems actually learn? In Proceedings of theConference of the North American Chapter of the
Association for Computational Linguistics: Human Language Technologies, Volume (Short Papers), pages–, New Orleans, Louisiana. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova.
.
BoolQ: Exploring the surprising difficulty of natural yes/no questions. In Proceedings of theConference of
the North American Chapter of the Association for Computational Linguistics: Human Language Technologies,
Volume (Long and Short Papers), pages–, Minneapolis, Minnesota. Association for Computational
Linguistics.
/

References II
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner.
. Quoref: A reading
comprehension dataset with questions requiring coreferential reasoning. In Proceedings of theConference on Empirical Methods in Natural Language Processing and the th International Joint Conference
on Natural Language Processing (EMNLP-IJCNLP), pages–, Hong Kong, China. Association for
Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner.
. DROP: A
reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of theConference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume (Long and Short Papers), pages–, Minneapolis, Minnesota. Association for
Computational Linguistics.
Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel Bowman, and Noah A. Smith.
.
Annotation artifacts in natural language inference data. In Proceedings of theConference of the North
American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume(Short Papers), pages– . Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil
Blunsom.
. Teaching machines to read and comprehend. In Advances in Neural Information Processing
Systems (NIPS), pages–.
/

References III
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi.
. Cosmos QA: Machine reading
comprehension with contextual commonsense reasoning. In Proceedings of theConference on Empirical
Methods in Natural Language Processing and the th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP), pages–, Hong Kong, China. Association for Computational Linguistics.
Peter Jansen, Niranjan Balasubramanian, Mihai Surdeanu, and Peter Clark.
. What’s in an explanation?
characterizing knowledge and inference requirements for elementary science exams. In Proceedings of COLING, the th International Conference on Computational Linguistics: Technical Papers, pages–,
Osaka, Japan. The COLING
Organizing Committee.
Robin Jia and Percy Liang.
. Adversarial examples for evaluating reading comprehension systems. In
Proceedings of theConference on Empirical Methods in Natural Language Processing, pages–
Association for Computational Linguistics.

.

Divyansh Kaushik and Zachary C. Lipton.
. How much reading does reading comprehension require? a criticalinvestigation of popular benchmarks. In Proceedings of theConference on Empirical Methods in Natural
Language Processing, pages–. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth.
. Looking beyond thesurface: A challenge set for reading comprehension over multiple sentences. In Proceedings of theConference of the North American Chapter of the Association for Computational Linguistics: Human Language
Technologies, Volume (Long Papers), pages–. Association for Computational Linguistics.
/

References IV
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward
Grefenstette.
. The NarrativeQA reading comprehension challenge. Transactions of the Association for
Computational Linguistics, :–.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle
Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei
Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov.
. Natural questions: A benchmark forquestion answering research. Transactions of the Association for Computational Linguistics, :–.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy.
. RACE: Large-scale reading
comprehension dataset from examinations. In Proceedings of theConference on Empirical Methods in
Natural Language Processing, pages–. Association for Computational Linguistics.
Peter LoBue and Alexander Yates.
. Types of common-sense knowledge needed for recognizing textual
entailment. In Proceedings of the th Annual Meeting of the Association for Computational Linguistics: Human
Language Technologies, pages–, Portland, Oregon, USA. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng.
MARCO: A human generated machine reading comprehension dataset. CoRR, abs/
.
.

. MS

Yixin Nie, Yicheng Wang, and Mohit Bansal.
. Analyzing compositionality-sensitivity of NLI models. In
Proceedings of AAAI Conference on Artificial Intelligence.
/References V
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester.
. Who did What: A large-scaleperson-centered cloze dataset. In Proceedings of theConference on Empirical Methods in Natural
Language Processing, pages–. Association for Computational Linguistics.
Anusri Pampari, Preethi Raghavan, Jennifer Liang, and Jian Peng.
. emrQA: A large corpus for question
answering on electronic medical records. In Proceedings of theConference on Empirical Methods in
Natural Language Processing, pages–, Brussels, Belgium. Association for Computational Linguistics.
Pranav Rajpurkar, Robin Jia, and Percy Liang.
. Know what you don’t know: Unanswerable questions for
SQuAD. In Proceedings of the th Annual Meeting of the Association for Computational Linguistics (Volume :Short Papers), pages–. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang.
. SQuAD:,+ questions for machinecomprehension of text. In Proceedings of theConference on Empirical Methods in Natural Language
Processing, pages–. Association for Computational Linguistics.
Siva Reddy, Danqi Chen, and Christopher D. Manning.
. CoQA: A conversational question answering challenge.
Transactions of the Association for Computational Linguistics, :–.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan.
. DuoRC: Towards complex
language understanding with paraphrased reading comprehension. In Proceedings of the th Annual Meetingof the Association for Computational Linguistics (Volume : Long Papers), pages–. Association for
Computational Linguistics.
/

References VI
Roy Schwartz, Maarten Sap, Ioannis Konstas, Leila Zilles, Yejin Choi, and Noah A. Smith.
. Story cloze task: UW
NLP system. In Proceedings of the nd Workshop on Linking Models of Lexical, Sentential and Discourse-level
Semantics, pages – , Valencia, Spain. Association for Computational Linguistics.
Kai Sun, Dian Yu, Jianshu Chen, Dong Yu, Yejin Choi, and Claire Cardie.
. DREAM: A challenge data set andmodels for dialogue-based reading comprehension. Transactions of the Association for Computational
Linguistics, :–.
Simon Suster and Walter Daelemans.
. CliCR: a dataset of clinical case reports for machine reading
comprehension. In Proceedings of theConference of the North American Chapter of the Association for
Computational Linguistics: Human Language Technologies, Volume (Long Papers), pages–.
Association for Computational Linguistics.
Richard Sutcliffe, Anselmo Peñas, Eduard Hovy, Pamela Forner, Álvaro Rodrigo, Corina Forascu, Yassine Benajiba,
and Petya Osenova.
. Overview of QA MRE main task at CLEF
. Working Notes, CLEF.
Sowmya Vajjala and Detmar Meurers.
. On improving the accuracy of readability classification using insightsfrom second language acquisition. In Proceedings of the Seventh Workshop on Building Educational
Applications Using NLP, pages–. Association for Computational Linguistics.

/

References VII
Eric Wallace, Shi Feng, Nikhil Kandpal, Matt Gardner, and Sameer Singh.
. Universal adversarial triggers for
attacking and analyzing NLP. In Proceedings of theConference on Empirical Methods in Natural Language
Processing and the th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP),
pages–, Hong Kong, China. Association for Computational Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel.
. Constructing datasets for multi-hop reading
comprehension across documents. Transactions of the Association for Computational Linguistics, :–.

Qizhe Xie, Guokun Lai, Zihang Dai, and Eduard Hovy.
. Large-scale cloze test dataset created by teachers. In
Proceedings of theConference on Empirical Methods in Natural Language Processing, pages–,
Brussels, Belgium. Association for Computational Linguistics.
Semih Yagcioglu, Aykut Erdem, Erkut Erdem, and Nazli Ikizler-Cinbis.
. RecipeQA: A challenge dataset for
multimodal comprehension of cooking recipes. In Proceedings of theConference on Empirical Methods in
Natural Language Processing, pages–, Brussels, Belgium. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D.
Manning.
. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings oftheConference on Empirical Methods in Natural Language Processing, pages–. Association for
Computational Linguistics.
Weihao Yu, Zihang Jiang, Yanfei Dong, and Jiashi Feng.
. Reclor: A reading comprehension dataset requiring
logical reasoning. In International Conference on Learning Representations.
/