What Makes Reading Comprehension Questions Easier?

Saku Sugawara♠ , Kentaro Inui♣♦ ,Satoshi Sekine♦ , Akiko Aizawa♥♠♠

University of Tokyo, ♣ Tohoku University,♦
RIKEN AIP, ♥ National Institute of Informatics
November 4, EMNLP 2018

1/21
1 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2/21
2 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTest2015

bAbI

QA4MRE

CNN/Daily Mail

CBT

2/21
3 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2015

bAbI

CNN/Daily Mail

2016

SQuAD

WikiReading

CBT
LAMBADA

Who-did-What

NewsQAMS MARCO

2/21
4 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2015

bAbI

CNN/Daily Mail

2016

SQuAD

WikiReading

2017

TriviaQA

Quasar

CBT
LAMBADA

AddSent

RACE

Who-did-What

NewsQAMS MARCO

QAngaroo

NarrativeQA

MCScript

2/21
5 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2015

bAbI

CNN/Daily Mail

2016

SQuAD

WikiReading

2017

TriviaQA

Quasar

CBT
LAMBADA

AddSent

RACE

Who-did-What

NewsQAMS MARCO

QAngaroo

NarrativeQA

MCScript

2018

2/21
6 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2015

bAbI

CNN/Daily Mail

2016

SQuAD

WikiReading

2017

TriviaQA

Quasar2018

ARC CliCR MultiRC ProPara SQuAD2.0 DuoRC TextWorldsQA CLOTH
emrQA RecipeQA OpenBookQA HotpotQA ShARC QuAC CoQA

CBT
LAMBADA

AddSent

RACE

Who-did-What

NewsQAMS MARCO

QAngaroo

NarrativeQA

MCScript

2/21
7 / 57

Machine Reading Comprehension & Related Datasets
Many many datasets!
2013

MCTestQA4MRE

2015

bAbI

CNN/Daily Mail

2016

SQuAD

WikiReading

2017

TriviaQA

Quasar2018

ARC CliCR MultiRC ProPara SQuAD2.0 DuoRC TextWorldsQA CLOTH
emrQA RecipeQA OpenBookQA HotpotQA ShARC QuAC CoQA

CBT
LAMBADA

AddSent

RACE

Who-did-What

NewsQAMS MARCO

QAngaroo

NarrativeQA

MCScript

EMNLP2018> 30 datasets!
2/21
8 / 57

Reading Comprehension !!!
ID: MCTest, mc160.dev.29
Context: The princess climbed out thewindow of the high tower and climbed
down the south wall when her motherwas sleeping. She wandered out a good
ways. Finally she went into the forest
where there are no electric poles butwhere there are some caves.
Question: Where did the princess wander
to after escaping?
Answer: A) Mountain *B) Forest C)
Cave D) Castle

Skills:Coreference resolution
Commonsense reasoning
Discourse understanding
Spatiotemporal reasoning
Logical reasoning
Mathematical reasoning
Causal reasoning
Meta reasoning...
Cf. Weston+ (2015; bAbI)
Boratko+ (2018; analysis on ARC)
Sugawara+ (AAAI2017, ACL2017)
3/21
9 / 57

Reading Comprehension ???

Do MRC datasets really require comprehension?
' What kind of comprehension is required by MRC datasets?

4/21
10 / 57

Motivation: Understanding of Understanding
Annotation artifacts
Natural language understanding tasks contain unintended patterns
SNLI/MultiNLI: Gururangan+ (2018), Poliak+ (2018), Glockner+ (2018)
StoryClozeTest: Schwartz+ (2018)→ Word patterns/features specific to certain answer classes

5/21
11 / 57

Motivation: Understanding of Understanding
Annotation artifacts
Natural language understanding tasks contain unintended patterns
SNLI/MultiNLI: Gururangan+ (2018), Poliak+ (2018), Glockner+ (2018)
StoryClozeTest: Schwartz+ (2018)→ Word patterns/features specific to certain answer classes

Adversarial examples
MRC systems are fooled by manually injected distracting sentences
SQuAD + AddSent: Jia and Liang (2017)

5/21
12 / 57

Motivation: Understanding of Understanding
Annotation artifacts
Natural language understanding tasks contain unintended patterns
SNLI/MultiNLI: Gururangan+ (2018), Poliak+ (2018), Glockner+ (2018)
StoryClozeTest: Schwartz+ (2018)→ Word patterns/features specific to certain answer classes

Adversarial examples
MRC systems are fooled by manually injected distracting sentences
SQuAD + AddSent: Jia and Liang (2017)→ we don’t know exactly what kind of understanding is required
5/21
13 / 57

Example from SQuAD
What kind of understanding actually happens?
Question: When did hackers get into the Sony Pictures e-mail system?
Context: In November 2014, Sony Pictures Entertainment was targeted by hackers who
released details of confidential e-mails between Sony executives regarding several highprofile film projects. Included within these were several memos relating to the productionof Spectre , claiming that [...]. Eon Productions issued a statement [...].
Answer: November 2014

6/21
14 / 57

Example from SQuAD
What kind of understanding actually happens?
Question: When did hackers get into the Sony Pictures e-mail system?
Context: In November 2014, Sony Pictures Entertainment was targeted by hackers who
released details of confidential e-mails between Sony executives regarding several highprofile film projects. Included within these were several memos relating to the productionof Spectre , claiming that [...]. Eon Productions issued a statement [...].
Answer: November 2014

1. Recognizing entity type
Single candiadte answer November 2014 for when

6/21
15 / 57

Example from SQuAD
What kind of understanding actually happens?
Question: When did hackers get into the Sony Pictures e-mail system?
Context: In November 2014, Sony Pictures Entertainment was targeted by hackers who
released details of confidential e-mails between Sony executives regarding several highprofile film projects. Included within these were several memos relating to the productionof Spectre (2015), claiming that [...]. In February 2015, Eon Productions issued a statement [...].
Answer: November 2014

1. Recognizing entity type
Single candiadte answer November 2014 for when

6/21
16 / 57

Example from SQuAD
What kind of understanding actually happens?
Question: Whem did hackers get into the Sony Pictures e-mail system?
Context: In November 2014, Sony Pictures Entertainment was targeted by hackers who
released details of confidential e-mails between Sony executives regarding several highprofile film projects. Included within these were several memos relating to the productionof Spectre (2015), claiming that [...]. In February 2015, Eon Productions issued a statement [...].
Answer: November 2014

1. Recognizing entity type
Single candiadte answer November 2014 for when

2. Attending words between Context and Question
Sony, Pictures, hackers, emails, Sony...
6/21
17 / 57

What We Did in This Work

1. Propose two heuristics to identify Easy & Hard questions with regard to thebaseline performance
Entity type-based heuristic
Attention-based heuristic

2. Analyze these subsets by annotating with varidity and requisite skills
Validity: solvability, multiple candidates answers, unambiguity
Skills: word matching, knowledge reasoning, mathematics, etc.

7/21
18 / 57

What We Did in This Work

1. Propose two heuristics to identify Easy & Hard questions with regard to thebaseline performance
Entity type-based heuristic
Attention-based heuristic

2. Analyze these subsets by annotating with varidity and requisite skills
Validity: solvability, multiple candidates answers, unambiguity
Skills: word matching, knowledge reasoning, mathematics, etc.

7/21
19 / 57

What We Did in This Work

1. Propose two heuristics to identify Easy & Hard questions with regard to thebaseline performance
Entity type-based heuristic
Attention-based heuristic

2. Analyze these subsets by annotating with varidity and requisite skills
Validity: solvability, multiple candidates answers, unambiguity
Skills: word matching, knowledge reasoning, mathematics, etc.

7/21
20 / 57

Two Heuristics

8/21
21 / 57

Two Heuristics
A. Entity type-based heuristic
to filter questions solved by recognizing the entity type of the answer or typicalwords around the answer
Q: How many questions are solved only with the first k tokens?
& Is there a moderate gap of performance between full and k = small settings?
e.g., will I qualify for OSAP if I’m new in Canada? (k = 3) (from MS MARCO)

8/21
22 / 57

Two Heuristics
A. Entity type-based heuristic
to filter questions solved by recognizing the entity type of the answer or typicalwords around the answer
Q: How many questions are solved only with the first k tokens?
& Is there a moderate gap of performance between full and k = small settings?
e.g., will I qualify for OSAP if I’m new in Canada? (k = 3) (from MS MARCO)

B. Attention-based heuristic
to filter questions solved by matching words between question and context
Q: How many questions have their answers in the most similar sentence?
We just compute unigram overlap to get intuitive results
8/21
23 / 57

Datasets
1. SQuAD (v1.1) [Rajpurkar et al., 2016]
2. AddSent [Jia and Liang, 2017]
3. NewsQA [Trischler et al., 2017]
4. TriviaQA (Wikipedia set) [Joshi et al., 2017]
5. QAngaroo (WikiHop) [Welbl et al., 2018]
6. MS MARCO (v2) [Nguyen et al., 2016]
7. NarrativeQA (summary) [Kočiský et al., 2018]
8. MCTest (160 + 500) [Richardson et al., 2013]
9. RACE (middle + high) [Lai et al., 2017]
10. MCScript [Ostermann et al., 2018]
11. ARC Easy [Clark et al., 2018]
12. ARC Challenge [Clark et al., 2018]
9/21
24 / 57

Datasets
1. SQuAD (v1.1) [Rajpurkar et al., 2016]
2. AddSent [Jia and Liang, 2017]
3. NewsQA [Trischler et al., 2017]
4. TriviaQA (Wikipedia set) [Joshi et al., 2017]
5. QAngaroo (WikiHop) [Welbl et al., 2018]

Answerextraction

6. MS MARCO (v2) [Nguyen et al., 2016]
7. NarrativeQA (summary) [Kočiský et al., 2018]

Description

8. MCTest (160 + 500) [Richardson et al., 2013]
9. RACE (middle + high) [Lai et al., 2017]
10. MCScript [Ostermann et al., 2018]
11. ARC Easy [Clark et al., 2018]
12. ARC Challenge [Clark et al., 2018]

Multiplechoice9/21
25 / 57

Baseline Models
Bi-directional Attention Flow (BiDAF) [Seo et al., 2017]
for the answer extraction and description styles
previous SOTA on SQuAD

Gated Attention Reader (GAR) [Dhingra et al., 2017]
for the multiple choice styleprevious SOTA on CNN/Daily Mail [Hermann et al., 2015] and Who-did-What [Onishi
et al., 2016]

Hyperparameters are tuned on each dataset
10/21
26 / 57

Analysis: Questions with the first k tokens

F1/Rouge-L/Acc

Full Q

k=4

k=2

k=1

RACE MCScript ARC-E

ARC-C

80
60
40
20

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

11/21
27 / 57

Analysis: Questions with the first k tokens

F1/Rouge-L/Acc

Full Q

k=4

k=2

k=1

RACE MCScript ARC-E

ARC-C

80
60
40
20

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

QAngaroo : a small gap between Full and k = 1
The multiple choice datasets: not enough high performance to see its
difference?
11/21
28 / 57

Analysis: Answer in the most similar sentence
F1/Rouge-L/Acc

Baseline (= Full Q)

Ans in most sim sent (% below)

Ans NOT in most sim sent

80
60
40

SQuAD AddSent NewsQA TriviaQA QAnga
In (%) 76.3%

65.7%

46.3%

20.5%

4.2%

MARCO NarraQA MCTest

RACE MCScript ARC-E

ARC-C

18.6%

23.2%

34.8%

52.6%

33.5%

17.7%

48.7%

12/21
29 / 57

Analysis: Answer in the most similar sentence
F1/Rouge-L/Acc

Baseline (= Full Q)

Ans in most sim sent (% below)

Ans NOT in most sim sent

80
60
40

SQuAD AddSent NewsQA TriviaQA QAnga
In (%) 76.3%

65.7%

46.3%

20.5%

4.2%

MARCO NarraQA MCTest

RACE MCScript ARC-E

ARC-C

18.6%

23.2%

34.8%

52.6%

33.5%

17.7%

48.7%

Answer in the most similar sentence: improve the performance in somedatasets

12/21
30 / 57

Analysis: Answer in the most similar sentence
F1/Rouge-L/Acc

Baseline (= Full Q)

Ans in most sim sent (% below)

Ans NOT in most sim sent

80
60
40

SQuAD AddSent NewsQA TriviaQA QAnga
In (%) 76.3%

65.7%

46.3%

20.5%

4.2%

MARCO NarraQA MCTest

RACE MCScript ARC-E

ARC-C

18.6%

23.2%

34.8%

52.6%

33.5%

17.7%

48.7%

Answer in the most similar sentence: improve the performance in somedatasets
SQuAD AddSent NewsQA NarraQA and ARC-E: >45%
12/21
31 / 57

Easy and Hard Subsets

Score on the first twoquestion tokens (k = 2)
Heuristics
>0
0
Yes
Easy
Easy
Answer inmost sim sentence No
Easy
Hard

13/21
32 / 57

Easy and Hard Subsets
Easy

Hard

RACE MCScript ARC-E

ARC-C

41.2
58.8

33.6
66.4

Full
F1/Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
84.3
15.7

74.6
25.4

70.0
30.0

40.2
59.8

63.1
36.9

MARCO NarraQA MCTest
87.5
12.5

71.8
28.2

35.8
64.2

72.9
27.1

46.1
53.9

14/21
33 / 57

Easy and Hard Subsets
Easy

Hard

RACE MCScript ARC-E

ARC-C

41.2
58.8

33.6
66.4

Full
F1/Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
84.3
15.7

74.6
25.4

70.0
30.0

40.2
59.8

63.1
36.9

MARCO NarraQA MCTest
87.5
12.5

71.8
28.2

35.8
64.2

72.9
27.1

46.1
53.9

Full → Hard : the degration ranges from 4 to 47% !

14/21
34 / 57

Easy and Hard Subsets
Easy

Hard

RACE MCScript ARC-E

ARC-C

41.2
58.8

33.6
66.4

Full
F1/Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
84.3
15.7

74.6
25.4

70.0
30.0

40.2
59.8

63.1
36.9

MARCO NarraQA MCTest
87.5
12.5

71.8
28.2

35.8
64.2

72.9
27.1

46.1
53.9

Full → Hard : the degration ranges from 4 to 47% !
QAngaroo and some multiple choice datasets: unbalanced
14/21
35 / 57

Easy and Hard Subsets
Easy

Hard

RACE MCScript ARC-E

ARC-C

41.2
58.8

33.6
66.4

Full
F1/Rouge-L/Acc

100
80
60
40
20
0
SQuAD AddSent NewsQA TriviaQA QAnga
84.3
15.7

74.6
25.4

70.0
30.0

40.2
59.8

63.1
36.9

MARCO NarraQA MCTest
87.5
12.5

71.8
28.2

35.8
64.2

72.9
27.1

46.1
53.9

Full → Hard : the degration ranges from 4 to 47% !
QAngaroo and some multiple choice datasets: unbalanced
Hard questions: multiple choice > answer extraction & description
14/21
36 / 57

Annotation: Motivations

15/21
37 / 57

Annotation: Motivations

1. How many questions are valid in each dataset?
E.g. Hard questions are really hard... or unanswerable?

15/21
38 / 57

Annotation: Motivations

1. How many questions are valid in each dataset?
E.g. Hard questions are really hard... or unanswerable?

2. What kinds of reasoning skills explain the Easy & Hard questions?

15/21
39 / 57

Annotation: Motivations

1. How many questions are valid in each dataset?
E.g. Hard questions are really hard... or unanswerable?

2. What kinds of reasoning skills explain the Easy & Hard questions?
3. Are there any differences among the question styles?

15/21
40 / 57

Anntation: Procedure
Sampled 30 questions from each Easy & Hard subset
Annotated with validity → requisite skills
Used a skill classification in previous work [Trischler et al., 2017; Lai et al., 2017]16/21
41 / 57

Anntation Results: Validity

Skill proportion (%)

100

E H

E H

E H

E H

E H

E H

Valid

Unsolvable

E H

E H

E H

Single cand
E H

Ambiguous
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

17/21
42 / 57

Anntation Results: Validity

Skill proportion (%)

100

E H

E H

E H

E H

E H

E H

Valid

Unsolvable

E H

E H

E H

Single cand
E H

Ambiguous
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

TriviaQA, QAngaroo and ARCs: relatively high unsolvability by the inherent
unrelatedness between the questions and their context.

17/21
43 / 57

Anntation Results: Validity

Skill proportion (%)

100

E H

E H

E H

E H

E H

E H

Valid

Unsolvable

E H

E H

E H

Single cand
E H

Ambiguous
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

TriviaQA, QAngaroo and ARCs: relatively high unsolvability by the inherent
unrelatedness between the questions and their context.
Other multiple choice datasets: high validity!
17/21
44 / 57

Annotation Results: Requisite Skills
Word match
Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

18/21
45 / 57

Annotation Results: Requisite Skills
Word match
Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

Word matching: Easy > Hard

18/21
46 / 57

Annotation Results: Requisite Skills
Word match
Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

Word matching: Easy > Hard
Knowledge reasoning: Hard > Easy

18/21
47 / 57

Annotation Results: Requisite Skills
Word match
Skill proportion (%)

100

E H

E H

E H

E H

E H

Paraphrase
E H

E H

Knowledge
E H

Meta/Whole
E H

E H

Math/Logic
E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

Word matching: Easy > Hard
Knowledge reasoning: Hard > Easy
Meta/whole & math/logic:
multiple choice > answer extraction & description
18/21
48 / 57

Annotation Results: Multiple sentence reasoning
Multi sent
Skill proportion (%)

100

E H

E H

E H

E H

E H

E H

E H

E H

E H

E H

E H

E H

RACE MCScript ARC-E

ARC-C

80
60
40
20
0

SQuAD AddSent NewsQA TriviaQA QAnga

MARCO NarraQA MCTest

10 datasets: Hard > Easy
19/21
49 / 57

Related Work

20/21
50 / 57

Related Work
Macro perspective:
How Much Reading Does Reading Comprehension Require? A Critical
Investigation of Popular Benchmarks [Kaushik and Lipton, 2018] (EMNLP)
Context & question ablations
On 5 datasets: CBT, CNN, Who-did-What, bAbI, SQuAD

20/21
51 / 57

Related Work
Macro perspective:
How Much Reading Does Reading Comprehension Require? A Critical
Investigation of Popular Benchmarks [Kaushik and Lipton, 2018] (EMNLP)
Context & question ablations
On 5 datasets: CBT, CNN, Who-did-What, bAbI, SQuAD

Micro perspective:
A Systematic Classification of Knowledge, Reasoning, and Context within
the ARC Dataset [Boratko et al., 2018] (MRQA Workshop)
Detailed classifications of knowledge and reasoning
On the ARC dataset [Clark et al., 2018]

20/21
52 / 57

Related Work
Macro perspective:
How Much Reading Does Reading Comprehension Require? A Critical
Investigation of Popular Benchmarks [Kaushik and Lipton, 2018] (EMNLP)
Context & question ablations
On 5 datasets: CBT, CNN, Who-did-What, bAbI, SQuAD

Micro perspective:
A Systematic Classification of Knowledge, Reasoning, and Context within
the ARC Dataset [Boratko et al., 2018] (MRQA Workshop)
Detailed classifications of knowledge and reasoning
On the ARC dataset [Clark et al., 2018]

Evaluation Metrics for Machine Reading Comprehension: Prerequisite Skills
and Readability [Sugawara et al., 2017b,a] (AAAI & ACL)
A set of requisite skills as evaluation metrics
On 6 datasets: QA4MRE, MCTest, SQuAD, Who-did-What, MARCO, NewsQA

20/21
53 / 57

Summary and Conclusions

21/21
54 / 57

Summary and Conclusions
Proposing two heuristics to identify easy/hard questions
The baseline performances: Easy > Hard

21/21
55 / 57

Summary and Conclusions
Proposing two heuristics to identify easy/hard questions
The baseline performances: Easy > Hard
Annotating with varidity and skills
Knowledge reasoning & multi sentence reasoning: Hard > Easy
Kinds of reasoning skills: multiple choice > answer extraction & description

21/21
56 / 57

Summary and Conclusions
Proposing two heuristics to identify easy/hard questions
The baseline performances: Easy > Hard
Annotating with varidity and skills
Knowledge reasoning & multi sentence reasoning: Hard > Easy
Kinds of reasoning skills: multiple choice > answer extraction & description→ Let’s focus on Hard questions!
https://github.com/Alab-NII/mrc-heuristics
Looking for next summer internship position & full-time research position from 2020!

Thank you
21/21
57 / 57

21/21References I
Michael Boratko, Harshit Padigela, Divyendra Mikkilineni, Pritish Yuvraj, Rajarshi Das, Andrew McCallum, Maria Chang,Achille Fokoue-Nkoutche, Pavan Kapanipathi, Nicholas Mattei, Ryan Musa, Kartik Talamadupula, and Michael Witbrock.
2018. A systematic classification of knowledge, reasoning, and context within the arc dataset. In Proceedings of theWorkshop on Machine Reading for Question Answering, pages 60–70, Melbourne, Australia. Association for
Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018.
Think you have solved question answering? Try ARC, the AI2 reasoning challenge. CoRR, abs/1803.05457.
Bhuwan Dhingra, Hanxiao Liu, Zhilin Yang, William Cohen, and Ruslan Salakhutdinov. 2017. Gated-attention readers fortext comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics(Volume 1: Long Papers), pages 1832–1846. Association for Computational Linguistics.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil
Blunsom. 2015. Teaching machines to read and comprehend. In Advances in Neural Information Processing Systems(NIPS), pages 1693–1701.
Robin Jia and Percy Liang. 2017. Adversarial examples for evaluating reading comprehension systems. In Proceedings ofthe 2017 Conference on Empirical Methods in Natural Language Processing, pages 2011–2021, Copenhagen,Denmark. Association for Computational Linguistics.
22/21
59 / 57

References II
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised
challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 1601–1611. Association for Computational Linguistics.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? a criticalinvestigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural
Language Processing, pages 5010–5015, Brussels, Belgium. Association for Computational Linguistics.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette.
2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational
Linguistics, 6:317–328.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale reading comprehension
dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language
Processing, pages 796–805, Copenhagen, Denmark. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A
human generated machine reading comprehension dataset. CoRR, abs/1611.09268.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did What: A large-scaleperson-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language
Processing, pages 2230–2235. Association for Computational Linguistics.
23/21
60 / 57

References III
Simon Ostermann, Ashutosh Modi, Michael Roth, Stefan Thater, and Manfred Pinkal. 2018. MCScript: A novel dataset forassessing machine comprehension using script knowledge. In Proceedings of the Eleventh International Conferenceon Language Resources and Evaluation (LREC 2018). European Language Resources Association (ELRA).
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machinecomprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing,pages 2383–2392. Association for Computational Linguistics.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain
machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language
Processing, pages 193–203.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machinecomprehension. In International Conference on Learning Representations.
Saku Sugawara, Yusuke Kido, Hikaru Yokono, and Akiko Aizawa. 2017a. Evaluation metrics for machine reading
comprehension: Prerequisite skills and readability. In Proceedings of the 55th Annual Meeting of the Association for
Computational Linguistics (Volume 1: Long Papers), pages 806–817. Association for Computational Linguistics.
Saku Sugawara, Hikaru Yokono, and Akiko Aizawa. 2017b. Prerequisite skills for reading comprehension:Multi-perspective analysis of mctest datasets and systems. In AAAI Conference on Artificial Intelligence, pages
3089–3096.
24/21
61 / 57

References IV

Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017.
NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for
NLP, pages 191–200. Association for Computational Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading
comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287–302.

25/21
62 / 57