Benchmarking Machine Reading Comprehension:
A Psychological Perspective
Saku Sugawara1, Pontus Stenetorp2, Akiko Aizawa1
1National Institute of Informatics, 2University College London

saku@nii.ac.jp

MRC for Research on NLU¡ Natural language understanding (NLU) research aims to model a machine that canunderstand natural language (e.g., WSC (Levesque 2012), RTE (Dagan 2005))¡ Machine reading comprehension (MRC) is one of NLU tasks, with a general form

Context:

The princess climbed out the window of the high towerand climbed down the south wall when her mother wassleeping.
She wandered out.
Finally she went into the forest where there are noelectric poles.

Question: Where did the princess wander to after escaping?
Answer: A) Mountain *B) Forest C) Cave D) CastleCoreferenceCommonsensereasoning

Temporal relation

2

Trends and Progress of MRC
05&'DWDVHWVYV<HDU

05&'DWDVHWV

HumanPerformance



















<HDU

¡ ＞70 datasets (personal estimate)¡ Outperforms human in SQuAD v1.1

¡ Various formats, domains, skills...

= MRC is “solved”?
3Benchmarking Issues: Analytic Studies¡ Models for SQuAD are easily fooled by manually injecteddistracting sentences (Jia & Liang 2017)¡ Questions are solvable even after shuffling context words ordropping content words (Sugawara+ 2020)¡ ＞90% of the questions in SQuAD v1.1 require reading only one

sentence in passage (Min+ 2017)¡ Questions are solvable only with a few question tokens (or none)(Sugawara+ 2018, Feng+ 2018, Mudrakarta+ 2018, Kaushik & Lipton 2018)¡ Multi-hop reasoning datasets do not necessitate multi-hopreasoning (Min+ 2019, Chen & Durrett 2019)Q: What understanding is required by the datasets
and is actually achieved by models?

Adversarial example (Jia & Liang 2017)
Paragraph: there are many persons who cannot afford
to buy books, but who usually go to libraries and spend
hours reading something that interests them a lot. From
my point of view, literature is very important in our life.
For example, reading is a means of gaining culture andenriching our knowledge in different areas .
Q: People who are fond of literature are those that ____ .
A: have much interest in reading (multiple choice)Content word dropping (Sugawara+ 2020)
4

Assumptions, Goal, and Research Questions
Assumptions¡ To benchmark NLU, we need to ensure the explainability of tasks in human terms¡ Interpreting models may be insufficient for explaining how the task is accomplished

Goal

From a top-down perspective(Bender & Koller 2020)¡ Investigate a theoretical foundation for better benchmarking of MRC (or NLU)

Research Questions¡ Q: What does reading comprehension involve?

→ Computational model of reading comprehension in psychology
What¡ Q: How can we evaluate reading comprehension?

→ Validity of interpreting measurements in psychometrics
HowFutureDirections
5FutureWhat

What Question: Text Comprehension in Psychology
Construction-Integration Model (Kintsch 1986) / Situation Model (Zwaan & Radvansky 1998)

1.

Construction¡ Given raw textual input, create a propositional network inwhich propositions are adjacently connected & elaborated
2.

Integration¡ Using the propositions, create a coherent representation(situation model) where propositions are organized globally,sometimes grounded to other media
Hernández-Orallo (2017): (successful) comprehension is theprocess of searching for a situation model that best explains thegiven text and the readerʼs background knowledge
From Wharton & Kintsch (1991)

6

WhatRepresentation Levels and NLP Tasks
1.

Surface Structure Level -> “checklist” approach (Ribeiro+ 2020)¡ Linguistic propositions from the textual input¡ Skills: syntactic parsing, POS tagging, SRL, and NER

Situation
Model

2.

¡ Local relations of propositions¡ Skills: recognizing relations between entities and sentences such asTextbase
Surface Structure

Textbase Level -> “skill set” approach (e.g., Rogers+ 2020, Wang+ 2019)

coreference, factual knowledge, and discourse relations

3.

Situation Model Level -> Future directions (discuss later)¡ Global structure of propositions¡ Skills: creating a coherent representation and grounding it to othermedia (sound, image, ...)
7

What

Representation Levels: Example
Q1. Surface structure level

Passage: The princess climbed out the window of the hightower and climbed down the south wall when her motherwas sleeping. She wandered out. Finally, she went into theforest where there are no electric poles.

¡ Understanding the subject of the first event.

Q2. Textbase level¡ Understanding of relations among described

Q1. Who climbed out of the high tower?
A1. Princess

entities and events:

Q2. Where did the princess wander after escaping?
A2. Forest¡ Commonsense: “escaping” = the first event

Q3. What would happen if her mother was not sleeping?
A3. The princess would be found soon (multiple choice)¡ Coreference: “she” = “princess”¡ Solvable only by looking for a place specified

by where ? -> validation needed!

Q3. Situation model level¡ Imagining a different situation
8

How

How Question: Construct Validity in Psychometrics
Construct (psychology): an abstract concept used
to facilitate understanding of human behavior

Construct Validity (Messick 1995)¡ Evidence (or criteria) that is necessary to validate the interpretation ofoutcomes of psychological experiments.
Six Aspects of Validity in MRC (or NLU?) Six aspects of validity (Messick 1995)
1.

Content aspect

4.

¡ Reliability of evaluation metrics¡ Wide coverage of representations

2.

Substantive aspect

5.

Structural aspect¡ Structured evaluation metrics

External aspect¡ Consistency with external variables¡ Evaluation of the internal process3.

Generalizability aspect

6.

Consequential aspect¡ Robustness to adversarial attacks andreducing social biases

9

HowA Rubric Matters!
What is a rubric?
¡ A scoring guide used for assessments in education (Popham 1997)¡ Including evaluation criteria, quality definitions, and scoring strategy

Ideally, a rubric needs to cover... (for example)¡ (1) Content aspect¡Does the task have sufficient coverage of linguistic phenomena over the representation levels?

¡ (2) Substantive and (3) structural aspects¡

Are questions ensured to evaluate the internal process and/or have corresponding metrics?

¡ (4) Generalizability and (5) external aspects¡Are models performing well on your dataset good at out-of-domain datasets and other NLU tasks?

¡ (6) Consequential aspect¡

Does the task check modelsʼ robustness to adversarial inputs and their unintended biases?

10FutureFuture Directions (1) Situation Model
1. Evaluating Situation Model¡ Context-dependent situations¡ Representations are constructed distinctively depending on the given context¡ Defeasibility: if-then reasoning (Sap+ 2019), abductive reasoning (Bhagavatula+ 2020)¡ Novelty: StrategyQA (Geva+ 2021)¡ Grounding in non-textual information¡ Images: Visual MRC (Tanaka+ 2021), Visual Commonsense Reasoning (Zellers+ 2019),

Science textbooks (Kembhavi+ 2019), FigureQA (Kahou+ 2018), (but more focus on text?)¡ Structured data: HybridQA (tabular) (Chen+ 2020), Knowledge Base (many...)11

FutureFuture Directions (2) Substantive Validity
2. Assuring Substantive Validity: Better Evaluation of Internal Processes¡ Collecting high-quality, shortcut-free questions¡ Removing shortcuts (Geirhos+ 2020) by post-processing (e.g., in ReClor)¡ Alleviating dataset-specific and word-association biases (Sakaguchi+ 2020)¡ Formulating an explanation-by-design task
¡ Introspective explanation: R4C (Inoue+ 2020), which consists of “derivation” foranswering questions̶not only supporting sentences (e.g., HotpotQA)¡ Creating question dependency: ProPara (Dalvi+ 2018), CoQA (Reddy+ 2019)12

SummaryBackground & Motivation¡ MRC could be a good benchmarking task for natural language understanding¡ What understanding is required by the datasets and is achieved by models?

Research Questions and Tentative Answers¡ Q: What does reading comprehension involve?
¡ A: Process of creating a situation model that best explains context;
What

Computational model of human text comprehension may be useful¡ Q: How can we evaluate reading comprehension?
¡ A: Designing a rubric that covers various aspects of construct validity
How13