ACL 2022 #1566

What Makes Reading Comprehension
Questions Difficult?

Saku Sugawara,1 Nikita Nangia,2 Alex Warstadt,2 Samuel R. Bowman2
1National Institute of Informatics, 2 New York University

saku@nii.ac.jp

Our Goal

We want large-scale NLU benchmark data that is:¡ Difficult enough to discriminate state of the art models¡ Linguistically diverse to validate good performanceThis is a part of the criteria for better NLU benchmarking in Bowman & Dahl (2021)2

Background: Crowdsourcing NLU Data

¡ Protocols of worker handling and feedback (Nangia et al., 2021)¡ Design of the collection task (Ning et al., 2020; Rogers et al., 2020)

What aspects of text sources affect thedifficulty and diversity of examples?

3

Motivation

Children-level story→ Easy to read& factoid / simple math..?.

Technical document→ Difficult to read& logical reasoning...?

¡ The more difficult a passage to read, the more challenging a question about it?
¡ Are specific domains useful for collecting specific types of questions?

4This Study: What Passage Sources are Useful?
Research Question¡ Are difficult passages more suitable for crowdsourcing challenging questions?

Method¡ Crowdsource reading comprehension questions using passages taken fromseven different passage sources¡ Analyze how question difficulty and type are affected by linguistic aspects ofpassages (e.g., source, readability, syntactic & lexical surprisal, and vocabulary)¡ Bonus: what if we do the same data collection with model-in-the-loop?
5

Passage Sources
1. MCTest (childrenʼs stories Richardson et al., 2013)
2. Project Gutenberg (fictions: novel, narrative, story)
3. Slate (online magazine articles in Open ANC; Ide and Suderman, 2006)
4. RACE (middle- and high-school English exams; Lai et al., 2017)
5. ReClor (exercise questions for GMAT and LSAT exams; Yu et al., 2020)
6. Science articles in Wikipedia
7. Arts articles in Wikipedia
6

Crowsdourcing: Question Writing and Validation¡ We ask crowdworkers to write a question with four options given a passage¡ Workers are assigned to either the standard or adversarial data collection¡ Five different workers validate each question
Standard
direct submit

Adversarial

passage

Sources

validatedWritersubmitcorrect

incorrect

System (UnifiedQA)

Five votes
The writer can give up fooling thesystem after three attempts

Accepted

7

Statistics & Experiments
Dataset¡ Initially collected: 4,340 questions¡ 620 Qs * 7 sources, 310 each for the standard and adversarial methods¡ Validated ≈ 90%¡ High-agreement ≈ 65%

Systems (8 models)¡ RoBERTa large (fine-tuned on RACE) * 4 different models¡ DeBERTa large & xlarge (fine-tuned on MNLI or not): 4 models
8Results: Human‒Model Performance Gap¡ Small variation across sources (Δ=14.93.6)¡ Largest gap = MCTest (Childrenʼs stories)¡ larger than Gutenberg (adultsʼ stories)!
¡ Human performance may correspond to thepassage difficulty (e.g., MCTest & RACE vsGutenberg & Slate), but this trend doesnʼt applyto machine performance9Analysis: Correlation with Linguistic Aspects

Δ = human - systemeasy: Δ ≤ 20% acc
hard: Δ ≥ 40% acc

No statistically significant correlations with question difficulty!

10Analysis: Question Types¡ Hard questions seem to be generic,not specific to given passages (e.g.,“which of the following is correct?”)¡ Many “how many” questions in HardEasy questions

Hard questions

Standard Collection

Adveresarial Collection¡ Questions in Easy are more balanced(because the standard Qs are?)¡ This trend is probably because theworkers focus on writing specific types(i.e., generic and numeric) of questionsin the adversarial data collection11Analysis: Comprehension Types vs Difficulty¡ We annotated 980 example with sevencomprehension types¡ Numeric, spatial/temporal, and logicalquestions appear more often in thehard subset in both collection methods

12Analysis: Comprehension Types vs Sources
We observe some trends:¡ Technical documents (ReClor & Slate)→ Logical reasoning questions¡ Subjective or argumentative topics(Gutenberg, Slate, & ReClor)→ Gestalt/authorʼs attitude questions¡ Numbers in passages (MCTest, Wiki arts)→ Num reasoning in the adv. collection(Consistent with Kaushik et al. (2021)ʼs observations)13

Summary & Takeaway¡ Passage difficulty does not affect question difficulty¡ Selecting a diverse set of passages can help ensure a diverse rangeof reasoning types¡ Adversarial data collection has a risk to encourage workers to focus

on writing only a few specific types of questions

Our data is available at https://github.com/nii-cl/qa̲text̲source̲comparison
14