SlowGuess/ACL_2020_OCR huggingface dataset:
Jesse Dunietz,\* Gregory Burnham,\* Akash Bharadwaj, Owen Rambow, Jennifer Chu-Carroll, and David Ferrucci
Elemental Cognition
{jessed,gregb,akashb,owenr,jenniferc,davef}
@elementalcognition.com
Many tasks aim to measure MACHINE READING COMPREHENSION (MRC), often focusing on question types presumed to be difficult. Rarely, however, do task designers start by considering what systems should in fact comprehend. In this paper we make two key contributions. First, we argue that existing approaches do not adequately define comprehension; they are too unsystematic about what content is tested. Second, we present a detailed definition of comprehension—a TEMPLATE OF UNDERSTANDING—for a widely useful class of texts, namely short narratives. We then conduct an experiment that strongly suggests existing systems are not up to the task of narrative understanding as we define it.
Over the past few years, neural models (e.g., Chen et al., 2016; Devlin et al., 2019; Liu et al., 2019) have begun to match or even exceed human performance on MACHINE READING COMPREHENSION (MRC) benchmarks. In these tasks, systems demonstrate their comprehension of a passage by answering questions about it. Yet despite recent successes, MRC appears far from solved: systems continue to make basic, sometimes baffling mistakes, and they fail to generalize to new data. Such shortcomings have motivated a flurry of new MRC tasks, each designed to confront systems with questions deemed challenging for current methods. For example, tasks may ask questions requiring commonsense reasoning (Huang et al., 2019), multi-hop reasoning (Welbl et al., 2018), or inferences based on a second passage (Lin et al., 2019).
This line of research assumes that ever-more "difficult" question-answering tasks will ultimately lead to more robust and useful reading comprehension. We argue that, while the question-answering
format can be a fine choice for how to test comprehension, using difficulty as the basis for what to test is fundamentally flawed. To put it provocatively, the dominant MRC research paradigm is like trying to become a professional sprinter by glancing around the gym and adopting any exercises that look hard. The training may end up exercising some relevant muscles, but it is far too haphazard to achieve the ultimate goal.
Like athletic training, MRC tasks are not an end in themselves; ultimately, they are meant to lead to real-world applications. Current tasks may suffice for sufficiently similar applications—e.g., chatbots that look up customer questions in product documentation. But many proposed NLP applications hinge on deeper comprehension. Early work (e.g., Dyer, 1982) pointed to examples like assistance with legal disputes and service contracts; more recent work suggests applications such as summarizing a patient's clinical timeline (Jung et al., 2011). For such complex applications, machines will need to manipulate rich models of the world evoked by the text—e.g., to compare a claimant's narrative to legal standards, or to build a causal model of a patient's condition. From this broader perspective, the current paradigm falls short.
Specifically, we claim that in the quest for difficulty, task designers overlook the issue of what content—what information expressed, implied, or relied on by the passage—systems should comprehend. MRC datasets are usually constructed by having humans cast about for supposedly tricky questions, most often questions based on reasoning. But the questions that result are scattershot, offering little assurance that even a high-scoring system has achieved a useful and robust understanding.
We advocate for a different approach. We propose that the first step in defining MRC tasks should be specifying what content a system would likely need to understand for a given class of applica
7839
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7839-7859 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
tions. Only then can tasks systematically compile questions to probe for the internal model that the machine ought to have constructed.
This paper demonstrates such an approach for applications that involve understanding narratives. After reviewing existing approaches to constructing MRC datasets (§2), we argue for narratives as a valuable MRC testbed (§3.1). Then, inspired by cognitive science research on reading comprehension, we propose a "template of understanding" (ToU) for stories—an account of what an internal model of a story should minimally contain (§3.2). We also suggest ways to operationalize our ToU as a story comprehension task (§4). Finally, we show evidence from a pilot ToU-based task that current MRC models are not up to the challenge (§5).
2 Existing MRC dataset designs
This paper addresses how MRC tests can be made more systematic. Accordingly, we review existing tasks grouped by their data collection methods. We argue that each category falls short of testing a useful body of content in a satisfying way.
2.1 Manually written questions
By far the most popular strategy for generating MRC questions is to have humans—usually crowd workers, but sometimes trained annotators—think of questions about each passage.
The most straightforward version of this method gives annotators little to no guidance regarding what questions to ask. One early example is the TREC-8 dataset (Voorhees and Tice, 2000). In the more recent SNLI (Bowman et al., 2015) and MNLI (Williams et al., 2018) entailment tasks, the only constraint on crowd workers was that they produce one entailed, one contradicted, and one neutral hypothesis for each premise sentence. Similarly, the workers who assembled NewsQA (Trischler et al., 2017) were told only that the questions had to be answerable with short phrases, and workers for SQuAD (Rajpurkar et al., 2016) were simply given a "good" and a "bad" example and encouraged to use original wording.
The problem with such an open-ended generation process is that, absent stronger guidance, people tend to write simple questions that can be answered using lexical cues. (See, e.g., the dataset analysis in Rajpurkar et al., 2016.) This makes the tasks questionable measures of comprehension.
The dominant solution is to incorporate trickier twists. NarrativeQA (Kočiský et al., 2018) and DuoRC (Saha et al., 2018) reduce lexical similarity between questions and passages by showing annotators only a second passage about the same events. Other datasets emphasize reasoning presumed to be difficult, such as incorporating information from multiple parts of the text. MCTest (Richardson et al., 2013) and MultiRC (Khashabi et al., 2018) ask for questions that rely on multiple sentences; ROPES (Lin et al., 2019) has annotators apply information from one passage to write questions on a second; and HotpotQA (Yang et al., 2018b) and QASC (Khot et al., 2019) require multi-hop reasoning. Other forms of reasoning tested include coreference resolution (Quoref, Dasigi et al., 2019; Winograd Schema Challenge, Levesque et al., 2012), numerical reasoning (DROP, Dua et al., 2019), and commonsense reasoning (Cosmos QA, Huang et al., 2019). Tasks can also be made harder with devices such as unanswerable questions (SQuADRUn, Rajpurkar et al., 2018; NewsQA; CosmosQA) and filtering questions with an adversarial baseline (DROP; Quoref; QASC).
These twists do make MRC harder. But to pursue hard questions is to overlook why easy questions seemed inadequate in the first place: MRC tasks are a means to an end, namely useful applications, and easy questions—e.g., questions that depend only on lexical cues—do not suffice for that end. The techniques above may help by guiding annotators to a different space of questions: intuition suggests that some of these harder questions are indeed useful ones. But such techniques are an incomplete solution, as difficulty is a weak proxy for utility. What matters is not the system's sophistication per se; it is the alignment between the questions the system can answer and the ones a given application needs it to. Designing for difficulty still gives little assurance of such alignment.
Perhaps a truly random walk through question space would eventually cover a representative set of useful questions, but annotators are biased toward questions that humans find interesting (see Gordon and Van Durme, 2013; Misra et al., 2016; Zhang
<sup>1</sup>We will use "narrative" and "story" interchangeably, roughly following the Wikipedia definition: "A narrative or story is an account of a series of related events, experiences, or the like, whether true...or fictitious."
2Parts of the original RTE datasets (Dagan et al., 2006, etc.) were generated more systematically, but only in the sense that the outputs of NLP tools (e.g., translation or information extraction systems) were recorded as correct/incorrect examples of entailment. Little attention was paid to subject matter.
7840
et al., 2017). They do not think to ask questions whose answers seem obvious, even when those answers are essential to comprehension. If we do not delineate such facts and evaluate systems' ability to manipulate them, we will never be satisfied that the systems have adequately understood the text.
2.2 Naturally occurring questions
A second approach is to find questions "in the wild," then retrospectively collect documents containing the answers. This is the approach of BoolQ (Clark et al., 2019) and MS MARCO (Nguyen et al., 2016), which compile search engine queries, and of ELI5 (Fan et al., 2019), which harvests questions from Reddit's "Explain Like I'm Five" forum.
Such datasets are clearly useful for answering common queries, a valuable application class in its own right. For more complex applications, however, common queries are, if anything, less thorough than annotators at probing important elements of understanding (particularly aspects humans find obvious). The mismatch between questions and passage content is exacerbated by finding the passages retrospectively: the questions do not even attempt to test most of what each passage discusses, making them an insufficient measure of MRC.
2.3 Questions from tests designed for humans
The third strategy is to pull questions from tests written for humans. Examples include the early "Deep Read" corpus (Hirschman et al., 1999); the more recent TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) datasets, which mine collections of trivia questions; the AI2 Reasoning Challenge (ARC; Clark et al., 2018), which asks questions from standardized science tests; and RACE (Lai et al., 2017), which draws from English learning materials for Chinese school students.
Our chief concern about this approach echoes our concerns from §2.1: tests designed for humans rarely bother to test content that most humans find obvious. Accordingly, they gloss over vast swaths of understanding that machines do not yet have but which may be critical to applications. In addition, SearchQA, TriviaQA, and ARC find passages retrospectively, so again, the questions they ask only tangentially graze the content of each passage.
2.4 Automatically generated questions
Several projects generate questions algorithmically. The CNN/Daily Mail datasets (Hermann et al., 2015) and ReCoRD (Zhang et al., 2018) produce
cloze-style questions over news passages by masking out entities from summaries and below-the-fold sentences. ComplexWebQuestions (CWQ; Talmor and Berant, 2018) and WikiHop (Welbl et al., 2018) test for multi-hop reasoning by walking a structured knowledge base. Finally, bAbI (Weston et al., 2016) generates short texts and questions from a simple simulation of characters moving around.
Each algorithm encodes assumptions about what is worth asking. In theory, then, the algorithmic approach could produce a satisfying MRC test: given appropriate inputs, the algorithm could aim to generate questions that cover important content. Indeed, our proposal in §4.1 can be seen as a question generation algorithm to be run by humans.
In practice, however, algorithmic approaches have de-emphasized content. CNN/Daily Mail and ReCoRD capture explicit assertions about maskable entities, which do not amount to a principled body of content. The algorithms behind CWQ and WikiHop at least take as input some body of content, namely knowledge graphs. But the graphs include only a fraction—again, not a principled one—of the associated documents' content, and the questions are further restricted to rely on multi-hop reasoning. Multi-hop reasoning is no doubt a major error source for MRC, but applications are driven by what propositions must be extracted; whether each proposition takes zero inference steps or seven is immaterial. Accordingly, multi-hop questions are worth investigating, but they are not a sufficiently well-motivated body of content to constitute a measure of reading comprehension.
Similar remarks can be made about most of bAbI's 20 "tasks": grounded in simulations, their question generation algorithms start from known content, but target forms of reasoning. However, the tasks concerning time, positions, sizes, pathfinding, and motivations are closer to our content-first question generation strategy. These tasks are not driven by applications, and their synthetic passages are unrealistically simple, but among existing datasets, they are closest to our proposal.
2.5 Summary: What is missing
The most clear-cut way to test reading comprehension would be to select passages, describe what should be comprehended from them, and design tests for that understanding. Yet few MRC datasets have even approximated this approach. Many impose little structure on what content is tested; the
7841
rest pick some "difficult" form(s) of analysis or linguistic phenomena, but rarely consider downstream goals to determine what the questions should be about. Metrics for difficult reasoning and linguistic phenomena (see, e.g., Gardner et al., 2019) are useful, but only as tools for error analysis and mitigation; they are not top-line performance metrics.
In addition, many datasets to date suffer from two other problems: 1) they select passages after the questions are asked, meaning the questions test comprehension of only small portions of the passages; and/or 2) they ask very few questions whose answers are obvious to humans.
These issues of content scope also intersect with issues of format. Many tasks have adopted a span extraction format, including TREC QA, NewsQA, and (most notably) SQuAD and its successors. This format immediately rules out questions about inferred events or entities, which may be essential to a complete interpretation. The main alternative is multiple choice (MC), used in tasks such as Cosmos QA, RACE, ARC, WikiHop, and every task in GLUE (Wang et al., 2019b) and SuperGLUE (Wang et al., 2019a). But MC has its own problem of providing extra hints via answer choices.
We will return to the format issue in  $\S 4$ . But first, we propose a more systematic approach to constructing MRC datasets.
3 Defining deep story understanding
Our approach starts from the content of a passage, which we define as the information it expresses, implies, or relies on. Specifically, we propose that task designers lay out a minimal body of content that MRC systems should demonstrate they understand. Exactly what that content is will vary from passage to passage, of course, but the key is to define a TEMPLATE OF UNDERSTANDING (ToU): a set of question templates that can be filled in with specific events and entities for any given passage. The answers to the fleshed-out questions will constitute a floor of understanding for the passage—a plausible lower bound on what content machines ought to comprehend.
The natural next question is what content the ToU should cover. System needs will vary by application. To advance MRC writ large without limiting ourselves to a single application, we propose selecting a class of texts where one could reasonably predict a priori what content would be useful for applications. In the rest of this section, we endorse
fictional narratives as a particularly promising class of texts and propose a ToU for them. $^3$
3.1 The case for stories
Stories have several convenient properties that recommend them as a testbed for MRC.
Most importantly, applications that involve comprehending stories are numerous and diverse. Consider a legal aid tool: to assess whether a lawsuit may be warranted, it would have to comprehend an account of the events in question. Likewise, a tool that finds candidates for medical trials would need to read each patient history. (Appendix A fleshes out these scenarios.) These examples are not exceptional; applications in other domains will depend on stories in customer complaints, intelligence dispatches, financial news, and many other document types. Humans tend to think and communicate in terms of stories (see, e.g., Haidt, 2013; Mateas and Sengers, 1999; Bruner, 1991; Eck, 2006), so it is unsurprising that stories are ubiquitous in the content we want NLU tools to help us with.
Additionally, stories come with a strong prior from cognitive science about what elements of understanding will be useful. Research on human reading comprehension (e.g., Graesser et al., 1994; Zwaan et al., 1995) suggests that humans attend primarily to the timeline of events, to the locations of entities and events, and to the causes and motivations of events and actions. For applications that involve story comprehension, we can expect that machines will need to understand these same dimensions. We can thus design a principled ToU for stories even without specifying an application.
Stories' content also makes them a particularly compelling demonstration of understanding, for two reasons. First, cognitive science suggests that humans make more inferences when reading narrative text than expository text (Graesser et al., 1994). In particular, a story entails a highly structured network of relations (timelines, causality, etc.). Thus, stories do exercise abilities beyond simple factoid extraction. Second, stories rely on a large body of implicit world knowledge. If a system is able to use and express that knowledge when reading stories, it will likely be able to apply the same knowledge even when comprehending other kinds of texts.
Among stories, fictional ones offer the strongest test of comprehension: their contents cannot be
3To be clear, we are not claiming that fictional narratives are themselves an application; only that they are a class of texts that are useful for many applications.
7842
found in corpora, so systems must rely on comprehending the text (Richardson et al., 2013). Accordingly, we suggest using fictional narratives as the basis for developing a ToU and evaluating MRC.
3.2 A ToU for stories
We propose four overlapping clusters of questions for story comprehension, corresponding to the four elements identified by Zwaan et al. (1995) as the ones humans attend to when reading stories. Further support for these questions, particularly the last two, comes from early work in computational story understanding: Schank and Abelson (1977) identify causal chains, plans and goals as crucial elements of understanding multi-sentence stories.
1. Spatial: Where are entities positioned over time, relative to landmarks and each other? How are they physically oriented? And where do events take place?
2. Temporal: What events and sub-events occur, and in what order? Also, for what blocks of that timeline do entities' states hold true?
3. Causal: How do events and states lead mechanistically to the events and states described or implied by the text?
4. Motivational: How do agents' beliefs, desires, and emotions lead to their actions?
These question templates form the ToU. Systems should ideally be able to answer them about all entities and events that the story mentions or implies (though of course some entities/events are more important than others; see §4.1). We do not have a separate category for "who did what to whom" information, but we expect strong performance on the ToU to hinge on such analysis. In particular, much of this information is captured in the characterization of events for temporal questions.
Of course, these four facets do not cover everything one might comprehend. They include nothing about the story's message, or how it resembles other stories, or even most counting questions. The ToU merely provides a lower bound on what is needed. That said, many forms of reasoning (e.g., counting) can be reduced to deterministically manipulating the answers to multiple ToU questions.
4 Towards a story understanding task
Our ToU provides a conceptual framework for stating what a machine should understand from a story.
Spatial (sample entries):
- Rover is in the yard from when he runs out the door until he runs inside.
- Rover is in the house from when he runs inside until the end of the story.
Temporal (sample entries):
- Allie arrives just before Rover runs outside.
Rover barks just before he runs inside.
- It is still raining at the end of the story.
Motivational (sample entry):
Rover runs inside, rather than staying put, because:
- If he runs inside, he will be inside, whereas if he does not he will be outside, because:
* Rover is outside.
* Running to a place results in being there.
- If Rover is inside, he will not get rained on, whereas if he is outside he will, because:
* It is raining.
* When it is raining, things that are outside tend to get rained on, whereas things inside do not.
Rover would prefer not getting rained on to getting rained on, because:
* Most dogs prefer not to get rained on.
Figure 1: A partial RoU for the following simple story fragment: ... One day, it was raining. When Allie arrived, Rover ran out the door. He barked when he felt the rain. He ran right back inside.
However, there remains the challenge of operationalizing the framework-i.e., of rigorously assessing whether a machine has that understanding.
We do not claim to have solved this problem, but in this section we discuss two broad directions for further development: evaluating based on annotated answers to ToU questions and asking untrained humans to rank different answers. These approaches might even be combined to offer complementary perspectives on system performance.
4.1 Approach 1: Annotating ToU answers
One class of approaches starts with trained annotators writing plain-English answers to each ToU question. The annotators are given guidelines for instantiating the ToU on new stories and for making answers detailed and thorough. We call an annotator's answer document a RECORD OF UNDERSTANDING (RoU); see Figure 1 for an example.
Conceptually, answering temporal and spatial questions is straightforward, but the causal and motivational questions require more definition. People accept many kinds of answers to such questions. It is therefore important to clarify what a good answer should include—i.e., what causal or motivational
7843
facts an MRC system should comprehend.
We base our account of these questions on the philosophical literature on causality (see Schaffer, 2016) and on the social science literature on what explanations people seek (see Miller, 2019). Following this scholarship, we conceptualize a causal or motivational question as asking what root cause led the event or state from the story to happen rather than some alternative outcome. For example, in a story about Rover the dog, the question of why Rover came inside is taken to mean: Why did Rover come inside, rather than remaining where he was?4
The answer to such a question is a CAUSAL CHAIN tracing from the root cause to the event or state described in the story (see Figure 2 for examples). The links in the chain walk in lock-step through two parallel worlds: the REALIZED WORLD, where the root cause held true and led to the observed outcome; and an ALTERNATIVE WORLD, where the root cause would have been changed and led to some alternative outcome.
For mechanistic causation, each link in the chain ends in an event that helped bring about the outcome described in the story. For example, two mechanistic links from Figure 2a are the plant looks brown (rather than green) because it is unhealthy (rather than healthy) and the plant is unhealthy because it has little light (rather than lots of light).