SlowGuess/ACL_2020_OCR huggingface dataset:
For motivations, the structure is slightly different. Rather than the final link being an event that happened in the story, it is a statement of the agent's preferences (in Figure 2b, Rover would prefer not being rained on to being rained on). The links leading to it are the future causes and effects that the agent imagines will lead from their action to their preferred outcome (e.g., going inside leading to being inside leading to not getting rained on).
The causal chain provides the backbone of an explanation for an event or action, but the full explanation should recursively explain each link (e.g., Rover would prefer not being rained on to being rained on). Recursive explanations appeal to some combination of general knowledge about the world (e.g., Most dogs prefer not to get rained on) and
story-specific SUPPORTING FACTS—e.g., the fact that Rover is outside. Supporting facts generally need to be recursively explained, as well.
Even with guidelines, different annotators may give substantively different answers. In particular, they may drill down to different levels of detail in a causal chain before bottoming out in general knowledge—e.g., rather than stopping at dogs disliking rain, one annotator might explain that Rover disprefers rain because he dislikes getting wet, which in turn is because dogs often dislike getting wet. To handle such disagreements, we can adopt the pyramid method (Nenkova and Passonneau, 2004) from abstractive summarization, another task where annotators may provide different but equally sensible ground truths. Under this method, a reconciler merges RoUs into a single rubric by identifying shared content “nuggets” (e.g., that it is raining) and weighting each by how many annotators cited it. (See Voorhees [2004] for more on nuggets.)
4.1.1 Preliminary notes on RoU agreement
We conducted a small pilot study on RoU annotation: with the help of 5 annotators, we iteratively crafted guidelines and tested them on 12 stories. Here we share some initial qualitative observations.
For spatial annotations, agreement improved when annotators first drew a simple sketch of each scene, then translated their sketches into statements. This process seemed to help annotators notice implicit spatial facts. Some annotators also reported that sketches lowered the cognitive burden.
For temporal annotations, annotators generally agreed on what events took place and the temporal relations between them. Disagreements stemmed mainly from choices of which implicit occurrences to annotate. We are exploring ways to promote consistency, including having annotators draw timelines to draw attention to missing events. We are also looking to incorporate prior art (e.g., TimeML; Pustejovsky et al., 2003) into our guidelines.
On causal and motivational questions, we were pleasantly surprised by the conceptual consistency between annotators. Annotators appealed to similar causal assertions, even bottoming out in similarly detailed general rules. What was less consistent was structure—how causal chains were carved into links and how bullets were nested. Annotators also occasionally omitted self-evident general rules or supporting facts. We are optimistic that both issues can be improved by more examples and training.
As expected, annotators occasionally differed on
${}^{4}$  Causality as contrast may seem unintuitive,particularly since "why" questions tend not to state a contrasting outcome. But the audience generally just infers a reasonable default. Beyond its support in the literature, contrast offers several advantages. It makes it far easier to match intuitions about what should factor into a causal explanation. It also naturally handles relative preferences, and allows explaining multiple aspects of an event-e.g.,John walking carefully can be explained in contrast to both staying put and walking normally.
7844
(a) A mechanistic causal chain for the question, "Why did the plant turn brown?"
(b) A motivational causal chain for the question, "Why did Rover the dog run back inside when it started raining?"
Figure 2: Example causal chains answering causal (above) and motivational (below) ToU questions.
which causal contrasts to include. Such borderline judgments of salience may be inevitable, and seem to warrant use of the pyramid method.
4.1.2 Free-text evaluation
It is difficult to evaluate a system directly on an RoU or a rubric, as they are written in plain English. One option is to pose broad ToU questions (e.g., "What events happened and in what order?") and then to automatically compare systems' full fre-text answers to annotators'. But this would require an automated comparison metric, and existing metrics such as ROUGE and BLEU are concerned only with lexical similarity. Their correlation with humans' quality judgments is substantial but not stellar (Callison-Burch et al., 2006), and high scores do not always indicate good answers in MRC (see Yang et al., 2018a; Nema and Khapra, 2018). Superficial similarity measures may prove particularly weak given how open-ended ToU questions are.
Alternatively, human evaluators could read both the RoU-derived rubric and the system output and decide whether the output adequately covers each nugget from the rubric. This is how the pyramid method is typically applied in summarization.
Still a third possibility is to have human evaluators ask targeted questions about each nugget from the rubric. The evaluators could then judge whether the system's shorter free-text answers reflect a consistent understanding of that nugget. Such evaluation would be especially powerful if the evaluators knew the NLP systems' typical shortcuts and could reword a given question accordingly: a suspicious evaluator could query for the same fact in multiple ways to verify that the system consistently gets it right. This would make results more satisfying than many MRC evaluations, as systems couldn't rely
on terse answers being interpreted charitably.
Of course, using humans for the final evaluation is expensive, even if automated metrics are used during model development. Human evaluators also add variability and subjectivity, as they may probe differently for the same knowledge or find a given answer more or less convincing. Still, new tasks often start with human evaluation while the community fine-tunes what is worth measuring, and only later to progress to automated metrics that approximate human judgment. Such were the trajectories of topic model coherence (see Lau et al., 2014), summarization (see Yang et al., 2016), and machine translation (see Papineni et al., 2002), so it is a plausible pathway for RoU evaluation, too.
4.1.3 Thorough multiple-choice evaluation
Free-response is a compelling format that is tricky to evaluate. Multiple-choice inverts the trade-off: it is less compelling, but much easier to evaluate.
With the help of the ToU, a multiple-choice (MC) test can be fairly comprehensive. Question writers would first write out RoUs for a story, and perhaps reconcile them into a weighted rubric. They would then write MC questions targeting each nugget in the rubric: What goal is Rover pursuing by running inside rather than staying put? Where was Rover after he ran through the door? How were Rover, the house, and the rain positioned at the end of the story? Etc. Such a thorough MC test based on RoUs would be a step up from current tasks.
The downside of an MC task is that, though easy to evaluate, it would be questionable as a measure of comprehension. All MC tasks suffer from the same lack of naturalness: questions do not normally come with candidate answers, and ranking candidates is simply easier than the tasks MRC
7845
should ultimately support. Furthermore, systems learn to exploit incidental surface features in the question, sometimes performing well even without seeing the passage (Kaushik and Lipton, 2018). When humans take MC tests, we can make strong assumptions about what they must know or do to succeed; an NLP system offers no such assurances.
In the long run, then, we do not see multiple choice as an adequate format for demonstrating MRC. Still, such tests offer some leverage for progress in the short term.
4.2 Approach 2: Competing to satisfy judges
The RoU guidelines put a stake in the ground as to how ToU questions should be answered. But as noted above, ToU questions, particularly "why" questions, admit many good answers. The ones canonicalized by the guidelines and by annotators following them may not always be the most useful.
Consequently, it may prove beneficial to appeal directly to human intuition about what understanding entails. We have assumed that what lets humans perform story-related tasks is that they possess some internal answers to the ToU. If we further assume that humans can be led to favor machine answers that resemble their own internal ones, then humans should make good judges of answer quality even without the guidance of RoUs.
Accordingly, we could let humans judge system's full free-text answers based only on intuitive preferences. Evaluators could still be guided to ask ToU questions thoroughly, but extensive guidelines would not be needed: neither asking questions nor recognizing good answers demands nearly as much specification as stating canonical answers.
Whereas the approaches in §4.1 must strive for replicability in humans' answers, this approach seeks replicability only in humans' judgments of answers. We suggest two ways to achieve this.
First, in the absence of a rubric, we suspect that answers would best be judged via pairwise comparisons. For free-text writing, humans generally find comparative assessment easier than absolute scoring (Pollitt, 2012), and comparison is already used to evaluate natural-language generation (see, e.g., Yatskar et al., 2014). Comparisons also mitigate the difficulty of spotting errors of omission: when evaluators see an incomplete answer in isolation, they may gloss over or mentally fill in what was left unsaid. Comparing against a more complete competing answer makes it easier to notice gaps.
Second, evaluators can be guided to tease apart their judgments into several desirable dimensions of explanations—e.g., accuracy, depth, and coherence—just as is often done for natural language generation. Pilot studies would be required to refine the dimensions and their specifications.
5 Current MRC systems do not comprehend stories
If current systems performed well on the ToU, our argument would be moot. This section presents evidence that they do not.
5.1 Data and experimental setup
To test existing systems, the questions must be presented in a form the systems can handle. Many systems were designed for span extraction, but the ToU does not lend itself to answering with text spans. Instead, we report on experiments with a pilot version of the MC task described in §4.1.3.
To construct the test, we selected the first two narrative stories in the dev set of RACE (Lai et al., 2017). Based on our preliminary annotation guidelines, one annotator read both stories, drafted an RoU for each, and wrote a question for each statement in the rough RoUs. The annotator then collaborated with several others to write distractor answers, each characterized by one or more of the following: small surface variations on the correct answer that change the meaning; language from the passage, especially words that appear near words from the question; and language that might plausibly collocate with words from the question.
As an additional test for robustness, questions came in "variant groups": each question was paired with a variant, or occasionally more than one, that asks for the same information in a different way (see Figure 3). The distractors were often altered as well. We then evaluated accuracy in two ways: counting each question independently and counting each variant group as one unit. In the latter method, the group is marked correct only if both variants were answered correctly. This simulates a suspicious evaluator re-asking the question and deducting points if the model does not consistently exhibit the desired understanding.
The resulting dataset contains a total of 201 questions (98 variant groups).  $29\%$  are spatial or temporal; the remaining  $71\%$  are causal or motivational. The questions average 5.1 options, with a minimum of 4. (Including many distractors somewhat
7846
Q) What actually happened when Mr. Green and the man drove together?
A) They came to a small house.
B) They came to a hotel.
C) They traveled around the country.
D) They stopped several times at the side of the road.
Q) How did the man's directions actually turn out?
A) The directions the man gave led to where the man wanted to go.
B) The directions the man gave led to where Mr. Green wanted to go.
C) The directions Mr. Green gave led to where the man wanted to go.
D) The directions Mr. Green gave led to where Mr. Green wanted to go.
Figure 3: An example variant group from our ToU-based questions; correct answers in italics. In the associated RACE story, a man tricks Mr. Green into driving him home under the pretense of guiding Mr. Green to a hotel. See Appendix B for the full story text.
mitigates the weaknesses of the MC format.) All questions are included in the supplementary materials; Appendix B shows many examples.
For validation, the questions were presented to two colleagues with non-technical degrees. They scored  $96\%$  and  $91\%$  (measured on variant groups), suggesting that motivated, well-educated humans have little trouble with our questions.
Finally, we put the questions to XLNet (Yang et al., 2019), a large, transformer-based language model trained with generalized autoregression on BooksCorpus and English Wikipedia. After finetuning, the model achieves  $81.75\%$  on the original RACE task (within 5 points of the best nonensemble model at the time of the experiments).
Our results (Table 1) show that XLNet performs poorly. On individual questions, it scores just  $37\%$ , closing less than a third of the gap between chance and human performance. This strongly suggests that whatever XLNet is doing, it is not learning the ToU's crucial elements of world understanding. Furthermore, the system's performance is brittle, with many correct answers attributable to luck and/or unreliable cues: when moving from questions to variant groups, human performance falls just 3 points. XLNet's performance, on the other
All	By question type
Spatial + Temporal	Causal + Motivational
Per question	37%	33%	38%
Chance	15%	20%	13%
Human (avg.)	96%	93%	97%
Per variant group	20%	14%	23%
Chance	4%	5%	5%
Human (avg.)	93%	90%	95%
Table 1: XLNet accuracy on our ToU-based questions.
hand, falls 17 points, which leaves the system closing just  $18\%$  of the chance-vs.-human gap.
Although we tested only XLNet, all the other models that currently dominate the leaderboards are similar pre-trained language models; none has any distinguishing characteristic that might be expected to produce dramatically better results on our dataset. Likewise, no existing dataset is so much more systematic than RACE that fine-tuning on it should dramatically improve results on our dataset. Especially given that multiple-choice tests are artificially easy for systems (see §4.1.3), our pilot experiment offers strong evidence that existing MRC systems do not succeed on the ToU.
6 Taking the ToU idea forward
Our ToU for stories is a first attempt at defining what MRC systems should comprehend in a principled, systematic way. Drawing on work in psychology, philosophy, and pedagogy, we have argued for the ToU as a minimal standard and a valuable target for MRC. We have also shown it to be beyond the reach of current systems.
We therefore suggest that the NLP community further build on our ToU. This includes refining and perhaps expanding the questions; better defining the answers and evaluation procedures; building MRC corpora based on the ToU; and developing better-performing systems. We ourselves are working on all four, and we welcome collaboration.
But even beyond our ToU, the broader point stands: existing MRC approaches are not satisfactorily testing for a systematic set of content. Our efforts demonstrate that it is possible, with a sufficiently interdisciplinary approach, to define a plausible floor for comprehension for a given class of applications. If MRC is to achieve its ultimate goals, we—the NLP community—owe it to ourselves to ensure that our reading comprehension tests actually test for the comprehension we desire.
For questions with more than four answers, we split the answers across multiple sub-questions, all of whose answer sets contained the correct answer. We counted the question correct only if that answer was chosen across all answer sets. Chance performance was adjusted accordingly.
7847
Samuel R. Bowman, Gabor Angeli, Christopher Potts, and Christopher D. Manning. 2015. A large annotated corpus for learning natural language inference. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing, pages 632-642, Lisbon, Portugal. Association for Computational Linguistics.
Jerome Bruner. 1991. The narrative construction of reality. Critical Inquiry, 18(1):1-21.
Chris Callison-Burch, Miles Osborne, and Philipp Koehn. 2006. Re-evaluating the role of BLEU in machine translation research. In 11th Conference of the European Chapter of the Association for Computational Linguistics, Trento, Italy. Association for Computational Linguistics.
Danqi Chen, Jason Bolton, and Christopher D. Manning. 2016. A thorough examination of the CNN/ Daily Mail reading comprehension task. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2358-2367, Berlin, Germany. Association for Computational Linguistics.
Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. 2019. *BoolQ: Exploring the surprising difficulty of natural yes/no questions.* In *Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies*, Volume 1 (Long and Short Papers), pages 2924–2936, Minneapolis, Minnesota. Association for Computational Linguistics.
Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. 2018. Think you have solved question answering? Try ARC, the AI2 reasoning challenge. CoRR, arXiv:1803.05457v1 [cs.AI].
Ido Dagan, Oren Glickman, and Bernardo Magnini. 2006. The PASCAL recognising textual entailment challenge. In Proceedings of the First International Conference on Machine Learning Challenges: Evaluating Predictive Uncertainty Visual Object Classification, and Recognizing Textual Entailment, MLCW'05, pages 177-190, Berlin, Heidelberg. Springer-Verlag.
Pradeep Dasigi, Nelson F. Liu, Ana Marasovic, Noah A. Smith, and Matt Gardner. 2019. Quoref: A reading comprehension dataset with questions requiring coreferential reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 5927-5934, Hong Kong, China. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of
deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, and Matt Gardner. 2019. DROP: A reading comprehension benchmark requiring discrete reasoning over paragraphs. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 2368-2378, Minneapolis, Minnesota. Association for Computational Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V. Ugur Güney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. CoRR, arXiv:1704.05179v3 [cs.CL].
Michael George Dyer. 1982. In-depth Understanding: A Computer Model of Integrated Processing for Narrative Comprehension. Ph.D. thesis, Yale University, New Haven, CT, USA. AAI8220836.
Jill Eck. 2006. An analysis of the effectiveness of storytelling with adult learners in supervisory management. Master's thesis, University of Wisconsin-Stout, Menomonie, WI.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Matt Gardner, Jonathan Berant, Hannaneh Hajishirzi, Alon Talmor, and Sewon Min. 2019. On making reading comprehension more comprehensive. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 105-112, Hong Kong, China. Association for Computational Linguistics.
Jonathan Gordon and Benjamin Van Durme. 2013. Reporting bias and knowledge acquisition. In Proceedings of the 2013 Workshop on Automated Knowledge Base Construction, AKBC '13, pages 25-30, New York, NY, USA. ACM.
Arthur C Graesser, Murray Singer, and Tom Trabasso. 1994. Constructing inferences during narrative text comprehension. Psychological Review, 101(3):371.
Jonathan Haidt. 2013. The Righteous Mind: Why Good People are Divided by Politics and Religion, chapter 12. Vintage Books.
7848