SlowGuess/ACL_2020_OCR huggingface dataset:
Karl Moritz Hermann, Tomáš Kočisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the 28th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, pages 1693-1701, Cambridge, MA, USA. MIT Press.
Lynette Hirschman, Marc Light, Eric Breck, and John D. Burger. 1999. Deep Read: A reading comprehension system. In Proceedings of the 37th Annual Meeting of the Association for Computational Linguistics, pages 325-332, College Park, Maryland, USA. Association for Computational Linguistics.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Hyuckchul Jung, James Allen, Nate Blaylock, Will de Beaumont, Lucian Galescu, and Mary Swift. 2011. Building timelines from narrative clinical records: Initial results based on deep natural language understanding. In Proceedings of BioNLP 2011 Workshop, BioNLP '11, pages 146-154, Stroudsburg, PA, USA. Association for Computational Linguistics.
Divyansh Kaushik and Zachary C. Lipton. 2018. How much reading does reading comprehension require? A critical investigation of popular benchmarks. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 5010-5015, Brussels, Belgium. Association for Computational Linguistics.
Daniel Khashabi, Snigdha Chaturvedi, Michael Roth, Shyam Upadhyay, and Dan Roth. 2018. Looking beyond the surface: A challenge set for reading comprehension over multiple sentences. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 252-262, New Orleans, Louisiana. Association for Computational Linguistics.
Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2019. QASC: A dataset for question answering via sentence composition. CoRR, arXiv:1910.11473 [cs.CL].
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Jey Han Lau, David Newman, and Timothy Baldwin. 2014. Machine reading tea leaves: Automatically evaluating topic coherence and topic model quality. In Proceedings of the 14th Conference of the European Chapter of the Association for Computational Linguistics, pages 530-539, Gothenburg, Sweden. Association for Computational Linguistics.
Hector J. Levesque, Ernest Davis, and Leora Morgenstern. 2012. The Winograd Schema Challenge. In Proceedings of the Thirteenth International Conference on Principles of Knowledge Representation and Reasoning, KR'12, pages 552-561. AAAI Press.
Kevin Lin, Oyvind Tafjord, Peter Clark, and Matt Gardner. 2019. Reasoning over paragraph effects in situations. In Proceedings of the 2nd Workshop on Machine Reading for Question Answering, pages 58-62, Hong Kong, China. Association for Computational Linguistics.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. CoRR, arXiv:1907.11692v1 [cs.CL].
Michael Mateas and Phoebe Sengers. 1999. Narrative intelligence. In *Narrative Intelligence: Papers from the 1999 Fall Symposium*, pages 1-10, Stanford, CA, USA.
Tim Miller. 2019. Explanation in artificial intelligence: Insights from the social sciences. Artificial Intelligence, 267:1-38.
Ishan Misra, C Lawrence Zitnick, Margaret Mitchell, and Ross Girshick. 2016. Seeing through the human reporting bias: Visual classifiers from noisy human-centric labels. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 2930-2939.
Preksha Nema and Mitesh M. Khapra. 2018. Towards a better metric for evaluating question generation systems. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 3950-3959, Brussels, Belgium. Association for Computational Linguistics.
7849
Ani Nenkova and Rebecca Passonneau. 2004. Evaluating content selection in summarization: The pyramid method. In Proceedings of the Human Language Technology Conference of the North American Chapter of the Association for Computational Linguistics: HLT-NAACL 2004, pages 145-152, Boston, Massachusetts, USA. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majumder, and Li Deng. 2016. MS MARCO: A human generated machine reading comprehension dataset. CoRR, arXiv:1611.09268v3 [cs.CL].
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. BLEU: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, pages 311-318, Philadelphia, Pennsylvania, USA. Association for Computational Linguistics.
Alastair Pollitt. 2012. Comparative judgement for assessment. International Journal of Technology and Design Education, 22(2):157-170.
James Pustejovsky, José M Castano, Robert Ingria, Roser Saurí, Robert J Gaizauskas, Andrea Setzer, Graham Katz, and Dragomir R Radev. 2003. TimeML: Robust specification of event and temporal expressions in text. In New directions in question answering: Papers from the AAAI Spring Symposium, volume 3, pages 28-34, Stanford, CA, USA.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computational Linguistics.
Amrita Saha, Rahul Aralikatte, Mitesh M. Khapra, and Karthik Sankaranarayanan. 2018. DuoRC: Towards complex language understanding with paraphrased reading comprehension. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1683-1693, Melbourne, Australia. Association for Computational Linguistics.
Jonathan Schaffer. 2016. The metaphysics of causation. In Edward N. Zalta, editor, *The Stanford Encyclopedia of Philosophy*, fall 2016 edition. Metaphysics Research Lab, Stanford University.
Roger C. Schank and Robert P. Abelson. 1977. Scripts, Plans, Goals and Understanding. Lawrence Erlbaum Associates, Hillsdale, NJ.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP, pages 191-200, Vancouver, Canada. Association for Computational Linguistics.
Ellen M. Voorhees. 2004. Overview of the TREC 2003 question answering track. In Proceedings of the Twelfth Text Retrieval Conference (TREC 2003), pages 54-68. National Institute of Standards and Technology.
Ellen M. Voorhees and Dawn M. Tice. 2000. Building a question answering test collection. In Proceedings of the 23rd Annual International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR '00, pages 200-207, New York, NY, USA. ACM.
Alex Wang, Yada Pruksachatkun, Nikita Nangia, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. 2019a. SuperGLUE: A stickier benchmark for general-purpose language understanding systems. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 3266-3280. Curran Associates, Inc.
Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019b. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In 7th International Conference on Learning Representations (ICLR 2019), New Orleans, LA, USA. Open-Review.net.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.
Jason Weston, Antoine Bordes, Sumit Chopra, and Tomas Mikolov. 2016. Towards AI-complete question answering: A set of prerequisite toy tasks. In
7850
4th International Conference on Learning Representations, ICLR 2016, San Juan, Puerto Rico, May 2-4, 2016, Conference Track Proceedings.
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122. Association for Computational Linguistics.
An Yang, Kai Liu, Jing Liu, Yajuan Lyu, and Sujian Li. 2018a. Adaptations of ROUGE and BLEU to better evaluate machine reading comprehension task. In Proceedings of the Workshop on Machine Reading for Question Answering, pages 98-104, Melbourne, Australia. Association for Computational Linguistics.
Qian Yang, Rebecca J Passonneau, and Gerard De Melo. 2016. PEAK: Pyramid evaluation via automated knowledge extraction. In Proceedings of the Thirtieth AAAI Conference on Artificial Intelligence, pages 2673-2680. AAAI Press.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019. XLNet: Generalized autoregressive pretraining for language understanding. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems 32, pages 5753-5763. Curran Associates, Inc.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018b. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Mark Yatskar, Michel Galley, Lucy Vanderwende, and Luke Zettlemoyer. 2014. See no evil, say no evil: Description generation from densely labeled images. In Proceedings of the Third Joint Conference on Lexical and Computational Semantics (*SEM* 2014), pages 110-120, Dublin, Ireland. Association for Computational Linguistics and Dublin City University.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. ReCoRD: Bridging the gap between human and machine commonsense reading comprehension. CoRR, arXiv:1810.12885v1 [cs.CL].
Sheng Zhang, Rachel Rudinger, Kevin Duh, and Benjamin Van Durme. 2017. Ordinal common-sense inference. Transactions of the Association for Computational Linguistics, 5:379-395.
Rolf A. Zwaan, Mark C. Langston, and Arthur C. Graesser. 1995. The construction of situation models in narrative comprehension: An event-indexing model. Psychological Science, 6(5):292-297.
7851
A Examples of applying the ToU to stories for applications
In the main text (§3.1), we suggested that many advanced applications hinge on understanding the elements captured by our ToU for stories. Here we offer several examples from two domains.
A.1 Law
For the foreseeable future, legal decision-making will be the province of lawyers, not AI. However, one plausible use for MRC in a legal setting is as a screening tool for helping non-lawyers determine whether a case has enough merit to bother bringing in a lawyer.
For example, consider the first-person narrative below (fictional, but based on an amalgam of several real news stories):
My property borders on public lands where hunting is allowed. Last month, a hunter tracked a buck onto my property. He claims he didn't see my boundary sign. He ended up stepping up onto the remains of an old stone wall, which crumbled, and he broke his wrist. Now he's saying I can give him $10K now and he'll walk away, or else he's going to sue me for much more.
Before contracting a lawyer, the property owner may want to assess whether there is any merit to the threat. On the other side of the deal, a law firm that offers free initial consultations may wish to avoid wasting time on cases that are clear non-starters.
A second legal application for NLU tools might be helping a lawyer search for precedents. For instance, a tool could help with the narrative above (or perhaps a third-person version of it) by looking for cases with similar elements—e.g., an accidental trespass resulting in injury.
To assist in such application scenarios, a system would of course need information about legal codes. But it would also have to understand what happened in the cases it is trying to analyze. To that end, the answers to ToU questions would be essential, as demonstrated in Table 2. The table shows ToU questions and answers that would be key to understanding the landowner's situation. (These questions are ones the system would answer for itself while reading, not necessarily questions it would be asked by a user.)
A.2 Medicine
Medicine also offers ample opportunity for an MRC system competent in the narrative ToU to assist doctors and researchers. Narratives pervade electronic health records in the form of doctors' notes, which record information ranging from patient history to detailed descriptions of surgical procedures.
One narrative-based medical application is helping doctors understand a prior doctor's rationale. Currently, doctors often spend time sifting through a patient's records to understand why a prior doctor made a certain decision. The reasoning is often explained, but many documents must be searched to find the relevant note.
For example, consider the real medical note below, recorded after a routine follow-up appointment following breast cancer treatment:
She underwent radiation treatment ending in May 2008. She then started on Arimidex, but unfortunately she did not tolerate the Arimidex and I changed her to Femara. She also did not tolerate the Femara and I changed it to tamoxifen. She did not tolerate the tamoxifen and therefore when I saw her on 11/23/09, she decided that she would take no further antiestrogen therapy. She met with me again on 02/22/10, and decided she wants to rechallenge herself with tamoxifen. When I saw her on 04/28/10, she was really doing quite well with tamoxifen. She tells me 2 weeks after that visit, she developed toxicity from the tamoxifen and therefore stopped it herself. She is not going take to any further tamoxifen.
A future doctor may wonder why the patient is not on hormone therapy, which would be standard procedure. This explanatory note may be hard to find amongst the many notes in the patient's record.
A second medical application is finding patients who qualify for medical trials. For instance, a pharmaceutical company might develop a new anti-estrogen drug that they believe has milder side effects. They would then want to find patients who had already tried several anti-estrogen drugs,
${}^{6}$  Quoted from https://www.mtsamples.com/ site/pages/sample.asp?type=96- & Sample= 1939-Breast%20Cancer%20Followup%20-%201
7852
Table 2: Example ToU questions and answers for a legal application.
7853
Question type	ToU question	Example (partial) answer to ToU question	Significance to medical application
Temporal	When did the patient start and stop taking tamoxifen?	Multiple times: She started taking it sometime after May 2008 and stopped taking it by 11/23/09. Then, she started taking it again on 02/22/10, and stopped taking it by mid-May 2010.	A clinical trial may be seeking patients who kept stopping and starting a specific drug. It may also be important how long the side effects took to develop.
Also note that if the question of interest is really a counting question (“how many times”), this relies most of all on an underlying temporal understanding like the one captured by the ToU.
Causal/Motivational	Why is the patient not taking an anti-estrogen drug (rather than taking one)?	She was taking Arimidex, and it caused strong side effects (rather than her having mild or no side effects). Preferring fewer side effects, she therefore tried Femara (rather than continuing with Arimidex). Femara also caused side effects, so for the same reasons as before, she tried switching to tamoxifen (rather than continuing the Femara), but it also caused side effects. The patient preferred not experiencing the side effects to having the medical benefits of the drugs, so she decided not to take any such drug (rather than continuing with one of the above).	A future doctor may expect the patient to be on an anti-estrogen drug, as that is standard for someone with her history of breast cancer. Understanding that the patient has tried many drugs and decided to stop them may inform the doctor's course of action. The doctor might proceed differently if he determined that she had stopped for some other reason—e.g., that she simply lapsed in a prescription.
Also, a clinical trial may be seeking patients who stopped taking a drug because of side effects. Furthermore, the trial might be seeking specifically patients who stopped taking the drug at the advice of the doctor.
Table 3: Example ToU questions and answers for a medical application.
perhaps multiple times, and had toxicity problems with all of them. Currently, research hospitals find patients for a given clinical trial by employing humans to read through the hospital's database of medical notes and determine which patients meet the trial's criteria.
To assist in such application scenarios, an automated system would have to understand medical notes like the one above. In the rationale-finding application, it would have to interpret the note well enough to recognize that it explains the current medical regimen; in the patient-finding application, the system would have to recognize that this patient went on and off of several anti-estrogen drugs because of side effects. Again, understanding the answers to ToU questions would be essential, as demonstrated in Table 3.
B Example ToU-based multiple-choice questions on a RACE story
B.1 The story
Mr. Green was traveling around the country in his car. One evening he was driving along a road and looking for a small hotel when he saw an old man at the side of the road. He stopped his car and said to the old man, "I want to go to the Sun Hotel. Do you know it?"
"Yes." The old man answered. "I'll show you
the way."
He got into Mr. Green's car and they drove for about twelve miles. When they came to a small house, the old man said, "Stop here."
Mr. Green stopped and looked at the house. "But this isn't a hotel." He said to the old man.
"No," the old man answered, "This is my house. And now I'll show you the way to the Sun Hotel. Turn around and go back nine miles. Then you'll see the Sun Hotel on the left."
B.2 RACE's original questions
Answers marked correct by RACE are italicized.
Q1. Where did Mr. Green want to sleep that night?
A) In his car.
B) In his own house.
C) In a hotel.
D) In the old man's house.
Q2. Why did Mr. Green stop his car?
A) Because he found a hotel.
B) Because the lights were red.
C) Because he saw an old man.
D) Because he saw a friend.
Q3. Where did the old man promise to take Mr. Green?
A) To Mr. Green's house.
B) To the old man's house.
C) To the SunHotel. [sic]
D) To the country.
Q4. Why didn't the old man stop Mr. Green when they passed the hotel?
A) Because he wanted Mr. Green to sleep in his house.
B) Because he wanted to get home.
C) Because he didn't see the hotel.
D) Because he didn't know the hotel.
Q5. How far was it from the place where Mr. Green met the old man to the Sun Hotel?
A) About nine miles.
B) About three miles.
C) About twenty-one miles.
D) About twelve miles.
B.3 A sampling of our ToU-based questions
Correct answers are italicized. Questions are numbered with the IDs used in our dataset, which is available in this paper's supplementary data. The first number in each question ID indicates the variant group; the second number is a group-independent question index.
B.3.1 Causal chains
The questions below target different parts of causal chains explaining why the agents in the story took the actions that they did. The first five ask about why Mr. Green stopped his car (vs. continuing to drive); the next five ask about why the old man said he would show Mr. Green the way (vs. just giving him directions).
Q1-1. Why did Mr. Green stop his car the first time?
A) Because if he stopped his car, he could ask the man something.
B) Because if he stopped his car, he could make a new friend.
C) Because if he stopped his car, the old man could get in.
D) Because the directions he asked for said to stop the car.
E) Because if he stopped his car, he could drive for about twelve more miles.
F) Because he got a flat tire.
G) Because he was driving along a road.
H) Because he was traveling around the country.
I) Because he said, "I want to go to the Sun Hotel".
Q2-3. Why did Mr. Green want to ask the man something?
A) Because there was something he didn't know.
B) Because he liked to ask questions.
C) Because there was a chance to make a friend.
D) Because he didn't want to drive past the man without helping him.
E) Because if he stopped his car, he could drive for about twelve miles.
F) Because he got a flat tire.
G) Because he was driving along a road.
H) Because he said, "I want to go to the Sun Hotel".
Q3-7. Before they spoke at all, what did Mr. Green hope the man would be able to do?
A) Tell him where the hotel was.
B) Tell him where the small house was.
C) Get in his car.
D) Drive for about twelve miles.
E) Take him to his house.
F) Take him to the hotel.
G) See an old man.
Q4-9. What did Mr. Green hope the conversation with the old man would enable him to do?
A) Get where he was going
B) Travel around the country
C) See what he was seeing
D) Stop and look at a house
E) Drive with the old man
F) Come to a small house
G) Turn around and go back
7855
Q5-11. What was Mr. Green trying to do throughout the story?
A) To stay at the small hotel
B) To drive along a road
C) To pass the small hotel
D) To come to a small house
E) To see the old man
F) To stop at the side of the road
G) To speak with the old man
Q6-12. Why did the old man make his initial offer to Mr. Green?