SlowGuess/ACL_2020_OCR huggingface dataset:
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ Questions for Machine Comprehension of Text. In Proc. of EMNLP, pages 2383-2392.
Saku Sugawara, Kentaro Inui, Satoshi Sekine, and Akiko Aizawa. 2018. What Makes Reading Comprehension Questions Easier? In Proc. of EMNLP, pages 4208-4219.
James Thorne and Andreas Vlachos. 2018. Automated Fact Checking: Task formulations, methods and future directions. In Proc. of COLING, pages 3346-3359.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing Datasets for Multi-hop Reading Comprehension Across Documents. Trans. of ACL, 6:287-302.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering. In Proc. of EMNLP, pages 2369-2380.
A Crowdsourcing interface
Fig. 4 shows the instruction of our annotation task to crowdworkers. Fig. 5 shows the interface of the question-answering task.
6746
Artificial Intelligence (AI) has become fast and accurate. However, it still falls short when it comes to human-level reasoning. In this task, you are helping us educate an AI how to reason. Specifically, we will ask you to verify whether you can answer a question from articles and describe the steps you used to find the answer.
View instructions (for first time workers)
1. Read a given question and related articles.
2. Answer to the question solely based on the information from each article.
3. Describe your reasoning on how to reach the answer. Each reasoning step needs to be in a simple subject-verb-object form (see example below). Your reasoning must include sentences containing your answer.
You can consult each section for further instructions.
When is Barack Obama's wife's birthday?
- Article 1: Barack Obama is a former US president. His wife is Michelle Obama.
- Article 2: Michelle Obama is a former US first lady (born January 17, 1964).
January 17, 1964
See both a good response and bad response below:
Good/Bad
- Article 1, second sentence => [Barack Obama] 's wife is [Michelle Obama].
Article 2, first sentence => [Michelle Obama] is born on [January 17, 1964].
Good Note that the second reasoning step includes [January 17, 1964], the answer.
Article 2, first sentence  $\Rightarrow$  [Michelle Obama] is born on [January 17, 1964].
BAD The reasoning does not state the relation between [Michelle Obama] and [Barack Obama]. It is obvious for us, but it is not for Als. Do not forget write down such things in this task!
- Article 1, second sentence => [Barack Obama] 's wife is [Michelle Obama].
BAD The second reasoning step includes [on January 17, 1964], which is not an object.
Article 2, first sentence => [Michelle Obama] is born [on January 17, 1964].
Figure 4: Task instruction.
6747
1. Read
Read the following question and related articles carefully. Important sentences are highlighted.
Article 2: Annette Bening
Annette Carol Bening (born May 29, 1958) is an American actress.
She began her career on stage with the Colorado Shakespeare Festival company in 1980, and played Lady Macbeth in 1984 at the American Conservatory Theatre.
She was nominated for the 1987 Tony Award for Best Featured Actress in a Play for her Broadway debut in "Coastal Disturbances".
She is a four-time Academy Award nominee; for "The Grifters" (1990), "American Beauty" (1999), "Being Julia" (2004) and "The Kids Are All Right" (2010).
In 2006, she received a star on the Hollywood Walk of Fame.
2. Answer
Based upon the related articles, answer to the question.
If you found multiple possible answers, please choose one of them and write your reasoning for it below.
- If the question contains a typo(s): (i) if you can pretend that the typos are not there, answer the question. (ii) if you cannot (e.g. the question does not make sense), select "NONE OF THE ABOVE".
Choose your answer:
Shane Stanley
Figure 5: Task interface for the first question answering phase. The reasoning annotation interface shown in Fig. 3 follows after this interface.
6748
B Example annotations
Table 4 shows examples of crowdsourced annotations.
6749
Question	Were Scott Derrickson and Ed Wood of the same nationality?
Supporting Art. 1	[1] Scott Derrickson (born July 16, 1966) is an American director, screenwriter and producer.[2] He lives in Los Angeles, California.[3] He is best known for directing horror films such as "Sinister", "The Exorcism of Emily Rose", and "Deliver Us From Evil", as well as the 2016 Marvel Cinematic Universe installment, "Doctor Strange."
Supporting Art. 2	[1] Edward Davis Wood Jr. (October 10, 1924 December 10, 1978) was an American filmmaker, actor, writer, producer, and director.
Derivation step 1	[1, 1] [Scott Derrickson] [is] [an American director]
Derivation step 2	[1, 1] [Ed Wood] [was] [an American filmmaker]
Question	The director of the romantic comedy "Big Stone Gap" is based in what New York city?
Supporting Art. 1	[1] Big Stone Gap is a 2014 American drama romantic comedy film written and directed by Adriana Trigiani and produced by Donna Gigliotti for Altar Identity Studios, a subsidiary of Media Society.[2] Based on Trigiani's 2000 best-selling novel of the same name, the story is set in the actual Virginia town of Big Stone Gap circa 1970s.[3] The film had its world premiere at the Virginia Film Festival on November 6, 2014.
Supporting Art. 2	[1] Adriana Trigiani is an Italian American best-selling author of sixteen books, television writer, film director, and entrepreneur based in Greenwich Village, New York City.[2] Trigiani has published a novel a year since 2000.
Derivation step 1	[1, 1] [Big Stone Gap] [is directed by] [Adriana Trigiani]
Derivation step 2	[2, 1] [Adriana Trigiani] [is from] [Greenwich Village, New York City.]
Question	The arena where the Lewiston Maineiacs played their home games can seat how many people?
Supporting Art. 1	[1] The Lewiston Maineiacs were a junior ice hockey team of the Quebec Major Junior Hockey League based in Lewiston, Maine.[2] The team played its home games at the Androscoggin Bank Colise.[3] They were the second QMJHL team in the United States, and the only one to play a full season.[4] They won the President's Cup in 2007.
Supporting Art. 2	[1] The Androscoggin Bank Colise (formerly Central Maine Civic Center and Lewiston Colisee) is a 4,000 capacity (3,677 seated) multi-purpose arena, in Lewiston, Maine, that opened in 1958.[2] In 1965 it was the location of the World Heavyweight Title fight during which one of the most famous sports photographs of the century was taken of Muhammed Ali standing over Sonny Liston.
Derivation step 1	[1, 2] [Lewiston Maineiacs] [play in the] [Androscoggin Bank Colise]
Derivation step 2	[2, 1] [Androscoggin Bank Colise] [is an] [arena]
Derivation step 3	[2, 1] [Androscoggin Bank Colise] [has a seating capacity of] [3,677 seated]
Table 4: Example of annotation results of derivations. Each derivation step is in the following format: [article ID, SF] [Head entity] [Relation] [Tail entity].
6750