SlowGuess/NAACL_2022_OCR huggingface dataset:
Adina Williams, Nikita Nangia, and Samuel Bowman. 2018. A broad-coverage challenge corpus for sentence understanding through inference. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 1112-1122, New Orleans, Louisiana. Association for Computational Linguistics.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. 2020. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pages 38-45, Online. Association for Computational Linguistics.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369-2380, Brussels, Belgium. Association for Computational Linguistics.
Fengbin Zhu, Wenqiang Lei, Chao Wang, Jianming Zheng, Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.00774.
A Details on Writing, Speed Validation, and Untimed Validation
A.1 Writing
A.1.1 Writer Recruitment
We need writers who have good reading comprehension skills and/or writers who have experience constructing reading comprehension questions (e.g., literature teachers who have experience writing tests for their students). We hire two groups of writers on the freelancing platform Upwork (the second group two months after the first group, after we decided to increase the final size of the dataset).
For each group, we advertise a task titled Writing college-level reading comprehension questions. The job post is visible to all U.S. Upwork freelancers, and we specifically send out job invitations to promising freelancers who are writers or teachers, or who have college-level degrees in English, Literature, Creative Writing, Philosophy, Education, or similar fields. In the original job ad, we explain who we are and tell them how we will use their data. Specifically, we include the following phrase: "The data we collect through this project will be made publicly available for AI research. We will not distribute any identifying information about you, the writers."16
For the first group, we received 104 applications in the span of two weeks. Of those, we selected 26 people to complete a qualification task as a paid interview. For the second group, we received 65 applications and interviewed 11. The interview task consists of (i) reading through detailed instructions, (ii) reading through a tutorial example passage with 10 example questions, each with an explanation of what made it a good or a bad question, and (iii) writing 10 reading comprehension questions for a new passage; regardless of whether we eventually hire them, we pay workers \(30 and estimate that this task takes 2 hours to complete. Three authors (of this paper) then assess each writer's work using the following criteria: (i) whether the writer-provided correct answers are actually the correct answers, (ii) whether the questions are answerable and unambiguous, and (iii) whether more than just a few sentences of context are needed to correctly answer the questions. Based on these criteria, we select the top performing 15 writers to continue on to the main task in the first group, and 7 in the second
Of the 22 writers we hire after the interview, 15 have a college degree in English, literature, philosophy, creative writing, or education; 4 of these 15 writers are Ph.D. students or graduates. 11 of the 22 writers have taught high-school or college-level English or literature classes; among these 11 writers, 7 have  $5+$  years of teaching experience. 2 of the 22 writers mention that they write novels.
A.1.2 Writing Task
Each writer constructs 10 questions for a given passage, completing 6-30 passages in a given round and continuing for three complete rounds.[17] Each round is followed by feedback (detailed below) to allow writers to improve for the next round. Writers earn  $12.50 per passage and receive a bonus of$ 1.20 for each question that meets the following criteria: (i) the majority of facilitators agree with the writer's original label, (ii) the majority of facilitators rated the question as answerable and unambiguous, and (iii) the majority of facilitators answered the question incorrectly in the speed validation task (\$2.3.1). On average, writers receive bonuses on 4.2 questions per passage, resulting in average earnings of $17.54 per passage. Based on writer self-reports, the median time to complete one writing task is about 50 minutes, for an effective rate of $21.05/hr. Upwork charges fees on the workers' end. We account for this by adding an extra 20% to their pay, bringing our final cost to $2.10 per question.
Besides using the monetary bonus as an incentive for writing answerable, unambiguous, and difficult questions, we also instruct writers that their questions should use the entire context. Throughout the course of data collection, we provide writers with detailed feedback based on validations (detailed in §2.3.2) and this feedback includes information about how much of the passage needed to be read in order to answer the question. We monitor the proportion of questions that require more than a few paragraphs of context to answer correctly; if this rate significantly lags behind other writers, we inform the writers that their work is falling below expectations and ask them to be more careful with this issue in the next round. We also
17On average, group 1 writers complete 6, 14, 30 articles for the three batches, respectively; group 2 writers complete 6, 14, 20 articles for the three batches, respectively. The writing time limits for batches 1, 2, 3 are around 1, 2, 3 weeks, respectively. Validation for any batch takes less than a week.
16We later obtained consent from writers who chose to allow us to name them in the Acknowledgments section.
Your task is to write challenging reading-comprehension questions for the given passage. Please spend 15-20 minutes reading the provided passage, and write 10 challenging multiple-choice questions about the passage. A good question will require the reader to read the whole passage carefully and consolidate information across multiple parts of the passage. Please make sure to read the full instructions before working on the task (click on the Instructions button above). Estimated time: 40-50 minutes. Thank you!
Passage 1/1
Question Writing
thinning on top, his small blue eyes glittering preternaturally from the lenses fitted over his eyeballs.
Michael rubbed his fingers ruefully over the blond stubble on his scalp and wished he had waited until his tonsure were fully grown before he had ventured out into the world. But he had been so impatient to leave the Lodge, so impatient to exchange the flowing robes of the Brotherhood for the close-fitting breeches and tunic of the outer world that had seemed so glamorous and now proved so itchy.
"Yes," he replied courteously, for he knew the first rule of universal behavior, "I have been a Brother."
"Now why would a good-looking young fellow like you want to join a Brotherhood?" his shelf companion wanted to know. "Trouble over a female?"
Michael shook his head, smiling. "No, I have been a member of the Angeleno Brotherhood since I was an infant. My father brought me when he entered."
The other man clucked sympathetically. "No doubt he was grieved over the death of your mother."
Michael closed his eyes to shut out the sight of a baby protruding its fat face at him three-dimensionally, but he could not shut out its lisping voice: "Does your child refuse its food, grow wizened like a monkey? It will grow plump with oh-so-good Meaty Mush from Nunki."
"No, sir," Michael replied. "Father said that was one of the few blessings that brightened an otherwise benighted life."
Horror contorted his fellow traveller's plump features. "Be careful, young man!" he warned. "Lucky for you that you are talking to someone as broad-minded as I, but others aren't. You might be reported for violating a tabu. An Earth tabu, moreover."
"An Earth tabu?"
input goes here
1. input goes here
2. input goes here
3. input goes here
4. input goes here
Figure 3: The writing UI.
encourage writers to write difficult distractors, and the feedback we provide also contains what annotators think is the most difficult distractor for each question (§2.3.2).
If writers have fewer than  $40\%$  of questions meet the above three bonus criteria and fewer than  $75\%$  of questions meet criteria (i) and (ii), we exclude them from future writing rounds: One writer was excluded after batch 1, and one writer was excluded after batch 2, for this reason. We also exclude two writers who missed deadlines by significant margins. Two other writers voluntarily left the project before finishing all three batches.
A.1.3 The Writing UI
Figure 3 shows our writing UI. A writer creates 10 multiple-choice questions with four answer options each on each page. Before the interview task and each batch of data collection, we explain our bonus structure to the writers. In order to encourage writers towards writing the types of questions that require understanding of the general context from the passage, we provide the following examples of themes that questions can target in order to spur writers' creativity and provide suggestions if they have trouble coming up with difficult questions; however, they do not have to follow our suggestions.
- Characters' feelings and motivations
- Causes and consequences of described events
- Definitions, properties, and processes explained in a passage
- The summary and lesson of a passage
- What would have happened had a character made a different choice
We also allow writers to skip a given passage in case they find that they would be unable to write high-quality questions for that passage. Specifically, we tell writers the following.
If a passage is too difficult to write questions for, you can skip the article by choosing another URL to work on. We recommend that you do this if: (1) The text is hard to read due to major formatting issues. (2) The text is very technical or relies on cultural knowledge that you're unfamiliar with. (3) You think the passage is much too boring. We ideally want you to write questions for passages you find interesting!
A.2 Validation
A.2.1 Annotator Recruitment
We recruit annotators via Amazon Mechanical Turk (MTurk). We use a qualification task to identify annotators with good reading comprehension skills.
I finished reading the instruction, the question, and the choices. Show me the article (please click)!
45 SECONDS LEFT
- the passage will disappear when there are 5 seconds left
- if time runs out, the page will auto-submit
What advantage did the rebels have over the Venus Consolidated police?
1. Stashes of weaponry in caves
2. Detailed knowledge of the area
3. Fast atmospheric ships
4. Secret supporters on the inside
Go to the next passage
Figure 4: The speed validation UI before clicking the top-right button.
Passage 2/10
By Vic Phillips and Scott Roberts
Sheer efficiency and good management can
make a monopoly grow into being. And once
it grows, someone with a tyrant mind is
going to try to use it as a weapon if he can—
[Transcriber's Note: This etext was produced from
Astounding Science-Fiction April 1942.
Extensive research did not uncover any evidence that
the U.S. copyright on this publication was renewed.]
"That all, chief? Gonna quit now?"
Brian Hanson looked disgustedly at Pete Brent, his lanky assistant. That was the first sign of animation he had displayed all day.
"I am, but you're not," Hanson told him grimly. "Get your notes straightened up. Run those centrifuge tests and set up the still so we can get at that vitamin count early in the morning."
"Tomorrow morning? Aw, for gosh sakes, chief, why don't you take a day off sometime, or better yet, a
32 SECONDS LEFT
the passage will disappear when there are 5 seconds left
- if time runs out, the page will auto-submit
Figure 5: The speed validation UI after clicking the top-right button.
This task is open to all workers with more than 1000 tasks (HITs) accepted and a HIT accept rate of at least  $98\%$ . We pay $5 for completing the qualification task, plus a $5 bonus for passing it. The task consists of a ~3000-word passage with 10 multiple choice questions written or reviewed by the authors, each with a series of evaluation questions asking about the quality of that question. Of the 10 questions, 2 are intentionally ambiguous<sup>18</sup> in order to test if workers can accurately identify poorer quality questions.
In order to pass the qualification, workers need to (i) get at least 6/7 or 7/8 of the unambiguous questions correct, (ii) correctly identify at least one of the two ambiguous questions as ambiguous/unanswerable, and (iii) correctly identify at least half of the unambiguous questions as unam-
biguous. A total of 148 crowdworkers completed the task, and 45 of them passed (30.4%). All workers who pass the qualification are invited to complete tasks as part of both the speed validation and the untimed validation. We make it clear in this task as well as the main speed validation and untimed validation tasks who we are and which research group we are affiliated with. In order to help workers understand that we plan to use their data for research purposes related to language technologies, we also include the following in the FAQ section of each hit: "With your help, we think we'll be able to build some pretty exciting technologies to help computers better understand human language."
A.2.2 Speed Validation
Catch Questions We expect accuracy in the speeded task to be fairly low, so we construct catch questions to ensure that workers are not randomly guessing without attempting to find the correct an-
18 We later found that one question was unintentionally ambiguous; we do not use this question in assessing whether workers pass the qualification.
swer. These trials are written by the authors and are designed to be answerable with only 45 seconds of access to the passage. For example, a catch question may ask who spoke a quote, like "Who said 'You're a wizard Harry'?" where a single ctrl+F search of the quote gives the annotator the answer. Another catch question may have four options, three of which are clearly improbable. We do not validate the catch questions for correctness in the untimed validation, and so we do not include them in the final dataset, but we release them as a supplemental file for reproducibility.
Payment For most of the tasks, we pay workers  $2.25 per HIT and award a bonus of$ 0.20 for each correct answer. However, during the first of six rounds, we paid $2.00 per HIT with a $0.18 bonus for each correct answer. After asking workers for feedback about the task via a survey, we decided to increase the rate of pay because workers reported spending slightly longer on the task than we originally estimated.
Task Procedure Each MTurk task consists of 10 speed validation questions from different randomly chosen articles. In each task, once the annotator clicks into the page, they have unlimited time to read the question and the answer options, but the article is not shown (Figure 4). Then, the annotator clicks the button that says "I finished reading the instruction, the question, and the choices. Show me the article (please click)!" As soon as the annotator clicks the button, the countdown clock of 45 seconds starts, and the article appears (Figure 5). The annotator can make the choice and submit at any time.
When there are only 5 seconds left, the article hides itself. The annotator has 5 seconds to make the choice. If the time expires, the page autobmits, and we record that the annotator did not make a choice, which we score as incorrect. The exact instructions that the annotator sees are as follows:
In this task, you will see a long text passage and a multiple choice question that can be answered from that text. Read the question and select the best answer option. You only have 45 seconds to choose an answer, so this is not enough time to read the whole passage. We encourage you to skim and use keyword-based searches (e.g., using ctrl+F) during
this time. Even if you are unsure of the answer, you should make an educated guess.
After 45 seconds, your answer will be locked in and submitted. If you have not provided an answer at the end of 45 seconds, you will not be able to answer this question and will be automatically moved to the next question. We will not reject your work for a couple of blank answers, but excessive failures to answer will result in a loss of the qualification to complete these HITs.
You will not be penalized for wrong answers. We will give you a bonus of $0.20 for each correct answer. Thus, it is in your best interest to attempt each question, even if it is just a guess. A few of the questions will be answerable in just 45 seconds, and we do expect you to get these right reasonably often.
Annotator Performance Individual annotators consistently scored well above the chance rate of  $25\%$  on the catch questions. In all cases where an annotator's accuracy fell below  $50\%$  in a round, they were removed from future rounds. Two annotators fell below this threshold, though in that round they had also performed below threshold in the untimed validation. No annotators needed to be removed solely based on performance in the speed validation task. Average overall accuracy on the catch questions was  $83.8\%$ , indicating that most workers were able to develop a strategy for finding a correct answer when it could be found.
Accuracy on the questions written by Upwork writers was  $48.2\%$  overall, but annotators got better at this task over time, likely by developing new strategies to search for answers. Average accuracy was  $39.5\%$  in the first round, rising to  $58.4\%$  in the final round of data collection. When the majority of annotators (at least 3/5) are able to answer questions correctly in this setting, we exclude that question from the HARD subset.
A.2.3 Untimed Validation
Figure 6 shows the UI for untimed validation. As two writers each write 10 questions for the same article, there are 20 unique questions per article. Each validation UI page contains all 20 sets of questions, and each set of questions contains the
THE LOST TRIBES OF VENUS
By ERIK FENNEL
On mist-shrouded Venus, where hostile
swamp meets hostile sea ... there did
Barry Barr-Earthman transmuted-swap
his Terran heritage for the deep dark
waters of Tana; for the strangely
beautiful Xintel of the blue-brown skin.
Planet Stories May 1954.
Evil luck brought the meteorite to those particular space-time coordinates as Number Four rode the downhill spiral toward Venus. The football-sized chunk of nickel-iron and rock overtook the ship at a relative speed of only a few hundred miles per hour and passed close enough to come within the tremendous pseudo-gravitic fields of the idling drivers.
It swerved into a paraboloid course, following the flux lines, and was dragged directly against one of the three projecting nozzles. Energy of motion was converted to heat and a few meteoric fragments fused themselves to the nonmetallic tube casing.
In the jet room the posttronic line accelerator for that particular driver fouled under the intolerable overload, and the backsurge sent searing heat and deadly radiation blasting through the compartment before the main circuit breakers could clack open.
The bellow of the alarm horn brought Barry Barr fully awake, shattering a delightfully intimate dream of the dark-haired girl he hoped to see again soon in Venus Colony. As he unbuckled his bunk straps and started aft at a floating, bounding run his weightlessness told him instantly that Number Four was in free fall with dead drivers.
Red warning lights gleamed wickedly above the safety-locked jet room door, and Nick Podtiaguine, the air machines specialist, was manipulating the emergency controls with Captain Reno at his elbow. One by one the crew crowded into the corridor and watched in tense silence.
The automatic lock clicked off as the jet room returned to habitable conditions, and at Captain Reno's gesture
What happens if this problem is not repaired.
1. Nothing. Everything will opprate as usual.
2. It will leave the ship vulnerable to a hostile takeover.
3. The foreign material will cause the ship to become extremely difficult to maintain safely.
4. The ship will loose oxygen, and the crew will die
Is the question answerable and unambiguous?
Yes, there is a single answer choice that is the most correct.
No, two or more answer choices are equally correct.
No, it is unclear what the question is asking, or the question or answer choices are unrelated to the passage.
How much of the passage/text is needed as context to answer this question correctly?
Only a sentence or two of context
At least a long paragraph or two of context
At least a third of the passage for context
Most or all of the passage for context
Which of the options that you did not select was the best "distractor" item (i.e., an answer choice that you might be tempted to select if you hadn't read the text very closely)?
Option 1
Option 2
Option 3
Option 4
Why does Dorothy feel apprehensive of Hind?
reading comprehension question and the three additional evaluation prompts. Therefore, in total, there are 80 prompts on each page. The annotator has to complete all 80 before they can submit the page and complete the task. The exact instructions that the annotators read for this task are as follows:
In this task, you will answer multiple choice questions corresponding to a long article. Each passage comes with 20 sets of questions. Each set contains a comprehension question and three evaluation questions. Please read each passage carefully before answering the questions for that passage. Estimated time: 45-55 minutes.
If it is impossible to say which of the other answer options is correct, then select the answer option that is closest to correct.
You will receive a bonus of $0.50 for each reading comprehension question that you correctly answer. We consider the answer to be correct if your response of the reading comprehension question and the first evaluation question both agree with the most common answer from other workers and the original writer. If there's no agreement on an answer, we count it as correct for everyone.
This means that it's possible to receive a total bonus of $10.00 on this HIT.
We expect you to answer most of the questions correctly; however, we understand that some questions may be difficult, ambiguous, or mal-formed. If you answer a large number of questions incorrectly, we will disqualify you from future work on this task.
For the second and third evaluation questions, we will check if your choices agree with the majority of other workers who work on the same task, but your bonus is not dependent on these answers. The results will also be used to determine whether you retain the qualification for future batches of this task, but to a lower threshold because these questions tend to be more subjective.
Annotator Performance Individual annotator agreement with the gold label is  $91.2\%$  for all data collected in the main data collection (not including the responses collected to measure human accuracy described in  $\S 3.1$ ). Throughout the course of the study, workers need to maintain at least  $75\%$  accuracy each round to keep the qualification and continue to the next round. In a few cases, we identify passages that are themselves ambiguous or