SlowGuess/NAACL_2022_OCR huggingface dataset:
Richard Yuanzhe Pang* Alicia Parrish* Nitish Joshi* Nikita Nangia Jason Phang Angelica Chen Vishakh Padmakumar Johnny Ma Jana Thompson He He Samuel R. Bowman
{yzpang,alicia.v.parrish}@nyu.edu
To enable building and testing models on long-document comprehension, we introduce QuALITY, a multiple-choice QA dataset with context passages in English that have an average length of about 5,000 tokens, much longer than typical current models can process. Unlike in prior work with passages, our questions are written and validated by contributors who have read the entire passage, rather than relying on summaries or excerpts. In addition, only half of the questions are answerable by annotators working under tight time constraints, indicating that skimming and simple search are not enough to consistently perform well. Our baseline models perform poorly on this task (55.4%) and significantly lag behind human performance (93.5%).
Most of the best models for natural language understanding are restricted to processing only a few hundred words of text at a time, preventing them from solving tasks that require a holistic understanding of an entire passage. Moving past this limitation would open up new applications in areas like news comprehension, summarization, or applied question answering. We think that new benchmark datasets will help us do this. Most existing datasets (Rajpurkar et al., 2018; Fan et al., 2019; Lelkes et al., 2021) use shorter contexts that humans can read within a few minutes. While there are open-domain QA datasets that require longer contexts (Joshi et al., 2017; Zhu et al., 2021), finding a short excerpt that answers the questions at hand often suffices; however, long-document QA requires understanding a long context as a whole to correctly answer questions.
NarrativeQA (Kočisky et al., 2018) is the most established existing long-text benchmark for language understanding. It's a free-text-response QA
Figure 1: The crowdsourcing pipeline with an example. One writer reads the passage and writes 10 questions. Each question is validated by three or five annotators who read the full article, plus five more who have only 45 seconds per question. Writers receive feedback from both validations between writing batches. If a majority of timed annotators get the question wrong, but the untimed annotators get it right, we classify the example as HARD and give the writer a bonus.
dataset built around movie scripts and books, with an average of about 63k tokens of input per question. The authors creatively use summaries of the texts as the basis for their questions to make data collection relatively efficient. This protocol leads to short answers (avg. 4.7 tokens), and few questions require complex explanation-based reasoning:  $>60\%$  are what/who questions and  $<10\%$  are why/reason questions. Further, the sources are usually famous, such that they are analyzed and discussed widely in the training data used by large language models. Additionally, the generation-based format comes with the hurdle of determining how
Human Language Technologies, pages 5336 - 5358
Source	Dif.	Question	Answer Options	Label
Gutenberg	Hard	Why was the Volpla vocabulary limited when the narrator took a few into the valley?	(a) They had not been alive long enough to learn enough English to communicate well (b) They were encountering concepts that were unfamiliar from the lab environment (c) They are not smart enough to have a fully developed language, no matter how hard they try (d) They were confusing their own language with English, having trouble keeping the languages separate	b
Easy	What is Russell's greatest fear?	(a) Being disappointed (b) Losing his mind (c) Being lost and alone (d) Living forever	c
Slate	Hard	Which is NOT a reason why the narrator is concerned with the antichrist?	(a) Evangelical Christians are preaching that the end of the world is coming soon. (b) He is concerned that Christians will become violent toward Jews. (c) He thinks his life will be more important and influential than the average person. (d) He is conducting research for his dissertation.	d
Easy	Why does the author tell a story about his vehicle?	(a) To talk about how fast he drives (b) To make a point about what has the most impact on the economy (c) To talk about safe driving speeds (d) To make a point about how many different things impact the unemployment rate	b
Misc.	Hard	How does Sara feel about the Chevrolet ad?	(a) She thinks it's a final chance to bond with her father (b) She is sorry she did not watch the whole ad before she reacted to it (c) She is upset at the glorification of the military (d) he is frustrated that it tokenized a Mexican family	b
Easy	Why did Birmingham build over the Victorian era relics?	(a) To create space for a Maglev train (b) To erase their history (c) They were running out of room (d) To make technological progress	d
Table 1: Representative examples randomly selected from the training and dev sets in QuALITY.
to fairly assess accuracy, as metrics like BLEU, ROUGE, or BERTScore may not accurately convey the quality of generations (Wang et al., 2020; Durmus et al., 2020). To ease the burden of evaluation, we opt for a multiple-choice format to evaluate a model's long-document understanding ability.
We introduce our dataset QuALITY, Question Answering with Long Input Text, Yes! This is a multiple-choice QA dataset that uses English source articles of 2k-8k tokens. We collect this dataset using a creative crowdsourcing pipeline that ensures the examples have unambiguous answers but are still challenging. We instruct example writers to carefully read the full source article before writing questions, and to then write questions that are unambiguous and require consolidating information from multiple parts of the text. Then, to ensure our questions require readers to understand the larger context from the passage, in addition to running standard validation where annotators read the text and answer the questions, we also run speed validation (§2.3). In speed validation, annotators only have access to the text for 45 seconds,
so they can only skim or search for phrases to answer the question. If a question is unanswerable in this setting but unambiguous and answerable in the standard untimed setting, we use it as a signal for question difficulty. This crowdsourcing process is slow and expensive ($9.10/question),<sup>4</sup> but we successfully collect a challenging, high-quality, long-document multiple-choice QA dataset. QuALITY has 6,737 questions in total, of which 3,360 questions are in the difficult subset, QuALITY-HARD. Table 1 shows representative EASY and HARD examples from different types of source texts.
We test the Longformer (including the LED variant), RoBERTa, DeBERTaV3, and T5 models, using as much of the full source text as possible. In particular, for models whose context lengths are much shorter than the average article length, we test two-step systems with an extraction step that passes shorter contexts to the QA model. For text extraction, we use ROUGE-1 recall, fastText, or DPR based matching with the questions. The best model performance is achieved by DeBERTaV3-large with DPR-based extraction, with an accuracy of  $55.4\%$ . The best model's accuracy on QuALITY-HARD is  $46.7\%$ . Model accuracy is far below human accuracy on QuALITY, where human accuracy is  $93.5\%$  on the full dataset and  $89.1\%$  on
This includes the cost of question writing and validation for questions that were discarded after validation.
2The data is available, along with links to code and additional resources, at https://github.com/nyu-mll/ quality. Results on QuALITY will be collected on a dedicated leaderboard at https://nyu-mll.github.io/ quality/ and through the multitask SCROLLS benchmark (Shaham et al., 2022).
${}^{3}$  1.5k-6k words,not counting punctuation.
QuALITY-HARD.
Sources In order to create a dataset that is both broadly usable and meets the goal of containing long input texts, we use only sources that are licensed under CC-BY (or more permissive licenses) and contain articles of at least 2k tokens that are likely to allow for complex questions. We ultimately use Project Gutenberg fiction stories (mostly science fiction),[5] Slate magazine articles from the Open American National Corpus (Fillmore et al., 1998; Ide and Suderman, 2004), and other nonfiction articles taken from The Long+Short,[6] Freesouls,[7] and the book Open Access (Suber, 2012). Table 3 shows how many articles and questions come from each. Most of the Gutenberg texts are from the 1950s-1970s, while the other texts are mostly from the 1990s and after.
Texts are provided with the original HTML tags indicating paragraph breaks and basic formatting (e.g., italics), and it is in this format, with images removed, that we present the texts to our writers and annotators. In our dataset release, we also include a version of each file with this information stripped away, as current models, including our baselines, are not trained to consume these tags.
We set a maximum length for the texts at 6k words using word-level tokenization without counting HTML tags. For around  $40\%$  of the Gutenberg articles, the full text data is much longer; in these cases, we truncate the texts and manually check to make sure the truncation happens at a reasonable location (i.e., not in the middle of a paragraph).
Stages of Data Collection We collect data over several rounds to provide writers with feedback throughout the process. We iterate through the following pipeline each round: (i) we assign writers a set of passages, and they write 10 questions for each ( $\S 2.2$ ), (ii) annotators complete speed validation ( $\S 2.3.1$ ), (iii) annotators complete untimed validation ( $\S 2.3.2$ ), and (iv) we award writers bonuses and send feedback based on the annotations.
2.2 Question Writing
We hire 22 writers—most with degrees or professional experience in literature or teaching—from the freelancing platform Upwork and design a multi-part incentive structure to encourage difficult yet answerable questions. Details about hiring and writer qualifications are in Appendix A.1.1.
The Writing Task We design a feedback and incentive structure to encourage writers towards questions that are answerable, unambiguous, and difficult. Writers construct examples over multiple rounds, and they receive (i) detailed feedback based on the two validation tasks and (ii) bonuses based on how many of their questions met our criteria for HARD questions. Each writer constructs 10 questions with four answer options for a given passage, and they complete 6-30 such passages each round. Each passage is assigned to two writers, so there are 20 questions for each passage before filtering. Writers earn an average rate of $21.05/hr, after bonuses. Details about this process and the timeline are in Appendix A.1.2.
2.3 Data Validation
We use two validation tasks to evaluate if (i) the questions are difficult by testing if they are answerable under strict time constraints (speed validation) and (ii) the questions have a single correct answer (untimed validation). We recruit 45 annotators via Amazon Mechanical Turk (MTurk); details on the qualification process are in Appendix A.2.1.
2.3.1 Speed Validation
We want to ensure that the questions require understanding of the full text to answer correctly. If a person can quickly identify the answer to a question, such as through skimming or ctrl-F-style in browser search, then the question does not require broader understanding of the passage, and a model is likely to be able to identify the correct answer via extractive methods. More precisely, we aim to collect questions for which annotators, in the aggregate, are unable to select the correct answer under strict time constraints, and we construct a speed validation task to test this. Questions that pass this bar make up the HARD subset of QuALITY.
To our knowledge, this is a novel data collection method—it is inspired by adversarial collection methods (Nie et al., 2020; Bartolo et al., 2020) as a way of collecting more challenging data. As model performance on our dataset is very low, a true ad
5http://www.gutenberg.org
<sup>6</sup>http://thelongandshort.org
$^{7}$ http://freesouls.cc
With spaCy tokenization, the maximum number of tokens is larger (Figure 2).
versarial design would not be practical because model behavior would not provide enough signal to the crowdworkers, and we would risk limiting the usefulness of QuALITY as a test set for a full range of models (Bowman and Dahl, 2021). Thus, we design this task in a way that writers can reason about what would be difficult for another human as opposed to a model, and we award bonuses based on that metric.
Procedure We collect five annotations per question; within each task, questions appear one at a time to ensure the time limit is consistent for each question. The worker first reads the question and the four answer options without access to the passage. Then they press a button to reveal the passage, and they have 40 seconds to skim or search for keywords (e.g., with  $\mathrm{ctrl + F}$ ) to determine the correct answer. After the timer runs out, the passage disappears, and they have 5 more seconds to select an answer. Appendix A shows the user interface.
Each task consists of 10 questions from different passages, and the order of the answer options is randomized. Within each task, there are nine questions written by the Upwork writers and one question written by the authors as a catch question. We pay workers  $2.25 per task and award a bonus of$ 0.20 for each correct answer. On average, workers earn a bonus of $1.03 per task, and we estimate based on workers' survey responses that each task takes 11-12 minutes, for an effective rate of just over $17/hr. We use the catch questions to track annotator performance and ensure that all workers are performing well above chance on these examples, indicating that they are consistently making a faithful effort to find the answer in the text (see Appendix A.2.2 for additional details on the task, catch questions, and annotator performance).
2.3.2 Untimed Validation
To ensure all questions in QuALITY are correct and unambiguous, we conduct a validation task without a time limit, but with strong incentives towards accuracy. We collect three annotations for each example in the training set, and five annotations for each example in the dev and test sets.
Procedure Each task consists of one passage with all 20 questions created by the writers. Each of the 20 reading comprehension questions has three evaluation questions immediately below it. We instruct workers to first read the passage carefully and then answer all the questions. Each task pays $6.50,
Question	Answer Options
Q1. Is the question answerable and unambiguous?	○ Yes, there is a single answer choice that is the most correct.
○ No, two or more answer choices are equally correct.
○ No, it is unclear what the question is asking, or the question or answer choices are unrelated to the passage.
Q2. How much of the passage/text is needed as context to answer this question correctly?	○ Only a sentence or two of context
○ At least a long paragraph or two of context
○ At least a third of the passage for context
○ Most or all of the passage for context
Q3. Which of the options that you did not select was the best “distractor” item?	○ Option 1
○ Option 2
○ Option 3
○ Option 4
Table 2: Evaluation questions asked after each reading comprehension question during untimed validation.
with a  $0.50 bonus for each question in which both the reading comprehension question and evaluation question 1 (see below) agree with the majority vote label.\(^9$  We estimate based on survey responses that workers spend about 50–60 minutes on this task; the average bonus rate is \)8.13 per task, for an average rate of $15.96/hr.
Evaluation Questions We ask the three evaluation questions in Table 2 immediately following each reading comprehension question to assess question quality. Q1 is used to determine inclusion into the final dataset, as we exclude any questions for which the majority of annotators marked that the question was either ambiguous or unanswerable. Q2 and Q3 are used for feedback to the writers.
We find that responses to the evaluation questions slightly differ between the HARD and EASY subsets. For Q1, individual raters are less likely to rate a HARD question as answerable and unambiguous (92.8%) compared to an EASY question (95.1%). For Q2, in the HARD subset, 26.1% of the time, the question is rated as needing at least a third of the context or more (the 3rd and 4th options), compared to 21.7% of the time in the EASY subset. In the HARD subset, 81.3% of the questions are rated as needing at least a long paragraph or two of context, compared to 73.9% in the EASY subset.
9Bonus were \\(0.40 in round 1 and based only on reading comprehension questions. We updated both following our evaluation of the results and worker feedback.
Figure 2: Article length, question length, and option length in QuALITY. The average length of an article, question, and option is 5,159 tokens, 12.5 tokens, and 11.2 tokens, respectively. The maximum length of an article, question, and option is 7,759 tokens, 103 tokens, and 75 tokens, respectively. The histograms are truncated to only keep the visible mass.
Annotator Performance We track annotator performance throughout data collection and remove any workers whose accuracy falls below  $75\%$  in any given round. Annotator agreement on the reading comprehension questions for each passage is high, with a median Krippendorff's alpha of 0.71. Agreement on Q1 is also high, with  $92.6\%$  individual agreement with the majority vote, with the two 'No' options collapsed for analysis (alpha values are less valuable on such a skewed question). As Q2 and Q3 are more subjective, responses are noisy, with median alpha values of 0.12 and 0.21, respectively. Additional details and our protocol for reannotating data are in Appendix A.2.3.
3 Dataset Information and Analysis
After aggregating the labels assigned via untimed validations with the original writer's label, we calculate the gold label via majority vote of annotators.10 We only keep questions for which (i) a majority vote label (strictly larger than  $50\%$  ) can be assigned and (ii) the majority of annotators rate the questions as answerable and unambiguous. 6,737 out of 7,620  $(88.4\%)$  questions meet these inclusion criteria. The HARD subset corresponds to questions that the majority of the annotators answer incorrectly in the speed validation setting, and this constitutes  $49.9\%$  of the final dataset.
3.1 Human Accuracy
We estimate human accuracy on QuALITY on a random sample of 20 passages (367 questions). Each question is annotated by 3 new annotators who had not previously annotated that passage, and
whose labels do not contribute to the assignment of the gold label. We calculate the majority vote of the annotators, which yields an accuracy of  $93.5\%$  relative to the gold label. This breaks down to  $89.1\%$  on the HARD subset and  $97.0\%$  on the EASY subset. Annotators marked  $98.5\%$  of questions as answerable and unambiguous.
3.2 Size and Splits
We split the data into train/dev/test sets<sup>11</sup> such that there is minimal overlap in question writers among train/dev/test sets. This ensures that a model will not be rewarded for overfitting to any idiosyncrasies of a single writer's style. Table 3 shows the number of articles in QuALITY and HARD questions for each of the split. Gutenberg sources result in the highest proportion of HARD questions, and misc. sources result in the lowest proportion.
3.3 Length
Figure 2 shows the article lengths. The two peaks in the histogram correspond to articles from Slate and Gutenberg. The average context length is 5,159 tokens, much longer than other existing challenging QA datasets—CosmosQA (Huang et al., 2019) and RACE (Lai et al., 2017) contain an average context length of 70 and 322 tokens, respectively. We plot question length and option length in Figure 2 as well. The average question length is 12.5 tokens, and the average option length is 11.2 tokens.
3.4 Lexical Overlap
Prior work has shown that a lot of questions in existing datasets such as SQuAD can be answered by exploiting lexical overlap of the question with the article (Weissenborn et al., 2017). To under
<sup>11</sup>The test labels are not publicly released. The test set performance can be obtained by submitting to a leaderboard discussed in footnote 2.
This gold label calculation follows MNLI (Williams et al., 2018) but is more conservative as the writer's label is never a tie-breaking vote. The gold label and writer's label, both provided in the dataset, differ for  $\sim 4\%$  (274/7620) of questions.
Split	Gutenberg	Slate	Misc	All
Art.	All Qs	HARD Qs	% HARD	Art.	All Qs	HARD Qs	% HARD	Art.	All Qs	HARD Qs	% HARD	Art.	All Qs	HARD Qs	% HARD
Train	118	2000	1056	52.8	22	355	142	40.0	10	168	53	31.5	150	2523	1251	49.5
Dev	86	1552	873	56.2	19	351	149	42.5	10	183	43	23.5	115	2086	1065	51.1
Test	81	1486	828	55.7	25	450	170	37.8	10	192	46	24.0	116	2128	1044	49.1
All	285	5038	2757	54.7	66	1156	461	40.0	30	543	142	26.2	381	6737	3360	49.9
stand how effective this heuristic is in QuALITY, we compute the lexical overlap between the options and the article in QuALITY. The lexical overlap is computed as the fraction of the tokens in the option which are present in the article. Figure 7 in the Appendix plots the distribution of lexical overlap for the correct options and the incorrect options—since each question has three incorrect options, we use the maximum lexical overlap among the three. Simply predicting the option with the highest lexical overlap achieves only  $26.6\%$  accuracy, so correct options do not have a higher lexical overlap than the incorrect options, making it difficult for models to rely on this heuristic.
3.5 Question Types
Table 3: Data splits within QuALITY. Items that did not pass untimed validation are excluded from this table. 'Art.' shows the number of articles. 'HARD Qs' is the number of questions in QuALITY-HARD.
Question Type	# EASY	# HARD	% total
what	1361	1471	42.2
why	832	825	24.6
how	385	416	11.9
which	253	244	7.4
who	151	132	4.2
how + meas.	51	75	1.9
yes/no	53	55	1.6
where	43	42	1.3
when	35	34	1.0
other	155	124	4.1
Table 4: Different question types in QuALITY, split by HARD and EASY subsets. 'How + meas.' collapses multiple questions with 'how' plus some measurement, such as 'how long' or 'how many.' Examples of each question type are in Appendix Table 7.
We analyze the proportion of question types by automatically categorizing each question based on the first question word it contains.12 Table 4 shows that QuALITY contains many questions that require complex responses about "how" and "why" an event happened in a greater proportion of cases
than similar datasets such as NarrativeQA. However, we do not observe that our measure of question difficulty varies by question type.
3.6 Reasoning Strategies
As a qualitative analysis, we manually annotate the reasoning strategy needed in each question and present the results in Table 5. We take a random subset of 500 questions from the full dataset and manually annotate them. Each question is annotated by two of the authors; any disagreements in categorization are resolved via discussion. As we do not read the full passages, it is not always possible to determine the reasoning strategy, but we consider both the question and answer options in categorizing each item.