SlowGuess/NAACL_2022_OCR huggingface dataset:
We find that many of the questions rely on (i) reasoning about the best description, (ii) determining the correct explanation for why something happened, or (iii) the reader making an interpretation or using symbolism. All three of these reasoning types are likely to rely on broader context from the passage, compared to questions about who did something or where something happened. For example, the question How do you think Meredith feels about the rest of the crew? requires a description of the character's feelings (description), and it also requires the reader to interpret the character's feelings (symbolism/interpretation) and reason about the relation between different characters (relation). We also find that, despite questions using "what" being the most frequent in the question-types analysis, very few of the questions in QuAL-ITY depend on reasoning about objects or entities. Rather, most of these "what" questions ask for the description of a person or situation, or they ask for an interpretation from the reader. Further details about this analysis, the categories used, and examples of each reasoning type are in Appendix B.2.
12 In cases where the question starts with an auxiliary verb, or where there is no question word but an auxiliary verb appears after a comma, we categorize the question as "yes/no."
Reasoning Type	# HARD/251	# EASY/249	% oftotal
Description	89	77	33.2
Why/reason	73	83	31.2
Symbolism/interpretation	76	63	27.8
How/method	25	19	8.8
Event	17	18	7.0
Person	11	17	5.6
Not/except	13	6	3.8
Relation	12	7	3.8
Entity	7	9	3.2
Finish the Phrase	3	12	3.0
Location	5	7	2.4
Numeric	5	6	2.2
Object	5	4	1.8
What if	3	4	1.4
Duration	1	2	0.6
Table 5: Qualitative assessment on a random 500 example subset of QuALITY, split by difficulty, and categorizing the different kinds of things that need to be reasoned about. Questions can require multiple reasoning types, so values do not add up to  $100\%$ .
4 Baseline Experiments
Long-Context Models We experiment with the Longformer model (Beltagy et al., 2020), which uses a combination of sliding-window local attention and global attention to encode long inputs. The Longformer encoder models support up to 4,096 tokens. We test Longformer because it is likely to fit most or all of the context needed to answer the questions for the majority of examples in QuALITY.[13] We also experiment with Longformer Encoder-Decoder (LED) which supports up to 16,384 encoder input tokens.[14]
Extractive Models As an alternative to feeding the whole input context into a transformer model or truncating, we also test retrieval methods to score and extract relevant sentences from the passage and feed only the selected sentences as inputs to a given model. We can thus use a wider range of higher-performing short-sequence transformer models, at the cost of missing some input context.
Using the question as a retrieval query, we score each sentence in the passage relative to the query. We then select sentences in order of descending relevance until we reach 300 words.[15] We then sort the selected sentences based on the original passage
order and use the concatenation as the 'passage' for that example.
We consider three scoring methods. First, we use ROUGE-1 recall relative to the query. Second, we use cosine similarity based on bag-of-words of fastText (Bojanowski et al., 2017) embeddings. Third, we use DPR (Karpukhin et al., 2020), a model trained for open-domain retrieval for QA. Because DPR tackles span-based question-answering, the reader model is unsuitable for our multiple-choice dataset. However, we can use the retriever model for extraction, using the separate question- and context-encoders to encode our question and context sentences to vector representations. We then score similarity based on the negative Euclidean  $(L^2)$  distance.
After extraction, we apply standard models for multiple-choice question-answering: RoBERTa (Liu et al., 2019) and DeBERTaV3 (He et al., 2021) encoder models, and the T5 (Raffel et al., 2020) encoder-decoder model. To establish an upper bound of how well extractive models can do, we also introduce an oracle baseline in which we apply the same extraction strategy described above, but we use the correct answer as the extraction query.
Question-Only Baselines To test for dataset artifacts, we consider a baseline where we only give the models the questions and answer options, leaving out the passage.
Supplementary Training Data To supplement the training examples in QuALITY, we incorporate additional training examples from the RACE task dataset (Lai et al., 2017). Like QuALITY, RACE is a passage-based, four-way multiple-choice question-answering dataset. Although the passages are much shorter (321.9 words on average), the training set is large ( $\sim$ 88k questions), so we can expect reasonable knowledge transfer from RACE to QuALITY. We use the full RACE dataset, including both middle-school and high-school questions, for our intermediate training.
We consider three fine-tuning formats: (1) fine-tuning on QuALITY data, (2) fine-tuning on RACE and zero-shot evaluating on QuALITY, and (3) applying intermediate training (Phang et al., 2018; Pruksachatkun et al., 2020) by first fine-tuning on RACE and then fine-tuning on QuALITY.
Table 6 shows model performance on the test set. The results on the development set and additional
13The question and answer options are visible to models, but the article is sometimes truncated. 14Hyperparameter details for models in this section can be found in Appendix D. 15Punctuation is not counted toward this limit.
Training Data	Model	Full	Extr: R-1	Extr: fastText	Extr: DPR	Question-Only
QuALITY	Longformer-base	30.7 / 29.3	-	-	-	-
LED-base	25.1 / 24.6	-	-	-	-
LED-large	24.2 / 24.5	-	-	-	-
RoBERTa-base	-	33.4 / 30.7	39.7 / 36.1	39.9 / 34.0	36.6 / 34.8
RoBERTa-large	-	29.4 / 28.0	42.7 / 35.7	26.2 / 25.1	26.4 / 25.7
DeBERTaV3-base	-	36.7 / 35.7	38.9 / 35.9	44.1 / 38.5	38.2 / 35.6
DeBERTaV3-large	-	46.5 / 39.3	45.5 / 40.2	49.0 / 41.2	39.7 / 35.2
T5-base	-	28.0 / 28.0	28.9 / 27.4	29.3 / 29.1	30.1 / 29.9
RACE	Longformer-base	39.5 / 35.3	-	-	-	-
↓	LED-base	37.2 / 33.8	-	-	-	-
QuALITY	LED-large	39.4 / 35.3	-	-	-	-
RoBERTa-base	-	42.1 / 38.3	43.0 / 40.1	44.3 / 39.8	38.1 / 37.5
RoBERTa-large	-	48.0 / 40.8	50.4 / 43.7	51.4 / 44.7	40.4 / 37.1
DeBERTaV3-base	-	46.8 / 38.7	49.8 / 43.2	51.2 / 42.4	41.4 / 37.9
DeBERTaV3-large	-	53.8 / 46.3	54.7 / 46.7	55.4 / 46.1	43.3 / 38.2
T5-base	-	41.1 / 40.1	40.8 / 40.1	41.6 / 39.8	36.4 / 35.9
-	Human Annotators	93.5 / 89.1	-	-	-	-
Table 6: Accuracy on the full QuALITY test set and the QuALITY-HARD subset (formatted as full / HARD). The "Full" column has results from training with the source inputs truncated to fit into memory. R-1 (ROUGE-1), fastText, DPR are three extraction ("Extr") methods (\$4.1) used to select relevant portions of the source text.
results from training just on RACE are in Appendix D.3. All results in Table 6 fall well below human performance. There is a gap of 38.1 points between our current best-performing model (DeBERTaV3-large trained on RACE  $\rightarrow$  QuALITY, using DPR-based extraction) and human performance on the full test set. On QuALITY-HARD, this gap increases to 42.4 points.
Comparing models using different training data, we see that the RACE  $\rightarrow$  QuALITY results outperform RACE results in most cases (Table 9). Finetuning on QuALITY contributes to a small performance gain. Both RACE and RACE  $\rightarrow$  QuALITY significantly outperform the QuALITY only results, likely because of the small size of the QuALITY training set, though this suggests that knowledge transfer from RACE is useful.
In terms of extraction strategies, DPR-based extraction almost always produces the best result. In terms of models, DeBERTaV3-large consistently performs best. Compared to the RoBERTa and DeBERTa models fined-tuned on short contexts, the Longformer and LED models appear to struggle to learn the task from the long inputs, underperforming even the RoBERTa-base extraction-based models. We speculate that a combination of more long-context training data and better long-context models may improve performance beyond the extraction-based models. As with other models, intermediate training on RACE improves performance on QuALITY.
Question-Only Baselines The best-performing question-only baseline is DeBERTaV3-large using the  $\mathrm{RACE} \rightarrow \mathrm{QuALITY}$  setting for training, achieving an accuracy of  $43.3\%$ . The corresponding performance is only 12.1 percentage points lower than the DeBERTaV3-large's performance with text excerpts from DPR. This small margin of improvement may indicate that current models are not effectively using the input contexts.
QuALITY-HARD Model performance is always lower on QuALITY-HARD than on the full test-set, even on the question-only baselines. This suggests that speed-validation filtering yields more challenging questions for human annotators and models.
Extraction by Oracle Answer We show in Appendix D.3, Table 11 the results of the oracle-answer-based extraction on the development set. Compared to Table 10, using the oracle answers for extraction improves performance significantly (topping out at  $78.3\%$ ), but is still below human performance by 15 points. This demonstrates that extracting relevant excerpts alone is insufficient to solve QuALITY questions, and that QuALITY questions require reasoning over the full passage.
Rogers et al. (2021) survey the QA dataset explosion of recent years and the many formats and types of QA datasets. TriviaQA (Joshi et al., 2017) and SearchQA (Dunn et al., 2017) contain questions
with more than one document as the context, but since the supporting documents are collected after writing the question-answer pairs, most questions can be answered after retrieving a short context. HotpotQA (Yang et al., 2018), QAngaroo (Welbl et al., 2018), and ComplexWebQuestions (Talmor and Berant, 2018) are constructed to have more challenging questions which require multi-hop reasoning across multiple paragraphs. However, there has been recent work (Jiang and Bansal, 2019; Min et al., 2019) showing that these datasets contain reasoning shortcuts and a large fraction of the questions can be answered with single-hop reasoning.
NarrativeQA (Kočisky et al., 2018), the most similar work to ours, uses entire Gutenberg books and film scripts as contexts, with an average length of 60k tokens. The authors creatively make data-collection tractable by using Wikipedia summaries for the books as context when crowdsourcing questions. Unlike QuALITY, NarrativeQA is a freeform generation-based task. While there are many existing multiple-choice QA datasets (Richardson et al., 2013; Hill et al., 2015; Lai et al., 2017; Bajgar et al., 2016; Huang et al., 2019), they use much shorter contexts (<500 tokens) than our dataset.
A primary challenge of building a long-document QA dataset like QuALITY or NarrativeQA is building a tractable crowdsourcing pipeline that enables collecting high-quality examples. Roit et al. (2020) collect a challenging QA-SRL dataset by carefully hiring and training crowdworkers, with a strict qualification followed by two hours of training with extensive feedback. Nangia et al. (2021) compare crowdsourcing methods for collecting high-quality QA data and find that a long training process with iterative feedback and qualifications is an effective strategy.
We introduce the long-document QA dataset QuALITY. This dataset was crowdsourced and validated by humans to ensure that the questions are answerable, unambiguous, and challenging. The QuALITY-HARD subset, comprising half the dataset, consists of questions that are unanswerable by annotators working under tight time constraints, helping ensure that skimming and simple search do not yield high performance.
We find that our baseline models significantly lag behind human performance on QuALITY, with a 38.1 percentage point gap between human an
notators and the best performing model. The gap is even wider on QuALITY-HARD, at 42.3 points. We hope that research that aims at this gap will contribute to expanding the scope of texts on which effective NLU systems can be applied.
Both the authors of our source texts and the authors of our questions are based primarily in the US, and represent a relatively privileged, educated population. A system that performs well on our dataset is, thus, only demonstrating its effectiveness on mainstream US English, and should not be presumed to be effective on text in other languages or language varieties.
- Locating appropriate passage sources: AP, JM, VP, AC, NN, NJ, JT
- Preprocessing passages: AC, NN, NJ, JP
Data collection protocol design: NN, AP, RP, SB, HH
Data collection user interface: RP, JT
Data collection backend infrastructure: AC
Data collection management and postprocessing: RP, AP, NJ
- Writing catch questions: VP, JM, JT, AP, NN, NJ, RP
Data analysis: AP, NJ, RP, VP
Modeling: JP, AC
- Writing: AP, RP, NN, NJ, JP, HH, SB
- Project management: RP, AP
- Advising: SB
This project has benefited from financial support to SB by Eric and Wendy Schmidt (made by recommendation of the Schmidt Futures program), Samsung Research (under the project Improving Deep Learning using Latent Structure), Samsung Advanced Institute of Technology (under the project Next Generation Deep Learning: From Pattern Recognition to AI), and Apple. This material is based upon work supported by the National Science Foundation under Grant Nos. 1922658 and 2046556. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the National Science Foundation.
We thank Jon Ander Campos, Alex Wang, Saku Sugawara, and Omer Levy for valuable discussion.
We thank the anonymous reviewers for useful feedback. Finally, we thank the writers who wrote our source texts (credited in the data itself) and the writers who wrote our questions: Megan Barbee, Bridget Barrett, Kourtney Bradley, Kyle J. Brown, Alicia Chatten, Christine D., Leah Dorschner-Karim, Bobbie Dunn, Charisse Hake, Javier Hernandez, Molly Montgomery, Carilee Moran, Tracy M. Snyder, Lorna Stevenson, Isaiah Swanson, Kyla Thiel, Lisa V., Ryan Warrick, Julia Williamson, and others who chose to remain anonymous.
Ondrej Bajgar, Rudolf Kadlec, and Jan Kleindienst. 2016. Embracing data abundance: Booktest dataset for reading comprehension. arXiv preprint arXiv:1610.00956.
Max Bartolo, Alastair Roberts, Johannes Welbl, Sebastian Riedel, and Pontus Stenetorp. 2020. Beat the AI: Investigating adversarial human annotation for reading comprehension. Transactions of the Association for Computational Linguistics, 8:662-678.
Iz Beltagy, Matthew E Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv preprint arXiv:2004.05150.
Piotr Bojanowski, Edouard Grave, Armand Joulin, and Tomas Mikolov. 2017. Enriching word vectors with subword information. Transactions of the Association for Computational Linguistics, 5:135-146.
Samuel R. Bowman and George Dahl. 2021. What will it take to fix benchmarking in natural language understanding? In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4843-4855, Online. Association for Computational Linguistics.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. SearchQA: A new Q&A dataset augmented with context from a search engine. arXiv preprint arXiv:1704.05179.
Esin Durmus, He He, and Mona Diab. 2020. FEQA: A question answering evaluation framework for faithfulness assessment in abstractive summarization. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5055-5070, Online. Association for Computational Linguistics.
Angela Fan, Yacine Jernite, Ethan Perez, David Grangier, Jason Weston, and Michael Auli. 2019. ELI5: Long form question answering. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3558-3567, Florence, Italy. Association for Computational Linguistics.
Charles Fillmore, Nancy Ide, Daniel Jurafsky, and Catherine Macleod. 1998. An American national corpus: A proposal. In Proceedings of the First Annual Conference on Language Resources and Evaluation, pages 965-969. CiteSeer.
Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. 2021. DeBERTa: Decoding-enhanced bert with disentangled attention. In International Conference on Learning Representations.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2015. The Goldilocks principle: Reading children's books with explicit memory representations. arXiv preprint arXiv:1511.02301.
Lifu Huang, Ronan Le Bras, Chandra Bhagavatula, and Yejin Choi. 2019. Cosmos QA: Machine reading comprehension with contextual commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2391-2401, Hong Kong, China. Association for Computational Linguistics.
Nancy Ide and Keith Suderman. 2004. The American national corpus first release. In Proceedings of the Fourth International Conference on Language Resources and Evaluation (LREC'04), Lisbon, Portugal. European Language Resources Association (ELRA).
Yichen Jiang and Mohit Bansal. 2019. Avoiding reasoning shortcuts: Adversarial evaluation, training, and model development for multi-hop QA. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 2726-2736, Florence, Italy. Association for Computational Linguistics.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781, Online. Association for Computational Linguistics.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. 2018. The NarrativeQA reading comprehension challenge. Transactions of the Association for Computational Linguistics, 6:317-328.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding comprehension dataset from examinations. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing, pages 785-794, Copenhagen, Denmark. Association for Computational Linguistics.
Adam D. Lelkes, Vinh Q. Tran, and Cong Yu. 2021. Quiz-style question generation for news stories. In Proceedings of the Web Conference 2021.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. RoBERTa: A robustly optimized BERT pretraining approach. arXiv preprint arXiv:1907.11692.
Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2019. Compositional questions do not necessitate multi-hop reasoning. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 4249-4257, Florence, Italy. Association for Computational Linguistics.
Nikita Nangia, Saku Sugawara, Harsh Trivedi, Alex Warstadt, Clara Vania, and Samuel R. Bowman. 2021. What ingredients make for an effective crowdsourcing protocol for difficult NLU data collection tasks? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers), pages 1221-1235, Online. Association for Computational Linguistics.
Yixin Nie, Adina Williams, Emily Dinan, Mohit Bansal, Jason Weston, and Douwe Kiela. 2020. Adversarial NLI: A new benchmark for natural language understanding. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 4885-4901, Online. Association for Computational Linguistics.
Jason Phang, Thibault Févry, and Samuel R Bowman. 2018. Sentence encoders on STILTs: Supplementary training on intermediate labeled-data tasks. arXiv preprint arXiv:1811.01088.
Yada Pruksachatkun, Jason Phang, Haokun Liu, Phu Mon Htut, Xiaoyi Zhang, Richard Yuanzhe Pang, Clara Vania, Katharina Kann, and Samuel R. Bowman. 2020. Intermediate-task transfer learning with pretrained language models: When and why does it work? In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5231-5247, Online. Association for Computational Linguistics.
Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21(140):1-67.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
Matthew Richardson, Christopher J.C. Burges, and Erin Renshaw. 2013. MCTest: A challenge dataset for the open-domain machine comprehension of text. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 193-203, Seattle, Washington, USA. Association for Computational Linguistics.
Anna Rogers, Matt Gardner, and Isabelle Augenstein. 2021. QA dataset explosion: A taxonomy of NLP resources for question answering and reading comprehension. arXiv preprint arXiv:2107.12708.
Paul Roit, Ayal Klein, Daniela Stepanov, Jonathan Mamou, Julian Michael, Gabriel Stanovsky, Luke Zettlemoyer, and Ido Dagan. 2020. Controlled crowdsourcing for high-quality QA-SRL annotation. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 7008-7013, Online. Association for Computational Linguistics.
Uri Shaham, Elad Segal, Maor Ivgi, Avia Efrat, Ori Yoran, Adi Haviv, Ankit Gupta, Wenhan Xiong, Mor Geva, Jonathan Berant, et al. 2022. **Scrolls: Standardized comparison over long language sequences. arXiv preprint arXiv:2201.03533.**
Peter Suber. 2012. Open Access. MIT Press.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641-651, New Orleans, Louisiana. Association for Computational Linguistics.
Alex Wang, Kyunghyun Cho, and Mike Lewis. 2020. Asking and answering questions to evaluate the factual consistency of summaries. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 5008-5020, Online. Association for Computational Linguistics.
Dirk Weissenborn, Georg Wiese, and Laura Seiffe. 2017. Making neural QA as simple as possible but not simpler. In Proceedings of the 21st Conference on Computational Natural Language Learning (CoNLL 2017), pages 271-280, Vancouver, Canada. Association for Computational Linguistics.
Johannes Welbl, Pontus Stenetorp, and Sebastian Riedel. 2018. Constructing datasets for multi-hop reading comprehension across documents. Transactions of the Association for Computational Linguistics, 6:287-302.