SlowGuess/NAACL_2022_OCR huggingface dataset:
Question Type	# EASY	# HARD	% total	Example from the Training Set
what	1361	1471	42.2	What is the immediate significance of Ed defending the ads on his Facebook?
why	832	825	24.6	Why does Howell not want Linton to approach Snead in the restaurant?
how	385	416	11.9	How does Barker view his own film?
which	253	244	7.4	Which word least describes McGill?
who	151	132	4.2	Who is the most hated celebrity of 1999?
how + meas.	51	75	1.9	How many caves had Garmon and Rolf traveled through before their crash?
yes/no	53	55	1.6	Was it Nelson's decision to become part of the military?
where	43	42	1.3	Where was the space craft heading in the end?
when	35	34	1.0	When did the Hanseatic League begin?
other	155	124	4.1	Dole's quote would have been perceived as __ if it had included included the exclamation points from his tone?
Table 7: Different question types in QuALITY, split by HARD and EASY subsets. 'How + meas.' collapses multiple questions with 'how' plus some measurement, such as 'how long' or 'how many.'
especially difficult. In these cases, we do not use those passages in computing by-round accuracy for the annotators. We exclude a total of 11 workers throughout the course of data collection for low accuracy, most of them after the first or second round.
Data Reannotation During each untimed validation round, we keep track of the rate at which each worker agrees with the original writers' labels for each question in order to quickly identify cases where either (i) a worker has misunderstood the passage, or (ii) a worker is putting insufficient effort towards the task. For any tasks where the individual annotator disagrees with the writer's labels on at least  $40\%$  of questions, we automatically re-post that passage for reannotation and replace the data, with the assumption that the annotator may have misunderstood something crucial in the passage.[19] After all the annotations are complete, we calculate the gold label answer via majority vote of annotators plus the original writer's label, and assess individual annotator accuracy. If any worker is excluded in a round for low accuracy (i.e., below  $75\%$  accuracy), we discard all of their responses from that round, reannote and replace their data, and re-calculate the gold label and accuracy scores.
Switchboard Data In order to increase the diversity of genres we use as context passages, we attempted to include Switchboard conversations. However, after presenting just 12 such conversa
Figure 7: Lexical overlap of the correct and incorrect options with the context. Since each question has three incorrect options, we use the option with the highest lexical overlap.
tions to our writers, we decided to discard all the Switchboard questions because many writers informed us that it was very difficult to come up with difficult questions for the Switchboard conversations. The writers indicated they found the Switchboard articles more difficult because the conversations are relatively short and usually involve very simple everyday topics, without the kinds of plot twists that are more common in short stories or complex details that are more common in longform articles.
Lexical Overlap We analyze the lexical overlap between the answer options and the passage text (detailed in §3.4. In Figure 7, we plot the lexical overlap of both the correct answer option and the incorrect answer option with the passage.
B.1 Question Types
As described in §3.5, we analyze the different question types in QuALITY, split by HARD and EASY subsets, and present these results along with examples in Table 7. Most of the questions in the 'other' category are finish-the-phrase style questions, and
<sup>19</sup>We identified 10 passages that, after multiple rounds of reannotation, were not passing this threshold. This may be due to one of the writers misunderstanding the passage and thereby creating several ambiguous questions, so for these 10 passages, we chose to include all annotations collected rather than replace annotators' data.
for the example in the table, the answer options are different ways that sentence could be completed. Note that in most of the yes/no questions, the answer options include the necessary reasoning to support the yes/no answer, meaning that these are often complex, multi-part questions. Examples shown in Table 7 are randomly selected from the training set, with the caveat that we selected the 'when' question by hand, since about half of the questions categorized as 'when' are referencing some timepoint (e.g., 'when X happened, what did ...')
B.2 Reasoning Types
For the purposes of this analysis, we define reasoning type as the category that needs to be reasoned about in selecting the correct answer option (e.g., 'person' is usually a 'who' question and corresponds to answer options that are characters or people) or the type of strategy that must be used in answering (e.g., 'symbolism/interpretation' requires the reader to extrapolate from the context or identify something not stated in a passage, like its theme). We identify 15 categories of reasoning types to include in our analysis. These categories are initially inspired by those used in NarrativeQA, but we adapt them to our dataset, as we find that many questions in QuALITY do not fit their categorization. These categories are not mutually exclusive, and nearly a third of the questions are categorized as two or more types.
Reasoning Type Definitions The following includes definitions of all the categories used, along with at least one hand-selected example to demonstrate a question belonging to that category. All in-text examples are selected out of the training set.
- Description: The question relies on the reader reasoning about which description is correct. Often these questions are about describing a character's feelings ('How do Lowry and the Exec feel about the Venusians?') or point of view ('How is the book "Living a Normal Sex Life" seen by these people?'), describing a feature of the story ('What makes Grannie Annie's writing remarkable?'), or describing an individual ('Which word least describes Don?')
- Why/reason: The question relies on the reader reasoning about the cause or explanation for something in the story. Most of
these questions begin with 'why' and ask about the cause of an event ('Why does the crew get off the ship with Moran?'), causes of characters' feelings ('Why does Ben take offence to Cobb's comments about space-men?', or characters' internal motivations ('Why does Joseph lie about the water supply?'), though other questions formulate this differently while still asking for the underlying reason ('What makes Gubelin an outlier in the present day?' and 'What is the purpose of a comanalysis?).
- Symbolism/interpretation: The question relies on the reader making an interpretation that goes beyond what is explicitly said in the story, or it asks about symbolism or themes from the story. Many questions explicitly ask the reader to interpret what message the author was trying to convey ('What point is being made by comparing Fight Club to the UFC?') or what tone the story takes ('What is the tone like throughout the story?'). Other questions require the reader to predict what will happen next ('What will happen next to Jery?') or ask about the use of literary cues like irony ('What is ironic about Earth's customer service policy?').
- How/method: The question relies on reasoning about how something happened or the method that was used. Most of these questions rely on the question word 'how' to ask about a process ('How did Meryl Streep prepare for the role of Roberta?'), the manner in which something happens ('How did Templin find about about Pendleton's death?'), or a method by which something happens ('How does the shape of Starre's ship benefit them?').
- Event: The question relies on reasoning about an event, or asks for an event as the answer option. The majority of these questions focus on what someone did/plans to do ('What did Joe and Glmpauszn plan to do?') or what happened to someone ('What happened to Morgan Brockman by the end of the passage?').
- Person: The question relies on reasoning about which person or people are involved. Most ask about a specific person ('Who is Owen Fiss and what did he do?'), though many questions of this type still require rea
Question	Answer Options	Reasoning Types
What would have happened if Click's camera broke in the crash?	(a) Irish would have died on impact. (b) They would have returned immediately to Luna Base. (c) They would have caught Gunther faster. (d) They would have continued to believe the monsters were real.	What if; Event
What isn't a reason for narrator to be so skeptical of Gorb?	(a) Gorb looked just like an Earthling (b) Gorb was asking for too much money (c) Gorb had no proof to back up his claims (d) he had never heard of Wazzenazz	Why/reason; not/except
How many caves had Garmon and Rolf traveled through before their crash?	(a) thirty seven (b) forty seven (c) thirty (d) forty	Numeric
How do you think Meredith feels about the rest of the crew?	(a) She has a close bond of respect and (platonic) love for the rest of the members (b) She respects and loves one person the most (c) She's become friends with them slowly over time and appreciates them all (d) She respects one person the most and loves another person the most	Description; symbolism/terpretation; relation
The less you share...	(a) ...the more privacy you have. (b) ...the more your intellectual property is protected. (c) ...the less power you have. (d) ...the less your cultural goods will be appropriated.	Symbolism/interpretation; finish the phrase
How did Meryl Streep prepare for the role of Roberta?	(a) She learned to play the violin without any former instrument training. (b) She began to act very helplessly and feeble around the rest of the cast. (c) She is a method actor and became very vulnerable. (d) She made herself look dumpy and thick-waisted.	How/method
Table 8: Full examples of the annotations from our analysis of reasoning types on a subset of questions from QuALITY. Examples are taken from analyzed examples from the training set. Examples are selected non-randomly and are intended to demonstrate a range of reasoning types observed.
soning about the entire passage to answer ('Who seems to have the least to hide in the text?').
- Not/except: The question requires the reader to select the answer option that least answers the question, flipping the typical way a multiple-choice task is performed. All of these questions use some word to indicate this flipping, such as 'least' ('Which word least describes McGill?'), 'not' ('What word doesn't describe the natives from Tunpesh?'), or 'except' ('Dole makes all of the following charges against the New York Times EXCEPT for:').
- Relation: The question relies on reasoning about the relationship between two or more characters, as in 'Who is Sporr and what is his authority in calling the narrator Yandro?' or questions that ask about how one character feels about another ('How does Jakdane feel about Trella?').
- Entity: The question relies on reasoning about a non-human entity or a group, as in 'We can assume that Saladin's army represents which group?'
- Finish the phrase: The form of the ques
tion requires either a fill-in-the-blank style response or is a partial phrase that must be completed by selecting the correct answer option. Often, these questions do not include an explicit question word. Some of them have a blank written in ('The film reviewer is generally _ the actors in "Princess Mononoke," and _ the actors in "The Limey," respectively:') and others are just a partial sentence ('The less you share...').
- Location: The question relies on reasoning about a place, as in 'What city is Temple-Tracy in?'
- Numeric: The question relies on finding or computing the correct numeric option, as in 'How many caves had Garmon and Rolf traveled through before their crash?'.
- Object: The question relies on reasoning about an object, as in 'What does Captain Hannah use as an organic processor?'
- What if: The question requires the reader to make an inference about what would have been true if some fact from the story were changed, and most of these questions explicitly set up the counterfactual scenario ('What
Training Data	Model	Full	Extraction Based on Qs	Question-Only
R-1	fastText	DPR
QuALITY	Longformer-base	30.7 / 29.3	-	-	-	-
LED-base	25.1 / 24.6	-	-	-	-
LED-large	24.2 / 24.5	-	-	-	-
RoBERTa-base	-	33.4 / 30.7	39.7 / 36.1	39.9 / 34.0	36.6 / 34.8
RoBERTa-large	-	29.4 / 28.0	42.7 / 35.7	26.2 / 25.1	26.4 / 25.7
DeBERTaV3-base	-	36.7 / 35.7	38.9 / 35.9	44.1 / 38.5	38.2 / 35.6
DeBERTaV3-large	-	46.5 / 39.3	45.5 / 40.2	49.0 / 41.2	39.7 / 35.2
T5-base	-	28.0 / 28.0	28.9 / 27.4	29.3 / 29.1	30.1 / 29.9
RACE	Longformer-base	35.2 / 30.8	-	-	-	-
RoBERTa-base	-	42.4 / 36.8	43.2 / 37.2	44.2 / 36.1	33.8 / 29.7
RoBERTa-large	-	47.0 / 37.5	47.9 / 40.2	48.7 / 40.2	36.6 / 33.1
DeBERTaV3-base	-	45.3 / 36.1	46.1 / 39.0	47.8 / 39.4	34.7 / 30.5
DeBERTaV3-large	-	52.9 / 43.4	51.2 / 42.4	53.0 / 44.4	36.5 / 30.0
T5-base	-	41.5 / 38.6	42.3 / 39.9	43.4 / 41.0	36.5 / 34.8
RACE	Longformer-base	39.5 / 35.3	-	-	-	-
↓	LED-base	37.2 / 33.8	-	-	-	-
QuALITY	LED-large	39.4 / 35.3	-	-	-	-
RoBERTa-base	-	42.1 / 38.3	43.0 / 40.1	44.3 / 39.8	38.1 / 37.5
RoBERTa-large	-	48.0 / 40.8	50.4 / 43.7	51.4 / 44.7	40.4 / 37.1
DeBERTaV3-base	-	46.8 / 38.7	49.8 / 43.2	51.2 / 42.4	41.4 / 37.9
DeBERTaV3-large	-	53.8 / 46.3	54.7 / 46.7	55.4 / 46.1	43.3 / 38.2
T5-base	-	41.1 / 40.1	40.8 / 40.1	41.6 / 39.8	36.4 / 35.9
-	Human Annotators	93.5 / 89.1	-	-	-	-
Table 9: Accuracy on the full QuALITY test set and the QuALITY-HARD subset (formatted as full / HARD). The "Full" column has results from training with the source inputs truncated to fit into memory. R-1 (ROUGE-1), fastText, DPR are three extraction methods (\$4.1) used to select relevant portions of the source text. Results for training on QuALITY and  $\mathrm{RACE}\rightarrow \mathrm{QuALITY}$  are identical to Table 6 in the main text, this table simply presents those results alongside results from training only on RACE.
would have happened if the Peace State had not crash landed?).
- Duration: The question relies on reasoning about how long something happened for or how much time passed between two events, as in 'How long did Maggie care for Ben before he finally awoke after rescuing him?'
Annotation Details Three authors of this paper analyze a set of 500 randomly selected questions. One author annotates all 500, and the other two annotators analyze 250 each, such that each example is annotated by two unique individuals. Following annotation, the authors discuss any disagreements and adjust their original coding once consensus is reached. Using this consensus approach allows for clarification of the categories during and after annotation, which leads to an internally consistent coding scheme.
Sample Annotations Table 8 shows a set of representative example annotations from this analysis, demonstrating several sentences that were categorized as more than one reasoning type.
C More Details on Analysis
Figure 8: Lexical overlap of all the correct and incorrect options with the article. The distribution is normalized since there are thrice as many incorrect options as there are correct options.
Lexical Overlap In addition to comparing lexical overlap of the correct option and the maximum lexical overlap of the incorrect option with the article (Section 3.4), we also plot a normalized distribution of lexical overlap for all the correct and incorrect options in Figure 8. Despite a higher fraction of the correct options having complete overlap
Training Data	Model	Full	Extraction Based on Qs	Question-Only
R-1	fastText	DPR
QuALITY	Longformer-base	33.7 / 32.6	-	-	-	-
LED-base	25.1 / 24.3	-	-	-	-
LED-large	25.1 / 25.6	-	-	-	-
RoBERTa-base	-	33.7 / 32.0	39.2 / 37.2	40.0 / 36.4	36.0 / 35.6
RoBERTa-large	-	30.0 / 28.7	42.7 / 38.3	26.7 / 24.0	26.7 / 23.0
DeBERTaV3-base	-	34.7 / 33.0	38.0 / 35.3	41.8 / 37.4	36.9 / 34.3
DeBERTaV3-large	-	44.0 / 37.6	44.3 / 37.9	45.1 / 39.2	38.1 / 33.7
T5-base	-	27.3 / 26.6	27.6 / 25.5	28.3 / 29.4	27.9 / 28.7
RACE	Longformer-base	34.5 / 31.6	-	-	-	-
RoBERTa-base	-	43.7 / 38.2	43.3 / 38.9	44.1 / 37.3	36.8 / 34.6
RoBERTa-large	-	48.6 / 41.7	48.4 / 42.3	50.9 / 45.3	37.2 / 35.3
DeBERTaV3-base	-	46.5 / 38.7	44.8 / 37.9	48.8 / 41.7	35.8 / 31.6
DeBERTaV3-large	-	51.2 / 43.9	50.5 / 43.8	53.5 / 47.3	38.3 / 34.3
T5-base	-	39.0 / 37.7	39.7 / 39.2	39.9 / 38.5	37.2 / 35.6
RACE	Longformer-base	38.1 / 32.8	-	-	-	-
↓	LED-base	35.6 / 32.0	-	-	-	-
QuALITY	LED-large	39.9 / 39.6	-	-	-	-
RoBERTa-base	-	43.7 / 38.2	41.7 / 36.2	43.8 / 37.2	37.4 / 36.6
RoBERTa-large	-	47.7 / 42.5	46.8 / 43.1	50.8 / 46.2	39.1 / 37.8
DeBERTaV3-base	-	45.5 / 40.0	46.6 / 40.1	46.7 / 40.9	39.6 / 35.2
DeBERTaV3-large	-	51.7 / 44.7	50.7 / 43.3	53.6 / 47.4	41.4 / 39.2
T5-base	-	40.0 / 38.2	40.4 / 38.6	39.0 / 37.9	37.1 / 36.1
Table 10: Accuracy on QuALITY development set (full / HARD).
with the article, models would not be able to exploit this heuristic, since other incorrect options for the same question may have complete overlap. This is demonstrated by the plot in Figure 7 and the fact that a baseline which relies on the lexical overlap heuristic only achieves  $26.6\%$  accuracy.
D More Details on Modeling
D.1 Extraction
For ROUGE-1 scoring, we use the rouge-score Python package.[20]
For fastText scoring, we use SpaCy with the en_core_web_sm model for tokenization, and use embeddings trained on Common CWEl,  $^{21}$  using the top 300k words in the vocabulary.
For DPR, we use the Transformers package (Wolf et al., 2020), using the facebook/dpr-ctxEncoder-multiset-base and facebook/dpr-questionEncoder-multiset-base models for encoding the context and query respectively.
D.2 Training
The full sets of hyperparameters used for tuning our baselines are shown in Table 12 and Table 13.
For RoBERTa, DeBERTaV3 and Longformer models, we train on QuALITY for 20 epochs. Where we do intermediate training on RACE, we do so for 3 epochs. Warmup is set to  $10\%$  of the full training steps.
Table 10 shows the results on development set. Table 11 shows the results using oracle-answer-based extraction. Please refer to the discussion in Section 4.2.
20https://github.com/google-research/ google-research/tree/master/rouge
$^{21}$ https://dl.fbaipublicfiles.com/fasttext/vectors-english/crawl-300d-2M.vec.zip
Training Data	Model	Extraction Based on Oracle Answers
R-1	fastText	DPR
QuALITY	RoBERTa-base	69.1/67.3	61.3/57.2	78.3/77.7
RoBERTa-large	67.5/64.2	63.9/60.0	75.3/73.7
DeBERTaV3-base	70.8/68.5	65.5/60.8	76.4/74.9
DeBERTaV3-large	68.9/65.2	66.6/60.8	77.8/75.1
RACE	RoBERTa-base	54.5/49.9	56.1/49.7	53.4/47.9
RoBERTa-large	59.2/53.7	58.2/52.1	56.9/51.5
DeBERTaV3-base	56.5/51.5	55.9/48.7	52.0/45.0
DeBERTaV3-large	59.8/54.1	59.8/53.5	57.5/49.6
RACE	RoBERTa-base	67.6/62.8	64.6/58.8	70.2/67.2
↓	RoBERTa-large	68.9/63.7	66.6/60.5	64.1/60.0
QuALITY	DeBERTaV3-base	69.6/64.4	68.1/62.5	66.9/61.2
DeBERTaV3-large	71.0/66.5	68.2/62.9	71.9/67.1
Table 11: Oracle accuracy on the full QuALITY development set and on the QuALITY-HARD subset (full/HARD) with models using the correct answers as queries to retrieve relevant excerpts. These results are meant to show the relative contribution of the retrieval and reading components of the two-stage models. Caution: These results rely on answers at test time, which are not available to any model during a conventional deployment or test set evaluation, and so are of very limited value in conventional comparisons.
Model	Learning rate	Training time	Max. sequence length	Batch size	Warmup steps
RoBERTaBASE	1e-5	3160 steps	512	16	316
RoBERTaLARGE	1e-5	3160 steps	512	16	316
DeBERTaV3BASE	1e-5	3160 steps	512	16	316
DeBERTaV3LARGE	1e-5	3160 steps	512	16	316
LongformerBASE	1e-5	3160 steps	4096	16	316
T5BASE	1e-4	40000 steps	512	128	0
LEDBASE	1e-5	3160 steps	16384	16	316
LEDLARGE	1e-5	780 steps	16384	32	78
Table 12: Hyperparameters used for fine-tuning models on QuALITY.
Model	Learning rate	Training time	Max. sequence length	Batch size	Warmup steps
RoBERTaBASE	1e-5	16473 steps	512	16	1647
RoBERTaLARGE	1e-5	16473 steps	512	16	1647
DeBERTaV3BASE	1e-5	16473 steps	512	16	1647
DeBERTaV3LARGE	1e-5	16473 steps	512	16	1647
LongformerBASE	1e-5	16473 steps	512	16	1647
LEDBASE	1e-5	16473 steps	512	16	1647
LEDLARGE	1e-6	13727 steps	512	32	1372
Table 13: Hyperparameters used for fine-tuning models on RACE.