SlowGuess/NAACL_2022_OCR huggingface dataset:
Haonan Li
Maria Vasardani
Martin Tomko
Timothy Baldwin
$\spadesuit$  School of Computing and Information Systems, The University of Melbourne
Department of Infrastructure Engineering, The University of Melbourne
Department of Geospatial Science, RMIT University
Department of Natural Language Processing, MBZUAI
haonanl5@student.unimelb.edu.au, maria.vasardani2@rmit.edu.au
tomkom@unimelb.edu.au, tb@ldwin.net
Most existing reading comprehension datasets focus on single-span answers, which can be extracted as a single contiguous span from a given text passage. Multi-span questions, i.e., questions whose answer is a series of multiple discontinuous spans in the text, are common in real life but are less studied. In this paper, we present MultiSpanQA $^1$ , a new dataset that focuses on questions with multi-span answers. Raw questions and contexts are extracted from the Natural Questions (Kwiatkowski et al., 2019) dataset. After multi-span re-annotation, MultiSpanQA consists of over a total of 6,000 multi-span questions in the basic version, and over 19,000 examples with unanswerable questions, and questions with single-, and multi-span answers in the expanded version. We introduce new metrics for the purposes of multi-span question answering evaluation, and establish several baselines using advanced models. Finally, we propose a new model which beats all baselines and achieves the state-of-the-art on our dataset.
The task of reading comprehension, where models are required to process a text and answer questions about it, has seen rapid progress in recent years. As systems have increasingly matched humans on popular datasets (Rajpurkar et al., 2016, 2018), researchers have developed newer, more complex formulations of the task, such as very long contexts and answers (Kwiatkowski et al., 2019), multi-hop reasoning (Yang et al., 2018), and discrete operations over the content of paragraphs (Dua et al., 2019). One thing these datasets have in common is that the answer is constrained to be a single span that can be extracted or computed from the context.
However, in practice, the answer to a question will often consist of multiple parts. As in the example in Figure 1, the answer set contains 10 countries,
Question: Which countries does the Danube River flow through?
Passage: ... Originating in Germany, the Danube flows southeast for 2,850 km (1,770 mi), passing through or bordering Austria, Slovakia, Hungary, Croatia, Serbia, Romania, Bulgaria, Moldova and Ukraine before draining into the Black Sea. ...
Answer set: {Germany, Austria, Slovakia, Hungary, Croatia, Serbia, Romania, Bulgaria, Moldova, Ukraine}
Figure 1: Example of a multi-span question and answer pair.
some of which are discontinuous in the passage. Such cases are largely ignored in existing reading comprehension research, in part because there are no datasets of multi-span questions.
In this paper, we introduce MultiSpanQA, a new reading comprehension dataset consisting of 6,536 multi-span examples. The raw questions and passages are extracted from Natural Questions ("NQ": Kwiatkowski et al. (2019)), a large-scale open-domain QA dataset. Trained annotators were asked to identify question-passage pairs where the answer was multi-span, and annotate the spans. In addition to the basic version of the dataset consisting entirely of multi-span answers, we also prepare an expanded version with a selection of unanswerable questions, and questions with single- and multi-span answers, intended to reflect a more realistic QA setup.
We further classify answer semantics into 5 categories, and manually label the logical structure of the answer spans. We introduce metrics to evaluate multi-span QA systems across these different tasks.
We propose several baselines, and a new model which casts the task as a sequence tagging problem. The proposed model combines a sequence tagger with a span number predictor, span structure predictor, and span adjustment module. Experimental results show that the proposed model surpasses
Available at: https://multi-span.github.io
Human Language Technologies, pages 1250 - 1260
all baselines and achieves  $59.28\%$  exact-match F1 score and  $76.50\%$  partial-match F1 score.
- A new reading comprehension dataset containing 6.5k high-quality multi-span answers, along with analysis and metrics for multi-span QA.
- A novel label set for capturing the semantics of multi-span answers, with annotations.
- A new model for multi-span reading comprehension which achieves state-of-the-art results on our dataset.
2.1 Question Answering Datasets
Extractive QA Most existing extractive QA datasets such as SQuAD (Rajpurkar et al., 2016), SQuAD2.0 (Rajpurkar et al., 2018), SearchQA (Dunn et al., 2017), and QuAC (Choi et al., 2018) restrict the answer passage to a single span of text. SQuAD and SQuAD 2.0 limit the answer passage to a short paragraph from Wikipedia; the best-performing systems have now exceeded human performance on these datasets. QuAC frames the task in a dialogue setting by introducing a teacher and student, where the student repeatedly asks the teacher questions about a topic and the teacher tries to find answers from the given passage. That is, it supports information seeking through multi-turn conversation. TriviaQA (Joshi et al., 2017) and HotpotQA (Yang et al., 2018) extend the answer context from single passage to multiple passages, while HotpotQA further requires reasoning over multiple passages to answer the question. However, all of these datasets limit the answer to a single text span from the provided answer context.
DROP (Dua et al., 2019) requires systems to resolve (possibly multiple) references in a question, and perform discrete operations (such as addition, sorting, or counting) over them. However, because these operations are mostly numeric, the spans are almost exclusively semantically homogeneous and related to numeric values. MASH-QA (Dua et al., 2019) extends the answer space to texts that span across a longer document, but this dataset is highly domain-specific, in the healthcare domain. Quoref (Dasigi et al., 2019) and Natural Questions ("NQ": Kwiatkowski et al. (2019)) both contain multi-span answers. Quoref requires systems to resolve coreference among entities, to aid in span-selection. NQ is a large-scale dataset that provides questions with
very long answer contexts. The proportion of multispan answers is around  $10\%$  and  $2\%$  in Quoref and NQ, respectively. However in each case, multispan answers are captured as a single span, with no annotation of the internal structure of the component spans. WikiHowQA and WebQA (Cui et al., 2021) both focus on non-factoid (e.g., how, why) questions, with answers mostly being long spans or full sentences.
Generative QA Generative QA datasets usually require systems to answer questions in the form of several sentences, either selected from the provided answer context or generated based on it. WikiQA (Yang et al., 2015) and MS Marco (Nguyen et al., 2016) are two open-domain generative QA datasets, where answers in WikiQA are mostly sentences from the answer passage, while answers in MS Marco are free-form sentences generated by crowd workers. NarrativeQA (Kocisky et al., 2018) is a dataset of movie and book summaries. SearchQA (Dunn et al., 2017), ELI5 (Fan et al., 2019), and CoQA (Reddy et al., 2019) are three multiple-document datasets. SearchQA is constructed from question-answer pairs crawled from Jeopardy!, and most questions can be answered with a short (99% less than 5 tokens) extractive span from a single document. ELI5 requires systems to generate paragraph-length answers by summarizing information from multiple documents. CoQA contains conversational questions, with freeform text as answers.
Cloze style Cloze datasets such as CNN/Daily Mail (Hermann et al., 2015), Children's Book Test (CBT) (Hill et al., 2016), and BookTest (Bajgar et al., 2016) require systems to predict a missing word from a passage. However, researchers have shown that this task is artificial, and can be largely solved with simple methods and relatively little reasoning (Chen et al., 2016).
2.2 Multi-span Models
Dua et al. (2019) proposed to predict the number of output spans for each question, by applying a single-span predictor recursively, making training complex. Segal et al. (2020) first proposed to treat multi-span QA as a sequence tagging task, in the form of a multi-head architecture (Dua et al., 2019) to perform arithmetic operations between the predicted spans. Hu et al. (2019) applied the non-maximum suppression (NMS) algorithm (Rosenfeld and Thurston, 1971) to prune redundant
bounding boxes from the top- $k$  predicted spans of a single-span predictor. Pang et al. (2019) proposed HAS-QA, which supports multi-span prediction by computing answer probabilities at the question, paragraph and span levels. A common feature of these works is that the predicted spans are fed into an aggregation module, and the answers are usually a single span chosen from the prediction, or a number computed from them. Cui et al. (2021) proposed a model which can extract list-form answers across multiple spans. Their work mainly focuses on capturing the sequential and progressive relationships between long-span descriptions.
3 Dataset Construction and Composition
In this section, we describe how we construct MultiSpanQA, and provide a statistical breakdown of its composition.
3.1 Data Collection and Preprocessing
The question-passage pairs were selected from Natural Questions (NQ: Kwiatkowski et al. (2019)), a large-scale open-domain QA dataset made up of (question, passage, long answer, short answer) quadruples where: the questions are real queries issued to the Google search engine; the passage is a Wikipedia page which may or may not contain the information required to answer the question; the long answer is a paragraph from the page containing all information required to infer the answer; and the short answer is one or more text spans that answer the question. Both long and short answers can be NULL if no viable answer candidate exists on the page.
To create MultiSpanQA, we first extract NQ questions annotated with multiple short answers, and consider the long answer to be the answer passage. We then remove paragraphs that don't contain any question part, to eliminate the information-retrieval component of NQ and focus more on the short answer extraction problem. To make the dataset easy to use, we strip HTML from the passages, so that they only contain plain text. As table structure cannot be captured in the plain text after removing HTML, we remove the passages that contain tables. Ultimately, around 6700 candidates remain where each candidate is a triple of (question, passage, set of answer spans).
To aid the annotation process, we classify the samples into 5 categories according to the expected answer type of questions using a BERT-based clas
Answer type	%	Example
DESCRIPTION	16.4	other gases
LOCATION	18.6	Vermont
HUMAN	46.1	George Benson
NUMERIC	7.3	9,677 ft
OTHER ENTITY	15.4	Torah
Table 1: Proportion and examples of answer types in MultiSpanQA.
sifier trained on the TREC Question Classification dataset (Li and Roth, 2002). The classes are DESCRIPTION, LOCATION, HUMAN, NUMERIC, and OTHER ENTITY. Table 1 shows the breakdown and an example of each answer type class.
3.2 Issues in Existing Dataset
NQ was originally annotated by around 50 annotators, with an average annotation time of 80 seconds per instance. However, we found a number of issues with the dataset: (1) grammatical errors in questions, due to them being actual queries submitted to the Google search engine by real users; (2) answer boundary inconsistencies or errors, such as the entity University of Melbourne being annotated as an answer in one example but The University of Melbourne being annotated in another; and (3) wrong or incomplete answers: some questions are not answered or are answered incompletely in the annotated answer span, for example, to answer the question Which countries does the River Danube flow through?, 10 countries should be included in the answer span while only 9 are annotated. These issues are relatively uncommon overall in the dataset, but occur disproportionately in multi-span answers.
3.3 High Quality Re-annotation
We (re-)annotated all the data using the Brat annotation tool (Stenetorp et al., 2012).<sup>2</sup> Three annotators were provided with a category-specific annotation guide (broken down across the 5 predicted answer types), and annotated the data on a per-category basis.<sup>3</sup>
For each annotation instance, we show the question, passage, and the original multiple answer spans to the annotator. The first-pass annotation was according to the following four categories:
$^{2}$ http://brat.nlplab.org
3The annotation guide is available in the github repository along with the data.
Answer structure	%	Example
Conjunction	82.6	Q: What purpose do aircraft carriers serve for aircraft?
Passage: carrying, arming, deploying, and recovering
Multi-part-disjunction (Redundant)	4.5	Q: When does the Force Unleashed 2 take place?
Passage: The game takes place approximately six months after the events of the first game, and a year before the first Star Wars.
Multi-part-disjunction (Non-Redundant)	9.6	Q: When was the last year they made the Toyota Matrix?
Passage: Sales of the Matrix were discontinued in the United States in 2013, and in Canada in 2014.
Complex	3.1	Q: When was the Battle of Dien Bien Phu and what was the result?
Passage: The battle occurred between March and May 1954 and culminated in a comprehensive French defeat that influenced negotiations underway at Geneva among several nations over the future of Indochina.
Shared Structure	0.2	Q: What does Triangle Transit offer?
Passage: scheduled, fixed-route regional and commuter bus service
Table 2: Answer structure breakdown and examples.
- Good example: the question is clear, and the answer spans are labelled consistent with the annotation guide, in which case accept the instance as is.
- Bad question: the question is ungrammatical or not aligned with the passage content, in which case rewrite the question while preserving its original intended meaning where possible (otherwise reject).
- Bad answer span(s): the answer span(s) are incorrect or incomplete, in which case remove the inappropriate spans and select the correct spans.
- Bad question-answer pair: the question doesn't align with the passage content (e.g. there is no answer there) or there are not multiple answer spans in the passage (e.g. there is only a single answer span), in which case reject the instance.
Although all examples in our dataset contain multiple answer spans, the semantic structure varies considerably. We hand-announce this via a novel 5-way annotation scheme, as follows (see Table 2 for examples):
1. CONJUNCTION: Each span is part of the answer, and the answer is complete only when all of the spans are combined
- MULTI-PART-DISJUNCTION: Each span is a complete (but independent) answer to the question, with one of the following structures: - 2. REDUNDANT: the multiple spans re
fer to the same concept or entity. For example, in the example in Table 2, each span is a full answer to the question, specified using different temporal reference points.
- 3. NON-REDUNDANT: the different spans refer to different concepts or entities, each of which is independently correct in its respective context. For example, in the example in Table 2 each span is independently correct in the context of a particular national market.
- 4. COMPLEX: The question is complex (made up of multiple sub-parts), and each span is an answer to a different sub-part, the internal logic of which is not enumeration. For example, in the example in Table 2, the two spans are independent answers to the two subquestions in the original question.
- 5. SHARED STRUCTURE: Spans are enumerated in the form of a syntactically-coordinated structure, sharing either a modifier or a head (i.e. the first word(s) of the first span or last word(s) of the last span). For example, in the example in Table 2, the three spans share the syntactic head bus service, and the full answer is equivalent to scheduled bus service + fixed-route regional bus service + commuter bus service.
#Spans	2	3	4–5	6–8	9–12	13–21
Count	3,791	1,414	915	337	71	8
Table 3: Number of answer spans in MultiSpanQA.
The annotation was performed by three trained annotators with an average annotation time of 70 seconds per instance. To test the inter-annotator agreement (IAA), we randomly selected 100 instances for each pairing of the three annotators to annotate. The same annotation (of all spans) of an instance is considered as an agreement, and any difference in one instance is considered as a disagreement. The average pairwise IAA is 0.86 for answer spans and 0.94 for answer structures (both based on macroaveraged exact match F1 score), with some disagreements between CONJUNCTION and MULTIPART-DISJUNCTION (NON-REDUNDANT). To better understand the composition of MultiSpanQA, we compare our annotations with those in NQ, and provide some basic statistics. Compared to the original annotations in NQ, the annotators rejected  $3.1\%$  of instances, re-wrote the question for  $5.6\%$  of instances, and modified the answer span annotations for  $22\%$  of instances.
MultiSpanQA contains 6,536 instances with 5,230 for training, 653 for validation, and 653 for test. Table 3 provides the distribution of the number of answer spans in the dataset, from which we see the number of spans ranges from 2 to 21, but  $80\%$  of instances contain 2 or 3 spans, and only about  $1\%$  of instances contain more than 9 spans.
3.5 Dataset Expansion
In its basic form, the MultiSpanQA dataset contains only multiple-span answers, and the correct answer can always be located in the passage (in the form of multiple answer spans). However, in a real-world QA scenario, single-span answer questions and unanswerable questions (i.e. the answer is not contained in the passage) would realistically exist. To create a more realistic and challenging variant of the dataset, we add a comparable number of single-span question-answer pairs and unanswerable instances to MultiSpanQA, by randomly sampling from NQ and applying the same preprocessing. The total size of the expanded dataset is 19,608 instances (three times the basic version, partitioned similarly to the basic version).
Formally, given a question and passage pair  $\langle q,p\rangle$  the task of multi-span QA involves finding all answer spans  $s_1,s_2,\ldots s_n$ , which are neither duplicated nor overlap with each other, as well as predict the answer structures.
Single-span Baseline Because most existing reading comprehension datasets only have single-span answers, single-span architectures are widely used in reading comprehension research. Usually, a pre-trained model is used to encode the question and passage, and output a contextualised representation for all input tokens. Then two feed-forward networks are used to compute a score for each token which indicates whether the token is the start or end of the answer. Finally, a softmax layer followed by an argmax function is used to produce the start and end positions of the answer.
To make MultiSpanQA trainable for a single-span architecture, we experimented with two preprocessing methods, and created two baselines accordingly:
1. Mark the start of the answer as the start position of the first answer span and mark the end of the answer as the end position of the last answer span. In this way, the model can learn to find the shortest span that includes all answer spans. We select the best prediction for evaluation.
2. Suppose an instance has  $n$  answer spans, we replace the instance with  $n$  instances, one for each span with a single-span answer.
In this way, we can apply single-span answer models to our dataset.
For evaluation, to enable multi-span prediction, we output the 20 highest-scoring predictions, and tune a threshold  $t$  to select the answer spans with a confidence score larger than  $t$  that optimises performance on the training set. We remove overlapping predictions based on confidence scores, rejecting predictions with lower confidence scores. Note that for both baselines, we apply the pre-processing to the training data only.
Sequence Tagging Baseline Following Segal et al. (2020), we cast question answering as a sequence tagging task, predicting for each token whether it is part of an answer. In our experiments, we use the popular IOB tagging scheme to mark
Figure 2: Proposed multi-span QA model architecture.
answer spans in the passage where  $\mathbb{B}$  denotes the first token of an answer span,  $\mathbb{I}$  denotes subsequent tokens within a span, and  $\mathbb{O}$  denotes tokens that are not part of an answer span.
4.2 Proposed Model
By investigating the failures of the sequence tagging baseline, we find there is an issue that the model struggles to capture global information. For example, the number of answer spans may be specified in the question, but cannot be imposed as a constraint on the tagger. To better use such global information, we propose a span encoder, a number of span predictor, an answer structure predictor, and a span adjustment module (as in Figure 2), which can be combined with any on-the-fly sequence tagger (encoder).
Contextualiseed Encoder Given a pair of question  $q$  and passage  $p$ , we first encode the question and context together using a sequence pair encoder as:
$$
H = \operatorname {E n c o d e r} (\langle q, p \rangle) \in \mathbb {R} ^ {l \times h} \tag {1}
$$
where  $H = [H_{[CLS]};H_q;H_p]$  is the contextualised token representation of all input tokens with a pooled global token  $[CLS]$ ,  $h$  is the hidden-layer size, and  $l$  is the input length.
After encoding, we fetch the hidden states of the context tokens and input them to a linear classifier to perform a preliminary token-level answer span
prediction, as:
$$
T _ {p} = F F N \left(H _ {p}\right) \in \mathbb {R} ^ {l _ {p} \times t} \tag {2}
$$
where  $l_{p}$  denotes the length of passage, and  $t$  denotes the number of labels ( $t = 3$  in for IOB tagging scheme).
Span Encoder According to the argmax of preliminary predictions  $T_{p}$ , we take the continuous token representations of the predicted spans as span representation  $s_1, s_2, \ldots, s_n$ , where  $s_i = [H_{s_i}, H_{s_i + 1}, \ldots, H_{s_i + k - 1}] \in \mathbb{R}^{k \times h}$ ,  $k$  is the length of the span, which varies across spans. Average pooling is then applied to the span representations  $s_i$  to generate a fixed-length span representation  $S_i \in \mathbb{R}^{1 \times h}$ .
We then concat the hidden state of [CLS] token  $H_{[CLS]}$  with the span representations  $S_{i}$  as  $I_{span} = [H_{[CLS]}, S_{1}, \ldots, S_{n}]$ , and input them into a span encoder as:
$$
I = \operatorname {S p a n E n c o d e r} \left(I _ {\text {s p a n}}\right) \in \mathbb {R} ^ {(n + 1) \times h} \tag {3}
$$
Objective Function We fetch the hidden state of span-level [CLS] token  $I_{[CLS]}$  and input it to two feed-forward networks to predict the number of answer spans and the answer structure, respectively, as below:
$$
P _ {n u m} = F F N \left(I _ {[ C L S ]}\right) \tag {4}
$$
$$
P _ {\text {s t r u c t u r e}} = F F N \left(I _ {[ C L S ]}\right) \tag {5}
$$
We use cross-entropy loss for answer span and structure prediction, and mean-square loss for span number regression. For training, we use the weighted sum of the three losses:
$$
\mathcal {L} = \mathcal {L} _ {\text {s p a n s}} + \lambda_ {1} \mathcal {L} _ {\text {n u m}} + \lambda_ {2} \mathcal {L} _ {\text {s t r u c t u r e}} \tag {6}
$$
Finally, a span adjustment module is used to explicitly combine the predicted span number with the span texts. We first assign a confidence score to each label of the preliminary classification using a softmax layer:
$$
\alpha_ {c o n f} = \operatorname {s o f t m a x} \left(U _ {p}\right) \in \mathbb {R} ^ {l _ {p} \times t} \tag {7}
$$