SlowGuess/ACL_2020_OCR huggingface dataset:
Michael Glass, $^{1}$  Alfio Gliozzo, $^{1}$  Rishav Chakravarti, $^{1}$  Anthony Ferritto, $^{1}$  Lin Pan, $^{1}$  G P Shrivatsa Bhargav, $^{2}$  Dinesh Garg, $^{1}$  Avirup Sil $^{1}$
<sup>1</sup> IBM Research AI
$^{2}$  Dept. of CSA, IISC, Bangalore
mrglass@us.ibm.com, gliozzo@us.ibm.com,
rchakravarti@us.ibm.com, aferritto@ibm.com, panl@us.ibm.com,
bhargavs@iisc.ac.in, garg.dinesh@in.ibm.com,avi@us.ibm.com
BERT (Bidirectional Encoder Representations from Transformers) and related pre-trained Transformers have provided large gains across many language understanding tasks, achieving a new state-of-the-art (SOTA). BERT is pretrained on two auxiliary tasks: Masked Language Model and Next Sentence Prediction. In this paper we introduce a new pre-training task inspired by reading comprehension to better align the pre-training from memorization to understanding. Span Selection Pre-Training (SSPT) poses cloze-like training instances, but rather than draw the answer from the model's parameters, it is selected from a relevant passage. We find significant and consistent improvements over both  $\mathrm{BERT}_{\mathrm{BASE}}$  and  $\mathrm{BERT}_{\mathrm{LARGE}}$  on multiple Machine Reading Comprehension (MRC) datasets. Specifically, our proposed model has strong empirical evidence as it obtains SOTA results on Natural Questions, a new benchmark MRC dataset, outperforming  $\mathrm{BERT}_{\mathrm{LARGE}}$  by 3 F1 points on short answer prediction. We also show significant impact in HotpotQA, improving answer prediction F1 by 4 points and supporting fact prediction F1 by 1 point and outperforming the previous best system. Moreover, we show that our pre-training approach is particularly effective when training data is limited, improving the learning curve by a large amount.
State-of-the-art approaches for NLP tasks are based on language models that are pre-trained on tasks which do not require labeled data (Peters et al., 2018; Howard and Ruder, 2018; Devlin et al., 2018; Yang et al., 2019; Liu et al., 2019; Sun et al., 2019). Fine tuning language models to downstream tasks, such as question answering or other natural language understanding tasks, has been shown to be a general and effective strategy. BERT is a recently
introduced and highly successful model for language understanding.
The general BERT adaptation approach is to alter the model used for pre-training while retaining the transformer encoder layers. The model discards the layers used for the final prediction in the pretraining tasks and adds layers to predict the target task. All parameters are then fine tuned on the target task.
BERT is based on the transformer architecture (Vaswani et al., 2017), and trained on the following two unsupervised tasks:
- Masked Language Model (MLM): predicting masked word pieces from the surrounding context
- Next Sentence Prediction (NSP): predicting if the two provided sequences follow sequentially in text or not
The masked LM or "cloze" task (Taylor, 1953) and next sentence prediction are auxiliary tasks (Ando and Zhang, 2005) requiring language understanding, and therefore train the model to acquire effective representations of language. However, the cloze pre-training task often poses instances that require only shallow prediction, or else require memorized knowledge. For many cloze instances the model simply requires syntactic or lexical understanding to answer. For example, in the cloze instances in Table 1 the first two rows require syntactic and lexical understanding respectively. Other cloze instances mainly require completing collocations, as in the third example. However, some cloze instances require memorized knowledge, as in the last instance, which essentially asks where Hadrian died.
Other language models face the same challenge. In GPT-2 (Radford et al., 2019) the entities present in a language generation prompt are expanded with
Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 2773-2782 July 5 - 10, 2020. ©2020 Association for Computational Linguistics
Type	Cloze
Syntactic	In the 15th century, the blast furnace spread into what is now Belgium where it was improved.
Lexical	Akebia quinata grows to 10 m (30 ft) or more in height and has compound leaves with five leaflets.
Collocation	Apollo 11 was launched by a Saturn V rocket from Kennedy Space Center on Merritt Island, Florida
Memorized Knowledge	Hadrian died the same year at Baiae, and Antoninus had him deified, despite opposition from the Senate.
Table 1: Cloze instances of different types
related entities. For example, in a prompt about nuclear materials being stolen on a Cincinnati train, GPT-2 references "Ohio news outlets", "U.S. Department of Energy", and "Federal Railroad Administration" in ways consistent with their real world relationships to the entities in the prompt.
As the preceding examples illustrate, in many cloze and conventional language model prediction instances, the correct prediction depends on a specific, narrowly relevant, bit of knowledge. Further, pre-trained transformer models do indeed encode a substantial number of specific facts in their parameter matrices, enabling them to answer questions directly from the model itself (Radford et al., 2019). However, because the computational cost of transformers scales at least linearly with the number of parameters, it is expensive to encode all the facts that would enable the correct predictions. Encoding a large amount of rarely useful information in parameters that are used for every instance is an inefficient use of model capacity if it is not needed for the downstream task.
As the performance gains from GPT to GPT-2 and  $\mathrm{BERT}_{\mathrm{BASE}}$  to  $\mathrm{BERT}_{\mathrm{LARGE}}$  show, increasing model capacity continues to provide gains. Previous work also found seemingly limitless improvements from increasing model capacity (Shazeer et al., 2017), possible through sparse activation. Our hypothesis is that making more efficient use of a fixed number of parameters can provide analogous gains. In MRC tasks, the model does not need to generate an answer it has encoded in its parameters. Instead, the task is to use a retrieved passage, or passage set to extract an answer to the
To better align the pre-training with the needs of the MRC task, we use span selection as an additional auxiliary task. This task is similar to the cloze task, but is designed to have a fewer simple instances requiring only syntactic or collocation understanding. For cloze instances that require specific knowledge, rather than training the model to encode this knowledge in its parameterization, we provide a relevant and answer-bearing passage paired with the cloze instance.
We provide an extensive evaluation of the span selection pre-training method across four reading comprehension tasks: the Stanford Question Answering Dataset (SQuAD) in both version 1.1 and 2.0; followed by the Google Natural Questions dataset (Kwiatkowski et al., 2019) and a multihop Question Answering dataset, HotpotQA (Yang et al., 2018). We report consistent improvements over both  $\mathrm{BERT}_{\mathrm{BASE}}$  and  $\mathrm{BERT}_{\mathrm{LARGE}}$  models in all reading comprehension benchmarks.
The rest of the paper is structured as follows. In section 2 We describe earlier work on similar tasks and relate our extended pre-training to the broader research efforts on pre-training transformers. To provide context for our contribution, we review the most relevant parts of BERT in Section 3. Next, we describe and formalize our pre-training task and the architectural adjustments to BERT in Section 4. Finally we provide an extensive empirical evaluation in MRC tasks, describing benchmarks in Section 5 and evaluating our approach in Section 6. Section 7 concludes the paper highlighting interesting research direction for future work.
Since the development of BERT there have been many efforts towards adding or modifying the pretraining tasks. Joshi et al. (2019) introduced SpanBERT, a task that predicts the tokens in a span from the boundary token representations. Note that, unlike span selection, there is no relevant passage used to select an answer span. ERNIE 2.0 (Sun et al., 2019) trained a transformer language model with seven different pre-training tasks, including a variant of masked language model and a generalization of next-sentence prediction. XLNet (Yang et al., 2019) introduced the permuted language model task, although it is not clear whether the success of the model is due to the innovative pre-training or larger quantity of pre-training.
In this paper we focus on a pre-training task that has been specifically designed to support QA applications. Previous related work has explored tasks similar to span selection pre-training. These are typically cast as approaches to augment the training data for question answering systems, rather than alleviating the pressure to encode specific facts in the pre-training of a language model.
Hermann et al. (2015) introduces a reading comprehension task constructed automatically from news articles with summaries. In this view the constructed dataset is used both for training and test. Also, entities were replaced with anonymized markers to limit the influence of world knowledge. Unlike our span selection pre-training task, this requires summaries paired with articles and focuses only on entities. A similar approach was taken in Dhingra et al. (2018) to augment training data for question answering. Wikipedia articles were divided into introduction and body with sentences from the introduction used to construct queries for the body passage. Phrases and entities are used as possible answer terms.
Onishi et al. (2016) constructed a question answering dataset where answers are always people. Unlike other work, this did not use document structure but instead used a search index to retrieve a related passage for a given question. Because the answers are always people, and there are only a few different people in each passage, the task is multiple choice rather than span selection. Self training (Sachan and Xing, 2018) has also been used to jointly train to construct questions and generate self-supervised training data.
BERT was trained for one million batches, with 256 token sequences in each. Although this is already a considerable amount of pre-training, recent research has shown continued improvement from additional pre-training data. XLNet (Yang et al., 2019) used four times as much text, augmenting the Wikipedia and BooksCorpus (Zhu et al., 2015) with text from web crawls, the number of instances trained over was also increased by a factor of four. RoBERTa (Liu et al., 2019) enlarged the text corpus by a factor of ten and trained over fifteen times as many instances. This, along with careful tuning of the MLM task resulted in substantial gains. Unfortunately, these very large-scale pre-training approaches require significant hardware resources. We restrict our experiments to extended pre-training with less than half the steps
of BERT (390k batches of 256).
In this section, we give the readers a brief overview of the BERT (Devlin et al., 2018) pre-training strategy and some details which we modify for our novel span selection auxiliary task.
3.1 Architecture and setup
BERT uses a transformer (Devlin et al., 2018) architecture with  $L$  layers and each block uses  $A$  self-attention heads with hidden dimension  $H$ . The input to BERT is a concatenation of two segments  $x_{1}, \ldots, x_{M}$  and  $y_{1}, \ldots, y_{N}$  separated by special delimiter markers like so:  $[CLS], x_{1}, \ldots, x_{M}, [SEP], y_{1}, \ldots, y_{N}, [SEP]$  such that  $M + N < S$  where  $S$  is the maximum sequence length allowed during training<sup>1</sup>. This is first pre-trained on a large amount of unlabeled data and then fine-tuned on downstream tasks which has labeled data.
3.2 Objective functions
BERT used two objective functions during pretraining: masked language modeling and next sentence prediction. We discuss them in brief.
Masked Language Model (MLM): A random sample of the tokens in the input sequence is replaced with a special token called [MASK]. MLM computes a cross-entropy loss on predicting these masked tokens. Particularly, BERT selects  $15\%$  of the input tokens uniformly to be replaced.  $80\%$  of these selected tokens are replaced with [MASK] while  $10\%$  are left unchanged, and  $10\%$  are replaced with random token from the vocabulary.
Next Sentence Prediction (NSP): This is a binary classification loss that predicts if two sentences follow each other in the original text. The examples are sampled with equal probability such that positive examples are consecutive sentences while negatives are artificially created by adding sentences from different documents.
4 Span Selection Pre-training
In the previous section we briefly discussed the BERT framework along with its objective functions. In this section, we will propose a novel pre-training task for bi-directional language models called span selection.
<sup>1</sup>We follow standard notation here as in previous work.
4.1 Span Selection
Span selection is a pre-training task inspired both by the reading comprehension task and the limitations of cloze pre-training. Figure 1 illustrates an example of a span selection instance. The query is a sentence drawn from a corpus with a term replaced with a special token: [BLANK]. The term replaced by the blank is the answer term. The passage is relevant as determined by a BM25 (Robertson et al., 1995)  $(\mathrm{k}1 = 1.2, \mathrm{b} = 0.75)$  search, and answer-bearing (containing the answer term).
Query “In a station of the metro” is an Imagist poem by [BLANK] published in 1913 in the literary magazine Poetry
Passage ... Ezra Pound's famous Imagist poem, "In a station of the metro", was inspired by this station ...
Answer Ezra Pound Term
Unlike BERT's cloze task, where the answer must be drawn from the model itself, the answer is found in a passage using language understanding.
Figure 1: Example Span Selection Instance
Figure 2: Span Selection Training Generation
Figure 2 outlines the process of generating span selection pre-training data. The input is an unlabeled corpus, which is then split into passages and indexed. We used passages from Wikipedia<sup>2</sup> 300 to 2000 characters long, split on paragraph boundaries, and Lucene<sup>3</sup> 7.4.0 as the search engine. In addition to the text of the passage, we store the document ID, so that we may filter passages that occur in the same document as the query.
To gather queries, we iterate over the sentences in the corpus between 50 and 250 characters long.
Type	Span Selection Instance
Phrase Multiple Choice	Q: The year 1994 was proclaimed [BLANK] of the Family by the United Nations General Assembly.
P: The International Year for the Culture of Peace was designated by the United Nations as the year 2000, with the aim of celebrating and encouraging a culture of peace.
...
Suggestive Inference	Q: On the island of Kaja in [BLANK], a male orangutan was observed using a pole apparently trying to spear or bludgeon fish.
P: ...Although similar swamps can be found in Borneo, wild Bornean orangutans have not been seen using these types of tools.
Justified Inference	Q: The company's headquarters are located in the city of Redlands, California, 50 miles east of [BLANK].
P: Redlands (Serrano: Tukut) is a city in San Bernardino County, California, United States. It is a part of the Greater Los Angeles area. ...
Table 2: Span Selection instances of different types
For each sentence, we choose an answer term to replace with a blank. We used a set of simple heuristic criteria to identify answer terms that are likely to result in queries that require deep understanding to answer: the term should be between 4 and 30 characters and either a single token from an open class part-of-speech (20%) or a noun phrase or entity (80%), as detected by a part-of-speech pattern and ClearNLP NER.
To identify the passages, we use the generated query, with the answer term removed, as a bag-of-words query to search into the passage index. The top ten results were searched for an answer-bearing passage; if none were found the query was either discarded or sampled to maintain a  $30\%$  composition of impossible span selection instances. The impossible instances are those that do not have the answer-term in the provided passage. We further required a minimum BM25 score of 25 (tuned manually to reflect high relevance). If the answer term was part of a longer sequence of tokens shared by the query and passage, we extended the answer term to be the longest such sequence. This avoids cases where the answer term can be found through trivial surface-level matching.
Table 2 shows examples of span selection instances of different types. Rather than discreet types, these are best understood as a continuum.
2December 2018 snapshot
<sup>3</sup>http://lucene.apache.org/
Comparing to the cloze types in Table 1, we see an analogy between the lexical cloze type and phrase multiple choice. These two types involve understanding what words (or phrases) are reasonable in the context from the set of wordpieces (or possible spans). The memorized knowledge cloze type contrasts with the suggestive or justified inference span selection types. Because a suggestive or justifying passage is present, the model is trained only to understand language, rather than memorize facts. Simple syntactic instances are largely eliminated because closed class words are not possible answer terms. Also, since answer terms are expanded to the longest shared subsequence between query and passage, collocation instances are not a concern.
4.2 Extended Pre-training
Rather than training a transformer architecture from scratch, we initialize from the pre-trained BERT models (Devlin et al., 2018) and extend the pre-training with the span selection auxiliary task. We refer to the resulting models as  $\mathrm{BERT}_{\mathrm{BASE}} + \mathrm{SSPT}$  (Span Selection Pre-Training) and  $\mathrm{BERT}_{\mathrm{LARGE}} + \mathrm{SSPT}$ . We used batch sizes of 256, and a learn rate of 5e-5. All models were trained over 100 million span selection instances. We found continued improvement from 50 million to 100 million and have not yet tried larger pre-training runs. Unlike the efforts of XLNet or RoBERTa which increased training by a factor of ten relative to BERT, the additional data in SSPT represents less than a  $40\%$  increase in the pre-training of the transformer. This pre-training is also done over Wikipedia, adding no new text to the pre-training.
Figure 3 illustrates the adaptation of BERT to SSPT. The query and passage are concatenated
Figure 3: BERT for QA with is-possible prediction
in the standard two sequence representation, with a preceding [CLS] token and a separating [SEP] token, producing a sequence of tokens  $T$ . BERT produces output vectors for these tokens to obtain a sequence  $\{v_i\}_{i=1}^{|T|}$  of  $d$  dimensional vectors.
In span selection extended pre-training, we alter the vocabulary of the tokenizer, introducing the new special token: '[BLANK]'. We use the BertForQuestionAnswering $^4$  model, which uses a pointer network to find the answer location. The pointer network applies a simple fully connected network to predict the probability of start and end span pointers at each token position, using the output of the final transformer layer at that position. The loss in training is the cross entropy of these predictions with the true positions of the start and end.
Formally, The start of the answer span is predicted as  $p(i = \langle start \rangle) = \text{softmax}(\pmb{w}_{\langle start \rangle}^{\top} \pmb{v} + b_{\langle start \rangle})_i$ , where  $\pmb{w}_{\langle start \rangle} \in \mathbb{R}^d$ ,  $b_{\langle start \rangle} \in \mathbb{R}$  are trainable parameters. Then end of the span is predicted the same way:  $p(i = \langle end \rangle) = \text{softmax}(\pmb{w}_{\langle end \rangle}^{\top} \pmb{v} + b_{\langle end \rangle})_i$ .
Span selection pre-training may optionally include a classifier for answerability. If the answerability classifier is included in the pre-training then the presence of the answer span in the passage is predicted with probability given by:  $p(\text{possible}) = \text{sigmoid}(\boldsymbol{w}_{CLS}^{\top}\boldsymbol{v}_{CLS} + b_{CLS})$ . If it is not included, for impossible instances the target prediction is for both start and end to be position zero, the [CLS] token. We train models for QA without the answerability classifier for 100 million instances. This took approximately seven days on 16 P100 GPUs.
Training data and code to extend pre-training is available as open source<sup>5</sup>.
5 MRC Tasks
We follow previous work and evaluate our SSPT architecture on several downstream tasks. Our primary motivation is to improve question answering by improving the pre-trained language model. Our QA benchmarks are the following:
1. Stanford Question Answering Dataset (SQuAD) v1.1 (Rajpurkar et al., 2016) and v2.0 (Rajpurkar et al., 2018)
https://github.com/IBM/ span-selection-pretraining
Dataset	Context	Answer Types	Question Creation	Training Size	Dev Size	Test Size	Gap to Human Performance†
SQuAD 1.1	passage	span	generated	88k	11k	10k	< 0%
SQuAD 2.0	passage	span, impossible	generated	130k	12k	9k	< 0%
Natural Questions	document	span, yes, no, impossible	natural	307k	8k	8k	15%
HotpotQA	passage set	span, yes, no	generated	91k	7k	7k	8%
Table 3: Comparison of QA Datasets.
†As of Dec. 2019
2. Natural Questions (NQ) (Kwiatkowski et al., 2019)
3. HotpotQA (Yang et al., 2018)
The three datasets provide different characteristics of question answering and machine reading comprehension tasks as well as an opportunity to compare results with active leaderboards. Table 3 provides a summary comparison. We briefly discuss them here:
5.1 SQuAD
SQuAD provides a paragraph of context and asks several questions about it. The task is extractive QA where the system must find the span of the correct answer from the context. We evaluate on two versions of SQuAD: v1.1 and v2.0. In v1.1 the context always contains an answer. However, in v2.0 the task contains additional questions to which the given context does not have the correct answer.
Just as in Figure 3, the question and passage are concatenated with the separators ([CLS] and [SEP]) to form the input to the pre-trained BERT. The final token representations are then used to predict the probability for each token that it is the start or end of the answer span. The span with the highest predicted probability is then the predicted answer.
5.2 Natural Questions
NQ is a dataset of over 300,000 queries sampled from live users on the Google search engine for which a Wikipedia article is contained in the top ranking search results. Crowd sourced annotators are then tasked with highlighting a short answer span to each question<sup>6</sup>, if available, from the
Wikipedia article as well as a long answer span (which is generally the most immediate HTML paragraph, list, or table span containing the short answer span), if available.
Similar to SQuAD 2.0 the NQ dataset forces models to make an attempt at "knowing what they don't know" in order to detect and avoid providing answers to unanswerable questions. In addition, the fact that the questions were encountered naturally from actual users removes some of the observational bias that appears in the artificially created SQuAD questions. Both these aspects along with the recency of the task's publication means that this is still a challenging task with lots of headroom between human performance and the best performing automated system.