SlowGuess/ACL_2020_OCR huggingface dataset:
Experiments on the NQ dataset use the strategies and model described by Alberti et al. (2019b) to fine tune a  $\mathrm{BERT}_{\mathrm{LARGE}}$  model with a final layer for answerability prediction as well as sequence start/end prediction. Similar to their best performing systems, the model is first trained using the SQuAD v1.1 data set and then subsequently trained on the NQ task<sup>7</sup>. The hyperparameters follow Alberti et al. (2019b) with the exception of learning rate and batch size which are chosen according to the approach outlined by Smith (2018) using a  $20\%$  sub-sample of the data for each experimental setting.
5.3 HotpotQA
Recently, Yang et al. (2018) released a new dataset, called HotpotQA, for the task of reading compre
their small proportion, the models in this paper do not produce Yes/No answers
7Skipping the SQuAD v1.1 fine-tuning step for the NQ task leads to the same conclusions with respect to SSPT pre-training, but decreases the overall performance for both  $\mathrm{BERT}_{\mathrm{LARGE}}$  and  $\mathrm{BERT}_{\mathrm{LARGE}} + \mathrm{SSPT}$
Around  $1\%$  of the questions are answered as a simple Yes or No rather than a span of short answer text. Due to
Method	SQUAD 1.1	SQUAD 2.0
F1	Exact	F1	Exact
BERTBASE	88.52	81.22	76.45	73.29
+SSPT	91.71	85.10	82.31	79.19
+SSPT-PN	91.60	84.94	82.34	79.32
BERTLARGE	90.97	84.20	81.50	78.41
+SSPT	92.75	86.86	85.03	82.07
Table 4: Dev Set Results on SQuAD
Method	Short Ans F1	Long Ans F1
BERTBASE	47.27	61.02
+SSPT	50.40	63.35
BERTLARGE	52.7	64.7
+SSPT	54.2	65.85
hension style extractive QA. Each training instance in the distractor setting of this dataset comprises a question, a set of ten passages, an answer, and a binary label for each sentence in the passage-set stating whether that sentence serves as a supporting fact (or not) to arrive at the correct answer. The task is to predict both the correct answer as well as the supporting facts for any given test instance. The signature characteristic of this dataset lies in the fact that each question requires a minimum of two supporting facts from two different passages in order to derive its correct answer. Thus, this dataset tests the cross-passage, multi-hop reasoning capability of a reading comprehension based question answering system.
Our system for HotpotQA uses a three-phase approach. First, representations of the individual passages are built with a pre-trained transformer encoder. Second, interactions between these passages are attended to using a relatively shallow global transformer encoder. The supporting facts are predicted from the sentence representations produced by this global layer. Finally, the predicted supporting facts are then merged into a pseudo-passage that is used by a slightly altered version of the model for SQuAD. The one addition is that this model also predicts an answer-type ( $\{yes, no, span\}$ ) from the [CLS] token vector.
Table 5: Dev Set Results on Natural Questions
Method	Facts	Answer
F1	Exact	F1	Exact
BERTBASE	84.00	53.15	73.86	59.97
+SSPT	85.13	56.58	77.25	63.31
BERTLARGE	85.27	55.99	75.48	61.62
+SSPT	86.17	57.57	79.39	65.87
Table 6: Dev Set Results on HotpotQA
Figure 4: Learning curve improvement for BERTLarge with SSPT
Tables 4, 5, and 6 show our results on the development set with extended span selection pre-training for BERT relative to the pre-trained BERT. We use the same hyperparameters on these tasks as the original BERT. The best results for each dataset are in bold when significant relative to the BERT baseline. The four question answering datasets are improved substantially with span selection pre-training.
6.1 SQuAD
Relative to  $\mathrm{BERT}_{\mathrm{BASE}}$  we find a 3 point improvement in F1 for SQuAD 1.1 and a nearly 6 point improvement for SQuAD 2.0. In terms of error rate reduction the improvement is similar,  $28\%$  and  $25\%$  respectively. The error rate reduction for  $\mathrm{BERT}_{\mathrm{LARGE}}$  is  $20\%$  and  $19\%$  for SQuAD 1.1 and 2.0 respectively.
In reading comprehension tasks, the pointer network for answer selection is pre-trained through the span selection task. We measure how much of the improvement is due to this final layer pre-training versus the extended pre-training for the transformer
encoder layers by discarding the pre-trained pointer network and randomly initializing. This configuration is indicated as  $\mathrm{BERT}_{\mathrm{BASE}} + \mathrm{SSPT - PN}$ . Surprisingly, the pre-training of the pointer network is not a significant factor in the improved performance on reading comprehension, indicating the improvement is instead coming through a better language understanding in the transformer.
Figure 4 shows the improvement from SSPT on SQuAD 1.1 and 2.0 as the amount of training data increases. While there is significant improvement at  $100\%$  training, the improvement is even more pronounced with less training data. We hypothesize that this is due to the close connection of span selection pre-training with reading comprehension. This effect is strongest for SQuAD 1.1, which like span selection pre-training always contains a correct answer span in the passage.
6.2 Natural Questions
The work of Alberti et al. (2019a), which gets the  $\mathrm{BERT}_{\mathrm{LARGE}}$  performance listed in Table 5, is the highest ranking single model submission that does not use data augmentation with a published paper. Our implementation of  $\mathrm{BERT}_{\mathrm{LARGE}} + \mathrm{SSPT}$ , therefore, provides a  $1.5\%$  improvement over the best BERT-for-QA model performance that we are aware of on the NQ data set. In future work, we intend to explore data augmentation on top of  $\mathrm{BERT}_{\mathrm{LARGE}} + \mathrm{SSPT}$  for further improvements.
6.3 HotpotQA
In HotpotQA, unlike the other QA datasets, multiple passages are provided. We use the BERT transformer in two places, for supporting fact prediction to build the representations of each passage, and in answer prediction as in the other QA tasks. We find the most substantial gains of almost 4 F1 points for answer selection, the QA task most similar to span selection pre-training. Interestingly, we also find improvement of almost one point F1 in supporting fact prediction, demonstrating that the learned representations can generalize well to multiple QA sub-tasks.
HotpotQA also comes with its own leaderboard (https://hotpotqa.github.io/). A good number of submissions on this leaderboard are based on BERT<sub>BASE</sub> or BERT<sub>LARGE</sub>. We made an initial submission to this leaderboard, called TAP, which occupied Rank-5 at the time of submission and the underlying architecture employed BERT<sub>BASE</sub>. Next, we replaced BERT<sub>BASE</sub> with BERT<sub>LARGE</sub>+SSPT,
Model	Passage	F1	Exact
BERTBASE+SSPT	Related	62.88	49.27
BERTBASE+SSPT	Unrelated	46.51	34.32
BERTLARGE+SSPT	Related	65.39	51.82
BERTLARGE+SSPT	Unrelated	50.98	38.97
Table 7: Comparison of performance of SSPT for related vs. unrelated passages
calling that model TAP-2. This change resulted in a  $7.22\%$  absolute gain in the Joint  $F1$  score. An ensemble version of TAP-2 further offered a gain of  $1.53\%$ . The SSPT augmented TAP-2 (ensemble) and TAP-2 (single model) achieved Rank-1 and Rank-2 on the leaderboard at the time of submission.
6.4 Exploration of SSPT Instance Types
In section 4.1 we enumerated three types of span selection instances. The first type, Phrase Multiple Choice, is the least interesting since the semantic correspondence between the query and the passage is not used. Instead, the instance is treated as a cloze with options provided as spans in the passage. Note that in this type of instance the relevance of the passage to the query is not important.
To explore how frequent this case might be we select 100 thousand new SSPT instances with a relevant passage and for each select an alternative, random, answer-bearing, passage. The unrelated passage is from a document different both from the query's document and from the relevant passage's document. We then apply the SSPT trained model to the instances both with the related and unrelated passage and evaluate its performance in terms of token-level F1 and exact span match.
Table 7 show the performance of our SSPT trained models on the SSPT queries with related vs. unrelated passages. The large accuracy gains when using relevant passages imply that for many passages "Phrase Multiple Choice" is not the method used by the model. Instead, the semantic connection of the passage to the query is used to select the appropriate span.
6.5 Comparison to Previous Work
We also compare our span selection pre-training data with the data distributed by Dhingra et al. (2018). This data consists of approximately 2 million instances constructed using the abstract and body structure of Wikipedia. In contrast, our ap
proach to pre-training can generate data in unlimited quantity from any text source without assuming a particular document structure. When only one million training steps are used, both sources of pre-training are equally effective. But when moving to ten million steps of training, our data produces models that give over one percent better F1 on both SQuAD 1.1 and 2.0. This suggests the greater quantity of data possible through SSPT is a powerful advantage.
Span selection pre-training is effective in improving reading comprehension across four diverse datasets, including both generated and natural questions, and with provided contexts of passages, documents and even passage sets. This style of pretraining focuses the model on finding semantic connections between two sequences, and supports a style of cloze that can train deep semantic understanding without demanding memorization of specific knowledge in the model. The span selection task is suitable for pre-training on any domain, since it makes no assumptions about document structure or availability of summary/article pairs. This allows pre-training of language understanding models in a very generalizable way.
In future work, we will address end-to-end question answering with pre-training for both the answer selection and retrieval components. We hope to progress to a model of general purpose language modeling that uses an indexed long term memory to retrieve world knowledge, rather than holding it in the densely activated transformer encoder layers.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019a. Synthetic QA corpora generation with roundtrip consistency. CoRR, abs/1906.05416.
Chris Alberti, Kenton Lee, and Michael Collins. 2019b. A bert baseline for the natural questions.
Rie Kubota Ando and Tong Zhang. 2005. A framework for learning predictive structures from multiple tasks and unlabeled data. Journal of Machine Learning Research, 6(Nov):1817-1853.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. arXiv preprint arXiv:1810.04805.
Bhuwan Dhingra, Danish Danish, and Dheeraj Rajagopal. 2018. Simple and effective semi-supervised question answering. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers), volume 2, pages 582-587.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In C. Cortes, N. D. Lawrence, D. D. Lee, M. Sugiyama, and R. Garnett, editors, Advances in Neural Information Processing Systems 28, pages 1693-1701. Curran Associates, Inc.
Jeremy Howard and Sebastian Ruder. 2018. Universal language model fine-tuning for text classification. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 328-339. Association for Computational Linguistics.
Mandar Joshi, Danqi Chen, Yinhan Liu, Daniel S Weld, Luke Zettlemoyer, and Omer Levy. 2019. Spanbert: Improving pre-training by representing and predicting spans. arXiv preprint arXiv:1907.10529.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, Kenton Lee, Kristina N. Toutanova, Llion Jones, Ming-Wei Chang, Andrew Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: a benchmark for question answering research. TACL.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized BERT pretraining approach. CoRR, abs/1907.11692.
Takeshi Onishi, Hai Wang, Mohit Bansal, Kevin Gimpel, and David McAllester. 2016. Who did what: A large-scale person-centered cloze dataset. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2230-2235.
Matthew Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Kenton Lee, and Luke Zettlemoyer. 2018. Deep contextualized word representations. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 2227-2237. Association for Computational Linguistics.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, and Ilya Sutskever. 2019. Language models are unsupervised multitask learners. OpenAI Preprint.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. arXiv preprint arXiv:1806.03822.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad: 100,000+ questions for machine comprehension of text. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383-2392.
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gatford, et al. 1995. Okapi at trec-3. Nist Special Publication Sp, 109:109.
Mrinmaya Sachan and Eric Xing. 2018. Self-training for jointly learning to ask and answer questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 629-640. Association for Computational Linguistics.
Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc V. Le, Geoffrey E. Hinton, and Jeff Dean. 2017. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer. CoRR, abs/1701.06538.
Leslie N. Smith. 2018. A disciplined approach to neural network hyper-parameters: Part 1 - learning rate, batch size, momentum, and weight decay.
Yu Sun, Shuohuan Wang, Yukun Li, Shikun Feng, Hao Tian, Hua Wu, and Haifeng Wang. 2019. Ernie 2.0: A continual pre-training framework for language understanding. arXiv preprint arXiv:1907.12412.
Wilson L Taylor. 1953. "Cloze procedure": A new tool for measuring readability. Journalism Bulletin, 30(4):415-433.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In I. Guyon, U. V. Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems 30, pages 5998-6008. Curran Associates, Inc.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. CoRR, abs/1906.08237.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William W. Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the Conference on Empirical Methods in Natural Language Processing (EMNLP).
Yukun Zhu, Ryan Kiros, Rich Zemel, Ruslan Salakhutdinov, Raquel Urtasun, Antonio Torralba, and Sanja Fidler. 2015. Aligning books and movies: Towards story-like visual explanations by watching movies and reading books. In Proceedings of the IEEE international conference on computer vision, pages 19-27.