SlowGuess/ACL_2020_OCR huggingface dataset:
As discussed in § 2, we generate multiple document spans by splitting the Wikipedia page with a sliding window. Following (Pan et al., 2019; Alberti et al., 2019b), the size and stride of the sliding window are set to 512 and 192 tokens respectively. The average number of document spans of one Wikipedia page is about 22. Since most of the document span does not contain the answer, the number of negative samples (i.e., no answer) and positive samples (i.e., has answers) is extremely imbalanced. We follow (Pan et al., 2019; Alberti et al., 2019b) to sub-sample negative instances for training, where the rate of sub-sampling negative instance is the same as in (Pan et al., 2019). As a result, there are 469,062 training instances in total.
We use Adam optimizer (Kingma and Ba, 2015) with a batch size of 36 for model training. The initial learning rate, the learning rate warmup proportion, the training epoch, the hidden size  $h$ , the number of blocks  $T$ , and the hyperparameter  $K$  are set to  $2 \times 10^{-5}$ , 0.1, 2, 1024, 2, and 256 respectively. Our model takes approximately 24 hours to train with 4 Nvidia Tesla P40. Evaluation completed in about 6 hours on the NQ dev and test set with a single Nvidia Tesla P100.
We use the Google released BERT-large model fine-tuned with synthetic self-training (Alberti et al., 2019a) to encode the document and question as described in § 3.1.1. We also compare the performance of RikiNet which uses the pre-trained RoBERTa large model (Liu et al., 2019). It should be noted that our RikiNet is orthogonal to the choice of a particular pre-trained language model.
We present a comparison between previously published works on the NQ task and our RikiNet. We report the results of the precision (P), the recall (R), and the F1 score for the long-answer (LA) and short-answer (SA) tasks on both test set and dev set in Tab. 1. The first two lines of Tab. 1 show the results of two multi-passage MRC baseline models presented in the original NQ paper (Kwiatkowski et al., 2019). The third to sixth lines show the results of the previous state-of-the-art models. These models all employ the BERTlarge model and perform better than that two baselines. Our RikiNet-BERTlarge also employs the BERTlarge model, and its single model has achieved a significant improvement over the previously published best model on the test set (LA from 66.8 F1 to 74.3 F1, and SA from 53.9 F1 to 57.9 F1). To the best of our knowledge, this is the first<sup>4</sup> single model that surpasses the single human performance (Kwiatkowski et al., 2019) on both LA and SA tasks. We also provide a BERTjoint (Alberti et al., 2019b) + RoBERTa large (Liu et al., 2019) baseline on NQ, which only replaces the BERTlarge in BERTjoint method with RoBERTa large. To be expected, the BERTjoint + RoBERTa large performs better than original BERTjoint. Furthermore, our single model of RikiNet-RoBERTa large which employs RoBERTa large model also achieves better performance on both LA and SA, significantly outperforming BERTjoint + RoBERTa large. These results demonstrate the effectiveness of our RikiNet.
Since most submissions on the NQ leaderboard are ensemble models, we also report the results of our ensemble model, which consists of three RikiNet-RoBERTa large models with different hyper-parameters. At the time of submission (29 Nov. 2019), the NQ leaderboard shows that our ensemble model achieves the best performance on both LA (F1 76.1) and SA (F1 61.3).
4The single RikiNet-BERTlarge model was submitted to the NQ public leaderboard on 7 Nov. 2019.
LA Dev	LA Test	SA Dev	SA Test
P	R	F1	P	R	F1	P	R	F1	P	R	F1
DocumentQA (Clark and Gardner, 2018)	47.5	44.7	46.1	48.9	43.3	45.7	38.6	33.2	35.7	40.6	31.0	35.1
DecAtt (Parikh et al., 2016) + DocReader (Chen et al., 2017)	52.7	57.0	54.8	54.3	55.7	55.0	34.3	28.9	31.4	31.9	31.1	31.5
BERTjoint (Alberti et al., 2019b)	61.3	68.4	64.7	64.1	68.3	66.2	59.5	47.3	52.7	63.8	44.0	52.1
BERTlarge + 4M synth NQ (Alberti et al., 2019a)	62.3	70.0	65.9	65.2	68.4	66.8	60.7	50.4	55.1	62.1	47.7	53.9
BERTjoint (Alberti et al., 2019b) + RoBERTa large (Liu et al., 2019)‡	65.6	69.1	67.3	-	-	-	60.9	51.0	55.5	-	-	-
BERTlarge + SQuAD2 PT + AoA (Pan et al., 2019)†	-	-	68.2	-	-	-	-	-	57.2	-	-	-
BERTlarge + SSPT (Glass et al., 2019)†	-	-	65.8	-	-	-	-	-	54.2	-	-	-
RikiNet-BERTlarge	73.2	74.5	73.9	74.2	74.4	74.3	61.1	54.7	57.7	63.5	53.2	57.9
RikiNet-RoBERTa large‡	74.3	76.4	75.3	-	-	-	61.4	57.3	59.3	-	-	-
RikiNet-BERTlarge (ensemble)	74.4	76.3	75.4	75.3	75.9	75.6	66.9	53.8	59.6	63.2	56.1	59.5
RikiNet-RoBERTa large (ensemble)	73.3	78.7	75.9	78.1	74.2	76.1	66.6	56.4	61.1	67.6	56.1	61.3
Single Human (Kwiatkowski et al., 2019)	80.4	67.6	73.4	-	-	-	63.4	52.6	57.5	-	-	-
Super-annotator (Kwiatkowski et al., 2019)	90.0	84.6	87.2	-	-	-	79.1	72.6	75.7	-	-	-
RikiNet consists of two key parts: DPDA reader and multi-level cascaded answer predictor. To get a better insight into RikiNet, we conduct an in-depth ablation study on probing these two modules. We report the LA and SA F1 scores on the dev set.
Ablations of DPDA Reader We keep the predictor and remove the component of the DPDA reader. The results are shown in Tab. 2. In (a), we remove the entire DPDA reader as introduced in § 3.1 except  $\mathrm{BERT}_{\text {large }}$ . In (b), (c), and (d), we remove the dual-attention layer, question self-attention layer, and paragraph dynamic self-attention layer as described in § 3.1.1 respectively. In (e) and (f), we remove the paragraph attention mask of Eq. (3) and the dynamic attention mask of Eq. (4) respectively. We can see that after removing the DPDA reader, the performance drops sharply. In addition, the paragraph dynamic self-attention layer has the greatest impact on performance. Moreover, both the paragraph attention mask and dynamic attention mask contribute to the performance improvement.
We also change the hyper-parameter  $K$  and the number of blocks  $T$ . Results show that the setting of  $K = 384$  performs better than  $K = 512$  (i.e., no dynamic attention mask), and  $K = 256$  performs best. For the number of DPDA blocks  $T$ , the model achieves the best performance when  $T = 2$ .
Ablations of Predictor On the predictor side, we further remove or replace its component and report the results in Tab. 3. In (1) we remove the whole DPDA reader and predictor. In (2), we re
Table 1: Performance comparisons on the dev set and the blind test set of the NQ dataset. We report the evaluation results of the precision (P), the recall (R), and the F1 score for both long-answer (LA) and short-answer (SA) tasks. We use background color to highlight the column of F1 results. † refers to the works that only provide the F1 results on the dev set in their paper. ‡ refers to our implementations where we only report the results on the dev set, due to the NQ leaderboard submission rules (each participant is only allowed to submit once per week).
Setting	LA F1	SA F1
RikiNet-BERTlarge (Full)	73.9	57.7
(a) - DPDA reader	70.7	55.9
(b) - Dual-attention layer	73.1	56.6
(c) - Question self-attention layer	73.5	57.5
(d) - Paragraph self-attention layer	72.2	56.3
(e) - Paragraph attention mask	73.2	57.1
(f) - Dynamic attention mask	72.9	56.8
RikiNet-BERTlarge (K=512)	72.9	56.8
RikiNet-BERTlarge (K=384)	73.7	57.3
RikiNet-BERTlarge (K=256)	73.9	57.7
RikiNet-BERTlarge (K=128)	73.7	56.9
RikiNet-BERTlarge (T=0)	70.7	55.9
RikiNet-BERTlarge (T=1)	73.6	57.6
RikiNet-BERTlarge (T=2)	73.9	57.7
RikiNet-BERTlarge (T=3)	73.5	57.1
RikiNet-BERTlarge (T=4)	73.0	56.9
Table 2: Ablations of DPDA reader on dev set of NQ dataset.
move the way of multi-level prediction (i.e., training the model to predict long and short answer jointly) described in § 3.2, and follow the previous work (Alberti et al., 2019b) to directly predict the short answer and then select its paragraph as the long answer. We can see that our multi-level prediction is critical to the long answer prediction. In (3) we only remove the cascaded structure but keep the multi-level prediction, which means that the prediction representations are no longer used as input for other predictions, the performance of both long and short answers drops about 1.0 F1 score. In (4) we change the ordering of cascaded process. That is instead of considering long an-
6768
Setting	LA F1	SA F1
RikiNet-BERTlarge (Full)	73.9	57.7
(1) - DPDA reader & Predictor	65.9	55.1
(2) - Multi-level prediction	70.9	57.1
(3) - Cascaded structure	73.0	56.7
(4) + S2L cascaded structure	73.6	57.5
(5) - Question embedding	73.4	57.4
(6) - Tanh dense prediction layer	73.2	57.3
(7) + Bi-LSTM prediction layer	73.3	57.4
(8) + Transformer prediction layer	73.5	57.5
(9) + GELU dense prediction layer	73.7	57.6
Table 3: Ablations of multi-level cascaded predictor on dev set of NQ dataset.
swer first and then short answer as described in § 3.2, we consider the cascaded structure of short answer first and then long answer. However, we get slightly worse results in this way. In (5), we remove the question embedding which is used for answer type prediction. It can be observed that the question embedding contributes to performance improvement. In the variants of (6)-(9), we remove the dense prediction layers with Tanh activation function and replace it with Bi-directional Long-Short Term Memory (Bi-LSTM) (Hochreiter and Schmidhuber, 1997; Schuster and Paliwal, 1997) layers, transformer self-attention blocks, and dense prediction layers with Gaussian Error Linear Unit GELU (Hendrycks and Gimpel, 2016) activation function but neither get better performance.
Overall, both proposed DPDA reader and multi-level cascaded answer predictor significantly improve the model performance.
Natural Questions (NQ) dataset (Kwiatkowski et al., 2019) has been recently proposed, where each question is paired with an entire Wikipedia page which is a long document containing multiple passages. Although BERT (Devlin et al., 2018) based MRC models have surpassed human performance on several MRC benchmark datasets (Lan et al., 2019; Devlin et al., 2018; Liu et al., 2019; Rajpurkar et al., 2018), a similar BERT method (Alberti et al., 2019b) still has a big gap with human performance on NQ dataset.
There are several recently proposed deep learning approaches for multi-passage reading comprehension. Chen et al. (2017) propose DrQA which contains a document retriever and a document reader (DocReader). Clark and Gardner (2018) in
troduce Document-QA which utilizes TF-IDF for paragraph selection and uses a shared normalization training objective. De Cao et al. (2019) employ graph convolutional networks (GCNs) for this task. Zhuang and Wang (2019) design a gated token-level selection mechanism with a local convolution. In contrast, our RikiNet considers multi-level representations with a set of complementary attention mechanisms.
To solve the NQ task, Kwiatkowski et al. (2019) adapt Document-QA (Clark and Gardner, 2018) for NQ, and also utilizes DecAtt (Parikh et al., 2016) for paragraph selection and DocReader (Chen et al., 2017) for answer prediction. BERT $_{\text{joint}}$  (Alberti et al., 2019b) modifies BERT for NQ. Besides, some works focus on using data augmentation to improve the MRC models on NQ. Alberti et al. (2019a) propose a synthetic QA corpora generation method based on roundtrip consistency. Glass et al. (2019) propose a span selection method for BERT pre-training (SSPT). More recently, Pan et al. (2019) introduce attention-over-attention (Cui et al., 2017) into the BERT model. Pan et al. (2019) also propose several techniques of data augmentation and model ensemble to further improve the model performance on NQ. Although the use of data augmentation and other advanced pre-trained language models (Lan et al., 2019) may further improve model performance, as this is not the main focus of this paper, we leave them as our future work. Our RikiNet is a new MRC model designed tailored to the NQ challenges and can effectively represent the document and question at multi-levels to jointly predict the answers, which significantly outperforms the above methods.
We propose the RikiNet, which reads the Wikipedia pages to answer the natural question. The RikiNet consists of a dynamic paragraph dual-attention reader which learns the token-level, paragraph-level and question representations, and a multilevel cascaded answer predictor which jointly predicts the long and short answers in a cascade manner. On the Natural Questions dataset, the RikiNet is the first single model that outperforms the single human performance. Furthermore, the RikiNet ensemble achieves the new state-of-the-art results at 76.1 F1 on long-answer and 61.3 F1 on short-answer tasks, which significantly outperforms all the other models on both criteria.
6769
This work is supported by National Natural Science Fund for Distinguished Young Scholar (Grant No. 61625204) and partially supported by the Key Program of National Science Foundation of China (Grant No. 61836006).
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019a. Synthetic qa corpora generation with roundtrip consistency. arXiv preprint arXiv:1906.05416.
Chris Alberti, Kenton Lee, and Michael Collins. 2019b. A bert baseline for the natural questions. arXiv preprint arXiv:1901.08634.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Benjamin Borschinger, Jordan Boyd-Graber, Christian Buck, Jannis Bulian, Massimiliano Ciaramita, Michelle Chen Huebscher, Wojciech Gajewski, Yannic Kilcher, Rodrigo Nogueira, and Lierni Sestorain Saralegu. 2019. Meta answering for machine reading. arXiv:1911.04156.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In ACL.
Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In ACL.
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In ACL.
Nicola De Cao, Wilker Aziz, and Ivan Titov. 2019. Question answering by reasoning across documents with graph convolutional networks. In NAACL-HLT.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2018. Bert: Pre-training of deep bidirectional transformers for language understanding. In *NAACL*.
Michael Glass, Alfio Gliozzo, Rishav Chakravarti, Anthony Ferritto, Lin Pan, GP Bhargav, Dinesh Garg, and Avirup Sil. 2019. Span selection pretraining for question answering. arXiv preprint arXiv:1909.04120.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep residual learning for image recognition. In CVPR.
Dan Hendrycks and Kevin Gimpel. 2016. Gaussian error linear units (gelus). arXiv preprint arXiv:1606.08415.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation.
Ying Ju, Fubang Zhao, Shijie Chen, Bowen Zheng, Xuefeng Yang, and Yunfeng Liu. 2019. Technical report on conversational question answering. arXiv preprint arXiv:1909.10772.
Diederik P Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In ICLR.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, et al. 2019. Natural questions: a benchmark for question answering research. TACL, 7:453-466.
Zhenzhong Lan, Mingda Chen, Sebastian Goodman, Kevin Gimpel, Piyush Sharma, and Radu Soricut. 2019. Albert: A lite bert for self-supervised learning of language representations. arXiv preprint arXiv:1909.11942.
Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach. arXiv preprint arXiv:1907.11692.
Lin Pan, Rishav Chakravarti, Anthony Ferritto, Michael Glass, Alfio Gliozzo, Salim Roukos, Radu Florian, and Avirup Sil. 2019. Frustratingly easy natural question answering. arXiv preprint arXiv:1909.05286.
Ankur P Parikh, Oscar Tackström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In EMNLP.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In ACL.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. Squad:  $100,000+$  questions for machine comprehension of text. In EMNLP.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2019. Coqa: A conversational question answering challenge. TACL, 7:249-266.
Mike Schuster and Kuldip K Paliwal. 1997. Bidirectional recurrent neural networks. IEEE Transactions on Signal Processing.
Minjoon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In ICLR.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In NIPS.
6770
Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In ICLR.
Caiming Xiong, Victor Zhong, and Richard Socher. 2018. Dcn+: Mixed objective and deep residual coattention for question answering. In ICLR.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Ruslan Salakhutdinov, and Quoc V Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. arXiv preprint arXiv:1906.08237.
Zhuosheng Zhang, Yuwei Wu, Junru Zhou, Sufeng Duan, and Hai Zhao. 2019. Sg-net: Syntax-guided machine reading comprehension. arXiv preprint arXiv:1908.05147.
Yimeng Zhuang and Huadong Wang. 2019. Token-level dynamic self-attention network for multipassage reading comprehension. In ACL.
6771