SlowGuess/ACL_2020_OCR huggingface dataset:
Since the fact that only a small fraction of generated instances are tagged as positive instances which contains a complete span of long or short answer, and that  $51\%$  of the documents do not contain the answers for the questions, We downsample about  $97\%$  of null instances to get about 660,000 training instances in which 350,000 has a long answer, and 270,000 has short answers.
We use three model settings for our experiments, which are: 1) Model-I: A refined BERT baseline on the basis of Alberti et al. (2019b); 2) Model-II: A pipeline model with only graph initialization method to get representation of sentence, paragraph, and document; 3) Model-III: Adding two layers of our graph encoder on the basis of Model-II.
Model-I improves the baseline in Alberti et al. (2019b) in two ways: 1) When training an instance with a long answer only, we ignore the loss of predicting the short answer span to "no-answer" because it would introduce distraction to the model. 2) We sample more negative instances.
We use three BERT encoders to initialize our token node representation: 1) BERT-base: a
6713
Model	LA. F1	SA. F1
BERT-base+Model-I	63.9	51.0
BERT-base+Model-II	67.7	50.9
BERT-base+Model-III	68.9	51.9
BERTjoint	64.7	52.7
BERT-large+Model-I	66.0	52.9
BERT-large+Model-II	70.3	53.2
BERT-large+Model-III	70.7	53.8
BERT-syn+Model-I	67.8	56.1
BERT-syn+Model-II	72.2	56.7
BERT-syn+Model-III	72.7	56.9
BERT-base-uncased model finetuned on SQuAD 2.0; 2) BERT-large: a BERT-large-uncased model finetuned on SQuAD 2.0; 3) BERT-syn: Google's BERT-large-uncased model pre-trained on SQuAD2.0 with N-Gram Masking and Synthetic Self-Training. Since the Natural Question dataset does not provide sentence-level information, we additionally use spacy (Honnibal and Montani, 2017) as the sentence segmentor to get the boundaries of sentences.
We trained the model by minimizing loss  $L$  from Section 3.4 using the Adam optimizer (Kingma and Ba, 2015) with a batch size of 32. We trained our model for 2 epochs with an initial learning rate of  $2 \times 10^{-5}$ , and we use a warmup proportion of 0.1. The training of our proposed model is conducted on 4 Tesla P40 GPUs for approximately 2 days. For each setting, the results are averaged over three models initialized with different random seeds to get a more solid comparison, which also suggests the improvements brought by our methods are relatively stable. The hidden size, the number of attention heads, and the dropout rate in our graph encoder are equal to the values in the corresponding BERT model.
4.3 Comparison
The main results are shown in Table 1. The results show that our best model BERT-syn+Model-III(ensemble 3 models) have gained improvement over the previous models by a large margin. Our ensemble strategy is to train three models with different random seeds. The scores of answer candi
Table 2: Comparison of different models with different BERT models on the development dataset.
Model	LA.F1	SA.F1
0-layer	67.7	50.9
1-layer	68.8	51.2
2-layer	68.9	51.9
3-layer	68.9	51.9
4-layer	68.9	51.7
Table 3: Influences of graph layer numbers on the development set.
dates are averaged over these three models. At the time of submission (Jun. 25th, 2019), this model has achieved the state-of-the-art performance on both long answer (F1 score of  $74.5\%$ ) and short answer (F1 score of  $58.7\%$ ) on the public leaderboard<sup>3</sup>. Furthermore, our model surpasses single human performance at both long and short answer criteria on the development dataset.
The comparison of different models with different BERT models is illustrated in Table 2. The results show that our approach significantly outperforms our baseline model on both the long answer and the short answer. For the BERT-base setting, our Model-II with a pipeline inference strategy outperforms our baseline by  $3.8\%$  on long answer F1 score while our Model-II with two graph layers further improves the performance by  $1.2\%$  and  $1.0\%$ . For the BERT-syn setting, the Model-III benefits less from the graph layers because the pretraining for this model is already quite strong. Our Model-III with BERT-large, compared to previously public model  $(\mathrm{BERT}_{\mathrm{joint}})$  also using BERT-large, improves long answer F1 score by  $6.0\%$  and short answer F1 score by  $1.1\%$  on the development set.
From Table 1 and Table 2, we can see that the ensemble of human annotators can lead to a massive improvement at both long and short answer criteria (from  $73.4\%$  to  $87.2\%$ ,  $57.5\%$  to  $75.7\%$ ). However, the improvement of assembling our BERT-based model is relatively smaller (from  $72.7\%$  to  $73.9\%$ ,  $56.9\%$  to  $59.1\%$ ). This suggests that the diversity of human annotators is a lot better than the same model structure with different random seeds. How to improve the diversity of the deep learning models for the open-domain datasets like NQ remains as a hard question.
<sup>3</sup>Since we can only make 10 submissions on the test dataset, we only submit and report the result of our best model. Due to the official attempts on the test dataset are given 24 hours. We can only ensemble 3 models at most.
2This model can be downloaded at https://bit.ly/ 2w7nUQK.
6714
Model	LA. F1	SA. F1
BERT-base+Model-III	68.9	51.9
-Graph module	63.9	51.0
-Long answer prediction	65.1	51.4
-Short answer prediction	68.2	-
-Relational embedding	68.8	51.7
-Graph integration layer	68.3	51.1
-Self-attention layer	68.4	51.2
Table 4: Ablation study on the development set.
We evaluate the influence of layer numbers, which is illustrated in Table 3. We can see the increase in the performance of our models when the number of layers increases from 0 to 2 (The 0-layer setting means that only the graph initialization module is used to obtain the graph representations). Then the model performance does not improve with the number of network layers increasing. We attribute it to the fact that the information between every two nodes in our proposed graph can be passed through in no more than two edges, and that increasing the size of randomly initialized parameters may not be beneficial for BERT fine-tuning.
To evaluate the effectiveness of our proposed model, we conduct an ablation study on the development dataset on the BERT-base setting. The results are shown in Table 4. First, we discuss the effect of the joint training strategy. We can see that the removal of either sub-task goals will bring decreases on both tasks. It suggests that the two-grained answers can promote each other with our multi-granularity representation. Then we remove the whole graph module, which means the inference process depends on the score of short answer spans because long answer candidates cannot be scored. We can see the decrease of both long and short answer performance by  $5.0\%$  and  $0.9\%$ , respectively, indicating the effectiveness of our proposed graph representations.
Finally, we investigate the effect of components in our graph encoder. In Table 4, we can see that without relational embedding, the performance on the long answer and short answer both slightly decrease. When removing the graph integration layer, the performance of long answer and short answer both become worse by  $0.6\%$  and  $0.8\%$ . At last, we remove the self-attention layer in the graph encoder, the performance of long answer and short
answer both become worse by  $0.5\%$  and  $0.7\%$ . The ablation study shows the importance of each component in our method.
Machine reading comprehension has been widely investigated since the release of large-scale datasets (Rajpurkar et al., 2016; Joshi et al., 2017; Lai et al., 2017; Trischler et al., 2017; Yang et al., 2018). Lots of work has begun to build end-to-end deep learning models and has achieved good results (Seo et al., 2017; Xiong et al., 2017; Cui et al., 2017; Devlin et al., 2019; Lv et al., 2020). They normally treat questions and documents as two simple sequences regardless of their structures and focus on incorporating questions into the documents, where the attention mechanism is most widely used. Clark and Gardner (2018) proposes a model for multiparagraph reading comprehension using TF-IDF as the paragraph selection method. Wang et al. (2018) focuses on modeling a passage at word and sentence level through hierarchical attention.
Previous work on document modeling is mainly based on a two-level hierarchy (Ruder et al., 2016; Tang et al., 2015; Yang et al., 2016; Cheng and Lapata, 2016; Koshorek et al., 2018; Zhang et al., 2019). The first level encodes words or sentences to get the low-level representations. Moreover, a high-level encoder is applied to obtain document representation from the low-level. In these frameworks, information flows only from low-level to high-level. Fernandes et al. (2018) proposed a graph neural network model for summarization and this framework allows much complex information flows between nodes, which represents words, sentences, and entities in the graph.
Graph neural networks have shown their flexibility in a variant of NLP tasks (Zhang et al., 2018c; Marcheggiani et al., 2018; Zhang et al., 2018b; Song et al., 2018). A recent approach that began with Graph Attention Networks (Velickovic et al., 2018), which applies attention mechanisms to graphs. Wang et al. (2019) proposed knowledge graph attention networks to model the information in the knowledge graph, (Zhang et al., 2018a) proposed gated attention networks, which use a convolutional sub-network to control each attention head's importance. We model the hierarchical nature of documents by representing them at four different levels of granularity. Besides, the relations between nodes are represented by different
types of edges in the graph.
In this work, we present a novel multi-grained MRC framework based on graph attention networks and BERT. We model documents at different levels of granularity to learn the hierarchical nature of the document. On the Natural Questions dataset, which contains two sub-tasks predicting a paragraph-level long answer and a token-level short answer, our method jointly trains the two sub-tasks to consider the dependencies of the two-grained answers. The experiments show that our proposed methods are effective and outperform the previously existing methods by a large margin. Improving our graph structure of representing the document as well as the document-level pretraining tasks is our future research goals. Besides, the currently existing methods actually cannot process a long document without truncating or slicing it into fragments. How to model long documents is still a problem that needs to be solved.
This work was supported by the National Natural Science Foundation of China (NSFC) via grant 61976072, 61632011 and 61772153.
Chris Alberti, Daniel Andor, Emily Pitler, Jacob Devlin, and Michael Collins. 2019a. Synthetic QA corpora generation with roundtrip consistency. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers.
Chris Alberti, Kenton Lee, and Michael Collins. 2019b. A BERT baseline for the Natural Questions. arXiv preprint arXiv:1901.08634.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading wikipedia to answer open-domain questions. In Proc. of ACL.
Jianpeng Cheng and Mirella Lapata. 2016. Neural summarization by extracting sentences and words. In Proc. of ACL.
Christopher Clark and Matt Gardner. 2018. Simple and effective multi-paragraph reading comprehension. In Proc. of ACL.
Yiming Cui, Zhipeng Chen, Si Wei, Shijin Wang, Ting Liu, and Guoping Hu. 2017. Attention-over-attention neural networks for reading comprehension. In Proc. of ACL.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: pre-training of deep bidirectional transformers for language understanding. In Proc. of NAACL.
Patrick Fernandes, Miltiadis Allamanis, and Marc Brockschmidt. 2018. Structured neural summarization. arXiv preprint arXiv:1811.01824.
Dan Hendrycks and Kevin Gimpel. 2016. Bridging nonlinearities and stochastic regularizers with gaussian error linear units. CoRR, abs/1606.08415.
Matthew Honnibal and Ines Montani. 2017. spacy 2: Natural language understanding with bloom embeddings, convolutional neural networks and incremental parsing. To appear.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proc. of ACL.
Diederik P. Kingma and Jimmy Ba. 2015. Adam: A method for stochastic optimization. In Proc. of ICLR.
Omri Koshorek, Adir Cohen, Noam Mor, Michael Rotman, and Jonathan Berant. 2018. Text segmentation as a supervised learning task. In Proc. of NAACL.
Tom Kwiatkowski, Jennimaria Palomaki, Olivia Rhinehart, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Matthew Kelcey, Jacob Devlin, et al. 2019. Natural Questions: a benchmark for question answering research.
Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. 2017. RACE: Large-scale ReAding Comprehension dataset from Examinations. In Proc. of EMNLP.
Shangwen Lv, Daya Guo, Jingjing Xu, Duyu Tang, Nan Duan, Ming Gong, Linjun Shou, Daxin Jiang, Guihong Cao, and Songlin Hu. 2020. Graph-based reasoning over heterogeneous external knowledge for commonsense question answering. In Proceedings of the Thirty-Fourth Conference on Association for the Advancement of Artificial Intelligence (AAAI).
Diego Marcheggiani, Joost Bastings, and Ivan Titov. 2018. Exploiting semantics in neural machine translation with graph convolutional networks. In Proc. of NAACL.
Ramesh Nallapati, Feifei Zhai, and Bowen Zhou. 2017. Summarunner: A recurrent neural network based sequence model for extractive summarization of documents. In Proc. of AAAI.
Shashi Narayan, Shay B. Cohen, and Mirella Lapata. 2018. Ranking sentences for extractive summarization with reinforcement learning. In Proc. of NAACL.
6716
Ankur P. Parikh, Oscar Tackström, Dipanjan Das, and Jakob Uszkoreit. 2016. A decomposable attention model for natural language inference. In Proc. of EMNLP.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proc. of EMNLP.
Sebastian Ruder, Parsa Ghaffari, and John G Breslin. 2016. A hierarchical model of reviews for aspect-based sentiment analysis. In Proc. of EMNLP.
Min Joon Seo, Aniruddha Kembhavi, Ali Farhadi, and Hannaneh Hajishirzi. 2017. Bidirectional attention flow for machine comprehension. In Proc. of ICLR.
Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In Proc. of NAACL.
Linfeng Song, Yue Zhang, Zhiguo Wang, and Daniel Gildea. 2018. N-ary relation extraction using graph-state LSTM. In Proc. of EMNLP.
Duyu Tang, Bing Qin, and Ting Liu. 2015. Document modeling with gated recurrent neural network for sentiment classification. In Proc. of EMNLP.
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. Newsqa: A machine comprehension dataset. In Proceedings of the 2nd Workshop on Representation Learning for NLP.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Proc. of NIPS.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. 2018. Graph attention networks. In Proc. of ICLR.
Wei Wang, Ming Yan, and Chen Wu. 2018. Multigranularity hierarchical attention fusion networks for reading comprehension and question answering. In Proc. of ACL.
Xiang Wang, Xiangnan He, Yixin Cao, Meng Liu, and Tat-Seng Chua. 2019. Kgat: Knowledge graph attention network for recommendation. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining (KDD).
Caiming Xiong, Victor Zhong, and Richard Socher. 2017. Dynamic coattention networks for question answering. In Proc. of ICLR.
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D Manning. 2018. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Proc. of EMNLP.
Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In Proc. of NAACL.
Haoyu Zhang, Yeyun Gong, Yu Yan, Nan Duan, Jianjun Xu, Ji Wang, Ming Gong, and Ming Zhou. 2019. Pretraining-based natural language generation for text summarization. arXiv preprint arXiv:1902.09243.
Jiani Zhang, Xingjian Shi, Junyuan Xie, Hao Ma, Irwin King, and Dit-Yan Yeung. 2018a. Gaan: Gated attention networks for learning on large and spatiotemporal graphs. In Proc. of UAI.
Yue Zhang, Qi Liu, and Linfeng Song. 2018b. Sentence-state LSTM for text representation. In Proc. of ACL.
Yuhao Zhang, Peng Qi, and Christopher D. Manning. 2018c. Graph convolution over pruned dependency trees improves relation extraction. In Proc. of EMNLP.
A Error Analysis
We provide an error analysis for our proposed models. We divide the results for instances in development dataset into five cases:
- Case 1: The question has a long (short) answer, and the predicted score is above the threshold.
- Case 2: The question does not have a long (short) answer, and the predicted score is below the threshold.
- Case 3: The question has a long (short) answer, and prediction is wrong.
- Case 4: The question has a long (short) answer, and the predicted score is below the threshold.
- Case 5: The question does not have a long (short) answer, and the predicted score is above the threshold.
The analysis results are shown in Table 5. For BERT-base+Model-III, we can see it outperforms other BERT-base models in the first four cases on the long answer and gets comparable results on Case 5. For the short answer, the improvement of our proposed model mainly comes from Case 1 and Case 4, which suggests that our approach
6717
Long Answer	Short Answer
Case1	Case2	Case3	Case4	Case5	Case1	Case2	Case3	Case4	Case5
BERT-base+Model-I	38.2	28.4	9.7	10.9	12.8	20.2	48.5	7.7	16.2	7.3
BERT-base+Model-II	40.8	28.6	8.4	9.7	12.5	20.0	49.0	7.7	16.4	6.9
BERT-base+Model-III	41.8	28.6	8.1	9.0	12.6	20.9	48.2	8.0	15.3	7.7
BERT-syn+Model-I	40.0	30.0	7.9	11.0	11.1	22.6	49.3	7.4	14.1	6.6
BERT-syn+Model-II	42.8	30.7	6.6	9.5	10.4	23.3	48.9	7.6	13.2	7.0
BERT-syn+Model-III	43.0	30.9	6.2	9.7	10.2	23.9	48.2	8.1	12.2	7.7
Table 5: Percentage of five categories for both long answer and short answer.
Question: what's the dog's name on tom and jerry
Long Answer: Tom ( named " Jasper" in his debut appearance) is a grey and white domestic shorthair cat. "Tom" is a generic name for a male cat. He is usually but not always, portrayed as living a comfortable, or even pampered life, while Jerry ...
Short Answer: Jasper	Long Answer: Spike, occasionally referred to as Butch or Killer, is a stern but occasionally dumb American bulldog who is particularly disapproving of cats, but a softie when it comes to mice ( though in his debut appearance, Dog Trouble, Spike goes after both Tom and Jerry ) ...
Short Answer: Spike, occasionally referred to as Butch or Killer
Question: when is a spearman correlation meant to be used instead of a pearson correlation
Long Answer: This method should also not be used in cases where the data set is truncated; that is, when the Spearman correlation coefficient is desired for the top X records ( whether by pre-change rank or post-change rank, or both ), the user should use the Pearson correlation coefficient formula given above .
Short Answer: where the data set is truncated	Long Answer: The Spearman correlation between two variables is equal to the Pearson correlation between the rank values of those two variables ; while Pearson 's correlation assesses linear relationships , Spearman 's correlation assesses monotonic relationships ( whether linear or not ) ...
Short Answer: assesses monotonic relationships ( whether linear or not )
Table 6: Case studies from the development dataset. The results of directly predicting short answer span are shown on the left, and the results on the right are predicted by a pipeline strategy.
helps the model do well in cases that have a short answer. Comparing Model-I and Model-III, we can see the significant improvement of our model lies in the long answer on Case 1 (From  $38.2\%$ ,  $40.0\%$  to  $41.8\%$ ,  $43.0\%$ , respectively).
For Case 2 and Case 5, our Model-III does not have significant improvement compared to Model-I. The reason is that, for instances with no answer or no apparent answers, fine-grained information is more crucial. Therefore, using the score of short answer spans might be more accurate than the long answer score from paragraph nodes, which are coarse-grained. Overall, our Model-III is better than the baseline Model-I, especially for examples with long or short answers.
We report two case studies on the development dataset shown in Table 6. In the first case, the former prediction finds a wrong short answer "Jasper" where the word-level information in question "name" and "tom" is captured within a minimal context. Our pipeline strategy can consider
the context of the whole paragraph, leading to a more accurate long answer along with its short answer. For the second case, the former prediction failed to capture the turning information while our pipeline model sees the whole context in the paragraph, which leads to the correct short answer. In both two cases, short answers on the left both have a larger score than those on the right. This suggests that for a model that learns a strong paragraph-level representation, we can prevent errors from short answers by constraining it to the selected long answer spans.