SlowGuess/ACL_2020_OCR huggingface dataset:
Considering that questions are highly correlated due to the existence of coreferential mentions across questions, we concatenate each question with as many of its previous questions as possible up to the length limit of 64 question tokens. The action space of the model strides is set as  $[-16, 16, 32, 64, 128]$  for CoQA and  $[-16, 32, 64, 128, 256]$  for QuAC considering that documents in CoQA documents are shorter than those in QuAC. The first segment always starts with the first token of the document, and the model will take stride action after the first segment.
Results. In Table 2, we present F1 scores achieved by our methods and the baselines. The performance of the BERT-Large model drops drastically as the maximum sequence length decreases. We see a drop of  $8.6\%$  in F1 score on the CoQA dataset and a drop of  $27.0\%$  on the QuAC dataset when the maximum input length decreases from 512 to 192.
Followed by the same BERT-Large reader, the sentence selector baseline that only considers the current question achieves better performance than the selector fed with the concatenation of the current question and its previous questions. The selector with the current question performs well in selecting sentences containing answers from documents. For  $90.4\%$  of questions in CoQA and  $81.2\%$  of questions in QuAC, the top-ranked 12 sentences in the documents can include complete
Dataset	CoQA	QuAC
# of Doc Tokens	<=200	(200, 300]	(300, 400]	>400	<=300	(300, 450]	(450, 600]	>600
Percentage (%)	15.3	63.3	18.9	2.5	20.5	52.0	19.7	7.8
BERT-Large	81.0	81.9	81.8	67.2	66.2	62.8	62.2	38.7
BERT-RCM
- Gated recurrence	81.1	82.1	82.3	74.5	66.1	62.6	63.6	43.2
- LSTM recurrence	81.1	82.0	82.3	74.7	66.4	62.6	63.0	41.3
answers. However, the selector does not improve upon BERT-Large despite its high precision in sentence selection. This might be because selected sentences do not provide sufficient contexts for a model to identify answers accurately.
Our model with recurrent chunking mechanisms BERT-RCM performs consistently better than both BERT-Large and BERT-Sent-Selector. On the CoQA dataset, BERT-RCM with gated recurrence improves upon the BERT-Large model by  $3.2\%$ ,  $3\%$ ,  $0.3\%$ , and  $0.4\%$  with maximum sequence length of 192, 256, 284, and 512, respectively. The improvement brought by LSTM recurrence and RL chunking is  $2.6\%$ ,  $3.3\%$ ,  $0.3\%$ ,  $0.4\%$  on CoQA. As for the QuAC dataset, gated recurrence combined with RL chunking leads to improvements of  $17.1\%$ ,  $4.6\%$ ,  $3.2\%$ ,  $0.5\%$ , and LSTM recurrence has gains of  $19.4\%$ ,  $5.0\%$ ,  $3.7\%$ ,  $0.3\%$  under different maximum sequence lengths. On the two datasets, the gains of BERT-RCM over BERT-Large are statistically significant at  $p = 0.05$  with both gated and LSTM recurrence. We notice that our model is less sensitive to the maximum sequence length, and LSTM recurrence has comparable performance to the gated recurrence.
The gain is more obvious with maximum sequence length (192, 256, 384), and relatively small under the length of 512. This is perhaps because most document lengths are smaller than 512 in CoQA and QuAC. Therefore, we report the performance of our proposed method on documents of different lengths in Table 3, where the maximum sequence length is set as 512. We observe that the gain is more obvious on longer documents. For documents with more than 400 words in the CoQA dataset, RL chunking with gated recurrence has an improvement of  $7.3\%$  over BERT-Large, and RL chunking with LSTM recurrence improves F1 score by  $7.5\%$ . As for QuAC, the improvement of gated recurrence with RL chunking is  $4.5\%$ , and the improvement of LSTM recurrence is  $2.6\%$ .
Ablation Analysis. We further study the effect of recurrence alone without RL chunking here. As shown in rows BERT-Large and Gated recurrence (no RL chunking) in Table 2, gated recurrence alone can improve F1 score by  $2.4\%$ , and LSTM recurrence leads to an improvement of  $2.3\%$  without RL chunking when the maximum sequence length is 256. However, we do not observe any improvement when the maximum sequence length is set to 384 or 512.
Table 3: F1 Score (%) on documents with different numbers of tokens (max sequence length is 512).
Algorithms	F1
BERT-Large	61.3
Sent-Selector	59.8
BERT-RCM
- Gated recurrence	62.9
- LSTM recurrence	62.3
Table 4: F1 score  $(\%)$  of different algorithms on the TriviaQA dataset.
3.4 Results on TriviaQA
We further evaluate the ability of our model in dealing with extremely long documents on the TriviaQA Wikipedia dataset.
Setting. We set the maximum sequence length as 512 for all models. The action space of our BERT-RCM model is set to  $[-64, 128, 256, 512, 1024]$ . The stride sizes are larger than those in CoQA and QuAC, since TriviaQA provides much longer documents. During training, the maximum number of segments our model can extract from a document is set to three in the TriviaQA dataset. Note that our model reads no more than  $512 \cdot 3 = 1536$  tokens from these three segments, which are much fewer than the average document length.
Results. We filter a small number of questions whose answers cannot be extracted from documents and keep 7,251 questions from a total of 7,993 questions. In Table 4, we present the F1 scores of different algorithms. Compared with
6757
Dataset	CoQA	QuAC
BERT-Large (Devlin et al., 2019)	Prediction Stride Size	Prediction Stride Size
Training Stride Size	16	32	64	128	16	32	64	128
16	80.8	80.9	80.8	80.7	60.6	60.7	60.7	60.8
32	81.1	81.1	81.1	81.1	60.7	60.7	60.9	61.0
64	81.4	81.4	81.4	81.3	61.0	61.0	61.4	61.4
128	81.0	81.1	81.1	81.1	60.8	60.8	60.8	61.2
BERT-Large, the BERT-RCM model achieves  $1.6\%$  gain with gated recurrence and  $1\%$  gain with LSTM recurrence. Also, both BERT-RCM and BERT-Large models beat the Sent-Selector model.
In this section, we will analyze the performance of the baseline BERT-Large model and our proposed recurrent chunking mechanisms.
4.1 Analysis of different Stride Sizes in BERT-Large
In Table 5, we give an analysis of how the performance varies with different stride sizes in BERT-Large model (the baseline) training and prediction. An interesting observation is that smaller stride size in prediction does not always improve the performance, sometimes even hurts as can be seen on the QuAC dataset. It suggests that BERT-Large performs badly on selecting good answers from multiple chunks. Smaller stride size in model training also leads to worse performance. A possible explanation is that smaller stride size would cause the significant distortion of training data distribution, since the longer question-document pairs produces more training samples than short ones.
4.2 Discussions of Recurrent Chunking Mechanisms
We now provide an insight into the recurrent mechanisms and chunking policy learned by our proposed model using quantitative analysis. For the clarity of our discussions, we use the following setting on the CoQA and QuAC datasets: the maximum chunk length is set to 256, and the stride size of BERT-Large model is 128.
Segment-Hit Rate. With the ability of chunking policy, BERT-RCM is expected to focus on those document segments that contain an answer.
To evaluate how well a model can capture good segments, we use hit rate, i.e., the percentage of segments that contain a complete answer among all extracted segments, as evaluation metric.
Table 5: F1 score (%) of the BERT-Large model with different training/prediction stride sizes on the CoQA and QuAC datasets.
Hit rate	CoQA	QuAC
BERT-Large	54.0	34.1
BERT-RCM
- Gated recurrence	73.1	44.9
- LSTM recurrence	79.7	42.8
Table 6: Comparison of BERT-Large and BERT-RCM on segment-hit rate  $(\%)$
As shown in Table 6, BERT-RCM significantly outperforms BERT-Large, which indicates that the learned chunking policy is more focused on informative segments.
Figure 3: The answer-segment center distance.
Answer-Chunk Center Distance. As discussed in Fig. 1, the answer's position with respect to a document segment is important for answer prediction. When an answer is centered within the document segment, sufficient contexts on both sides help a model make better predictions. In
6758
Figure 4: Example of generated document segments by BERT-RCM from a CoQA document.
Fig. 3, it presents the averaged center distances of the first three segments generated by BERT-Large and BERT-RCMs on the CoQA validation dataset. Since all models start from the beginning of a document in the first segment, their first answer-chunk center distances are the same: 96 tokens. But for the second and third segments generated by BERT-RCMs, the answer-chunk center distances are much smaller than BERT-Large.
In this section, we also illustrate the working flow of BERT-RCM with a case study.
Case Study. We show an example from a CoQA document in Figure 4 to illustrate the chunking mechanism of our BERT-RCM model with LSTM recurrence. The model starts with the beginning of the document as the first segment, where the answer span is close to its right boundary. The model moves forwards 128 tokens to include more right contexts and generates the second chunk. The stride size is a bit large since the answer is close to the left boundary of the second segment. The model then moves back to the left by 16 tokens and obtains its third segment. The chunking scorer assigns the three segments with the scores 0.24, 0.87, and 0.90, respectively. It suggests that the model considers the third segment as the most informative chunk in answer selection.
There is a growing interest in MRC tasks that require the understanding of both questions and reference documents (Trischler et al., 2017; Rajpurkar et al., 2018; Saeidi et al., 2018; Choi et al., 2018; Reddy et al., 2018; Xu et al., 2019). Recent studies on pre-trained language models (Radford et al., 2018; Devlin et al., 2019; Baker et al., 2019; Yang
et al., 2019b) have demonstrated their great success in fine-tuning on MRC tasks. However these pre-trained NLP models (e.g., BERT) only take as input a fixed-length text. Variants of BERT are proposed to process long documents in tasks such as text classification (Chalkidis et al., 2019). To deal with lengthy documents in machine reading comprehension tasks, some previous studies skip certain tokens (Yu et al., 2017; Seo et al., 2018) or select a set of sentences as input based on the given questions (Hewlett et al., 2017; Min et al., 2018; Lin et al., 2018). However, they mainly focus on tasks in which most of the answers to given questions are formed by a single informative sentence. These previous approaches are less applicable to deal with those complicated questions that demand cross-sentences reasoning or have much lexical variability from their lengthy documents.
In this paper, we propose a chunking policy network for machine reading comprehension, which enables a model learn to chunk lengthy documents in a more flexible way via reinforcement learning. We also add a recurrent mechanism to allow the information to flow across segments so that the model could have knowledge beyond the current segment when selecting answers. We have performed extensive experiments on three public datasets of machine reading comprehension: CoQA, QuAC, and TriviaQA. Our approach outperforms benchmark models across different datasets.
6759
Henrietta Baker, Matthew R. Hallowell, and Antoine J.-P. Tixier. 2019. Automatically learning construction injury precursors from text. arXiv preprint, cs.CL/1907.11769v3.
Ilias Chalkidis, Ion Androutsopoulos, and Nikolaos Aletras. 2019. Neural legal judgment prediction in english. In Proceedings of the ACL, pages 4317-4323, Florence, Italy.
Eunsol Choi, He He, Mohit Iyyer, Mark Yatskar, Wentau Yih, Yejin Choi, Percy Liang, and Luke Zettle-moyer. 2018. QuAC: Question answering in context. In Proceedings of the EMNLP, pages 2174-2184, Brussels, Belgium.
Christopher Clark and Matt Gardner. 2017. Simple and effective multi-paragraph reading comprehension. arXiv preprint, cs.CL/1710.10723v2.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the NAACL-HLT, pages 4171-4186, Minneapolis, MN.
Hongyu Gong, Suma Bhat, Lingfei Wu, JinJun Xiong, and Wen-mei Hwu. 2019. Reinforcement learning based text style transfer without parallel training corpus. In Proceedings of the NAACL-HLT, pages 3168-3180, Minneapolis, MN.
Karl Moritz Hermann, Tomas Kocisky, Edward Grefenstette, Lasse Espeholt, Will Kay, Mustafa Suleyman, and Phil Blunsom. 2015. Teaching machines to read and comprehend. In Proceedings of the NeurIPS, pages 1693-1701, Montreal, Canada.
Daniel Hewlett, Llion Jones, Alexandre Lacoste, et al. 2017. Accurate supervised and semi-supervised machine reading for long documents. In Proceedings of the EMNLP, pages 2011-2020, Copenhagen, Denmark.
Felix Hill, Antoine Bordes, Sumit Chopra, and Jason Weston. 2016. The goldilocks principle: Reading children's books with explicit memory representations. In Proceedings of the ICLR, Caribe Hilton, Puerto Rico.
Sepp Hochreiter and Jürgen Schmidhuber. 1997. Long short-term memory. Neural computation, 9(8):1735-1780.
Phu Mon Htut, Samuel R Bowman, and Kyunghyun Cho. 2018. Training a ranking function for open-domain question answering. In Proceedings of the NAACL-HLT, pages 120-127, New Orleans, LA.
Minghao Hu, Yuxing Peng, Zhen Huang, Nan Yang, Ming Zhou, et al. 2019. Read+ verify: Machine reading comprehension with unanswerable questions. In Proceedings of the AAAI, pages 6529-6537, Honolulu, HI.
Mandar Joshi, Eunsol Choi, Daniel S. Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In Proceedings of the ACL, pages 1601-1611, Vancouver, Canada.
Nitish Shirish Keskar, Bryan McCann, Caiming Xiong, and Richard Socher. 2019. Unifying question answering and text classification via span extraction. arXiv preprint, cs.CL/1904.09286v2.
Tomáš Kocisky, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gáabor Melis, and Edward Grefenstette. 2018. The narrativeqa reading comprehension challenge. TACL, 6:317-328.
Yankai Lin, Haozhe Ji, Zhiyuan Liu, and Maosong Sun. 2018. Denoising distantly supervised open-domain question answering. In Proceedings of the ACL, pages 1736-1745, Melbourne, Australia.
Sewon Min, Victor Zhong, Richard Socher, and Caiming Xiong. 2018. Efficient and robust question answering from minimal context over documents. In Proceedings of the ACL, pages 1725-1735, Melbourne, Australia.
Alec Radford, Karthik Narasimhan, Tim Salimans, and Ilya Sutskever. 2018. Improving language understanding by generative pre-training. In Preprint.
Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for squad. In Proceedings of the ACL, pages 784-789, Melbourne, Australia.
Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In Proceedings of the EMNLP, pages 2383-2392, Austin, TX.
Siva Reddy, Danqi Chen, and Christopher D Manning. 2018. CoQA: A conversational question answering challenge. TACL, 7:249-266.
Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Roktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In Proceedings of the EMNLP, pages 2087-2097, Brussels, Belgium.
Minjoon Seo, Sewon Min, Ali Farhadi, and Hannaneh Hajishirzi. 2018. Neural speed reading via SkimRNN. In Proceedings of the ICLR, New Orleans, LA.
Richard S Sutton and Andrew G Barto. 2018. Reinforcement learning: An introduction.
Richard S Sutton, David A McAllester, Satinder P Singh, and Yishay Mansour. 2000. Policy gradient methods for reinforcement learning with function approximation. In Proceedings of the NeurIPS, pages 1057-1063, Denver, CO.
6760
Adam Trischler, Tong Wang, Xingdi Yuan, Justin Harris, Alessandro Sordoni, Philip Bachman, and Kaheer Suleman. 2017. NewsQA: A machine comprehension dataset. In Proceedings of the RepL4NLP, pages 191-200, Vancouver, Canada.
Ronald J Williams. 1992. Simple statistical gradient-following algorithms for connectionist reinforcement learning. Machine learning, 8(3-4):229-256.
Hu Xu, Bing Liu, Lei Shu, and Philip S. Yu. 2019. Review conversational reading comprehension. arXiv preprint, cs.CL/1902.00821v2.
Wei Yang, Yuqing Xie, Aileen Lin, Xingyu Li, Luchen Tan, Kun Xiong, Ming Li, and Jimmy Lin. 2019a. End-to-end open-domain question answering with bertserini. In Proceedings of the NAACL-HLT, pages 72-77, Minneapolis, MN.
Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. 2019b. XLNet: Generalized autoregressive pretraining for language understanding. In Proceedings of the NeurIPS, pages 5754-5764, Vancouver, Canada.
Mark Yatskar. 2019. A qualitative comparison of CoQA, SQuAD 2.0 and QuAC. In Proceedings of the NAACL-HLT, pages 2318-2323, Minneapolis, MN.
Adams Wei Yu, Hongrae Lee, and Quoc Le. 2017. Learning to skim text. In Proceedings of the ACL, pages 1880-1890, Vancouver, Canada.
Sheng Zhang, Xiaodong Liu, Jingjing Liu, Jianfeng Gao, Kevin Duh, and Benjamin Van Durme. 2018. Record: Bridging the gap between human and machine commonsense reading comprehension. arXiv preprint, cs.CL/1810.12885v1.
6761