Venue: NAACL 2021

# Does Structure Matter? Encoding Documents for

Machine Reading Comprehension

Hui Wan Song Feng Chulaka Gunasekara Siva Sankalp Patel Sachindra Joshi Luis A. Lastras IBM Research AI {hwan@us, sfeng@us, chulaka.gunasekara@}.ibm.com {siva.sankalp.patel@, jsachind@in, lastrasl@us}.ibm.com

###### Abstract

Machine reading comprehension is a challenging task especially for querying documents with deep and interconnected contexts. Transformer-based methods have shown advanced performances on this task; however, most of them still treat documents as a flat sequence of tokens. This work proposes a new Transformer-based method that reads a document as tree slices. It contains two modules for identifying more relevant text passage and the best answer span respectively, which are not only jointly trained but also jointly consulted at inference time. Our evaluation results show that our proposed method outperforms several competitive baseline approaches on two datasets from varied domains.

+
Footnote †: _Corresponding of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 4626–4634 June 6-11, 2021. ©2021 Association for Computational Linguistics

## 1 Introduction

Machine Reading Comprehension (MRC) is the task of reading a given text and answering questions about it [5]. Some MRC tasks such as SQuAD [7, 8],and ShARC [9] provide a short text snippets as the context documents; while others such as TriviaQA [10], Natural Questions [11] and Doc2Dial [12] use full articles as documents. Most top performing models on MRC tasks use different variants of Transformers [23]. Transformer-based models typically only consider a certain number of tokens, utilize a sliding window approach [13] or segment the document into passages [10, 11] due to the constraint on the size of input sequence. More recent works explore how to scale up input length [11, 12, 13, 14] but still mainly focus on flat sequences. In addition to scaling up input length, ETC[1] also propose to deal with encoding structured inputs based on relative position encoding [14] through the global-local mechanism.

A series of recent work explores incorporating structured knowledge embedded in text into MRC [14, 15]. However, such kind of linking information for creating triples is not necessarily prominent in documents other than Wikipedia. Some works segment the document content based on its semantic structures and rank them based on their relevance to the query [14, 15, 16, 17, 18].

Another thread of works, on hierarchical document encoding [13, 16, 17, 18], first obtain sentence level representations then encode document based on the sentence vectors. Those works do not directly apply on fine-grained answer extraction across sentences.

In many online documents, certain important information unfolds through the semantic relations of hierarchical structures such as parent-child and siblings between different parts of the document. Figure 1 illustrates the difference when using a doc

Figure 1: A sample document segment with the hierarchical structure (left), the partial tree slices (right) and sample dialogue turns (bottom right).



ument with and without the structure information for a MRC task. For query U1, it is crucial to keep in mind we are in the context of "How to appeal a TVB ticket" and "Online" while reading the passage of "You will need" to find the answer to the user query. However, conventional Transformers fail to capture such contextual information when the text is too long to fit in the maximum sequence length allowed.

In this work, we explore the utilization of document structure for the focused task of fine-grained Machine Reading Comprehension on document. We propose a Transformer-based method that reads a document as tree slices; it jointly learns the relevance of paragraphs and spans, and then performs a cascaded inference to find the best answer span. Our work is intuitively inspired by how people read through documents Choi et al. (2017) based on structural cues such as titles and subtitles, and then focus on the relevant parts to search for an answer. We utilize the structural information naturally available in online documents for identifying tree slices. Each slice corresponds to nodes along a path from a root node to a lower level child node as illustrated by the right part of Figure 1. Thus, we are able to capture the essential structural information for the inference that could be outside of a conventional sliding window or text segment. Compared to approaches such as Longformer Beltagy et al. (2020) or ETC Ainslie et al. (2020), our approach can be directly applied to many existing pretrained models, and has a small GPU memory footprint. RikiNet Liu et al. (2020) employs a dynamic paragraph dual-attention reader and a multi-level cascaded answer predictor, while our tree slices consider hierarchical structures above paragraphs, and our cascaded inference is in beam search style rather than greedy decoding style in RikiNet.

We evaluate on two datasets with structured documents: one obtained from Natural Questions Kwiatkowski et al. (2019), which is based on Wikipedia articles, and one from Doc2Dial Feng et al. (2020), which is based on web pages of several domains. Our proposed method is compared with several baselines to see performance gain on both datasets. For example, our method achieves 4% gain of F1 on Doc2Dial, which shows its superiority on small-scaled dataset across multiple domains.

Our contributions can be summarized as follows:

(1) We propose a Transformer-based method that reads a document as a tree. It simultaneously identifies the relevance of paragraphs and finds the answer span via jointly trained models with cascaded inference. (2) Our method can utilize common structures as seen in many web documents. It allows Transformer models to read in more focused content but with deep context; thus it can be used to handle long documents in an efficient way. (3) Our proposed method outperforms several competitive baseline methods on two kinds of MRC tasks with documents from varied domains.

## 2 Approach

We adopt a Transformer-based document-tree-slice encoder with joint learning and cascaded inference. Our approach is influenced by the pattern of human behavior during reading Choi et al. (2017), which is to focus on a smaller portion at a time and favor the more relevant parts while looking for answer. This approach can also overcome the constraint on fixed-length input allowed by the common Transformer architecture Vaswani et al. (2017). More importantly, this enables us to always include important structural context information during encoding.

### Tree Slicing

To obtain the tree representation of a web page, we consider the different levels of HTML title tags as the main indicators of the hierarchical structures such as parent-child and siblings in Figure 1. More details are provided in Section 3.

Formally, we define an example in the dataset as \((Q,D,s,e)\) where \(Q\) is a question, \(D\) is a document, \(s\) and \(e\) denote the inclusive indices pointing to the start and end of the target answer span.

Suppose one does not consider the structure information, \(D\) is treated as a sequence and sent to Transformer encoder. For long documents, the sliding window approach is widely used to truncate \(D\) into \(m\) overlapping fragments \(D_{1},...D_{m}\), and \((Q,D,s,e)\) is converted to \(m\) training instances \((C_{i},s_{i},e_{i})\) where \(C_{i}=([\mathtt{CLS}],Q_{1},...,Q_{|Q|},[\mathtt{SEP}],D_{i,1},...,D_{i,|D_{ i}|},[\mathtt{SEP}])\), \(s_{i}\) and \(e_{i}\) are mapped indices in \(C_{i}\). If \(D_{i}\) does not contain the target answer, \(s_{i}\) and \(e_{i}\) are set to the index of the \([\mathtt{CLS}]\) token.

In our proposed approach to encode a document, we consider the structured information along with its content. Given a document \(D\), let \(k\) be 

the number of leaf nodes in its tree structure. We first convert \((Q,D,s,e)\) into \(k\) examples \((Q,A_{i},P_{i},s_{i},e_{i})\), where \(P_{i}\) is a leaf node, \(s_{i}\) and \(e_{i}\) are mapped indices in \(P_{i}\), and \(A_{i}\) denotes \(P_{i}\)'s ancestor chain in the document tree of \(D\). Each \((Q,A_{i},P_{i})\) is then encoded with Transformers as a sequence \(C_{i}=([\mathtt{CLS}],Q_{1},...,Q_{|Q|},\\ [\texttt{SEP}],A_{i,1},...,A_{i,|A_{i}|},[\texttt{SEP}],P_{i,1},...,P_{ i,|P_{i}|},[\texttt{SEP}])\). An example \(A_{i}\) in Figure 1 would be the list of {_How to appeal a TVB ticket conviction_', _Online_', _You will need_'}. Intuitively, the tree slice approach ensures that the most relevant structural information, the ancestor chain, is always taken into account and attended to with Transformer encoder, while this is unlikely to be guaranteed by the sliding window truncation.

### Joint Model with Cascaded Inference

With tree slicing approach, from each document we have many paragraphs to select the answer span from, as compared to the case of sliding windows. In order to teach the model to favor the candidates from the more relevant parts of the document, we train a joint model to simultaneously learn to identify the relevance of paragraphs and find the answer span. Then we perform a cascaded inference to first find the most relevant paragraphs and then find the best answer span from them, based on the scores from the joint model, as Figure 2 shows.

Joint modelThe encoded representation of \(C\) can be used to perform two tasks, each being handled by a separate module: 1) the pooler layer and the matching layer (both linear layers) predict how likely a paragraph \(P\) contains the answer; 2) the span selection layer (another linear layer) identifies the answer span from \(P\). Each training instance is converted to \((C,s,e,g)\) where \(g\in\{0,1\}\) denotes whether \(P\) contains the answer. We define the loss function to be

\[Loss(g,s,e,C;\theta) =\mathcal{L}_{\texttt{CE}}(f_{hit}(g,C;\theta))\] \[+\lambda*(\mathcal{L}_{\texttt{CE}}(f_{start}(s,C;\theta))\] \[\qquad+\mathcal{L}_{\texttt{GE}}(f_{end}(e,C;\theta)))\]

where \(\mathcal{L}_{\texttt{CE}}\) is the Cross Entropy loss function, \(\theta\) denotes the model parameters, and each \(f\) is the score obtained by the corresponding linear layer on top of the last layer representation of Transformer encoder: \(f_{hit}\) by the pooler layer and the matching layer, and \(f_{start}\) and \(f_{end}\) by the span selection layer.

Cascaded inferenceAfter the two modules of the model are jointly trained, we conduct a cascaded inference in a beam search style.

* First, from all the instances corresponding to tree slices of a single document, we select the top \(n\) instances ranked by \(f_{hit}(g=1,C;\theta)\). This is important for filtering out high scored spans from irrelevant tree slices.
* Then, from these top instances, each candidate document span is assigned a score attributed from both modules of the model: \[\texttt{Score}(C,s,e) =f_{hit}(g=1,C;\theta)\] \[+\gamma*(\texttt{Score}_{start}(s,C;\theta)\] \[\qquad+\texttt{Score}_{end}(e,C;\theta))\] where we adopted the trick from Alberti et al. (2019) to define \(\texttt{Score}_{start}(s,C;\theta)=f_{start}(s,C;\theta)-f_{start}(\texttt{ Idx}_{\texttt{CLS}},C;\theta)\), and \(\texttt{Score}_{end}(e,C;\theta)\) as \(f_{end}(e,C;\theta)-f_{end}(\texttt{Idx}_{\texttt{CLS}},C;\theta)\).
* Finally, we choose the document span with the highest \(\texttt{Score}(C,s,e)\) as the answer.

Given a document with tree slices, we would create more instances than the sliding window approach. However, with the joint training and cascaded inference, our model reaches better accuracy in less training time, as will be shown in Section 4.

## 3 Data

Our focused task is utilizing document structure in contextual representation for fine-grained MRC. Since there is very few prior MRC datasets that provides document structure information, we identified two public datasets where HTML markup tags are available in the document data together

Figure 2: Joint Model with Cascaded Inference.

 

with QA pairs, and extract tree structure out of the HTML documents for MRC. Data script could be found at [http://html2struct.github.io](http://html2struct.github.io).

Extract Tree StructureTo obtain the tree representation of documents from the two datasets, we first parse HTML files to get markup tags of the textual content elements, which corresponds to the titles, lists, tables and paragraphs. We consider the different levels of title tags as the main indicators of the hierarchical structures such as parent-child and siblings. Thus, the stem nodes are inherently section or subsection titles of the article and leaf nodes are typically paragraphs, list content or table content. We assign the article title as the tree root. Please refer to Appendix A for more details about the data statistics for the experiment.

NQStructNatural Question Kwiatkowski et al. (2019) provides QA pairs that are grounded in Wikipedia articles. The original task provides answers in two formats: _long answer_, typically a paragraph or table; _short answer_, typically one or more entities. In our task, we focus on identifying the short answer given the whole document as the input, and do not use the long answers data. We observe the bias on answers appearing in first paragraph, which is significant enough to serve as a baseline Kwiatkowski et al. (2019). Thus, we follow Geva and Berant (2018) to alleviate such bias by only considering the questions where the short answer does not come from the first paragraph. As a result, we derive a subset of 48K examples from about 100K examples with short answers from training and dev sets.

D2DStructDoc2Dial Feng et al. (2020) provides document-grounded dialogues with annotations of dialogue scenes, which allow us to identify question-answer pairs that are most related to our target task. Specifically, we combine each turn of the agent responding to a user query, together with the previous dialogue context, as a question. The public dataset contains over 4.1K documented-grounded dialogues based on about 450 documents from different domains, and we derive 9.3K QA pairs out of it.

## 4 Experiments and Results

We compare our proposed method (**TreeJC** in short) with several baseline methods. Next we describe the baselines, the experiment settings, and present the evaluation results.

Baselines Sliding Window (SW)is a popular question answering baseline that trains a span selection model with Transformer encoding document trunks as described in Section 2.

Longformeris a Transformer model that handles long documents Beltagy et al. (2020). We experiment the sliding window approach above with Longformer-base pretrained model with max sequence length of 4096 and a stride of 3072.

Ir+Swis a pipeline approach that first identifies small number of \(k\) candidate paragraphs (\(k=10\) in the experiments here) via an information retrieval mechanism BM25 Robertson et al. (1995), and then uses the SW approach. We consider it as a solution with reduced time complexity from the traditional SW approach for us to compare with.

LeafJcFor ablation study, we experiment with a variant of TreeJC approach that excludes ancestors during encoding. The other implementation and experimental details are similar to TreeJC.

### Experiment Settings

All models are implemented in PyTorch. Pretrained models are Roberta-base for SW, IR+SW, LeafJC and TreeJC, and Longformer-base for Longformer. Implementations of SW, Longformer and IR+SW are adapted from the SQuAD example code1 in HuggingFace Transformers Wolf et al. (2019).

Footnote 1: [https://github.com/huggingface/transformers/blob/master/examples/legacy/question-answering/](https://github.com/huggingface/transformers/blob/master/examples/legacy/question-answering/). It is customized to fit the data format and to fix a tokenization issue in the original HuggingFace code.

\begin{table}
\begin{tabular}{l c c c} \hline
**Model** & **F1** & **Exact Match** & **Train hrs** \\ \hline SW & \(49.6\pm 0.1\) & \(32.2\pm 0.2\) & 1.75 \\ Longformer & \(\mathbf{54.2\pm 0.5}\) & \(33.7\pm 0.2\) & 8 \\ IR+SW & \(40.3\pm 0.5\) & \(26.7\pm 0.5\) & 1 \\ LeafJC & \(53.1\pm 0.5\) & \(33.7\pm 0.8\) & 1.5 \\ TreeJC & \(53.7\pm 0.6\) & **34.5\(\pm\)****0.3** & 1.5 \\ \hline \end{tabular}
\end{table}
Table 1: Results on D2DStruct test set.

\begin{table}
\begin{tabular}{l c c c} \hline
**Model** & **F1** & **Exact Match** & **Train hrs** \\ \hline SW & \(52.4\pm 0.1\) & \(41.8\pm 0.3\) & 32 \\ Longformer & \(51.8\pm 0.2\) & \(41.4\pm 0.3\) & 46 \\ IR+SW & \(46.9\pm 0.4\) & \(35.6\pm 0.4\) & 9 \\ LeafJC & \(53.0\pm 0.4\) & \(42.6\pm 0.2\) & 15 \\ TreeJC & \(\mathbf{54.9\pm 0.4}\) & **44.2\(\pm\)****0.3** & 16 \\ \hline \end{tabular}
\end{table}
Table 2: Results on NQStruct test set.

 

2019). For a fair comparison, input to SW, Longformer and IR+SW is flattened equivalent of the tree input to LeafJC and TreeJC and does not include the HTML tags of web pages.

All experiments were done on a single V100 GPU. In order to encode long sequences, Longformer requires much larger GPU memory, only 2 instances could fit in one V100 GPU, whereas 27 instances could fit in one V100 GPU with Roberta. Please see Appendix B for more details about experiment setup.

For the training of our approach TreeJC, positive instances are up-sampled to reach a balanced proportion of positive and negative training instances. To avoid the consequent bias towards longer documents, the loss from each example (document QA pair) is scaled down by the number of training instances from this example. \(\lambda\) and \(\gamma\) in Section 2 is set to be 0.5 and 1, respectively.

### Results

For evaluation, we use exact match score and token-level F1 score Rajpurkar et al. (2018). Table 1 and Table 2 present the evaluation results on the test sets of D2DStruct and NQStruct respectively along with the training time. All numbers are in the form of mean \(\pm\) std, which is from three runs with different random seeds.

We observe consistent performance gains by TreeJC over almost all baselines. TreeJC shows a significant improvement over SW, which indicates the effectiveness of encoding the structure information with our joint model with cascaded inference. LeafJC performs better than SW but worse than TreeJC, which confirms the importance of including ancestor nodes during encoding. Longformer 2 serves as a competitive baseline and it achieves half a point higher F1 for D2DStruct dataset, however, at the cost of much longer training time. IR+SW method, on the other hand, shows high efficiency but suffers lower effectiveness, attributing to the fact that the IR method only achieves around 73% recall. In order to further examine how our approach performs on documents with different sizes, we break down the results on NQStruct dataset and compare the performances in Table 3. The results show that our approach has a clear gain on all document lengths over SW, especially on very long documents.

Footnote 2: 5 epochs were finished on NQStruct, as in experiments in Beltagy et al. (2020).

## 5 Conclusion

We introduce a new Transformer-based method with joint learning and cascaded inference inspired by the tree structures of documents for machine reading comprehension. It outperforms several competitive baselines on two datasets from multiple domains. In particular, our study demonstrates that the proposed model is effective to encode longer documents with deep contexts for MRC tasks.

## References

* Ainslie et al. (2020) Joshua Ainslie, Santiago Ontanon, Chris Alberti, Vaclav Cvieck, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Gifan Wang, and Li Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), Online. Association for Computational Linguistics.
* Alberti et al. (2019) Chris Alberti, Kenton Lee, and Michael Collins. 2019. A BERT baseline for the natural questions. CoRRabs/1901.08634.
* Beltagy et al. (2020) Iz Beltagy, Matthew E. Peters, and Arman Cohan. 2020. Longformer: The long-document transformer. arXiv:2004.05150.
* Choi et al. (2017) Eunsol Choi, Daniel Hewlett, Jakob Uszkoreit, Illia Polosukhin, Alexandre Lacoste, and Jonathan Berant. 2017. Coarse-to-fine question answering for long documents. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), Vancouver, Canada. Association for Computational Linguistics.
* Dhingra et al. (2020) Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, and William W. Cohen. 2020. Differentiable reasoning over a virtual knowledge base.
* Feng et al. (2020) Song Feng, Hui Wan, Chulaka Gunasekara, Siva Patel, Sachindra Joshi, and Luis Lastras. 2020. doc2dial: A goal-oriented document-grounded dialogue dataset. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language

\begin{table}
\begin{tabular}{l c c} \hline
**Doc length (# tokens)** & **SW** & **TreeJC** \\ \hline \(<0.5\text{K}\) & \(50.3/64.5\) & \(55.0/68.1\) \\
0.5\(\text{k-1k}\) & \(45.1/57.6\) & \(45.4/57.9\) \\
1\(\text{k-2k}\) & \(45.0/56.3\) & \(45.8/57.5\) \\
2\(\text{k-4k}\) & \(44.4/54.4\) & \(45.9/56.1\) \\
4\(\text{k-8k}\) & \(39.5/48.5\) & \(40.6/49.7\) \\ \(>8\text{k}\) & \(34.9/45.1\) & \(38.1/48.4\) \\ \hline \end{tabular}
\end{table}
Table 3: Breakdown of results in Exact_Match/F1 on NQStruct test set.

 

Processing (EMNLP)_, pages 8118-8128, Online. Association for Computational Linguistics.
* Geva and Berant [2018] Mor Geva and Jonathan Berant. 2018. Learning to search in long documents using document structure. In _Proceedings of the 27th International Conference on Computational Linguistics_, pages 161-176, Santa Fe, New Mexico, USA. Association for Computational Linguistics.
* Guo et al. [2019] Mandy Guo, Yinfei Yang, Keith Stevens, Daniel Cer, Heming Ge, Yun-hsuan Sung, Brian Strope, and Ray Kurzweil. 2019. Hierarchical document encoder for parallel corpus mining. In _Proceedings of the Fourth Conference on Machine Translation (Volume 1: Research Papers)_, pages 64-72, Florence, Italy. Association for Computational Linguistics.
* Hu et al. [2019] Minghao Hu, Yuking Peng, Zhen Huang, and Dongsheng Li. 2019. Retrieve, read, rerank: Towards end-to-end multi-document reading comprehension. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 2285-2295, Florence, Italy. Association for Computational Linguistics.
* Joshi et al. [2017] Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehension. In _Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)_, pages 1601-1611, Vancouver, Canada. Association for Computational Linguistics.
* Kitaev et al. [2020] Nikita Kitaev, Lukasz Kaiser, and Anselm Levskaya. 2020. Reformer: The efficient transformer. In _International Conference on Learning Representations_.
* Kwiatkowski et al. [2019] Tom Kwiatkowski, Jennimaria Palomaki, Olivia Redfield, Michael Collins, Ankur Parikh, Chris Alberti, Danielle Epstein, Illia Polosukhin, Jacob Devlin, Kenton Lee, Kristina Toutanova, Llion Jones, Matthew Kelcey, Ming-Wei Chang, Andrew M. Dai, Jakob Uszkoreit, Quoc Le, and Slav Petrov. 2019. Natural questions: A benchmark for question answering research. _Transactions of the Association for Computational Linguistics_, 7:452-466.
* Lee et al. [2018] Jinhyuk Lee, Seongjun Yun, Hyunjae Kim, Miyoung Ko, and Jaewoo Kang. 2018. Ranking paragraphs for improving answer recall in open-domain question answering. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 565-569, Brussels, Belgium. Association for Computational Linguistics.
* Li et al. [2015] Jiwei Li, Thang Luong, and Dan Jurafsky. 2015. A hierarchical neural autoencoder for paragraphs and documents. In _Proceedings of the 53rd Annual Meeting of the Association for Computational Linguistics and the 7th International Joint Conference on Natural Language Processing (Volume 1: Long Papers)_, pages 1106-1115, Beijing, China. Association for Computational Linguistics.
* Liu et al. [2020] Dayiheng Liu, Yeyun Gong, Jie Fu, Yu Yan, Jiusheng Chen, Daxin Jiang, Jiancheng Lv, and Nan Duan. 2020. RikiNet: Reading Wikipedia pages for natural question answering. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6762-6771, Online. Association for Computational Linguistics.
* Liu et al. [2019] Shanshan Liu, Xin Zhang, Sheng Zhang, Hui Wang, and Weiming Zhang. 2019. Neural machine reading comprehension: Methods and trends. _Applied Sciences_, 9(18):3698.
* Rajpurkar et al. [2018] Pranav Rajpurkar, Robin Jia, and Percy Liang. 2018. Know what you don't know: Unanswerable questions for SQuAD. In _Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers)_, pages 784-789, Melbourne, Australia. Association for Computational Linguistics.
* Rajpurkar et al. [2016] Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. SQuAD: 100,000+ questions for machine comprehension of text. In _Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing_, pages 2383-2392, Austin, Texas. Association for Computational Linguistics.
* Richardson et al. [2013] Matthew Richardson, Christopher JC Burges, and Erin Renshaw. 2013. Mctest: A challenge dataset for the open-domain machine comprehension of text. In _Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing_, pages 193-203.
* Robertson et al. [1995] Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulieu, Mike Gafford, et al. 1995. Okapi at trec-3. _Nist Special Publication Sp_, 109:109.
* Saeidi et al. [2018] Marzieh Saeidi, Max Bartolo, Patrick Lewis, Sameer Singh, Tim Rocktaschel, Mike Sheldon, Guillaume Bouchard, and Sebastian Riedel. 2018. Interpretation of natural language rules in conversational machine reading. In _Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing_, pages 2087-2097, Brussels, Belgium. Association for Computational Linguistics.
* Shaw et al. [2018] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. 2018. Self-attention with relative position representations. In _Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 2 (Short Papers)_, pages 464-468, New Orleans, Louisiana. Association for Computational Linguistics.
* Shen et al. [2020] Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020. Exploiting structured knowledge in text via graph-guided representation learning. In _Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)_, pages 8980-8994, Online. Association for Computational Linguistics 

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, L ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In _Advances in Neural Information Processing Systems_, volume 30, pages 5998-6008. Curran Associates, Inc.
* Wang et al. (2018) Shuohang Wang, Mo Yu, Jing Jiang, Wei Zhang, Xiaoxiao Guo, Shiyu Chang, Zhiguo Wang, Tim Klinger, Gerald Tesauro, and Murray Campbell. 2018. Evidence aggregation for answer re-ranking in open-domain question answering. In _International Conference on Learning Representations_.
* Wang et al. (2020) Shuohang Wang, Luowei Zhou, Zhe Gan, Yen-Chun Chen, Yuwei Fang, Siqi Sun, Yu Cheng, and Jingjing Liu. 2020. Cluster-former: Clustering-based sparse transformer for long-range dependency encoding.
* Wang et al. (2019) Zhiguo Wang, Patrick Ng, Xiaofei Ma, Ramesh Nallapati, and Bing Xiang. 2019. Multi-passage BERT: A globally normalized BERT model for open-domain question answering. In _Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP)_, pages 5878-5882, Hong Kong, China. Association for Computational Linguistics.
* Wolf et al. (2019) Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R'emi Louf, Morgan Funtowicz, and Jamie Brew. 2019. Huggingface's transformers: State-of-the-art natural language processing. _ArXiv_, abs/1910.03771.
* Yan et al. (2019) Ming Yan, Jiangnan Xia, Chen Wu, Bin Bi, Zhongzhou Zhao, Ji Zhang, Luo Si, Rui Wang, Wei Wang, and Haiqing Chen. 2019. A deep cascade model for multi-document reading comprehension. In _Proceedings of the AAAI Conference on Artificial Intelligence_, volume 33, pages 7354-7361.
* Yang et al. (2019) Zhilin Yang, Zihang Dai, Yiming Yang, Jaime G. Carbonell, Ruslan Salakhutdinov, and Quoc V. Le. 2019. Xlnet: Generalized autoregressive pretraining for language understanding. In _Advances in Neural Information Processing Systems 32: Annual Conference on Neural Information Processing Systems 2019, NeurIPS 2019, 8-14 December 2019, Vancouver, BC, Canada_, pages 5754-5764.
* Yang et al. (2016) Zichao Yang, Diyi Yang, Chris Dyer, Xiaodong He, Alex Smola, and Eduard Hovy. 2016. Hierarchical attention networks for document classification. In _Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies_, pages 1480-1489, San Diego, California. Association for Computational Linguistics.
* Zhang et al. (2019) Xingxing Zhang, Furu Wei, and Ming Zhou. 2019. HIBERT: Document level pre-training of hierarchical bidirectional transformers for document summarization. In _Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics_, pages 5059-5069, Florence, Italy. Association for Computational Linguistics.
* Zheng et al. (2020) Bo Zheng, Haoyang Wen, Yaobo Liang, Nan Duan, Wanxiang Che, Dexin Jiang, Ming Zhou, and Ting Liu. 2020. Document modeling with graph attention networks for multi-grained machine reading comprehension. In _Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics_, pages 6708-6718, Online. Association for Computational Linguistics.

 

Data

### Data

#### NQStruct

We derive a subset of 46k examples from about 100k training examples with short answers. From the set of 46K, we set aside a subset of 2.5K examples as our dev set. Similarly, we derive a subset of 2.3K examples from the original NQ dev set, and use that as our test set. In the cases that multiple short answers are available, we use the first one for training and evaluation. The number of examples in training/dev/test split is shown in Table 4. The distribution of document length is shown in Table 5.

D2DTructThe public release of Doc2Dial only provides train and dev sets 3. For filtering out the non-answer agent turns, we filter out the cases where agent turn is grounded on the (sub)section titles. We combine the two and then create train/dev/test splits as 70%, 15%, 15% where 50% of the dev/test set are from documents unseen in training set. The number of examples in training/dev/test split is shown in Table 4. The distribution of document length is shown in Table 5.

Footnote 3: [http://doc2dial.github.io/data.html](http://doc2dial.github.io/data.html)

## Appendix B Experiment Settings

The deep learning systems are in PyTorch, and use Transformer encoder from HuggingFace Transformers. We use Roberta-base pretrained model and max sequence length of 512 unless otherwise stated. All experiments were done with fp16, on a single V100 GPU. Table 8 presents generation configurations and hyper-parameters that are shared by both datasets. For evaluation, we use the evaluation script 2.0 of SQuAD.

Table 9 presents configuration specifics for D2DStruct. For each dialogue in Doc2Dial, we combine all previous turns in reverse order as a query. Table 10 presents configuration specifics for NQStruct.

## Appendix C Results on Dev Sets

Table 6 and 7 present evaluation results on the dev sets of D2DStruct and NQstruct datasets respectively.

\begin{table}
\begin{tabular}{l c l} \hline
**Model** & **F1** & **Exact Match** \\ \hline SW & \(53.0\pm 0.4\) & \(36.3\pm 0.4\) \\ Longformer & \(56.6\pm 0.8\) & \(37.0\pm 0.6\) \\ IR+SW & \(44.5\pm 0.4\) & \(31.3\pm 0.5\) \\ LeafJC & \(56.2\pm 0.3\) & \(39.7\pm 0.3\) \\ TreeJC & **57.3\(\pm\) 0.4** & **41.0\(\pm\) 0.3** \\ \hline \end{tabular}
\end{table}
Table 6: Results on D2DStruct dev set.

\begin{table}
\begin{tabular}{l c c c c} \hline
**Dataset** & \(\leq\)**0.5k** & **0.5k-1k** & **lk-2k** & \(>\)**2k** \\ \hline NQStruct & \(7\%\) & \(9\%\) & \(15\%\) & \(69\%\) \\ D2DD Struct & \(25\%\) & \(45\%\) & \(24\%\) & \(6\%\) \\ \hline \end{tabular}
\end{table}
Table 5: Statistics of document length (in tokens).

 



\begin{table}
\begin{tabular}{l c c c c c} \hline  & **SW** & **Longformer** & **IR+SW** & **LeafJC** & **TreeJC** \\ \hline
**\# epochs** & 8 & 5 & 8 & 1 & 1 \\
**\# instances per epoch** & \(879,249\) & \(84,788\) & \(275,261\) & \(3,232,359\) & \(3,241,152\) \\
**max sequence length** & \(512\) & \(4096\) & \(512\) & \(512\) & \(512\) \\
**Document stride** & \(256\) & \(3072\) & \(256\) & \(256\) & \(256\) \\
**max query length** & \(64\) & \(64\) & \(64\) & \(64\) & \(64\) \\
**max answer length** & \(30\) & \(30\) & \(30\) & \(30\) & \(30\) \\ \hline \end{tabular}
\end{table}
Table 10: Dataset-specific parameters for NQStruct experiments

\begin{table}
\begin{tabular}{l c c c c c} \hline  & **SW** & **Longformer** & **IR\(+\)SW** & **LeafJC** & **TreeJC** \\ \hline
**Total batch size** & \(54\) & \(64\) & \(54\) & \(54\) & \(54\) \\
**Batch size per GPU** & \(27\) & \(2\) & \(27\) & \(27\) & \(27\) \\
**Learning rate** & \(3\)e-\(5\) & \(3\)e-\(5\) & \(3\)e-\(5\) & \(3\)e-\(5\) & \(3\)e-\(5\) \\
**Weight decay** & \(0.01\) & \(0.01\) & \(0.01\) & \(0.01\) & \(0.01\) \\
**N best size** & \(20\) & \(20\) & \(20\) & \(5\) & \(5\) \\
**Warmup proportion** & \(10\%\) & \(10\%\) & \(10\%\) & \(10\%\) & \(10\%\) \\
**Checkpoint freq** & \(1/4\) epoch & \(1/4\) epoch & \(1/4\) epoch & \(1/10\) epoch & \(1/10\) epoch \\ \hline \end{tabular}
\end{table}
Table 8: General parameters

\begin{table}
\begin{tabular}{l c c c c c} \hline  & **SW** & **Longformer** & **IR\(+\)SW** & **LeafJC** & **TreeJC** \\ \hline
**\# epochs** & 8 & 8 & 8 & 1 & 1 \\
**\# instances per epoch** & \(43,961\) & \(7,247\) & \(16,441\) & \(266,853\) & \(266,951\) \\
**max sequence length** & \(512\) & \(4096\) & \(512\) & \(512\) & \(512\) \\
**Document stride** & \(128\) & \(3072\) & \(128\) & \(128\) & \(128\) \\
**max query length** & \(128\) & \(128\) & \(128\) & \(128\) & \(128\) \\
**max answer length** & \(30\) & \(30\) & \(30\) & \(30\) & \(30\) \\ \hline \end{tabular}
\end{table}
Table 9: Dataset-specific parameters for D2DStruct experiments