AttenWalker: Unsupervised Long-Document Question Answering via Attention-based Graph Walking
Year: 2023

Abstract:
Annotating long-document question answering (long-document QA) pairs is time-consuming and expensive. To alleviate the problem, it might be possible to generate long-document QA pairs via unsupervised question answering (UQA) methods. However, existing UQA tasks are based on short documents, and can hardly incorporate long-range information. To tackle the problem, we propose a new task, named unsupervised long-document question answering (ULQA), aiming to generate highquality long-document QA instances in an unsupervised manner. Besides, we propose AttenWalker, a novel unsupervised method to aggregate and generate answers with long-range dependency so as to construct long-document QA pairs. Specifically, AttenWalker is composed of three modules, i.e., span collector, span linker and answer aggregator. Firstly, the span collector takes advantage of constituent parsing and reconstruction loss to select informative candidate spans for constructing answers. Secondly, by going through the attention graph of a pre-trained long-document model, potentially interrelated text spans (that might be far apart) could be linked together via an attentionwalking algorithm. Thirdly, in the answer aggregator, linked spans are aggregated into the final answer via the mask-filling ability of a pre-trained model. Extensive experiments show that AttenWalker outperforms previous methods on Qasper and NarrativeQA. In addition, AttenWalker also shows strong performance in the few-shot learning setting.1

heading: 1 Introduction
text: Textual question answering (QA) is the task of answering questions given textual documents as the context. Previous works can be divided into short-document QA2 methods (Seo et al., 2017)
∗Corresponding author 1We have released our codes and data in https://github.
com/JerrryNie/Unsupervised-Long-Document-QA. 2Usually, the term ‘short-document QA’ is simplified as ‘QA’ in the literature, which refers to the QA task with a short
and long-document QA methods (Nie et al., 2022b). Short-document methods approach, and even outperform humans due to the availability of largescale short-document QA datasets (Rajpurkar et al., 2016). Despite that, long-document methods still lag behind humans by a large margin since annotating long-document QA datasets (Dasigi et al., 2021) is time-consuming and costly.
Intuitively, the high cost of annotating longdocument QA pairs can be alleviated in an unsupervised manner. However, there are only shortdocument unsupervised question answering (UQA) works (Lewis et al., 2019; Pan et al., 2021), which aim to construct a large number of short-document QA pairs in an unsupervised manner and train a QA
context. We emphasize ‘short-document’ QA in this work to distinguish it with ‘long-document’ QA.
13650
model with these QA pairs. Lewis et al. (2019) first propose the UQA task and use unsupervised neural translation to construct QA pairs in a short passage. Pan et al. (2021) raise the unsupervised shortdocument multi-hop question answering (UMQA) task and design a question generation method to build multi-hop questions within two short passages. To break the document length limitation and incorporate long-range information, we propose a more challenging task, i.e. unsupervised long-document question answering (ULQA) task, to generate high-quality long-document QA pairs and train a competitive QA model without any human-labeled long-document QA pairs.
The core challenge of this task is in the modeling of long-range dependency without supervision. To address this issue, we study an attention-driven method to incorporate meaningful long-range information in the constructed QA pairs. Figure 1 illustrates a motivating example of the attention flow in a long document. It is observed that, by walking through the attention edges of a pre-trained model, related spans would be linked and long-range dependency in the document could be constructed. Therefore, long-range information could be also incorporated into QA pairs through these walkable attention patterns among text spans. Thus, we propose AttenWalker, a novel unsupervised framework to generate long-range dependent answers in long-document QA pairs. Specifically, AttenWalker comprises three modules: span collector, span linker and answer aggregator. Firstly, the span collector takes advantage of the constituent parsing and reconstruction ability of a pre-trained model to select informative candidate spans. Secondly, related spans that might be far apart could be connected through local or global attention edges of a long-document pre-trained model. Thirdly, collected spans are aggregated through the reconstruction ability of a pre-trained model.
Extensive experiments on Qasper (Dasigi et al., 2021) and NarrativeQA (Kociský et al., 2018) show that the proposed AttenWalker can effectively model long-range dependency in long-document QA. Besides, AttenWalker also shows strong performance in the few-shot learning setting.
Our contributions are as follows:
• To the best of our knowledge, we are the first to explore unsupervised long-document QA.
• Without the human-annotated long-range knowledge, we propose AttenWalker, a novel
unsupervised long-document QA framework, which can incorporate long-range reasoning via attention-based graph walking.
• Extensive experiments show that AttenWalker outperforms previous methods in unsupervised and few-shot settings.

heading: 2 Related Works
text: Unsupervised Question Answering Unsupervised question answering (UQA) (Lewis et al., 2019) targets at alleviating the data scarcity problem in QA datasets. It focuses on generating QA pairs without supervision and training a QA model on them. Lewis et al. (2019) firstly propose the UQA task. Based on a pure short document, they extract answers via named entity tools and propose a novel cloze translation method to make alignment between cloze question and natural question so as to generate plenty of natural questions. Then, the constructed (context, question, answer) triples are used to train a QA model. Li et al. (2020) use cited documents to generate questions so that the overlapping problem between the generated question and the raw context could be alleviated. Nie et al. (2022a) propose to mine answers beyond named entities in the synthetic QA dataset and improve the model’s ability in dealing with diverse answers. Pan et al. (2021) propose the first unsupervised multi-hop QA framework via multi-hop question generation. However, most of these methods focus on the short-document scenario, while the longdocument setting is still unexplored.
Long-document Question Answering Longdocument question answering (long-document QA) aims to answer questions based on the understanding of a long sequence of text. Previous methods can be divided into end-to-end methods and selectthen-read methods. End-to-end methods (Dasigi et al., 2021) apply sparse attention models to directly answer the question given a long document. Dasigi et al. (2021) uses the Longformer-EncoderDecoder model to make long-range reasoning on a long document and then answer a question. Caciularu et al. (2022) uses a sequence-level objective to improve evidence verification. For the select-thenread methods, Nie et al. (2022b) propose a compressive graph selector network to select questionrelated snippets from the long document and then use the selected short snippets for answer generation. However, despite competitive performances
on long-document QA, these methods heavily rely on supervised QA data and can hardly apply to the low-resource setting.

heading: 3 AttenWalker
text: In this section, we first formalize the task of longdocument QA. After that, the proposed AttenWalker is described in detail.

heading: 3.1 Problem Formulation
text: The setup of long-document QA is as follows. Given a question q and a long document c, where c is often more than 10K tokens, the QA model pθ(a|c, q) needs to produce a free-formed answer a by understanding the long document c and aggregating question-related snippets from c.
In this paper, we consider an unsupervised setting, where only long document c is available. Our aim is to generate synthetic QA pairs (q′, a′) with long-range information and train a competitive long-document QA model via (c, q′, a′) triples.

heading: 3.2 Overview of the Method
text: The proposed AttenWalker focuses on incorporating long-range information via a well-designed answer generator. Specifically, AttenWalker comprises three modules: Span collector, Span linker, and Answer Aggregator. As shown in Figure 2, the Span Collector first partitions the Long Document into different spans via Constituent Parsing and T5 Reconstructor. Secondly, a Span Linker is used to capture long-range dependency among these Par-
titioned Spans via Attention Graph Walking. This module aims to walk through local and global attention edges to link semantically related spans (which could be far apart in the long text) for aggregating answers. Thirdly, an Answer Aggregator combines all the Linked Spans via the reconstruction ability of a BART model to generate the answer.

heading: 3.3 Span Collector
text: To determine the candidate spans for generating the answers, we propose a Span Collector. Specifically, as shown in Figure 3, it first seeks for candidate spans via constituent parsing and then reconstructs masked text via a pre-trained T5 model (Raffel
et al., 2020) to select informative spans for answer generation. Each masked text serves as an input to the T5 model3. The reconstruction loss is:
L = − 1 T
T∑
i=1
log(p(yi)) , (1)
where L is the reconstruction loss of the specific span. T is the number of tokens in the ground truth span and p(yi) is the T5 predicting probability of the i-th token yi in the ground truth span. As shown in Figure 3, “sentence encoder” has the largest reconstruction loss. Thus, we select it as one of the candidate spans. Meanwhile, its parent spans (i.e. “as sentence encoder”) and its child spans (“sentence” and “encoder”) will not be selected for redundancy concern.

heading: 3.4 Span Linker
text: The proposed Span Linker is to incorporate longrange information in AttenWalker. It can effectively incorporate long-range dependency through attention-based graph walking. The Span Linker is composed of two sub-modules: a Span Graph Constructor and an Attention-based Graph Walker.
Span Graph Constructor To explore possible relations among spans, token-level attention scores4 of the LED pre-trained model (Beltagy et al., 2020) can be used. As shown in Figure 1, based on the spans acquired in Section 3.3, we
3In practice, we use <extra_id_0> as the mask token. The <mask> token in Figure 3 is just for illustrative purpose.
4The token-level attention scores are acquired through the encoder part of the LED model. We consider each span graph for each Transformer layer and head.
build a span graph G via attention scores between each pair of tokens as shown in Figure 4. For span i and span j, where i, j ∈ G, if there are any attention edges from one of the tokens in span i to one of the tokens in span j, there is an edge from span i to span j. Motivated by the idea of max-pooling technique (Dumoulin and Visin, 2016), to obtain the most obvious relation in each pair of spans, the edge weight eij from span i to span j can be calculated by the maximum attention weight between any pair of tokens in between:
eij = max m∈Gi,n∈Gj ,(m,n)∈Gt wm,n , (2)
where Gi and Gj are tokens in span i and span j. (m,n) is an edge in token-level attention graph Gt. wm,n is the attention weight of the edge (m,n).
In the LED encoder, there are local and global attention weights among the tokens in a long document. Both two types of weights can serve as the token-level edge weights wm,n in Eqn 2. In this work, we propose to consider both types for span graph construction. If there is a local attention weight lm,n from token m to token n, we directly assign the value to wm,n. Otherwise, the global attention is considered: we insert a “
” at the beginning of each paragraph and set global attention for each of it (Appendix B). It means that each “
” can attend to every token in the long sequence and vice versa. Each “
” could serve as the representative of the paragraph that follows it. Therefore, “
” can be regarded as a bridge to two spans in different paragraphs, which could be far apart and could not be accessible to each other only through the local attention mechanism. To build the “bridge” from paragraph pi to paragraph pj , we first select one of the K tokens tpi with the maximum attention score to the representation of “
” spi . Next, for the representation of spi , L highest attention scores to other “
” tokens are selected. For one of the L “
” tokens spj in paragraph pj , we can access its maximum M attention weights to the corresponding M tokens (tpj ) in paragraph pj . For each tpi , its attention to the target token tpj can be:
gtpi ,tpj = 3 √ wtpi ,spi × wspi ,spj×, wspj ,tpj , (3)
where gtpi ,tpj is the global attention score from token tpi to token tpj . wtpi ,spi , wspi ,spj , wspj ,tpj are attention scores directly acquired according to the global attention in the LED model. Here,
we use the geometric mean of the attention edge weights from tpi to tpj as the approximate attention weight of the edge (tpi , tpj ). Thus, if there is no direct (local) attention from tpi to tpj but a global path, we can use gtpi ,tpj as the “lost” wtpi ,tpj .
Attention-based Graph Walker Span linking can be done via attention-based graph walking on the constructed span graph. Essentially, the proposed graph walker collects interrelated spans via traversing the span graph. Its main algorithm is based on the Depth First Search (Even, 2011). As shown in the lower half of Figure 5, starting from the first span “The main contributions”, graph walking continues searching for accessible span. Thus, it successfully links to the span “a single-layer forward recurrent neural network”. Then, starting from this linked span, “Long Short-Term Memory” is also linked because of the high weight 0.48 between it and “a single-layer forward recurrent neural network”. To decide whether the edge is of “high weight”, we set a pre-defined threshold τ on the edge weight. In other words, the original span graph G can be pruned as a new graph G′ via:
G′ = {e|e ∈ G, we > τ} , (4)
where we is the weight of edge e. Finally, spans on the walking path are clustered together, which will be used in the following section.

heading: 3.5 Answer Aggregator
text: The proposed Answer Aggregator produces the final answer by aggregating the linked spans in Section 3.4. To achieve this goal, we take advantage of the reconstruction ability of a BART model (Lewis et al., 2020). For instance, the linked spans in the lower half of Figure 5 can be formalized into the input to BART: “The main contributions <mask>
a single-layer forward recurrent neural network <mask> sentence information”. Finally, the output can be an integral text as the answer: “The main contributions were to develop a single-layer forward recurrent neural network for sentence information”.

heading: 3.6 Question Generation
text: Question generation (QG) is applied when we obtain the answer and all the sentences the linked spans from. We use the QG Operator in Unsupervised Multi-hop QA (Pan et al., 2021) as the QG module in our work. We concatenate the answer from Section 3.5 with all the aforementioned sentences into the QG module to generate a question.

heading: 3.7 Two-Pass Scheme for Long-Range Reasoning
text: In the pre-trained LED model, query, key, and value matrices of the global attention are just copied from the corresponding matrices in the local attention5. To further improve the ability of global attention in long-range reasoning, we design a two-pass scheme to construct long-document QA pairs as shown in Figure 6. In the first pass, only local attention is used in the proposed Span Graph Constructor. Then, an LED model is fine-tuned on these QA pairs with global and local attention as described in Appendix B. This step aims to improve the ability of the query, key, and value matrices, especially for global attention. In the second pass, based on the fine-tuned LED model, both local and global attention are considered to construct the span graph for attention walking. Hence, further knowledge with global attention is incorporated into the finally constructed QA pairs.
5https://github.com/allenai/longformer

heading: 4 Experimental Setup
text: We evaluate the proposed AttenWalker on Qasper (Dasigi et al., 2021) and NarrativeQA (Kociský et al., 2018). In particular, for Qasper, the answer types in this dataset can be extractive, abstraction, yes/no, or unanswerable. Yet, according to our analysis (Appendix A), QA instances with yes/no or unanswerable answers cannot properly evaluate the ability of long document reasoning. Therefore, we only focus on the extractive and abstractive QA instances in this work. The datasets splitting and processing details are in appendix C.1.
We use the documents in the Qasper training set to construct QA pairs for training the QA model and do Qasper-related experiments. The long documents in the training set of NarrativeQA are used similarly. The dataset construction details can be found in Appendix C.2. What’s more, the setting of the long document QA model trained on the constructed dataset can be referred to C.3.

heading: 5 Experiment
text: In this section, we first discuss the main results of AttenWalker on Qasper and NarrativeQA, and then further analyze the proposed method.

heading: 5.1 Main Results
text: Since there is no direct unsupervised method for long documents, we select competitive baselines from unsupervised short-document QA (UQA) and unsupervised short-document multi-hop QA (UMQA). The UQA works include UNMT (Lewis
et al., 2019), RefQA (Li et al., 2020), DiverseQA (Nie et al., 2022a). The UMQA work is MQAQG (Pan et al., 2021). The adaptation of them to long documents is described in Appendix E. Following Dasigi et al. (2021) and Kociský et al. (2018), we use answer F1 score (including extractive F1, abstractive F1 and overall F1 in this paper) as the evaluation metrics on Qasper dataset, while we use Bleu-1/4 (Papineni et al., 2002), Meteor (Denkowski and Lavie, 2011) and Rouge-L (Lin, 2004) for evaluation on NarrativeQA dataset.
As shown in Table 1, in the Supervised block, it can be found that an LED model trained on the synthetic dataset of AttenWalker can further make improvements when it is continuously fine-tuned on the supervised data, especially on Qasper, showing that the proposed method can effectively alleviate the data scarcity problem in Qasper. In the Unsupervised block, the proposed AttenWalker outperforms all baselines by a large margin in the fully unsupervised setting. showing a competitive performance of AttenWalker.

heading: 5.2 Ablation Study
text: We conduct an extensive ablation study on different components of AttenWalker. As shown in Table 2, the effectiveness of each component can be shown according to four observations.
Effects of the span collector. As shown in Table 2, the performance drop of “w/ Random Span Collector” illustrates that randomly selecting candidate spans could introduce much noise and harm the quality of the generated QA pairs.
Effects of the span linker. From the performance drop in setting “w/ Un-pre-trained LED” and “w/ Embedding Linker” as shown in Table 2, it can be known that the attention information stored in the LED parameters is rather useful for constructing high-quality long-document QA pairs. Besides, the competitive result of “w/ Embedding Linker” suggests that embedding information can benefit the QA pair construction. In addition, the performance of “w/o Global” illustrates that global attention is also an essential factor in improving the quality of the generated long-document QA pairs.
Effects of the answer aggregator. According to “w/ Answer Connector” in Table 2, the performance drops when simply connecting spans. It shows that connecting spans with proper transition words is crucial for generating a high-quality answer.
Effects of the two-pass scheme. The Two-Pass Scheme is helpful in improving the performance
of the model as shown in the “w/ Single Pass” and “w/ Single Pass + Global” setting from Table 2. It suggests that local and global attention can benefit from the parameters of a fine-tuned LED model.

heading: 5.3 Effects on Long-Range Modeling
text: AttenWalker aims to incorporate long-range information in the QA pair construction. To further understand it, an experiment with varied document lengths is conducted. As shown in Figure 7, in essence, “w/o Global” is only to use local attention while “w/ Embedding” denotes a situation that both global and local are not used. When the document length is small (1-2,000), the performances of different methods are comparable. However, with the increasing document length, the gap among methods becomes larger. It shows that AttenWalker can model long-range dependency effectively. Furthermore, it is observed that MQA-QG performs worse than “w/ Embedding” when the document length is large. It can be explained in two aspects. Firstly, MQA-QG could hardly capture long-range information. Secondly, MQA-QG is only a reduced version of “w/ Embedding”, which can only link two spans via literal matching (Section 5.6).

heading: 5.4 Effects of Attention Weights
text: We design three different span graph construction strategies to further investigate their influences on the proposed method. As shown in Table 3, the “Max-Pooling” strategy outperforms the other two strategies by large margins. It can be explained that the “Max-Pooling” strategy can capture the most obvious (and probably important) relation between two spans, which is useful in QA pair construction.

heading: 5.5 Few-Shot Learning
text: We conduct the few-shot learning experiment to explore the effectiveness of AttenWalker in different low-resource settings. As shown in Figure 8, with the increasing of the labeled training size, the performance of the model trained on the synthetic QA pairs from AttenWalker is consistently better than that of MQA-QG in Qasper and an LED model. It is because the Qasper dataset is quite small, which makes the synthetic dataset rather beneficial. Besides, in the NarrativeQA, AttenWalker reaches the best performance from 10 to 10,000 training sizes and then becomes comparable with MQA-QG. It can be explained that a large number of training sizes would narrow the gaps between them.

heading: 5.6 Case Study
text: In this section, we first analyze an example with the proposed two-pass scheme to explore the benefits of attention changes. Then, we compare two QA examples between AttenWalker and MQA-QG.
As shown in Figure 5, with an LED model, the spans “The main contributions” can be connected with “a single-layer forward recurrent neural network” and “[7]”. Yet, after fine-tuning the model with generated QA instances, a more reasonable path “The main contributions” -> “a single-layer
forward recurrent neural network” -> “Long ShortTerm Memory” is strengthened and the link to the trivial span “[7]” is weakened. It can be explained that after fine-tuning, noise in the LED attention edges is reduced, further improving the span linking and the quality of the generated QA instances.
In addition, as shown in Table 4, we compare two QA pairs generated by AttenWalker and the bestperformed baseline, MQA-QG. There are three key observations from the table. Firstly, AttenWalker can synthesize multiple spans into an answer whereas MQA-QG can only link the repeated text. Secondly, MQA-QG fails in long-range modeling since repeated spans could probably be in a short distance. Thirdly, the generated answer by AttenWalker is much more informative than MQAQG’s. In the long-document setting, answering a question might need synthesizing many pieces of information from different parts of the document. Therefore, the informativeness property of AttenWalker can be a better method for this setting.

heading: 6 Conclusion
text: We study a new task, named unsupervised longdocument question answering, and propose AttenWalker, an unsupervised method to incorporate long-range information in QA pairs via graph walking. Extensive experiments show the strong performance of the proposed method. We believe that this work can be an important step in the long-document reasoning with a low-resource setting.
Limitations
Despite the strong performance of the proposed AttenWalker. There is still large room for improving efficiency. For example, the time cost of our method is still high. Since we need to search for all Transformer layers and heads to find potentially re-
lated spans, the dataset construction could be quite time-consuming. Therefore, an algorithm could be designed in the future to pre-select proper layers and heads for attention-based graph walking, which would save much time in dataset construction.

heading: Acknowledgements
text: The work is supported by National Key R&D Plan (No. 2020AAA0106600), National Natural Science Foundation of China (No.62172039, U21B2009 and 62276110), in part by CCF-AFSG Research Fund under Grant No.RF20210005, and in part by the fund of Joint Laboratory of HUST and Pingan Property & Casualty Research (HPL). We thank the ACL reviewers for their helpful feedback. We would like to acknowledge Rong-Cheng Tu for the helpful discussions.

heading: A Analysis of Qasper Question Types
text: In this section, we analyze the contributions of the long document to different question types in the original Qasper dataset. As shown in Table 5, when the full text is absent from the input, the performance drops dramatically on the “Extractive” and “Abstractive” answer types. However, for “Yes/No” answers, the performance only drops a little, also keeping a competitive F1 score of 64.84. Besides, the performance of “Unanswerable” answers become unexpectedly better. Based on these observations, we argue that “Yes/No” and “Unanswerable” types are not suitable for testing the ability of long-range reasoning. Therefore, we only use “Extractive” and “Abstractive” in our experiments.

heading: B Details in Fine-Tuning the LED Model
text: Similar to the input setting in Dasigi et al. (2021), for a long document, we prepend a special token 
 before each paragraph. And then we send the preprocessed long document into an LED model. For example, assume that there is a long document: [t1,1, t1,2, ..., tp,1, tp,2, ..., tP,PN−1, tP,PN ], where ti,j is the i-th token in paragraph j, P is the number of paragraphs, PN is the number of tokens in paragraph P . After inserting the special token 
, the input can be [
, t1,1, t1,2, ..., 
, tp,1, tp,2, ..., tP,PN−1, tP,PN ].

heading: C Preprocessing Details of Qasper and NarrativeQA
text: C.1 Datasets We evaluate the proposed AttenWalker framework on two long-document QA datasets6: Qasper (Dasigi et al., 2021) and NarrativeQA (Kociský et al., 2018). Qasper 7 is a dataset (license: CC BY 4.0) for answering questions based on long scientific papers. The questions are annotated based on the abstract of a scientific paper and the answer is annotated by understanding the entire paper’s content. The answer types in this dataset can be extractive, abstraction, yes/no or unanswerable. Yet, according to our analysis (Appendix A), QA instances with yes/no or unanswerable answers cannot properly evaluate the ability of long
6The datasets used are originally created for research, which is consistent with our purpose.
7https://allenai.org/data/qasper
document reasoning. Therefore, we only focus on the extractive and abstractive QA instances in this work. NarrativeQA (license: Apache-2.0) is a QA dataset established upon books and movie scripts of long text sequences. Given summaries of the books/scripts, annotators need to generate corresponding QA pairs where answers are free-formed. Table 6 shows the statistics of these two datasets. We use version 0.3 of Qasper dataset8 for our experiment, where empty documents are removed. For NarrativeQA, we use the dataset9 provided in Huggingface, which is a well-formed dataset. Thus, no extra cleaning step is needed.
C.2 Unsupervised Long-Document QA Dataset Construction
The datasets constructing process is shown in Figure 6. Specifically, we first extract sentence constituents from a long document using Berkeley Neural Parser (Kitaev et al., 2019). Then, a t5-small model is used in reconstruction-based span selection. In the span linker, we use led-base-16384 to acquire the token-level attention graph for span linking. The threshold τ is set to 0.45. In the answer aggregator, we use the bart-large model to convert spans into an integral answer. Then, an operator 10 is used to generate questions. In the first pass, the generated dataset is used to train an led-base-16384 model. In the second pass, the trained LED model is first used to provide the token-level attention graph as mentioned above. Besides, the global attention scores are also used to complete the attention graph (described in the paragraph “Span Graph Constructor”). The globalattention-related hyperparameters K,L,M . are all set as 3. The construction of the Qasper-documentbased dataset costs 12 hours on 4 11GB GPUs while 15 hours on the NarrativeQA-documentbased dataset.
C.3 Long-Document QA Model Setting
We use led-base-16384 as the QA model throughout all of our experiments. The input format is described in Appendix B. We searched over batch sizes {2, 4, 8, 16, 32}, learning rates {3e-5, 5e-5, 8e-5, 1e-4}, warmup proportions {10%, 20%, 30%, 40%, 50%}, epochs {2, 4, 5, 6, 8, 10}. And the final batch size is 16, the learning rate is 5e-5, the
8https://allenai.org/data/qasper 9https://huggingface.co/datasets/narrativeqa
10https://github.com/teacherpeterpan/Unsupervised-Multihop-QA
warmup proportion is 30% and the epoch number is 5. We chunk the maximum input length into 13,000 tokens and set the attention window size to 640 so that the LED model in this configuration can be trained on four 11GB GPUs in 3 hours. Despite this relatively limited setting, we find that the performance of the LED model is comparable to the default configuration.

heading: D Statistics of the Generated Datasets
text: In this section, we summarize the long-document QA datasets generated by AttenWalker. For saving time in QA pair generation, for each document, we randomly sample at most 32 linked span sets for QA-pair generation. The final generated results are shown in Table 7.

heading: E Details in the Implementing of Baselines
text: Since current UQA methods cannot directly apply to the ULQA setting, we make further modifications and describe our implementation in detail.
UNMT (Lewis et al., 2019) To generate QA pairs with UNMT, each paragraph in the long document is used as a short context for QA generation. When training the LED model, the question generated by UNMT and the full long document is concatenated into a full sequence so as to train the model.
RefQA (Li et al., 2020) Similar to UNMT, each paragraph in the long document is separately used to generate QA pairs.
DiverseQA (Nie et al., 2022a) Similar to UNMT and RefQA, each paragraph is selected as a short document. And then, answers of diverse types are extracted from the document. Finally, each question is generated based on the answer and the short document.
MQA-QG (Pan et al., 2021) For MQA-QG, in a long document, two paragraphs are randomly sampled. These two paragraphs are then input into the MQA-QG for generating multi-hop QA pairs. Finally, the generated question is concatenated with the long document as the input to train the LED model.
ACL 2023 Responsible NLP Checklist

heading: A For every submission:
text: 3 A1. Did you describe the limitations of your work?
Section "Limitations".
A2. Did you discuss any potential risks of your work? Not applicable. Left blank.
3 A3. Do the abstract and introduction summarize the paper’s main claims? Section "Abstract" and "1. Introduction".
7 A4. Have you used AI writing assistants when working on this paper? Left blank.
B 3 Did you use or create scientific artifacts? Section "G.1 Datasets".
3 B1. Did you cite the creators of artifacts you used? Section "G.1 Datasets".
3 B2. Did you discuss the license or terms for use and / or distribution of any artifacts? Section "G.1 Datasets".
3 B3. Did you discuss if your use of existing artifact(s) was consistent with their intended use, provided that it was specified? For the artifacts you create, do you specify intended use and whether that is compatible with the original access conditions (in particular, derivatives of data accessed for research purposes should not be used outside of research contexts)? Section "G.1 Datasets".
B4. Did you discuss the steps taken to check whether the data that was collected / used contains any information that names or uniquely identifies individual people or offensive content, and the steps taken to protect / anonymize it? Not applicable. Left blank.
B5. Did you provide documentation of the artifacts, e.g., coverage of domains, languages, and linguistic phenomena, demographic groups represented, etc.? Not applicable. Left blank.
3 B6. Did you report relevant statistics like the number of examples, details of train / test / dev splits, etc. for the data that you used / created? Even for commonly-used benchmark datasets, include the number of examples in train / validation / test splits, as these provide necessary context for a reader to understand experimental results. For example, small differences in accuracy on large test sets may be significant, while on small test sets they may not be. Section "G.1 Datasets".
C 7 Did you run computational experiments? Left blank.
C1. Did you report the number of parameters in the models used, the total computational budget (e.g., GPU hours), and computing infrastructure used? Not applicable. Left blank.
The Responsible NLP Checklist used at ACL 2023 is adopted from NAACL 2022, with the addition of a question on AI writing assistance.
3 C2. Did you discuss the experimental setup, including hyperparameter search and best-found hyperparameter values? Section "G.2 Unsupervised Long-Document QA Dataset Construction" and "G.3 Long-Document QA Model Setting".
3 C3. Did you report descriptive statistics about your results (e.g., error bars around results, summary statistics from sets of experiments), and is it transparent whether you are reporting the max, mean, etc. or just a single run? Section "5.3 Effects on Long-Range Modeling", "A. Maximum Evidence Span Range Analysis" and "B. Multi-Hop Analysis"
3 C4. If you used existing packages (e.g., for preprocessing, for normalization, or for evaluation), did you report the implementation, model, and parameter settings used (e.g., NLTK, Spacy, ROUGE, etc.)? Section "5.1 Main Results".
D 7 Did you use human annotators (e.g., crowdworkers) or research with human participants? Left blank.
D1. Did you report the full text of instructions given to participants, including e.g., screenshots, disclaimers of any risks to participants or annotators, etc.? Not applicable. Left blank.
D2. Did you report information about how you recruited (e.g., crowdsourcing platform, students) and paid participants, and discuss if such payment is adequate given the participants’ demographic (e.g., country of residence)? Not applicable. Left blank.
D3. Did you discuss whether and how consent was obtained from people whose data you’re using/curating? For example, if you collected data via crowdsourcing, did your instructions to crowdworkers explain how the data would be used? Not applicable. Left blank.
D4. Was the data collection protocol approved (or determined exempt) by an ethics review board? Not applicable. Left blank.
D5. Did you report the basic demographic and geographic characteristics of the annotator population that is the source of the data? Not applicable. Left blank.

heading: Future Work
text: Therefore, an algorithm could be designed in the future to pre-select proper layers and heads for attention-based graph walking, which would save much time in dataset construction