Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Authors: Inderjeet Jayakumar Nair
Shwetha S
Apoorv Saxena
Koustava Goswami
Venue: EMNLP 2023 Findings (2023)
Keywords: Long Document Question Answering
Large Language Model
Zero-shot Prompting
Evidence Retrieval
TL;DR: We propose a zero-shot approach to retrieve relevant paragraphs from a document to answer a question.

Abstract: We address the task of evidence retrieval for long document question answering, which involves locating relevant paragraphs within a document to answer a question. We aim to assess the applicability of large language models (LLMs) in the task of zero-shot long document evidence retrieval, owing to their unprecedented performance across various NLP tasks. However, currently the LLMs can consume limited context lengths as input, thus providing document chunks as inputs might overlook the global context while missing out on capturing the inter-segment dependencies. Moreover, directly feeding the large input sets can incur significant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAI's GPT variants). To address these challenges, we propose a suite of techniques that exploit the discourse structure commonly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling a more comprehensive understanding and analysis of relationships between different parts. We retain $99.6$% of the best zero-shot approach's performance, while processing only $26$% of the total tokens used by the best approach in the information seeking evidence retrieval setup. We also show how our approach can be combined with *self-ask* reasoning agent to achieve best zero-shot performance in complex multi-hop question answering, just $\approx 4$% short of zero-shot performance using gold evidence.

Drilling Down into the Discourse Structure with LLMs for Long Document Question Answering
Inderjeet Nair*1 Shwetha Somasundaram #2 Apoorv Saxena', Koustava Goswami
+University of Michigan, Ann Arbor, MI
2 Adobe Research, India Anair@umic .dd
(shsomasu, apoorvs, wop.rrrs
Abstract
a) Long documents often exceed the maximum to- ken limit of existing transformer- Pretrained Language Models (PLMs) (Devlin et al., 2019; Liu et al., 2019; Lewis et al., 2020; Raffel et al., 2020), posing a challenge in directly processing their con- tent to extract pertinent information (Dong et al., 2023). b) The information required to answer a question is often dispersed across different sections or paragraphs within the document which may re- quire sophisticated reasoning process to identify and extract the relevant information (Nie et al., 2022). c) Processing the entire document to find answers can be computationally expensive and in- efficient (Dong et al., 2023).
We address the task of evidence retrieval for long document question answering, which in- volves locating relevant paragraphs within a document to answer question. We aim to assess the applicability of large language mod- els (S) in the task zero-shot long docu- ment evidence retrieval, owing their unprece- dented performance across various NLP tasks. However, currently the LLMs can consume lim- ited context lengths input, thus providing document chunks as inputs might overlook the global context while missing out capturing the inter-segment dependencies. Moreaver, di- rectly feeding the large imput sets can incur sig- nificant computational costs, particularly when processing the entire document (and potentially incurring monetary expenses with enterprise APIs like OpenAl's s GPT variants). To address these challenges, we propose a suite of tech- niques that exploit the discourse structure com- monly found in documents. By utilizing this structure, we create a condensed representation of the document, enabling more comprehen- sive understanding and analysis of relationships between different parts. We retain 99.6% of the best zero- shot approach's 's performance, while processing only 26% of the total tokens used by the best approach the information seeking evidence retrieval setup. We also show how approach can be combined with seif-ask reason- ing agent to achieve best zero-shot performance in complex multi-hop question answering, just 4% short of zero-shot performance using gold evidence.
One popular approach for LDQA is the retrieve- then-read method (Zheng et al., 2020; Gong et al., 2020; Nie et al., 2022: Ainslie et al., 2020, 2023), where relevant paragraphs document provide the answer. major draw- back of existing works is reliance on supervised fine-tuning for the evidence selection phase, ex- hibiting poor generalization on out-o--distribution data (Thakur et al., 2021).
retrieved from the
Given the remarkable few-shot/zero-shot perfor- mance and enhanced generalization capabilities demonstrated by Large Language Models (LLMs) across various Natural Language Generation and Understanding tasks (Brown al., 2020; Chen al., 2021; Rae al., 2022: Hoffmman et al., 2022; Chowdhery et al., 2022), we investigate the poten- tial of leveraging these LLMs for zero-shot evi- dence retrieval. Notably, LLMs that have been in- struction fine-tuned (Wei et al., 2022a; Chung et al., 2022) or trained using Reinforcement Learning with Human Feedback (Bai et al., 2022; Ouyang et al., 2022) exhibit exceptional generalization per- formance even on unseen tasks (Ouyang al., 2022: Min et al., 2022; OpenA1, 2023). Thus, we explore the feasibility of utilizing LLMMs for zero- shot evidence retrieval. However, LLMs, which are based on transformer architecture (Vaswani
1 Introduction
Long Document Question Answering (LDQA) is a complex task that involves locating relevant evi- dence from lengthy documents to provide accurate answers to specific questions (Dasigi et al., 2021).
LDQA challenging for the following reasons
Equal contribution
Work done at Adobe Rescarch, India

---

al., 2017), are limited by their context length and suffer from expensive inference times that in- crease quadratically with the number of tokens in the input. Additionally, utilizing enterprise LLM solutions such as OpenAT's 3pt-3.5- 5-turbo, dext-davinci-003, gpt-4, etc." to process an en- tire long document without optimizations would in- cur significant monetary costs. This highlights the need for LLM-based evidence retrieval solution that can achieve faster and more cost-effective infer- ence by selectively processing relevant portions of the document, without compromising downstream performance.
2 Related Work
2.1 LLMs in Retrieve-Then-Real Approaches
The retrieve-then-read (Green Jr et al., 1961; Chen et al., 2017; Wang al., 2018; Das et al., 2019; Guu et al., 2020) approach is widely adopted technique in open-domain (Voorhees et al., 1999; Dunn al., 2017: Joshi al., 2017: Zhu et al., 2021), multi- document question answering (Yang et al., 2018; Perez et al., 2020; Ferguson et al., 2020) and long- document question answering (Pereira et al., 2023), In this approach, LLMs are utilized specifically for the reader component, which generates responses based the relevant fragments retrieved by the triever (Pereira et al., 2023). Although LLMs have been utilized as recisionnmmakngg agents browser interactions for document retrieval (Nakano et al., 2022), their direct application for fine-grained evi- dence retrieval has not been extensively explored to the best of our knowledge. On that front, our paper is the first to evaluate the applicability of LLMs for evidence retrieval.
To overcome these challenges, we harness the inherent discourse structure commonly present in long documents. This structure encompasses the or- ganization of topics, semantic segments, and infor- mation flow, enabling effective information search and knowledge acquisition for question answer- ing, (Guthrie et al., 1991; Meyer et al., 1980; Taylor and Beach, 1984; Cao and Wang, 2022; Dong et al., 2023; Nair et al., 2023). Utilizing this valuable structure, we construct condensed representation of the document by replacing the content within each section with corresponding summary. This condensed representation is then fed to the L.MM, enabling efficient processing of tokens while al- lowing the model comprehensively analyze the entire input context for identifying relevant sec- tions. Thereafter, the content within each relevant section is further processed by the LLM for fine- grained evidence retrieval. We call our proposed approach 3 (Drilling Down into the Discourse) due to the nature of the solution described above.
2.2 Chaining LLMs Runs for Question Answering
Chaining in LLMs refers the task of breaking complex overarching task into a sequence of fine- grained targetted sub-tasks where the information generated by a particular run of the scquence is passed to the subsequent runs (Wu et al., 2022a,b), This allows for realizing powerful machine learn- ing applications without requiring any changes to the model architecture (Tan et al., 2021; Betz et al., 2021; Reynolds and McDonell, 2021). LangChain has implemented procedures using chaining for ev- idence retrieval and question answering in long documents. They employ three chaining variants (map-reduce, map-rerank, and refine) 2, which processes document chunks individually and ag- gregate the information from each chunk to derive the final answer. This implementation, however, processes the entire document input resulting in significant compute and monetary cost.
Our approach undergoes evaluation in two dis- tinct settings: Information Seeking and Multi-hop Reasoning in Question Answering. the infor- mation seeking experiments, our approach retains the best zero-shot -tate-of-the-art (SoTA) results, while only utilizing 26% of the tokens employed by the SoTA approach. Additionally, we exam- ine the robustness of our model across various document lengths and analyze the number of to- kens required and latency for different zero-shot approaches. Moreover, we explore the integration of our approach with other zero-shot techniques within an agent framework designed to break down intricate queries into a sequence of simpler follow- up queries.
2.3 Evidence Retrieval for LDQA
Prior evidence retrieval approaches typically em- ploy following two mechanims which are trained by supervised fine-tuning local processing to han- dle individual document chunks with occasional information flow between them (Gong et al., 2020)
'nttps://python httts:://yython..agghaiii mooinssddccumnnt.

---

thoune-tratur
the docarent
reprrsatation (623.1)
Selecing opt
rsteenat scetim from
Quesior
(Ira STEPS
proggaphe
ctisecctier: The sytrabices Iiclatee Nekk The al sttnkkees io
Erooent asccson
better anderttasd sear atterance
Exing the as costot, previde
Ir
rystens
...
est rexst toe aou the
"Wad are Us bactires facerrrrde by this
are basalinne putprreeree by vark?t
baselow todels (Tranee Traners Pporia
Fire aaraginss lis that contatios rateearr
List olt pectice rurce that mo relovant for
enipast with contereanaated rapare the ard rolomatic
irousde
Exosiaen Weat und busellno
paction the certisn vin intessnn
resganne it
Quatiio: What are the baselines ontperforaed by this work?
Retevant Seetioon
Wedevint Pargraah lDs
Asewar
Experarents, Recults and Anolysis
126, 28,411
TransferTransfo and Hyorid
Figure I: An illustration of the end-end pipeline of D Given a question and long document with discourse structure that indicates sections, subsections etc., we first identify sections that are relevant for answering the question. Following this step we select relevant paragraphs from the paragraphs in the relevant sections. In the final step. pass these relevant paragraphs LLM for question answering.
and global processing to aggregate the informa- tion from each chunk to identify relevant para- graphs (Zheng et al., 2020; Ainslic et al., 2020; Nie et al., 2022; Ainslie et al.. 2023). Inspired by this strategy, our method represent each section using its corresponding summary in local process- ing step and, in the global processing mechanism, we utilize a suitable verbalizer to concatenate the summaries from each section.
discourse that categorizes information coherently based on topical similarity (Cao and Wang, 2022; Nair et al., 2023; Dong et al., 2023). This inherent discourse structure serves as a valuable framework, enabling effective categorization of information and facilitating clear and logical flow of ideas within the document.
Drawing from these insights, we posit that en- capsulating the essence of a section through its name and content summary would yield valuable cues in determining its relevance for answering spe- cific questions. Thereafter, we emulate the above- described cognitive process by fine-grained anal- ysis of relevant sections to extract evidence para- graphs. This methodology offers three key advan-
3 Dd: Drilling Down into the Discourse
3.1 Problem Formulation
In LDQA, a question is asked for document D = [21, PPp] where (1 i S n) is the 7th paragraph in the natural reading order of D. The task of LDQA is retrieve set of rele- vant paragraphs Eg C D and generate a free-form answer a based on q and D (Dasigi et al., 2021; Nie et al., 2022). Due to the length of the docu- ments, often exceeding 5K tokens, we employ the retrieve-then-readd strategy. This approach involves first determining Eg and subsequently generating a using only q and Eq.
tages:
(1) By condensing each section with its name and summary, we can effectively reduce the document's 's token count, enabling LLMs to analyze the entire context and make accurate inferences.
(2) By efficiently filtering out irrelevant sections in the initial stage, our method reduces the number of tokens processed.
(3) Our method is applicable for any instruction- following LLMs (Ouyang et al., 2022; Min et al., 2022: OpenAl, 2023), enabling zero-shot applica- tion without the need for architectural modifica- tions.
3.2 Motivation
The cognitive strategy employed by humans to search for relevant information from document entails systematic approach of first categorizing the information within the document to determine relevant coarse segments and then conducting deeper analysis of the relevant categories extract fine- fine-grained segments (Guthrie and Kirsch, 1987; Guthrie et al., 1991; Guthrie and Mosenthal, 1987). Long documents often possess a well-structured
3.3 Methodology
Instead of representing a document D as a or- dered set of constituent paragraphs, we repre-

---

sent D [St, S SxI. where St(1 S S k) denotes 7th section, such that, name(S;) and paragraphs(S)) denotes its name heading and the list of constituent paragraphs respectively (paragraphs(S.) | where |Si| denotes number of constituent paragraphs). Note that, E = n. Inspired by the cognitive process of knowledge acquisition / /ifformtiin scarch for question answering, our approach first finds the relevant sections that may answer the question and then, analyses the paragraphs from the relevant sec- tions for fine-grained evidence paragraph retrieval. (Figure 1)
tiple zero-shot strategies to achieve this step. We, first, obtain set of all paragraphs Pa associated with Rg.
Pq U SERq
paragraphs(S)
Thereafter, one of the following strategy can be employed for fine-grained retrieval:
MONOT5: This employs MonoT5 (Nogueira et al., 2020), which sequence-to-seque model trained over the task of Document Re- ranking (Nguyen et al., 2016), to select the most relevant paragraphs from Pq.
3.3.1 Finding Relevant Sections
2. BASE: Each paragraph from Pa marked with an identifier and then, these identifier annotated paragraphs are concatenated with a newline separator. Thereafter, we prompt the LLM in a zero-shot manner to generate all paragraph identifiers whose corresponding paragraph is relevant to q. If the number of paragraphs in P. exceeds the maximum con- text length of LLM, we make multiple LLM calls. In each call, we fit the maximum num- ber of paragraphs that can fit into the con- text length, ensuring that paragraphs are not 'chopped'.
The crux of this step is to represent the con- tent in each section S; by the summary of paragraphs($t). Summarization (El-Kassas et al., 2021) refers to the task of generating a concise summary for given input that captures its main idea within limited number of tokens, effectively conveying its topical essence. We denote the sum- marization operation by S, for which we have used bart-large (Lewis et al., 2020) fine-tuned over CNN/Daily Corpus (Nallapati et al., 2016). Thereafter, we represent the entire document as follows:
Section: name(S)) (St)
((paragraph((S.) Section: name (Sa)
3. HIER BASE: In our approach, we adopt a two- step process to capture the essence of cach paragraph. Firstly, we represent paragraphs using their corresponding summaries obtained through S. Following that, employ the BASE strategy identify potentially relevant candidates. In the next stage, we apply the BASE technique once again, this time consid- cring the original content of the paragraphs, to pinpoint the most relevant ones.
S (pparagraphs(S.)
Section: name(Sk)
S(paragraphs(S.))
An instruction is passed to an LLM involving the above representation to identify all the sections that are relevant to q. Due to this condensed represen- tation, the LLM can process the entire document context enabling comprehensive analysis of long range dependencies for accurate inference. Let the set of sections identified as relevant be denoted by Rg C D.
In our experimental evaluation, we also ex- plore the effectiveness of chaining these strategies in succession. One such method, called BASE + MoNoT5, combines the BASE strategy with MoNoTS. This approach initially identifies a rel- evant set of paragraphs using the BASE strategy and subsequently employs MONoT5 to refine the selection further, retaining only the most relevant ones from the initially identified set.
3.3.2 Fine-Grained Evidence Retrieval
The objective of this step is to infer the set of rele- vant paragraphs from Rg. Here, we explain mul-
'In practice documents often have a hierarchical discourse structure, consisting of multiple levels of sections (Nair et al., 2023). To handle this, we can flatten the structure using a pre- order traversal approach. When verbalizing a specific section, we concatenate the names of all sections along the path from the root node to that particular node in the discourse structure. This flattening process allows us to represent the document as list of sections while considering the hicrarchical relationships among sections.
For most of the experiments presented in the upcoming sections, we use bart-large (Lewis et al., 2020) trained over the CNN/Daily-Mas Cor- pus (Nallapati et al., 2016) for S. For retrieval

---

Answering Performance Yes/No
Evidence FL
Tokens Prncessed
API
Appronch
Extracrive
Abstractive
Unanswcerable
Overall
Calls
HUMAN (Dusigi at.. 2021)
58.92
39.71
78.98
69.44
60.92
71.62
CGSN (Nic al. 2022)
34.75
14.39
71.84
53.98
LED* (Nie et al., 2022)
$2.41
23.44
76,96
77.91
52.87
got-3. 3-tuuboo
UNSUPERVISED
MONOTS (Nogucira et al., 2020)
42.84
82.23
69.09
47.21
34.23
DPR (Karpukhin et al., 2020)
31.58
18.57
78.46
84,33
42
19.52
cross-encoder- mis------rssss 1i-222-
PARAGRAPH
45.20
26.02
76.13
72.56
47.92
32.02
8519.37
47.24
CHUNK
45.96
29.61
84.30
65.57
49.00
35.59
5411.44
2.44
MAP-REDUCE
21.37
19.65
76.47
90.28
39.26
12.84
12730.13
48.24
MAP-REDUCE OPTIMIZED
47.45
26.05
82.79
71.32
50.13
50.11
7491.97
3.69
D BASE
42.90
23.65
74.35
79.61
47.45
49.92
1980.94
1.99
Table 1: Comparison of various zero-shot approaches against SoTA methods for QASPER dataset. The simplest algorithm from D family yields competitive value across several metrics while being zero-shot and requiring least number of tokens. *; Inference obtained using gold evidence.
and question answering, we utilize the highly ca- pable 3pt-3.5-turbe. 5-tuboo model, known for its re- markable performance across a wide range of NLP tasks, all while being more cost-effective (Ye et al., 2023) when compared against text-davinci-003. To identify the most relevant scctions$3.3.1, we prompt the LLM with the following instruction:
This model is a cross-encoder reranker which employs Siamese learning and BERT-like architecture. This model is offered the sentence-transformers (Reimers and Gurevych, 2019) library. While the library pro- vides many different model checkpoints, we chose rros--nnodder--sm-mrrotn-nnnn--
Document section structure:
[Condensed representation described $3.3.1)
yielded highest F1 score for evidence retrieval. PARAGRAPH: Every paragraph in D is processed by the LLM independently assess its relevance to through a boolean prompt.
Question:
(q)
List all section names that may be relevant for fnnswrrng the question. Respond with
semma-tep section name list. Provide an empty response none of the sections are relevant.
(5) CHUNK: The document is partitioned into consecutive fragments, aiming to accommodate as many paragraphs as possible within a predefined
For the BASE strategy described in $3.3.2, we em- ploy the following prompt:
token limit called chunk size (3500). Subse- quently, BASE is applied for each fragment.
(Paragraphs annotated with identifier ($3.3.2)
Question:
(6) MAP-REDUCE: This approach, widely adopted in the literature, involves evaluating the relevance of each paragraph to and subsequently processing the relevant paragraphs together in a single call to the LLM. However, we observed that using LangChain's implementation directly led to subpar performance in terms of evidence retrieval Fi-score and inference cost. This be attributed to the incompatibility of the prompt with the target domain, resulting in significant performance degradation due to 3. 5.t-3.s-turbo's high sensitivity to the prompt (Ye et al., 2023). (7) MAP-REDUCE OPTIMIZED: For better alignment with our target task, we made crucial modifications to the original implementation. Building upon the observation that CHUNK outperforms PARAGRAPH in terms of performance, decided process document chunks instead of individual paragraphs using the BASE tech-
(q)
Find paragraph ids that contains relevant information for answering the question. Respond with semaartee id list. Provide an empty
response if none the paragraphs are relevant.
4 Baselines:
We consider the following zero-shot approaches for performance comparison:
(1) MoNoT5: In this approach, MoNOT5 is directly applied to re-rank the paragraphs of D [11 PP..P...... Pa] based on
(2) DPR: Dense Passage Retrieval
(DPR) (Karpukhin et al., 2020) is retrieval approach that leverages dense representations. This method utilizes bi-encoder architecture to generate embeddings for both the documents and the query independently which are used for finding the most relevant documents.

---

nique. Following the initial stage, where relevant paragraphs are identified, we concatenate them and subject them to the same strategy (BASE) for further processing.
24iiii sandbing
Thetoaate Entoade
5 Experiments and Results
We mainly assess the applicability of our method in two scenarios: (1) $5.1: Information-sccking setting (Dasigi et al., 2021) and (2) $5.1: Multi- hop Reasoning in Question Answering (Yang et al.. 2018). Thereafter, we conduct an extensive analy- sis of our approach's performance across various configurations, discussed in 15.3.1, and exam- ine its effectiveness in different document length categories, as outlined in $5.3.2. Thereafter, we justify the need for the inclusion of global con- text modeling in $5.3.3, and highlight the signif- icance of incorporating discourse information in $5.3.4. Finally, $5.3.5 we compare our methods performance with the best performing zero-shot ap- proach for different categories and identify scope for improvement. In our analyses, will also include insights into the inference cost and latency associated with utilizing large language models (LLMs) for the evidence retrieval stage. This will encompass the monetary cost and the processing ef- ficiency measured in terms of the number of tokens processed and the number of LLM inferences. It is important to note that these measurements pertain exclusively to LLMs and do not encompass smaller fine-tuned models. Unless otherwise specified, would be using BASE approach for fine-grained retrieval and the overall approach would be called DA-BASE. Due to the monetary cost associated with gpt-based LLMs, we experiment with 150 randomly sampled documents for all the experi- ments.
Cechalccing relevv (loWT)))
PLPP pinn
XX Leghtin Ander
Figure 2: Qualitative comparison of D*-BASE with publicly available Zero-shot Approaches such as MoNOT5 and Langchain's MAP-REDUCE: "Tokens used" refers to the total number of tokens processed by the evidence retrieval and question answering stage to generate the final answer. Similarly "Total OpenAl Calls" also computes the number of API calls over both the tasks.
(SoTA) model (Nie et al., 2022), fine-tuned LED model evaluated gold evidence, and zero-shot gpt-3. 5-turbo model evaluated gold evidence. Among them, our simplest approach, 3 achieves competitive performance terms of Ev- idence F1 score. Notably, retains 99.6% the performance the best zero-shot approach, MAP- REDUCE OPTIMIZED, while processing only 26% of the tokens required by the latter.
The original implementation of MAP-REDUCE suffers from two main limitations. Firstly, it pro- cesses each paragraph independently, overlooking the effectiveness of chunk-level processing over paragraph-level processing. Secondly, it employs suboptimal few-shot prompting to retrieve relevant sources, resulting in increased processing costs and poor performance when the few-shot prompting is not well-aligned with the domain. Due to these significant processing costs and the underper- formance of PARAGRAPH and MAP-REDUCE, we exclude their evaluations from subsequent analyses. Similarly, among the non-LLM base- lines, exclude the evaluations of DPR and due their poor performance.
5.1 Performance for Information-Seeking Setting
We assess the performance of the models on the QASPER dataset (Dasigi et al., 2021), which com- prises iittommttion-seekiin questions designed for lengthy research papers. The dataset includes a set of ground truth evidence paragraphs and answers. The questions are categorized as extractive, abstrac- tive, yes/no, and unanswerable, and our proposed method must accurately discern the question's 's in- tent to generate a relevant response.
While it is possible to enhance the performance of our approach, 3 with alternative configura- tions (as shown in Section 5.3.1), it remains an excellent choice for rapid testing in different do- mains. Its cost-effectiveness and minimal latency in terms of API calls make it highly suitable for such purposes.
Table presents the results of various ap- proaches, including a fine-tuned state-of-the..

---

Approsch
Asswer F1
erating answers were preserved, which does not imply the irrelevance of other paragraphs addressing the question. This observation highlights the presence of this artifact.
Processed
CGSN 2022)
LED 2022)
got -tturbp
42.01
SHOT APPROACHES
Moxo15
62:35
37.91
Augmenting with self-ask boosts perfor- mance: This highlights the fact that zero-shot retrievers are better positioned retrieve frag- ments for simpler queries and self-ask effec- tively uses them to get better performance. fact, our approach D BASE is very close in performance to zero-shot question answering with gold evidence.
36.31
6702.78
MAP REPULE UPTINIED
8529.81
-BASE
31.05
23.55
3141.29
2.30
7ERO SERO-SROTTAA APPROACHEES ATICMENTE WITH S-AK
CRUNK
9522.49
MAP-REnOCR OPTINIZED
42
D
26,87 43.45
5376,29
5.37
Table 2: Comparison of various zero-shot approaches for HOTPOTQA-Doe Dataset. While directly apply- ing D*-BASE leads to poor performance, combining this with self-ask prompting methodology yields best performance while processing least number of tokens when compared against other zero-shot LLM-based methods. #; Inference obtained using gold evidence.
5.3 Ablations & Annalysss
5.3.1
Performance of different configurations of D
We investigated several configurations to determine possible directions to improve the performance. Following explorations were performed (Table 3):
5.2 Performance for Questions Requiring Multi-Hop Reasoning
Variations in fine-grained retrieval: Al- though employing MonoT5 can reduce infer- ence costs, it also adversely affects evidence retrieval performance, as evidenced by the table. Conversely, D HIERBASE demon- strates a enhancement in performance with only marginal increase in inference cost.
Here, we use HOTPOTQA-Doc dataset (Yang et al., 2018; Nie et al., 2022), where the objective to answer a complex query involving multi-hop rea- soming given two long documents. We have in- vestigated the performance of different seoo approaches using two schemes: (a) Direct Pro- cessing: Queries directly fed to the Zero-Shot retrievers to get the relevant evidences. (b) self-ask based Processing: By leveraging the power of elic- itive prompting (Yao et al., 2022; Press al., 2022: Wei et al., 2022b), we employ the technique of self-ask (Press et al., 2022). This approach entails decomposing a complex query into a series of sim- pler questions, which collectively form the basis for the final answer. Through iterative questioning, the agent analyzes prior answers and previously posed questions to generate subsequent inquiries. Leveraging the zero-shot retrieval approach, the agent obtains relevant answers for each question.
Using LLMs for summarization: We re- placed fine-tuned summarizer with enter- prise LLMs such as 3.tturb-t and text-davinci-oe (Ouyang et al., 2022) and open-source LLM, vicuna-13b (Chiang al., 2023). While the performance increased along several metries, there is additional pro- cessing cost to get the condensed represen- tation of the document (very minimal when smaller bart-large based summarizer was used).
Exploring alternative LLMs for retrieval: this analysis, we observe a decline in per- formance when utilizing other LLMs, high- lighting the superiority of gpt-3. 5- 5-rurbo as the optimal choice for retrieval tasks.
The results for this experiment are tabulated at Table 2. We note the following observations:
Evidence Fj poorly correlated with Answer F1: Considering same question answering model were used (gpt-3. 3..33.5-uuro 5-turbo) for each of the zero-shot approaches, answer performance of MAP-REDUCE OPTI- MIZED aligns with that of MONOT5, albeit with noticeably lower evidence retrieval effi- cacy. It's worth noting that during dataset con- struction, only the paragraphs utilized for gen-
Investigating alternative LLMs for ques- tion answering: We observe a performance boost when employing text-davinci-000 However, it is important to consider the higher monetary cost associated with using this API compared to 3 5-turbo.
find that the
httte:///oenaa..co/bllo/eeeeee

---

Answering Performance
Evidence F
Tokens Processed
API Calls
Approach
Extractive
Abstractive
YoeNo
Unamswerable
Overall
OUR PRIMARY APPROACH
D
42.90
23.65
74.35
79.61
47.45
49.92
1980.94
VARIATIONS IN GINE-GRAINED EVIDENCE RETRIEVAL
MONOTS
34.86
20.47
67.59
88.64
43.33
32.73
844.09
1.0
MONOTS+BAR
39.61
22.31
74.32
87.35
47.19
44,39
1520.35
1.95
BASE+MONOTS
39.62
23.66
74.54
82.33
46,42
40.23
1980.94
1.99
HIERBASE 45.48
24.14
71.55
86.18
49.48
50.09
2125.67
2.85
REPLACING FINE- TUNED SUMMARIZATION MODEL WITH INSTRUCTION ALIGNED LLM
gpt-3.5-turbe 5-trbo
43.89
27.31
79.81
75.55
49.28
51.19
2106.57
2.0
text-davinci-001
27.04
79.28
76.32
48.61
50.37
2239.97
vicuna-13b
43.03
27.70
75.16
79.55
49.09
50.09
2208.63
2.14
VARYING L.LMS FOR RETRIEVAL
-ext-davinci-003 vicuna-13b
41.45
24.12
79.08 74.07
77.72 86.88
47.11
37.53
2673.26
2.0
23.35
15.38
36,52
28.10
1810.48
1.85
VARYING LLMS FOR QUESTION ANSWERING
text-davinci-00 vicuna-13b
49.99
20.33
77.86 62.29
90.82
52.51
49,92
1980. 94
1.99
31.71
21.34
60.43
37.06
49.92
1980.94
1.99
Table Performance for different D configurations for QASPER dataset.
5.3.2 Performance across Different Document Length categories
the chunk-size increasing. This observation under- scores the fact that larger chunk sizes enable the model to capture longer-range relationships and contextual information, resulting in improved per- formance. By processing the entire document in a single pass, D benefits from accessing the global context and long-range range relationships, leading enhanced performance.
We divided the test set into different categories based on their respective lengths. We notice the advantage of our method along three aspects:
Evidence Fi: Our approach consistently achieves competitive performance in evidence retrieval across various document length cate- gories (Figure 3 (a)).
5.3.4 Role of Discourse Headings and Sub-headings
Evidence Retrieval Cost: Our approach sig- nificantly reduces the number processed tokens compared to other methods in all doc- ument length categories (Figure (b)). This cost-efficient characteristic makes it an excel- lent choice for minimizing both inference and monetary costs, regardless of the document's length.
Evidence Fi
Tokens Processed
Approach
Calls
DASE 49.92
1980.94
D'-BASE x
43.11
2183.55
1.88
Table 4: Performance of different methods with and without section information (denoted by x) for QASPER dataset.
Latency: Irrespective of the document's length, our approach maintains a minimal la- tency by making approximately API calls to the LLM (Figure 3 (c)). This efficient perfor- mance further highlights its desirability and suitability for various applications.
To assess the importance of section headings sub-headings in the discourse structure for retrieval, we replaced them with randomly initialized Univer- sally Unique Identifiers (UUIDs) and test evidence retrieval performance over the QASPER dataset (Table 4). The significant decrease in the perfor- mance shows that section headings sub-headings are crucial in conveying the topical essence of a section, which needed for accurate retrieval.
5.3.3
Need for Global Context Modeling
In this section, we investigate the effect of chang- ing the chunk size of the baseline CHUNK in terms of evidence precision, recall and F -sccc... As we see from Figure 4, the precision and Fi-score in- creases at the cost of modest decrease in recall as

---

cored firee
Drr Aed Sarvaii
Dramen (M0n
(a) Evidence Fj acToss different docu- ment length categories
(b) Tokens Processed for retrieval across different document length categories
(c) API calls for retrieval across different
document length categories
Figure 3: Analysing the performance of different approaches across different length categories along three metries for QASPER Dataset
Question Caeegory
Exidence Precision (D / MRR)
Exidence Recall
Evidence
Reslesse lisall chnnk
(D* / /RRO
(D* / MRO)
0.7
Extractive
52.63 52.19
71.9/8149
16.39/57.46
Abstrcttvve
41.78/50.12
Yes/No
43.6 38.79
59.6/55.31
45.53/ 40.18
Unanswerable
42.94/25.5
25.4/25.46
43.43/25.10
Rikkow Freaice
Table 5: Comparison of evidence retrieval perfor-
mance of D with the best performing zero- shot baseline MAP-REDUCE OPTIMIZED (MRO)
0.2
182
40
234
2000
1001
6 Conclusion
Clackib
Figure Evidence Precision, Recall and over different chunk lengths for QASPER Dataset
We demonstrated zero-shot approach for evidence retrieval which leverages the discourse structure of the document for information categorization which not only yielded competitive performance for infor- mation seeking and question answering with multi- hop reasoning setting, but also processed lowest number of tokens resulting in significant compute and cost savings. This approach demonstrates sev- eral desirable characteristics such as robustness in evidence retrieval performance, lower latency, etc. all across different document length ranges.
5.3.5 Evidence Retrieval Performance over Question Categories
In this section, compare 3-ABE with the best performing zero-shot baseline MAP-REDUCE OPTI- MIZED (MRO) over different question categories in QASPER to identify differences in model perfor- mance (Table 5). While our approach is precise in identifying the evidence paragraphs, it occasionally falls short in identifying all pertinent evidence (i.e. lower recall). This indicates that representing sec- tion with a summary leads to loss of information and the LLM may miscategorize its relevancy question due to this loss of information. Designing an effective strategy prevent loss of vital infor- mation for a particular question may be future research direction.
7 Limitations
Although our approach demonstrated compet- itive performance in the information seeking setup, there is room for improvement, partic- ularly when confronted with questions that require intricate multi-hop reasoning. Since we represent the document by summarizing each section, there is a potential loss of criti- cal information that essential for addressing complex queries necessitating multi-hop rea-

---

soning, Moving forward, we aim to explore methods that allow for accurate section se- lection while minimizing inference costs and mitigating information loss.
Ben Mann, and Jared Kaplan. 2022. Training a help- ful and harmless assistant with reinforcement learn- ing from human feedback.
Gregor Betz, Kyle Richardson, and Christian Voigt. 2021. Thinking aloud: Dynamic context generation improves zero-shot reasoning performance of gpt-2. arXiv preprint arXiv:2103. 103.3333
Our experimental analyses primarily focus on enterprise-leve language models, which re- quire enterprise compute credits. In the future, we plan to explore the capabilities of more advanced open-source models as they become available, which may offer enhanced perfor- mance and accessibility.
Michele Bevilacqua. Giuseppe Ottaviano, Patrick Lewis, Scott Yih, Sebastian Riedel, and Fabio Petroni. 2022. A Autoregressive search engines: Generating substrings as document identifiers. Advances in Neural Infor- mation Processing Systems, 33:31668-31666
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neclakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. 2020. Language models are few-shot learners. Advances in neural information processing systems, 1971
While our experiments have primarily cen- tered around single-document use cases, have yet delve into the realm of retrieval involving multiple documents or collections. This area remains unexplored, and we antici- pate investigating strategies and techniques to effectively handle such scenarios.
Shuyang Cao and Lu Wang. 2022. HIBRIDS: Atten-
tion with hicrarchical biases for structure-awarc long document summarization. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pages 786-807, Dublin, Ireland. Association for Computational Lin- guistics.
Although our evaluations provided insight into how various summarizers impacted the final downstream performance, the current study did not inherently assess the quality of sum- marization. In future work, we aim to assess the summarization's 's faithfulness the origi- nal content and its impact on end-to-end per- formance.
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. 2017. Reading Wikipedia to answer open- domain questions. In Proceedings of the 55th Anmual Meeting of the Association for Computational Lin- guistics (Volume I: Long Papers), pages 1870-1879, Vancouver, Canada. Association for Computational Linguistics.
Mark Chen, Jerry Tworek, Heewoo Jun, Qiming Yuan, Henrique Ponde de Oliveira Pinto, Jared Ka- plan, Harri Edwards, Yuri Burda, Nicholas Joseph, Greg Brockman, et al. 2021. Evaluating large language models trained on code. arXiv preprint arXiv:2107.03374
References
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyanskiy, David Uthus, Mandy Guo, James Lee-Thorp, Yi Tay, et al. 2023. Colt5: Faster long-range transform- ers with conditional computation. arXiv preprint arXiv:2303.09752.
Wei-Lin Chiang, Zhuohan Li, Zi Lin, Ying Sheng. Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang. Yonghao Zhuang, Joseph E Gonzalez, et al. 2023. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality. See https://vicuna. Imsys. org (accessed 14 April 2023).
Joshua Ainslie, Santiago Ontanon, Chris Alberti, Va- clav Cvicek, Zachary Fisher, Philip Pham, Anirudh Ravula, Sumit Sanghai, Qifan Wang, and Yang. 2020. ETC: Encoding long and structured inputs in transformers. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Process- ing (EMNLP), pages 268-284, Online. Association for Computational I Linguistics.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sehastian Gehrmann, Parker Schuh, Kensen Shi, Sasha Tsvyashchenko, Joshua Maynez, Abhishek Rao, Parker Barnes, Tay, Noam Shazeer, Vin- odkumar Prabhakaran, Emily Reif, Nan Du Ben Hutchinson, Reiner Pope, James Bradbury. Jacob Austin, Michael Isard, Guy Gur-Ari, Pengeheng Yin, Toju Duke, Anselm Levskaya, Sanjay Ghemawat, Sunipa Dev, Henryk Michalewski, Xavier Garcia, Vedant Misra, Kevin Robinson, Liam Fedus, Denny Zhou, Daphne Ippolito, David Luan, Hyeontaek Lim,
Yuntao Bai, Andy Jones, Kamal Ndousse, Amanda Askell, Anna Chen, Nova DasSarma, Dawn Drain, Stanislav Fort, Deep Ganguli, Tom Henighan, Nicholas Joseph, Saurav Kadavath, Jackson Kernion, Tom Conerly, Sheer El-Showk. Nelson Elhage, Zac Hatfield-Dodds Danny Hernandez, Tristan Hume, Scott Johnston, Shauna Kravec, Liane Lovitt, Neel Nanda, Catherine Olsson, Dario Amodei, Tom Brown, Jack Clark, Sam McCandlish, Chris Olah,

---

Barret Zoph, Alexander Spiridonov, Ryan Sepassi, David Dohan, Shivani Agrawal, Mark Omernick, An- drew M. Dai, Thanumalayan Sankaranarayana Pil- lai, Marie Pellat, Aitor Lewkowyez, Erica Moreira, Rewon Child, Oleksandr Polozov, Katherine Lee, Zongwei Zhou, Xuezhi Wang. Brennan Saeta, Mark Diaz, Orhan Firat, Michele Catasta, Jason Wei, Kathy Douglas Eck, Jeff Dean, Slav Petrov, and Noah Fiedel. 2022. Palm: Scaling language mod- cling with pathways.
dataset of incomplete information reading compre- hension questions. In Proceedings of the 2020 Con- ference on Empirical Methods in Natural Language Processing (EMNLP), pages 1137-1147, Online. As- sociation for Computational Linguistics.
Hongyu Gong, Yelong Shen, Dian Yu, Jianshu Chen, and Dong Yu. 2020. Recurrent chunking mecha- nisms for long-text machine reading comprehension. In Proceedings of the 58th Annual Meeting of the As- sociation for Computational Linguistics, pages 6751 6761, Online, Association for Computational Lin- guistics.
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph. Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani. Siddhartha Brahma, Al- bert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdh- ery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Zhao, Yanping Huang. Andrew Dai, Hongkun Yu, Slav Petrov, Ed H. Chi, Jeff Dean, Ja- cob Devlin, Adam Roberts, Denny Zhou, Quoe V. Le. and Jason Wei. 2022. Scaling instruction finetuned language models.
Bert F Green Jr. Alice K Wolf, Carol Chomsky, and Kenneth Laughery. 1961. Baseball: an automatic question-answerer. In Papers presented at the May 9-11, 1961, western joint IRE-AIEE-ACM computer conference conference, pages 219.224.
John T Guthrie, Tracy Britten, and K Georgene Barker. 1991. Roles of document structure, cognitive strategy, and awareness in searching for information. Reading Research Quarterly, pages 300-324.
restuuttoo-iinetuneed
Rajarshi Das, Shehzaad Dhuliawala, Manzil Zaheer, and Andrew McCallum. 2019. Multi-step retriever- reader interaction for scalable open-domane question answering. In International Conference on Learning Representations.
John T Guthrie and Irwin S Kirsch. 1987. Distinctions between reading comprehension and locating infor- mation in text. Journal of educational psychology. 79(3):220.
Pradeep Dasigi, Kyle Lo. Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. 2021. A dataset of information-ssekking questions and answers an- chored in research papers. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies. pages 4599-4610 On-
John T Guthrie and Peter Mosenthal. 1987. Literacy as multidimensional: Locating information and read- ing comprehension. Educational Psychologist, 22(3- 4):279-297.
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasu- pat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International confer- ence on machine learning, pages 1929-3934 PMLR.
line. Association for Computational Linguistics.
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and
Kristina Toutanova. 2019. BERT: Pre-training of deep bidirectional transformers for language under- standing. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, Volume 1 (Long and Short Pupers), pages 4171-4186, Minneapolis, Minnesota. Association for Computational Linguistics.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, Tom Hennigan, Eric Noland, Katie Millican, George van den Driessche, Bogdan Damoc, Aurelia Guy, Simon Osindero, Karen Si- monyan, Erich Elsen, Jack W. Rae, Oriol Vinyals. and Laurent Sifre. 2022. Training compute-optimal large language models.
Zican Dong, Tianyi Tang, Lanyi Li, and Wayne Xin Zhao. 2023. A survey on long text modeling with transformers. arXiv preprint arXiv:2302. 14502.
Mandar Joshi, Eunsol Choi, Daniel Weld, and Luke Zettlemoyer. 2017. TriviaQA: A large scale distantly supervised challenge dataset for reading comprehen- sion. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics ( (Vol- ume 1: Long Papers), pages 1601-1611, Vancouver. Canada, Association for Computational Linguisties.
Matthew Dunn, Levent Sagun, Mike Higgins, V Ugur Guney, Volkan Cirik, and Kyunghyun Cho. 2017. Searchqa: A new q&a dataset augmented with context from a search engine. arXiv preprint arXiv:/704.05179.
Wafaa S El-Kassas, Cherif R Salama, Ahmed A Rafea, and Hoda K Mohamed. 2021. Automatic text sum- marization: A comprehensive survey. Expert systems with applications, 165:113679.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open- domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pages 6769-6781. Online. Association for Computational Linguistics.
James Ferguson, Matt Gardner, Hannaneh Hajishirzi, Tushar Khot, and Pradeep Dasigi. 2020. IRCC: A

---

LangChain. Langchain.
2022 Conference on Empirical Methods in Natu- ral Language Processing, pages 5036-5047, Abu Dhabi, United Arab Emirates. Association for Com- putational Linguistics.
Mike Lewis. Yinhan Liu, Naman Goyal, Marjan Ghazvininejad, AbdeIrahman Mohamed, Omer Levy. Veselin Stoyanov, and Luke Zettlemoyer. 2020. BART: Denoising sqquence-to-sequenee pre-training for natural language generation, translation, and com- prehension. In Proceedings of the 58th Annual Meet- ing of the Association for Computationnl Linguistics, pages 7871-7880, Online. Association for Computa- tional Linguisties.
Rodrigo Nogueira, Zhiying Jiang, Ronak Pradeep, and Jimmy Lin. 2020. Document ranking with pre- trained scquence-to-sequene of the Association for Computational Linguistics: EMNLP 2020, pages 708-718, 788, Online. Association for Computational Linguistics.
model. In Findings
Yinhan Liu, Myle Ou, Naman Goyal, Jingfei Du, Man- dar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta:
OpenAL. 2023. Gpt-4 technical report.
Long Ouyang, Jeff Wu, Xu Jiang, Diogo Almeida, Car- roll L. Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray. John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder. Paul Christiano, Jan Leike, and Ryan Lowe. 2022. Training language models to follow instructions with human feedback.
robustly optimized bert pretraining ap- proach. arXiv preprint arXiv: 1907. 1692.
Bonnie J. F. Meyer, David M. Brandt, and George J. Bluth. 1980. Use of top-level structure in text: Key students.
for reading comprehension of Reading Research Quarterly, 16(1):72-103.
Sewon Min, Mike Lewis, Luke Zettlemoyer, and Han- nanch Hajishirzi. 2022. MetalCL: Learning to learn in context. In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Tech- nologies, pages 2791-2809, Seattle, United States. Association for Computational Linguisties.
Jayr Pereira, Robson Fidalgo, Roberto Lotufo, and Ro- drigo Nogueira. 2023. Visconde: Multi-document qa with gpt-3 and neural reranking. European Conference on Information Retrieval, pages 534-543, Springer.
Ethan Perez, Patrick Lewis, Wen-tau Yih, Kyunghyun Cho, and Douwe Kiela. 2020. Unsupervised question decomposition for question answering. In Proceed- ings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8864 8880, Online. Association for Computational Linguistics.
Inderject Nair, Aparna Garimella, Balaji Vasan Srini- vasan, Natwar Modani, Niyati Chhaya, Srikrishna Karanam, and Sumit Shekhar. 2023. A neural CRF- based hierarchical approach for linear text segmen- tation. In Findings of the Association for Compu- lational Linguistics: EACL 2023, pages 883-893, Dubrovnik, Croatia. Association for Computational Linguistics.
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A Smith, and Mike Lewis. 2022. Measuring and narrowing the compositionality gap in language models, arXiv preprint arXiv:2210.033: 03.05500.
Reicchiro Nakano, Jacob Hilton, Suchir Balaji, Jeff Wu, Long Ouyang. Christina Kim. Christopher Hesse, Shantanu Jain, Vineet Kosaraju, William Saunders, Xu Jiang, Karl Cobbe, Tyna Eloundou, Gretchen Krueger, Kevin Button, Matthew Knight, Benjamin Chess, and John Schulman. 2022. Webgpt: Browser- assisted pnestion-answeree with human feedback.
Jack W. Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring. Susan- nah Young, Eliza Rutherford, Tom Hennigan, Ja- cob Menick, Albin Cassirer, Richard Powell, George van den Driessche, Lisa Anne Hendricks, Mari- beth Rauh, Po-Sen Huang, Amelia Glaese, Jo- hannes Welbl, Sumanth Dathathri, Saffron Huang, Jonathan Uesato, John Mellor, Irina Higgins, Anto- nia Creswell, Nat McAleese, Amy Wu, Erich Elsen. Siddhant Jayakumar, Elena Buchatskaya, David Bud- den, Esme Sutherland, Karen Simonyan, Michela Pa- ganini, Laurent Sifre, Lena Martens, Xiang Lorraine Li, Adhiguna Kuncoro, Aida Nematzadeh, Iilena Gribovskaya, Domenic Donato. Angeliki Lazaridou. Arthur Mensch, Jean- -aptistt Lespiau, Maria Tsim- poukelli, Nikolai Grigorev, Doug Fritz, Thibault Sot- tiaux, Mantas Pajarskas, Toby Pohlen, Zhitao Gong, Daniel Toyama, Cyprien de Masson d' Autume, Yujia Li, Tayfun Terzi, Vladimir Mikulik, Igor Babuschkin, Aidan Clark, Diego de Las Casas, Aurelia Guy, Chris Jones, James Bradbury, Matthew Johnson. Blake Hechtman, Laura Weidinger, Iason Gabriel,
Ramesh Nallapati, Bowen Zhou, Cicero dos Santos, Çaglar Gulçehre, and Bing Xiang, 2016. Abstrac- tive text summarization using sequence-to-sequee sequence-to-sequence RNNs and beyond. In Proceedings of the 20th SIGNLL Conference on Computational Natural Lan- guage Learning, pages 280-290, Berlin, Germany. Association for Computational Linguistics.
Tri Nguyen, Mir Rosenberg, Xia Song, Jianfeng Gao, Saurabh Tiwary, Rangan Majunder, and Li Deng. 2016. Ms marco: A human generated machine read- ing comprehension dataset. choice, 2640:660.
Yuxiang Nie, Heyan Huang, Wei Wei, and Xian- Ling Mao. 2022. Capturing global structural information long document question answering with compres- sive graph selector network. In Proceedings of the

---

William Isaac, Ed Lockhart. Simon Osindero, Laura Rimell, Chris Dyer, Oriol Vinyals, Kareem Ayoub, Jeff Stanway, Lorrayne Bennett, Demis Hassabis, Ko- ray Kavukcuoglu, and Geoffrey Irving. 2022. Scaling language models: Methods, analysis insights from training gopher.
Shuohang Wang. Mo Yu, Xiaoxiao Guo, Zhiguo Wang, Tim Klinger, Wei Zhang. Shiyu Chang, Gerry Tesauro, Bowen Zhou, and Jing Jiang. 2018. R 3: Reinforced ranker-reader for open-domain question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 32.
Colin Raffel, Noam Shazeer. Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text trans- former. Journal of Machine Learning Research, 21:1- 67.
Jason Wei, Maarten Bosma, Vincent Zhao, Kelvin Guu, Adams Wei Yu, Brian Lester, Nan Du, Andrew M. Dai, and Quoc V Le. 2022a. Finetuned language models are zero-shot learners. In International Con- ference on Learning Representations.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi. Quoe V Le, and Denny Zhou. 2022b. Chain of thought prompt- ing elicits reasoning in large language models. In Advances in Neural Information Processing Systems.
Nils Reimers and Iryna Gurevych. 2019. Sentence- BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-JJCNLP), pages 3992-3992, Hong Kong, China, Association for Com- putational Linguistics.
Tongshuang Wu, Ellen Jiang. Aaron Donsbach, Jeff Gray, Alejandra Molina, Michael Terry, and Carrie Cai. 2022a. Promptchainer: Chaining large language model prompts through visual programming. In Ex- tended Abstracts of the 2022 CITI Conference on Human Factors in Computing Systems, CHI EA 222. New York. NY, USA. Association for Computing Machinery.
Laria Reynolds and Kyle McDonell. 2021. Prompt pro-
gramming for large language models: Beyond the few-shot paradigm. In Extended Abstracts of the 2021 CHI Conference on Human Factors in Com- puting Systems, CHI EA '21, New York, NY, USA. Association for Computing Machinery.
Tongshuang Wu, Michael Terry, and Carrie Jun Cai. 2022b. Ai chains: Transparent and controllable human- ai interaction by chaining large language model prompts. In Proceedings of the 2022 CHI Conference Human Factors in Computing Sys- zems, CHI '22, New York, NY, USA. Association for Computing Machinery.
Bowen Tan, Zichao Yang, Maruan Al-Shedivat, Eric Xing. and Zhiting Hu. 2021. Progressive generation of long text with pretrained language models. In Pro- ceedings of the 2021 Conference of the North Amer- ican Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 4313-4324, Online. Association for Computational Linguisties.
Zhilin Yang. Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christo- pher D. Manning. 2018. HotpoiQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empiri- cal Methods in Natural Language Processing, pages 2369-2388 Brussels, Belgium. Association for Com- putational Linguistics.
Yi Tay, Vinh Tran, Mostafa Dehghani, Jianmo Ni, Dara Bahri, Harsh Mehta, Zhen Qin, Kai Hui, Zhe Zhao, Jai Gupta, et al. 2022. Transformer memory as a differentiable search index. Advances in Neural In- formation Processing Systems, 35:21831-218 221443.
Barbara M. Taylor and Richard W. Beach. 1984. The ef- fects of text structure instruction on middle-grade stu- dents' comprehension and production of expository text. Reading Research Quarterly, (9(2):134-1)
Shunyu Yao. Jeffrey Zhao, Dian Yu, Izhak Shafran, Karthik R Narasimhan, and Yuan Cao. 2022. React: Synergizing reasoning and acting in language models. In NeurIPS 2022 Foundation Models for Decision Making Workshop.
Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych. 2021. BEIR: A heterogeneous benchmark for zero-shot evaluation of information retrieval models. In Thirty-fftth Con- ference on Neural Information Processing Systems Datasets and Benchmarks Track (Round 2).
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv: 10420.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30.
Bo Zheng. Haoyang Wen, Yaobo Liang. Nan Duan, Wanxiang Che, Daxin Jiang, Ming Zhou, and Ting Liu. 2020. Document modeling with graph attention networks for multi grained machine reading compre- hension. In Proceedings of the 58th Anmual Meet- ing of the Association for Computational Linguistics, pages 6708-6718, 6718, Online, Association for Computa- tional Linguistics.
Ellen M Voorhees al. 1999. The trec-8 question answering track report. In Trec, volume 99, pages 77-82.

---

Fengbin Zhu, Wenqiang Lei, Chao Wang. Jianming Zheng. Soujanya Poria, and Tat-Seng Chua. 2021. Retrieving and reading: A comprehensive survey on open-domain question answering. arXiv preprint arXiv:2101.0077.


Reviews:
[{"review_id": "cT8pTDsWjt", "reviewer": "Reviewer_4fXv", "summary": "", "questions": "Question A) Do you have results on using simpler baselines such as DPR to retrieve relevant chunks and pass into the LM ?", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and their positive feedback. In the following response, the comments are first stated and then followed by our point-by-point response\n\n> Question A) Do you have results on using simpler baselines such as DPR to retrieve relevant chunks and pass into the LM ?\n\n> I believe the paper is missing some even primitive baselines such as using DPR to find top-k relevant chunks and feeding them into the Language Model rather than prompting the LM for what passages are important.\n\nFor open domain question answering (ODQA), a standard pipeline uses a retriever (such as BM25 or DPR) to retrieve top-100 relevant passages from millions of candidates. These top-100 passages are then re-ranked again using methods like monoT5 to send the most relevant passages to the reader. In the long document question answering (LDQA) scenario, the retrieval happens over a much limited set of candidates (100 passages in total in documents) similar to the reranking step. Following this intuition we have compared our method with MonoT5, a reranking algorithm in the paper. \n\nWe had initially run experiments with simpler models (such as DPR) trained for the objective of semantic search. We first ran some experiments to determine the value of top_k that gives best performance for these methods (DPR, all-MiniLM-L6-v2) and found the ideal top_k=3. We then applied them for retrieval on QASPER dataset. The results of these experiments are tabulated below. As we can see, monoT5 achieved the best Evidence F1 performance, far higher than simpler baselines such as DPR. Hence we compared mainly with monoT5 in our paper (when comparing our method to embedding-based retrieval).\n\nFor comprehensive analysis, we are including the results of these experiments below:\n\nTable: Additional experiments on QASPER dataset using semantic search methods, monoT5, DPR\n\n|Method| Extractive F1 (Answering performance) | Abstractive F1 (Answering performance) | Yes/No F1 (Answering performance)| Unanswerable F1 (Answering performance) |Overall F1 (Answering performance) | Evidence F1\n|-------------------------------------------------|-----------|-----------|-----------|-----------|-----|-----------|\n|monoT5                                           |42.84      |25.84      |82.23      |69.09      |47.21|34.23      |\n|DPR (model trained on combination of multiple datasets)         |31.58      |18.57      |78.46      |84.33      |42.11|19.32      |\n|all-MiniLM-L6-v2   |34.28      |22.94      |69.67      |75.79      |41.03|24.89      |\n|msmarco-bert-base-dot-v5| 37.48      |26.47      |80.69      |73.06      |44.84|25.85      |\n|multi-qa-mpnet-base-cos-v1 |34.94| 23.72| 74.26 | 69.08 | 41.70 |26.38 |\n|cross-encoder-ms-marco-MiniLM-L-12-v2  |38.69      |23.25      |78.04      |71.42      |43.48|30.76      |\n\n> As per my knowledge, the current SOTA for the QASPER dataset is COLT5-XL with an F-1 score of 53.9. COLT5: Faster Long-Range Transformers with Conditional Computation https://arxiv.org/pdf/2303.09752v2.pdf\n\nWe thank the reviewer for pointing out missing references, we will include them along with the above table in the camera-ready version of the paper if our paper gets accepted.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper proposes a systematic way to leverage discourse structure for more efficient evidence retrieval. It shows Competitive zero-shot performance while using significantly fewer tokens. It also seems to be easily generalizable by replacing QA LLM, summarizer etc. ", "reasons_to_reject": "I believe the paper is missing some even primitive baselines such as using DPR to find top-k relevant chunks and feeding them into the Language Model rather than prompting the LM for what passages are important. ", "questions_to_authors": "Question A) Do you have results on using simpler baselines such as DPR to retrieve relevant chunks and pass into the LM ?", "paper_topic_and_main_contributions": "This paper introduces a method called D3 (Drilling Down into the Discourse) for long document question answering (LDQA) which uses the inherent discourse structure in long documents to create a condensed representation that enables more efficient processing. In D3 the document is divided into sections and each section is summarized. The section names and summaries are provided as context to a large language model (LLM), which identifies relevant sections. Then the paragraphs in the relevant sections are processed to retrieve fine-grained evidence. This allows focusing on pertinent portions of the document while capturing global context. Several zero-shot strategies are explored for evidence retrieval, including using the LLM directly or a re-ranker like MonoT5. The approach is evaluated on QASPER and HotpotQA and shows competitive performance while being computationally cheaper. \n", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Missing_References": "As per my knowledge, the current SOTA for the QASPER dataset is COLT5-XL with an F-1 score of 53.9. \nCOLT5: Faster Long-Range Transformers with Conditional Computation\nhttps://arxiv.org/pdf/2303.09752v2.pdf"}}, {"review_id": "y4T5dqemht", "reviewer": "Reviewer_SsPB", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and their feedback. In the following response, the comments are first stated and then followed by our point-by-point response\n\n> The proposed technique still needs to process all the tokens in the document via the local LLMs to condense the document content before making calls to the OpenAI LLM for additional processing. The impact of the proposed technique might be better understood if the authors presented comparisons based on end-to-end processing times of the various systems. \n\nWe would like to point out that the processing of the document to derive the condensed representation is done using a lightweight fine-tuned transformer model and thus this computation can be done in a fraction of a second. Moreover, once we get the condensed representation, we can answer all the questions pertaining to that document at once, thereby making the effective processing time per document even smaller. Since our observation revealed that the primary time consumption is attributed to LLM usage, we've reported the number of sequential API calls, reflecting latency. This metric has two benefits: firstly, it's deterministic, and secondly, for different LLMs, multiplying the API call count by their average turnaround time provides a quick approximation of the evidence retrieval time. \nThat being said, we agree with your point that the computation of the end-to-end processing time can better communicate the results. We computed the end-to-end processing time of the LLM based systems (which used gpt-3.5-turbo) in Table 1 (The langchain implementation of Map-reduce takes very long as the API calls ask LLMs to generate a longer sequence resulting in more latency). As we see in the following table, **our approach is significantly more time and token-efficient**. We will incorporate this change in the camera-ready version if our paper gets accepted to the proceedings.\n\nTable:  End-to-End processing time (per question) of LLM based systems on the QASPER dataset\n\n|Method              |Overall Answer F1|Evidence F1|Number of Tokens processed|Number of API Calls|Average Time (per question in seconds)|\n|--------------------|-----------------|-----------|--------------------------|-------------------|----------------------------------------|\n|Paragraph           |47.92            |32.03      |8519.37                   |47.24              |3.48                                    |\n|Chunk               |49.00            |35.59      |5411.44                   |2.44               |2.49                                    |\n|Map-reduce          |39.26            |12.84      |12730.13                  |48.24              |33.82                                   |\n|Map-reduce-optimized|50.13            |50.11      |7491.97                   |3.69               |3.59                                    |\n|D3 -BASE            |47.45            |49.92      |1980.94                   |1.99               |1.44                                    |\n\n\n\n> In the current environment, where new LLMs are constantly released (whether open sourced or via proprietary APIs) with larger contexts and response time SLAs, it is hard to evaluate relevance of this work in the near future.\n\nLLMs are increasingly being adopted for document based question answering. Many prior approaches for LDQA using LLMs do not consider crucial aspects such as latency, and compute cost and have primarily focused on achieving high performance in terms of accuracy and effectiveness. Our work makes a novel contribution in addressing the real-world practicalities involved in using LLMs for LDQA by finding efficient ways to save computational / monetary costs. If our paper gets accepted to the conference proceedings, the significance of cost-reduction techniques can reach a broader audience. This would subsequently foster increased attention of the researchers towards this problem, thereby making LLMs more applicable for document based question answering. Adapting our proposed algorithm will also give engineers   cushioning to apply LLMs without worrying much on cost and infrastructures.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "This work proposes a suite of techniques to the summarize document paragraphs to create a condensed representation of the document for a particular question using local (and less expensive) LLMs. The condensed paragraphs are then fed to OpenAI based LLMs to filter out irrelevant paragraphs from which the OpenAI based LLM can generate the final answer to the question. The paper presents evaluation showing that the proposed technique retain 99.6% of the best zero-shot approach’s performance, while requiring to process only 26% of the total documents tokens via the expensive OpenAI LLMs.", "reasons_to_reject": "Limited Impact: \n1.  The proposed technique still needs to process all the tokens in the document via the local LLMs to condense the document content before making calls to the OpenAI LLM for additional processing. The impact of the proposed technique might be better understood if the authors presented comparisons based on end-to-end processing times of the various systems.\n2. In the current environment, where new LLMs are constantly released (whether open sourced or via proprietary APIs) with larger contexts and response time SLAs, it is hard to evaluate relevance of this work in the near future.\n\nLimited Novelty: The authors propose a technique that summarizes a document using local LLMs to reduce the document that needs to be processed via the expensive OpenAI LLMs. In my opinion, this will have limited interest to the main conference audience and is more appropriate for workshop.", "questions_to_authors": "", "paper_topic_and_main_contributions": "This work proposes a suite of techniques to the summarize document paragraphs to create a condensed representation of the document for a particular question using local (and less expensive) LLMs. The condensed paragraphs are then fed to OpenAI based LLMs to filter out irrelevant paragraphs from which the OpenAI based LLM can generate the final answer to the question. The paper presents evaluation showing that the proposed technique retain 99.6% of the best zero-shot approach’s performance, while requiring to process only 26% of the total documents tokens via the expensive OpenAI LLMs.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "ZRdAfTXxOF", "reviewer": "Reviewer_yp1i", "summary": "", "questions": "One of the baselines, map-reduce that directly finds relevant paragraphs given a question (akin to the proposed method but excluding the first two steps), is implemented using the LangChain library, but it is specified (L443-L454) that the implementation uses a “suboptimal few-shot prompting”. Why is this implementation not consistent with other baselines? The idea seems straightforward for implementing without LangChain.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for the detailed review!  In the following response, the reviewer questions/comments are first stated, followed by our point-by-point responses.\n\n> One of the baselines, map-reduce that directly finds relevant paragraphs given a question (akin to the proposed method but excluding the first two steps), is implemented using the LangChain library, but it is specified (L443-L454) that the implementation uses a “suboptimal few-shot prompting”. Why is this implementation not consistent with other baselines? The idea seems straightforward for implementing without LangChain.\n\n* We used the LangChain library because it  has a standard, well-tested and widely used implementation of map-reduce. We noticed that the standard prompt used in Langchain for the reduce stage did not perform well. This is because of the incompatibility of the prompt with the target domain as it had 2 in context examples one from the contract domain and one from a transcript of a speech. We replace the combine_prompt with a zero-shot prompt for the map-reduce optimized baseline. \n* We have further optimized the map-reduce implementation by processing the input in chunks rather than paragraphs as mentioned in L383-L392. We have reported the scores for this as map-reduce-optimized in Table 1. \n\n> The summaries generated in the first step were not assessed to measure their impact on the overall performance. We can see small improvements when BART is replaced with GPT-3 or Vicuna. Questions including are the summaries mostly extractive or what happens if the summary is hallucinated or how the quality of summaries varies across different sections are worth a discussion in the paper.\n\n * We agree with the reviewer that the quality of summarization plays an important role in the downstream performance. While different summarizers were extrinsically evaluated in terms of final evidence retrieval performance in table 3, we would include an additional sub-section devoted towards more rigorous evaluation of the summarization quality. Particularly, we would be using reference-free evaluation metrics (such as Semantic Distribution Correlation which generate a document both with and without a summary as a prompt, and then computes the difference in information content between two generated documents for metric computation [Reference-free Summarization Evaluation via Semantic Correlation and Compression Ratio by Liu et. al. 2022]) and measure its correlation with the end-to-end retrieval performance. Similarly, we intend to evaluate the summarization faithfulness with respect to the original document content and assess how it affects the retrieval performance. We would be including these analyses in our final-version, which won’t require much edits and can be conveniently presented in a sub-section, if our paper gets accepted to the proceedings.\n\n> It is hard to figure out where the proposed method makes mistakes which can be essential for future work. An error analysis would strengthen the paper to highlight its limitations.\n\n* In the table below, we compare the evidence retrieval performance with map-reduce optimized over different question categories in QASPER to identify differences in model performance. While our approach is precise in identifying the evidence paragraphs, it occasionally falls short in identifying all pertinent evidence (i.e lower recall). This indicates that representing a section with a summary leads to loss of information and the LLM may miscategorize its relevancy to a question due to this loss of information. Designing an effective strategy to prevent loss of vital information for a particular question may be a future research direction. We will include this discussion in our camera-ready version and include qualitative examples if our paper gets accepted and provide researchers with a clearer perspective on potential extensions of our work.  Also the categorization of the results below based on question type will help researchers in effectively understanding the question categories where our performance falls short. We will also add some qualitative error analysis in our final version.\n\n\nTable: Evidence Retrieval Performance over different question types in the QASPER dataset\n\n| Type of question | Evidence Precision [D3/map-reduce-optimized] | Evidence Recall  [D3/map-reduce-optimized]| Evidence F1  [D3/map-reduce-optimized] |\n| ---------------- | ------------------------------------------ | --------------------------------------- | ----------------------------------- |\n| Extractive       | 52.63 / 52.19                              | 71.9 / 81.49.                                             | 56.39 / 57.46                       |\n| Abstractive      | 41.25 / 45.1 | 52.51 / 71.57  | 41.78 / 50.12 |\n| Yes/No           | 43.6 / 38.79 | 59.6 / 55.31  | 45.53 / 40.18 |\n| Unanswerable |  42.94/ 25.74 | 44.4/25.46 | 43.43/25.10 |\n\n\n\n> Details about how documents are split when the length of a prompt exceeds the LLM maximum context size is missing. In the paper, it is stated that paragraphs would be contiguously fragmented to fit into the context size (L305-L308). Which paragraph would be fragmented? One option is to evenly fragment all paragraphs and another option is to fragment the last paragraph. In any way, the prompt would be expanded to multiple prompts. Now, the newly created prompts may include chopped paragraphs. Would this affect the performance? Also, would other paragraphs that were not fragmented be repeated in the subsequent prompts?\n\n* If the number of paragraphs in Pq exceeds the maximum context length of LLM, we make multiple LLM calls. In each call, we fit the maximum number of paragraphs that can fit into the context length, ensuring that paragraphs are not ‘chopped’. We will make this more clear in the camera-ready version if our paper gets accepted.\n\n> The proposed method does not work well in multi-hop QA. On HotpotQA (Table 2), using a document ranking model (MonoT5) works better than the proposed method and its variants (20-30% lead). Even after applying self-ask, the gap is 4-6%, thus telling us simple methods are still useful.\n\n* An explanation of the performance of the baselines and the proposed method is mentioned in L485-489. \n* To summarize, due to the method by which the HotpotQADoc dataset was constructed, only the paragraphs utilized for generating answers were preserved, which does not imply the irrelevance of other paragraphs. This is reflected by the fact that the Evidence F1 score does not correlate with the Answer F1 score in Table 2 even if the same question answering model is used. In the zero-shot set of experiments, MonoT5 achieves the highest evidence performance (62.33 evidence F1) but performs the same as Map reduce Optimized (39.52 Evidence F1) in Answer F1.\n\n\n\n\n> One assumption of this work is the presence of sections in the documents. Many long documents such as news articles are not usually organized into sections. However, leveraging sections seems to play an important role in the proposed method. In Table 1, for instance, the difference between the evidence F of “chunk” and “base” is around 14%. On the other hand, the “chunk” baseline is more general in that it can be used without sections as well. It actually works well in multi-hop QA (Table 2), which raises the question whether the use of discourse as claimed in the paper is really impactful in the task.\n\n> In Table 1, evidence retrieval of the “paragraph” and “chunk” baselines are low (32.02 and 35.59), but their answer effectiveness (it is not explained what “answering performance” actually means) 47.92 and 49.00 becomes close to the best model (50.13). This observation needs an explanation.\n\n1. Performance of Chunk baseline and the competitive performance of Chunk / Paragraph baseline:  \n    * We agree with the reviewer that the chunk baseline used in the paper is applicable in cases where there is no discourse structure. However, the chunk baseline is akin to a high-recall brute-force solution for retrieval with LLMs. It can perform reasonably well however at a high cost and latency. \n\n2.  Performance of Chunk on HotpotQA dataset: \n        Please refer to the previous point in this response\n\n3.  Fine grained analysis of Chunk / Paragraph vs D3-base: \n       * In the below table we have listed the Evidence Precision, Evidence Recall, Evidence F1, Average Number of paragraphs retrieved (per question), Tokens Processed and Number of API calls of the Chunk / Paragraph baseline and our method D3-base. (on QASPER Dataset) When processing paragraphs using the chunk method, the LLM does not have a global view of the document. It has access only to the paragraphs present in the context. This suboptimal processing devoid of global context results in the observation that the chunk / paragraph method tends to categorize a substantial number of paragraphs as relevant to the question, as shown in the table below. This is the reason why the chunk baseline has high recall but low precision. As the recall is higher, the chunk based methods have access to more paragraphs that contain the ground truth answer and hence we can observe competitive answering performance. \n     * When employing a retrieval mechanism for LDQA, we would want the retriever to have both high precision and high recall, while minimising cost. Compared to the chunk baseline, our method D3-base has more balanced precision recall values while processing only 36% of the tokens.\n\nTable: Fine grained analysis of evidence performance using precision, recall, F1 and average number of paragraphs retrieved (per question). \n(QASPER dataset)\n| Method    | Evidence Precision | Evidence Recall | Evidence F1 | Average Number of paragraphs retrieved (per question) | Tokens Processed | Number of API calls |\n| --------- | ------------------ | --------------- | ----------- | ------------------------------------------------------- | ---------------- | ------------------- |\n| Gold      | \\-                 | \\-              | \\-          | 1.48                                                   | \\-               | \\-                  |\n| Chunk     | 27.82              | 71.96           | 35.59       | 6.72                                                  | 5411.44          | 2.44                |\n| Paragraph | 24.53              | 74.96           | 32.02        | 8.47                                                   | 8519.37          | 47.24               |\n| D3-base   | 47.53              | 62.43           | 49.92      | 2.92   |  1980.94 | 1.99\n\n> We also thank the reviewer for highlighting Presentation Improvements, we will make the corrections in the final version", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. QA over long documents poses challenges that Transformer models tend to struggle with, and the techniques introduced in this paper can shed light on how to leverage document structure to address such challenges.\n2. The proposed method is simple and effective, making it easy to adopt in the community.", "reasons_to_reject": "1. The summaries generated in the first step were not assessed to measure their impact on the overall performance. We can see small improvements when BART$_\\text{large}$ is replaced with GPT-3 or Vicuna. Questions including are the summaries mostly extractive or what happens if the summary is hallucinated or how the quality of summaries varies across different sections are worth a discussion in the paper.\n\n2. It is hard to figure out where the proposed method makes mistakes which can be essential for future work. An error analysis would strengthen the paper to highlight its limitations.\n\n3. One assumption of this work is the presence of sections in the documents. Many long documents such as news articles are not usually organized into sections. However, leveraging sections seems to play an important role in the proposed method. In Table 1, for instance, the difference between the evidence F$_1$ of “chunk” and “base” is around 14%. On the other hand, the “chunk” baseline is more general in that it can be used without sections as well. It actually works well in multi-hop QA (Table 2), which raises the question whether the use of discourse as claimed in the paper is really impactful in the task.\n\n4. The proposed method does not work well in multi-hop QA. On HotpotQA (Table 2), using a document ranking model (MonoT5) works better than the proposed method and its variants (20-30% lead). Even after applying self-ask, the gap is 4-6%, thus telling us simple methods are still useful.\n\n5. Details about how documents are split when the length of a prompt exceeds the LLM maximum context size is missing. In the paper, it is stated that paragraphs would be contiguously fragmented to fit into the context size (L305-L308). Which paragraph would be fragmented? One option is to evenly fragment all paragraphs and another option is to fragment the last paragraph. In any way, the prompt would be expanded to multiple prompts. Now, the newly created prompts may include chopped paragraphs. Would this affect the performance? Also, would other paragraphs that were not fragmented be repeated in the subsequent prompts?\n\n6. In Table 1, evidence retrieval of the “paragraph” and “chunk” baselines are low (32.02 and 35.59), but their answer effectiveness (it is not explained what “answering performance” actually means) 47.92 and 49.00 becomes close to the best model (50.13). This observation needs an explanation.", "questions_to_authors": "One of the baselines, map-reduce that directly finds relevant paragraphs given a question (akin to the proposed method but excluding the first two steps), is implemented using the LangChain library, but it is specified (L443-L454) that the implementation uses a “suboptimal few-shot prompting”. Why is this implementation not consistent with other baselines? The idea seems straightforward for implementing without LangChain.", "paper_topic_and_main_contributions": "The paper introduces a new method for the task of evidence retrieval in long document question answering. The main challenge in dealing with long documents is that their size usually surpasses the context size of Transformer-based language models. To tackle this challenge, the proposed method relies on the document structure by breaking down long documents into their constituent sections. The hierarchical structure of sections would be flattened by concatenating the parent section titles into their respective subsections. The method undertakes the task in three main steps: summarizing sections, finding relevant sections to an input question, and finally finding relevant paragraphs from relevant sections given a question. A key contribution of the proposed method is that it works in a zero-shot fashion with no labelled data requirements.\n\nSpecifically, each section is first summarized using BART$_\\textrm{large}$, fine-tuned on the CNN/DailyNews corpus. In the next step, given a question and a list of summarized sections, relevant sections are selected by eliciting a prompt from an LLM. Subsequently, using a similar method to the previous step, relevant paragraphs are determined by instructing an LLM to find relevance based on an input question and relevant sections that are split into paragraphs or contiguous chunks of text.\n\nThe method is tested on two benchmarks: QA over research papers (QASPER) and multi-hop QA (HotpotQA). The method is compared against its different variants as well as baselines from previous work. The paper also considers the number of processed tokens and the number of API calls to measure the cost of using LLMs. The results show that the proposed method achieves competitive performance on both tasks.", "excitement": 3, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {"Typos_Grammar_Style_and_Presentation_Improvements": "- Tables 1 and 2 provide “answer performance”, but how final answers are obtained after retrieving evidence from the document is only visualized in the appendix. It’d be better to clarify this in the text as well.\n- L163, allows realizing $\\rightarrow$ allows for…\n- L382, ito $\\rightarrow$ to\n- When making references to tables and figures, it is recommended to use capitalization, e.g. L484, L567.\n- Sections 5.3.2 and 5.3.4 do not contain any references to Figure 2 and Table 4, respectively."}}, {"review_id": "JYddAMgPoF", "reviewer": "Reviewer_nCeH", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their comments and positive feedback. In the following response, your comments are first stated and then followed by our point-by-point responses.\n> While the paper focuses on evidence paragraph selection, I think the scalability of the proposed method seems a bit limited. For example, how can the method be applied to an open-domain QA setup? It's not essential, but I would like to see a discussion regarding adapting the method to an open-domain setup (where the retrieval candidates are very large). Some reference papers regarding the issue are written in the \"Missing References\" section. \n\n>Missing References:\nhttps://arxiv.org/abs/2202.06991\nhttps://arxiv.org/pdf/2204.10628\n\nThe strategy to represent a document section by a summary to determine its potential relevance to a question can also be leveraged for open domain question answering (ODQA). Typical approaches for ODQA use an efficient retriever to get a high recall set and then a more compute intensive re-ranker to get the relevant documents from the high recall set. We can use our method as a re-ranker in this pipeline where a summary would be computed for each document returned by the retriever. Then this summary can be used for identifying potentially relevant documents. Thereafter, akin to D3, we will use the entire document content of the filtered set to identify the relevant paragraphs. We can attempt a limited discussion of such an approach in this work in the camera ready version upon acceptance, but a full exploration of open domain QA is beyond the current scope and will be looked at in a future work (and we thank the reviewer for their suggestion!) \n\nWe thank the reviewer for pointing out missing references, we will be adding them in the final version of the paper, if our paper gets accepted.\n\n> minor) It would be nice if the authors can highlight the main difference of the proposed method compared to baselines in Section 4. (summary and filtering method)\n\nThe main difference of the proposed method (D3-base) with the baselines mentioned in Section 4 is the use of discourse structure. All of the baselines retrieve relevant paragraphs by only looking at ‘local information’ i.e only looking at the paragraphs/document chunks present in one inference call. The proposed method D3-base creates a condensed representation of the document where each section is represented by a section summary. This condensed representation allows us to pass in a global view of the document to the LLM, enabling the LLM to make accurate inferences about the sections that need to be selected. We then perform fine grained retrieval and choose the relevant paragraphs from selected sections for answering the question.\n\nMain differences:\n1. Condensed representation of document presents global view of document to the LLM\n2. Coarse retrieval (section selection) followed by fine-grained retrieval (paragraph selection) exploits discourse structure, reduces tokens processed and reduces latency.\n \nWe will elaborate more on this comparison in the final version of the paper if our work gets accepted to the proceedings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper tackles an interesting problem, using LLM for fine-grained evidence retrieval. Compared to other baselines (exploiting LLM), the proposed method is effective in computational cost while maintaining the SOTA performance.\n\n- The authors compare their method to many other baselines. The main experimental results seem significant (accuracy, computational cost (tokens processed, API call)). Furthermore, the ablation study conducted in 5.3.1 would be helpful to readers for engineering.\n\n- The paper is well-organized, demonstrating figures and examples that help readers follow the paper.", "reasons_to_reject": "- While the paper focuses on evidence paragraph selection, I think the scalability of the proposed method seems a bit limited. For example, how can the method be applied to an open-domain QA setup? It's not essential, but I would like to see a discussion regarding adapting the method to an open-domain setup (where the retrieval candidates are very large). Some reference papers regarding the issue are written in the \"Missing References\" section.\n\n- minor) It would be nice if the authors can highlight the main difference of the proposed method compared to baselines in Section 4. (summary and filtering method)", "questions_to_authors": "", "paper_topic_and_main_contributions": "The study discusses the assessment of large language models (LLMs) for zero-shot long document evidence retrieval. LLMs' limited context length and high computational costs when processing large input sets are acknowledged as challenges. To overcome these issues, the authors propose techniques that exploit discourse structure in documents, leading to a condensed representation for a better understanding of relationships between different parts. Despite the reduction in token usage, the proposed approach retains almost all of the best zero-shot performance. Furthermore, the combination of their approach with a self-ask reasoning agent achieves excellent results in complex multi-hop question answering, coming close to the performance with gold evidence.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "- https://arxiv.org/abs/2202.06991\n- https://arxiv.org/pdf/2204.10628", "Typos_Grammar_Style_and_Presentation_Improvements": "n.a."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers agree this work proposes an interesting and useful new method for retrieval/QA from long documents, promising significant token efficiency, without sacrificing overall accuracy. The authors’ commitments to updating the paper with new results and analysis, requested by reviewers, will improve the applicability and clarity of this method to practitioners.
Decision: Accept-Findings