RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval
Authors: Parth Sarthi
Salman Abdullah
Aditi Tuli
Shubh Khanna
Anna Goldie
Christopher D Manning
Venue: ICLR 2024 poster (2024)
Keywords: Retrieval Augmented Language Models
Information Retrieval
summarization
QA
Primary area: generative models
TL;DR: RAPTOR improves LLM QA performance by constructing a hierarchical summarization tree for information retrieval, outperforming existing retrieval methods across various metrics and datasets.

Abstract: Retrieval-augmented language models can better adapt to changes in world state and incorporate long-tail knowledge.  However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic understanding of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing a tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over traditional retrieval-augmented LMs on several tasks. On question-answering tasks that involve complex, multi-step reasoning, we show state-of-the-art results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20\% in absolute accuracy.

Published as a conference paper at ICLR 2024
RAPTOR: RECURSIVE ABSTRACTIVE PROCESSING FOR TREE-ORGANIZED RETRIEVAL
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, Christopher D. Manning Stanford University
esarthi @cs. stannorr..... edu
ABSTRACT
language models can better adapt changes in world state and incorporate long-tail knowledge. However, most existing methods retrieve only short contiguous chunks from a retrieval corpus, limiting holistic under- standing of the overall document context. We introduce the novel approach of recursively embedding, clustering, and summarizing chunks of text, constructing tree with differing levels of summarization from the bottom up. At inference time, our RAPTOR model retrieves from this tree, integrating information across lengthy documents at different levels of abstraction. Controlled experiments show that retrieval with recursive summaries offers significant improvements over tra- ditional
LMs on several tasks. On question-answeriin tasks that involve complex, multi-step reasoning, we show state-of-the-at results; for example, by coupling RAPTOR retrieval with the use of GPT-4, we can improve the best performance on the QuALITY benchmark by 20% in absolute accuracy.
INTRODUCTION
Large Language Models (LLMs) have emerged as transformative tools showing impressive perfor- mance on many tasks. With the growing size of LLMs, they can serve standalone very effective knowledge stores, with facts encoded within their parameters (Petroni et al., 2019; Jiang et al., 2020; Talmor al., 2020; Rae al., 2021: Hoffmann et al., 2022: Chowdhery al., 2022: Bubeck al., 2023; Kandpal et al., 2023) and models can be further improved with fine-tuning on downstream tasks (Roberts et al., 2020). Nevertheless, even a large model does not contain sufficient domain- specific knowledge for particular tasks and the world continues to change, invalidating facts in the LLM. Updating the knowledge of these models through additional fine-tuning or editing is difficult, particularly when dealing with vast text corpora (Lewis et al., 2020; Mitchell et al., 2022). An alter- native approach, pioneered in open domain question answering systems (Chen et al., 2017; Yu et al., 2018), is to index large quantities of text, after splitting it into chunks (paragraphs), in a separate information retrieval system. Retrieved information is then presented to the LLM along with the question context ("retrieval augmentation", Lewis al., 2020; Izacard et al., 2022; Min al., 2023; Ram et al., 2023), making it easy to provide a system with current knowledge particular to some domain and enabling easy interpretability and provenance tracking, whereas the parametric knowledge of LLMS is opaque and difficult to trace back to its source (Akyurek et al., 2022).
Nevertheless, existing retrieval- -augeeted approaches also have flaws. The one we tackle is that most existing methods retrieve only few short, contiguous text chunks, which limits their ability to represent and leverage large-scale discourse structure. This particularly relevant for thematic questions that require integrating knowledge from multiple parts of a text, such as understanding an entire book, as in the NarrativeQA dataset (Kočiský et al., 2018). Consider the fairy tale of Cinderella, and the question "How did Cinderella reach her happy ending?". The top-k retrieved short contiguous texts will not contain enough context to answer the question.
To address this, we design indexing and retrieval system that uses a tree structure capture both high-level and low-level details about a text. As shown in Figure 1, our system, RAPTOR, clusters chunks of text, generates text summaries of those clusters, and then repeats, generating tree from the bottom up. This structure enables RAPTOR to load into an LLM's context chunks representing the text at different levels that it can effectively and efficiently answer questions at different levels.

---

Published as a conference paper at ICLR 2024
RAPTOR Tree
Formation of one tree layer
Contents a node
Index #t
Rco: layer
2. Summarization LLM
Child Nodes 2,3
Teat: sumrmary and
S
2
Text Embedding
Leaf layer
Clustering
Figure 1: Tree construction process: RAPTOR recursively clusters chunks of text based on their vector embeddings and generates text summaries those clusters, constructing a tree from the bottom up. Nodes clustered together are siblings; parent node contains the text summary of that cluster.
Our main contribution is the idea of using text summarization to allow retrieval augmentation of context at different scales, and to show its effectiveness in experiments on collections of long doc- uments. Controlled experiments with three language models (UnifiedQA (Khashabi et al., 2020), GPT-3 (Brown et al., 2020) and GPT-4 (OpenAI, 2023)) show that RAPTOR outperforms current retrieval augmentation. Moreover, RAPTOR coupled with GPT-, and sometimes even with Uni- fiedQA, gives new state-of-the art results three QA tasks: free text response questions books and movies (NarrativeQA, Kočiský et al. 2018), full-text NLP papers (QASPER, Dasigi et al. 2021), and multiple-choice questions based medium-length passages (QuALITY, Pang et al. 2022).
RELATED WORK
Why Retrieval? Recent advances in hardware and algorithms have indeed expanded the text lengths that models can handle, leading to questions about the need for retrieval systems (Dai al., 2019; Dao al., 2022; Liu et al., 2023). However, Liu al. (2023) and Sun et al. (2021) have noted, models tend to underutilize long-range context and see diminishing performance as con- text length increases, especially when pertinent information is embedded within a lengthy context. Moreover, practically, use of long contexts is expensive and slow. This suggests that selecting the most relevant information for knowledge-intems intensive tasks is still crucial.
Retrieval Methods Retrieval-augmeente language models (RALMs) have seen improvements in various components: the retriever, the reader, and end-to-end system training. Retrieval methods have transitioned from traditional term-based techniques like TF-IDF (Spärck Jones, 1972) and BM25 (Robertson et al., 1995; Roberts et al., 2020) to deep learning- strategies (Karpukhin al., 2020; Khattab Zaharia, 2020; Sachan et al., 2023). Some recent work proposes using large language models retrievers due to their ability to memorize extensive knowledge (Yu et al., 2022: Sun et al., 2022). Research on the reader component includes Iusion-in-Decodee (FID) (Izacard & Grave, 2022), which employs both DPR and BM25 for retrieval and processes passages independently in the encoder and RETRO (Borgeaud et al., 2022; Wang et al., 2023), which utilizes cross-chunked attention and chunkwise retrieval generate text grounded retrieved context.
End-to-end system training work includes Atlas (Izacard et al., 2022), which fine-tunes an encoder- decoder model in conjunction with the retriever; REALM (Guu al., 2020), bidirectional, masked LM fine-tuned for open-domain question answering; and RAG EnttrivaalAAumeenter Genera- tion) (Lewis et al., 2020), which integrates pre-trained sequence-to-sequenee models with neural retriever. Min et al. (2021) introduced Joint Passage Retrieval (JPR) model which uses a tree- decoding algorithm to handle passage diversity and relevance in multi-answer retrieval. Dense Hi- erarchical Retrieval (DHR) and Hybrid Hierarchical Retrieval (HHR) represent advancements retrieval accuracy by combining document and passage level retrievals and integrating sparse and dense retrieval methods, respectively (Liu et al., 2021; Arivazhagan et al., 2023).
We have released the code of RAPTOR https: / thub. wwtp....m/paatthaaaaeeeeeeeeeeee raptor.
2

---

Published as a conference paper at ICLR 2024
Despite diversity in methods, the retrieving components of models predominantly rely stan- dard approaches, i.c., chunking corpora and encoding with BERT-based retrievers. Although this approach is widely adopted, Nair et al. (2023) highlights a potential shortcoming: contiguous seg- mentation might not capture the complete semantic depth of the text. Reading extracted snippets from technical scientific documents may lack important context making them difficult to read or even misleading. (Cohan & Goharian, 2017; Newman et al., 2023; Zhang et al., 2023).
Recursive summarization as Context Summarization techniques provide a condensed view of documents, enabling more focused engagement with the content (Angelidis & Lapata, 2018). The summarization/snipen model by Gao et al. (2023) uses summarizations and snippets of passages, which improves correctness on most datasets but can sometimes be lossy means of compression. summarization model by Wu et al. (2021) employs task decomposition
The recursive-abstracti to summarize smaller text chunks, which are later integrated to form summaries larger sections. While this method is effective for capturing broader themes, it can míss granular details. Llamalndex (Liu, 2022) mitigates this issue by similarly summarizing adjacent text chunks but also retaining intermediate nodes thus storing varying levels of detail, keeping granular details. However, both methods, due to their reliance on adjacency for grouping or summarizing adjacent nodes, may still overlook distant interdependencies within the text, which find and group with RAPTOR.
3 METHODS
Overview of RAPTOR cal structures (Cao & Wang, 2022: Dong et al., 2023b), RAPTOR addresses the issue of semantic depth and connection in reading by building a recursive tree structure that balances broader thematic comprehension with granular details and which allows nodes to be grouped based on semantic sim- ilarity not just order in the text.
Building on the idea that long texts often present subtopics and hierarchi-
Construction of the RAPTOR tree begins with segmenting the retrieval corpus into short, contiguous texts of length 100, similar to traditional retrieval augmentation techniques. If sentence exceeds the 100-token limit, we move the entire sentence to the next chunk, rather than cutting mid-sentence. This preserves the contextual and semantic coherence of the text within each chunk. These texts are then embedded using SBERT, a BERT-based encoder (multi-Ga -i-iai--aai sareiii-aa-eeee--eeeeeeee mpnet-base-cos-- (Reimers & Gurevych, 2019). The chunks and their corresponding SBERT embeddings form the leaf nodes of our tree structure.
To group similar text chunks, employ clustering algorithm. Once clustered, Language Model used to summarize the grouped texts. These summarized texts are then re-embedded, and the cycle of embedding, clustering, and summarization continues until further clustering becomes infeasible, resulting in a structured, multi-layered tree representation of the original documents. An important aspect of RAPTOR is computational efficiency. The system scales linearly in terms of both build time and token expenditure, making it suitable for processing large and complex corpora. For comprehensive discussion RAPTOR's scalability, please refer to the Appendix A.
For querying within this tree, we introduce two distinct strategies: tree traversal and collapsed tree. The tree traversal method traverses the tree layer-by-layer pruning and selecting the most relevant nodes at each level. The collapsed tree method evaluates nodes collectively across all layers to find the most relevant ones.
Clustering Algorithm Clustering plays a key role in building the RAPTOR tree, organizing text segments into cohesive groups. This step groups related content together, which helps the subse- quent retrieval process.
One of the unique aspects of our clustering approach is the use of soft clustering, where nodes can belong to multiple clusters without requiring a fixed number of clusters. This ficxibility is essen- tial because individual text segments often contain information relevant various topics, thereby warranting their inclusion in multiple summaries.
Our clustering algorithm is based on Gaussian Mixture Models (GMMs), approach that offers both flexibility and probabilistic framework. GMMs assume that data points are generated from a mixture of several Gaussian distributions.
3

---

Published as a conference paper at ICLR 2024
Given set of N text segments, each represented as a d-dimensional dense vector embedding, the likelihood of a text vector, x, given its membership in the 1tth Gaussian distribution, is denoted by
P(x]k) (x: Mg. Ex). The overall probability distribution is a weighted combination P(x)
Σx= (x: pkk Σx), where Ti signifies the mixture weight for the t Gaussian distribution.
The high dimensionality of vector embeddings presents a challenge for traditional GMMs, as dis- tance metries may behave poorly when used to measure similarity high-dimensional spaces (Ag- garwal et al., 2001). To mitigate this, we employ Uniform Manifold Approximation and Projection (UMAP), a manifold learning technique for dimensionality reduction (MeInnes et al., 2018). The number of nearest neighbors parameter, n_neighhors, in UMAP determines the balance between the preservation of local and global structures. Our algorithm varies n. neighhrsss to create a hierar- chical clustering structure: it first identifies global clusters and then performs local clustering within these global clusters. This two-step clustering process captures a broad spectrum of relationships among the text data, from broad themes to specific details.
Should a local cluster's combined context ever exceed the summarization model's token threshold, our algorithm recursively applies clustering within the cluster, ensuring that the context remains within the token threshold.
To determine the optimal number of clusters, we employ the Bayesian Information Criterion (BIC) for model selection. BIC not only penalizes model complexity but also rewards goodness of fit (Schwarz, 1978). The BIC for a given GMM is BIC = In(N) - 2 In(L), where N is the number of text segments (or data points), k is the number of model parameters, and L is the maximized value of the likelihood function of the model. In the context of GMM, the number of parameters is a function of the dimensionality of the input vectors and the number of clusters.
With the optimal number of clusters determined by BIC, the Eopectatioo-Maxxiizzan is then used estimate the GMM parameters, namely the means, covariances, and mixture weights.
algorithm
While the Gaussian assumption in GMMs may not perfectly align with the nature of text data, which often exhibits sparse and skewed distribution, our empirical observations suggest that it offers an effective model for our purpose. We run an ablation comparing GMM Clustering with summarizing contiguous chunks and provide details in Appendix B.
Model-Based Summarization After clustering the nodes using Gaussian Mixture Models, the nodes in each cluster are sent to a language model for summarization. This step allows the model to transform large chunks of text into concise, coherent summaries of the selected nodes. For our experiments, we use gpt-3. 3.0t-3.5turxe 5-rubo to generate the summaries. The summarization step con- denses the potentially large volume of retrieved information into manageable size. We provide statistics on the compression due to the summarization in Appendix C and the prompt used for summarization in Appendix D.
While the summarization model generally produces reliable summaries, a focused amnotation study revealed that about 4% of the summaries contained minor hallucinations. These did not propagate to parent nodes and had no discernible impact on question-answerinn answering tasks. For an in-depth analysis of hallucinations. refer to the appendix E.
Querying In this section, elaborate the two querying mechanisms employed by RAPTOR: tree traversal and collapsed tree. These methods offer unique ways of traversing the multi-layered RAPTOR tree retrieve relevant information, each with its own advantages and trade-offs. We provide the pseudocode of both methods in Appendix F. Note that we embed all nodes using SBERT.
The tree traversal method first selects the top-k most relevant root nodes based on their cosine similarity to the query embedding. The children of these selected nodes are considered at the next layer and the top-k nodes are selected from this pool again based on their cosine similarity to the query vector. This process is repeated until we reach the leaf nodes. Finally, the text from all selected nodes is concatenated to form the retrieved context. The algorithm's steps are outlined below:
1. Start at the root layer of the RAPTOR tree. Compute the cosine similarity between the query embedding and the embeddings of all nodes present at this initial layer.
2. Choose the top-k nodes based on the highest cosine similarity scores, forming the set S1.
4

---

Published as a conference paper at ICLR 2024
A. Tree Traversal Retrieval
Overy
Quey
Anower
Giocdder
LLM
Thee Structure
Collapsed Tree Retrieval
Qaey
Query
Encoder
Retrieved Contest
Figure 2: Illustration of the tree traversal and collapsed tree retrieval mechanisms. Tree traver- sal starts at the root level of the tree and retrieves the top-k (here, top-1) node(s) based on cosine similarity to the query vector. At each level, it retrieves the top-k node(s) from the child nodes of the previous layer's top-k. Collapsed tree collapses the tree into a single layer and retrieves nodes until threshold number of tokens is reached, based cosine similarity to the query vector. The nodes on which cosine similarity search is performed are highlighted in both illustrations.
3. Proceed to the child nodes of the elements in set St. Compute the cosine similarity between the query vector and the vector embeddings these child nodes.
4. Select the top l child nodes with the highest cosine similarity scores the query, forming the set S2.
5. Continue this process recursively for d layers, producing sets S1, S.,..., Sa.
6. Concatenate sets S through Sa to assemble the relevant context to the query.
By adjusting the depth d and the number of nodes k selected at each layer, the tree traversal method offers control over the specificity and breadth of the information retrieved. The algorithm starts with a broad outlook by considering the top layers of the tree and progressively focuses on finer details as it descends through the lower layers.
The collapsed tree approach offers simpler way search for relevant information by considering all nodes in the tree simultaneously. as depicted in Figure 2. Instead of going layer-by-layer, this method flattens the multi-layered tree into single layer, essentially bringing all the nodes onto the same level for comparison. The steps for this method are outlined below:
1. First, collapse the entire RAPTOR tree into a single layer. This new set of nodes, denoted C, contains nodes from every layer of the original tree.
2. Next, calculate the cosine similarity between the query embedding and the embeddings of all nodes present in the collapsed set C.
3. Finally, pick the top-k nodes that have the highest cosine similarity scores with the query. Keep adding nodes to the result set until you reach predefined maximum number of tokens, ensuring you don't ton't exceed the model's input limitations.
We tested both approaches on 20 stories from the QASPER dataset. Figure shows the performance of tree traversal with different top- sizes and collapsed tree with different maximum token numbers. The collapsed tree approach consistently performs better. We believe collapsed tree retrieval better due to offering greater flexibility than tree traversal; i.e., by searching through all the nodes simultaneously. it retrieves information that is at the correct level of granularity for given question. In comparison, while using tree traversal with the same values of d and k, the ratio of nodes from each level the tree will be constant. So, the ratio of higher-order thematic information to granular details will remain the same regardless of the question.
5

---

Published as a conference paper at ICLR 2024
One drawback, however, of the collapsed tree approach is that it requires cosine similarity search to be performed on all nodes in the tree. However, this can be made more efficient with fast k-nearest neighbor libraries such as FAISS (Johnson et al., 2019).
Overall, given the collapsed tree approach's greater flexibility and its superior performance on the subset of the QASPER dataset, this is the querying approach with which we proceed. Specifically, we use the collapsed tree with 2000 maximum tokens, which approximately equates to retrieving the top-20 nodes. Using token-based approach ensures the context does not exceed model context constraints as token counts can vary across nodes. For experiments with the UnifiedQA model, we provide 400 to- kens of context, as UnifiedQA has a max con- text length of 512 tokens. We provide the same amount of tokens of context to RAPTOR and to the baselines.
The Traerrud
88
1op
Top
-2
500 1000
1800
2000
2500
Content Length
Figure 3: Comparison of querying methods. Results on 20 stories from the QASPER dataset using tree traversal with different top-k values, and collapsed tree with different context lengths. Collapsed tree with 2000 tokens produces the best results, so we use this querying strategy for our main results.
Qualitative Study We conduct a qualitative analysis to understand the benefits of RAP- TOR's retrieval process compared to Dense Passage Retrieval (DPR) methods. Our study focuses on thematic, multi-hop questions using 15500word Cinderella fairytale. As illustrated in Figure 4, RAPTOR's tree-based retrieval allows it to choose nodes from different tree layers, matching the question's detail level. This approach often yields more relevant and comprehensive information for downstream tasks than DPR. For detailed discussion and examples, including the text retrieved by both RAPTOR and DPR for specific questions, please refer to the appendix G.
4 EXPERIMENTS
Datasets We measure RAPTOR's performance across three question-answering datasets: Narra- tiveQA, QASPER, and QuALITY.
NarrativeQA dataset that comprises question-answer pairs based on the full texts of books and movie transcripts, totaling 1,572 documents (Kočiský et al., 2018; Wu al., 2021). The NarrativeQA-Story task requires a comprehensive understanding of the entire narrative in order to accurately answer its questions, thus testing the model's ability to comprehend longer texts in the literary domain. We measure performance on this dataset using the standard BLEU (B-1, B-4), ROUGE (R-L), and METEOR (M) metrics. Please appendix H for more details the Narra- tiveQA evaluation script used in our experiments.
The QASPER dataset includes 5,049 questions across 1,585 NLP papers, with each question probing for information embedded within the full text (Dasigi et al., 2021). The answer types in QASPER Yes/No, Abstractive, and Extractive. Accuracy is
are categorized as Answerablc/Unanswerri measured using standard FI.
Lastly, the QuALITY dataset consists of multiple-choice questions, each accompanied by context passages averaging approximately 5,000 tokens in length (Pang et al., 2022). This dataset calls for reasoning over the entire document for QA tasks, enabling us to measure the performance of our re- trieval system medium-length documents. The dataset includes a challenging subset, QuALITY- HARD, which contains questions that majority of human annotators answered incorrectly a speed-setting. We report accuracies for both the entire test set and the HARD subset.
Controlled Baseline Comparisons We first present controlled comparisons using the UnifiedQA 3B the reader, with SBERT (Reimers & Gurevych, 2019), BM25 (Robertson et al., 1995; 2009), and DPR (Karpukhin et al., 2020) as the embedding models with and without the RAPTOR tree structure, on three datasets: QASPER, NarrativeQA, and QuALITY. As shown in Tables and 2,

---

Published as a conference paper at ICLR 2024
0
2
3
6
7
8
9
10
11
12
13
14
15
RAPTOR retrieved Question
Question
Illustration of how RAPTOR retrieves information for two questions "What is the central theme of the story?" and "How did Cinderella find a happy ending?". Highlighted nodes indicate RAPTOR's selections, while arrows point to DPR's leaf nodes. Notably, RAPTOR's ' context often encompasses the information retrieved by DPR, either directly or within higher-layer summaries.
Figure 4: Querying Process:
about the Cinderella story:
our results demonstrate that RAPTOR, when combined with any retriever, consistently outperforms the respective retriever across all datasets."
Since RAPTOR with SBERT has the best performance, we use in all subsequent experiments. We now compare RAPTOR with BM25 and DPR, using three different LLMs: GPT-3, GPT-4, and UnifiedQA. shown in Table 3, RAPTOR consistently outperforms BM25 and DPR across all three Language Models on the QASPER dataset. RAPTOR'S s F-1 Match scores are 53.1 1%, 55.7%, and 36.6% when using GPT-3, GPT-4, and UnifiedQA, respectively. These scores surpass DPR by margins of 1.8, 2.7, and 4.5 points, and outdo BM25 by 6.5, 5.5, and 10.2 points across the respective LLMs. QASPER requires synthesizing information within NLP papers, so it is unsurprising that RAPTOR's higher-level summary nodes would allow it outperform methods that only extract the top-k most similar raw chunks of text, which may not contain the correct response in isolation.
Table I: NarrativeQA Performance With + Without RAPTOR: Performance comparison of various retrieval methods (SBERT, BM25, DPR) with and without RAPTOR on the NarrativeQA dataset, using UnifiedQA-3B as the language model. RAPTOR outperforms baselines of each re- spective retrieval method.
Model
ROUGE
BLEU-1
BLEU-4
METEOR
SBERT with RAPTOR
30.87%
23.50%
6.42%
19.20
SBERT without RAPTOR
29.26%
22.56%
5.95%
18.15%
BM25 with RAPTOR
27.93%
21.17%
5.70%
17.03%
BM25 without RAPTOR
23.52%
17.73%
4.65%
13.98%
DPR with RAPTOR
30.94%
23.51
6.45%
19.05 %
DPR without RAPTOR
29.56%
22.84%
6.12%
18.44%
Likewise, in the QuALITY dataset as shown Table 4, RAPTOR achieves an accuracy of 62.4%. 4%, which is 2% and 5. % improvement over DPR and BM25. Similar trends are observed when Uni- fiedQA employed, with RAPTOR outperforming DPR and BM25 by 2.7% and 6.7%, respectively.
Finally, in the NarrativeQA dataset, as presented in Table 6, RAPTOR excels across multiple met- rics. For ROUGE-I, surpasses BM25 and DPR by 7.3 and 2.7 points, respectively. In other metrics like BLEU-1, BLEU-4, and METEOR, RAPTOR outperforms BM25 and DPR by margins ranging from 1.7 to 5.8 and 0.7 to 2.1 points, respectively.
For the DPR experiments in Tables I and 2, we used the dpr-mult www.tttssebbbssseee base model opposed to dpr-singl e- le-nn-base base which used in rest of the experiments done earlier. This decision was based on the performance observed in Karpukhin al. (2020), where dpr-mule -bae showed superior results.

---

Published as a conference paper at ICLR 2024
Table 2: QuALITY and QASPER Performance With + Without RAPTOR: Performance com-
parison across the QuALITY and QASPER datasets of various retrieval methods (SBERT, BM25, DPR) with and without RAPTOR. UnifiedQA-3B used the language model. RAPTOR outper- forms baselines of each respective retrieval method for both datasets.
Model
Accuracy (QuALITY)
Answer F1 (QASPER)
SBERT with RAPTOR
56.6%
36.70%
SBERT without RAPTOR
54.9%
36.23%
BM25 with RAPTOR
52.1 %
27.00 %
BM25 without RAPTOR
49,9%
26.47%
DPR with RAPTOR
54.7%
32.23%
DPR without RAPTOR
53.1%
31.70%
Table 3: Controlled comparison of [-] scores on the QASPER dataset, using three different lan- guage models (GPT-3, GPT-4, UnifiedQA 3B) and various retrieval methods. The column "Title Abstract" reflects performance when only the title and abstract of the papers RAPTOR outperforms the established baselines BM25 and DPR across all tested language models. Specifically, RAPTOR's R-1 scores are at least 1.8% points higher than DPR and at least 5.3% points higher than BM25.
used for context.
BM25 GPT-3 Match
Retriever
GPT-4 F-1 Match
UnifiedQA F-1 Match
Title Abstract
25.2
22.2
17.5
46.6
50.2
26.4
DPR
51.3
53.0
32.1
RAPTOR
53.1
55.7
36.6
Comparison to State-of-the-art Systems Building upon controlled comparisons, we examine RAPTOR's performance relative other state-of-the-art models. As shown in Table 5, RAPTOR with GPT-4 sets a new benchmark on QASPER, with 55.7% F-1 score, surpassing the CoLT5 XL's score of 53.9%.
Table 4: Comparison of accuracies the QuAL- ITY dev dataset for two different language mod- els (GPT-3, UnifiedQA 3B) using various retrieval methods. RAPTOR outperforms the baselines of BM25 and DPR by at least 2.0% accuracy.
Model
GPT-3 Acc.
UnifiedQA Acc.
BM25
57.3
49.9
In the QuALITY dataset, as shown in Table 7, RAPTOR paired with GPT-4 sets a new state- of-the-art with an accuracy of 82.6%, surpass- ing the previous best result of 62.3%. ticular. it outperforms CoLISA by 21.5% QUALITY-HARD, which represents questions that humans took unusually long to correctly answer, requiring rereading parts of the text, difficult reasoning, both.
DPR RAPTOR
53.9 62.4
60.4
56.6
par-
Table 5: Results on F-1 Match scores of various models on the QASPER dataset.
Model
F-1 Match
LongT5 XL (Guo et al., 2022)
53.1
CoLT5 XL (Ainslic al., 2023)
53.9
For the NarrativeQA dataset, as represented in Table 6, RAPTOR paired with UnifiedQA sets a new state-of-the-art METEOR score. When compared to the recursively summarizing model by Wu et al. (2021), which also employs UnificdQA, RAPTOR outperforms it on all metrics. While Wu et al. (2021) rely solely on the summary in the top root node of the tree structure, RAPTOR benefits from its intermediate layers and clustering approaches, which allows capture range of information, from general themes to specific details, contributing to its overall strong performance.
RAPTOR GPT-4
55.7
4.1 CONTRIBUTION OF THE TREE STRUCTURE
We examine the contribution of each layer of nodes RAPTOR's 's retrieval capabilities. We hy- pothesized that upper nodes play a crucial role in handling thematic or multi-hop hop queries requiring broader understanding of the text.
8

---

Published as a conference paper at ICLR 2024
Table 6: Performance comparison on the NarrativeQA dataset across multiple models, focusing on four metrics: ROUGE-L, BLEU-1, BLEU-4, and METEOR. RAPTOR, when paired with Uni- fiedQA 3B, not only surpasses retrieval methods like BM25 and DPR but also sets new state-of- the-art in the METEOR metric.
Model
ROUGE-L
BLEU-
BLEU-
METEOR
BiDAF (Kočiský et al., 2018)
6.2
0.3
3.7
BM25 BERT (Mou et al., 2020)
15.5
14.5
1.4
5.0
Recursively Summarizing Books (Wu al., 2021)
21.6
22.3
4.2
10.6
Retriever Reader (Izacard & Grave, 2022)
32.0
35.3
7.5
11.1
RAPTOR UnifiedQA
30.8
23.5
19.1
Table 7: Accuracies of the QuALITY dataset on both the overall test set and the more challenging hard subset. GPT-4 with RAPTOR new state-of-the-art
Accuracy
Model
Test Set
Hard Subset
Longformer-bass (Beltagy et al., 2020)
39.5
35.3
DPR and DeBERTaV3 V--lagg (Pang et al., 2022)
55.4
46.1
CoLISA (DeBERTaV3 3-rger (Dong al., 2023a)
62.3
54.7
RAPTOR GPT-4
82.6
76.2
Table Performance of RAPTOR when querying different tree layers for Story from the QuAL-
ITY dataset. Columns represent different starting points (highest layer) and rows represent different numbers of layers queried.
Layers Queried Start Layer
Layer (Leaf Nodes)
Layer
Layer
layer
57.9
57.8
$7.9
layers
52.6
63.15
layers
73.68
We validated this hypothesis both quantitatively and qualitatively. We present qualitative analysis in appendix G. To quantitatively understand the contribution of the upper-level nodes, used stories from the QuALITY dataset. The RAPTOR tree is built for each of these stories, as described in Section 3. However, during retrieval, limit the search to different subsets of layers. For example, we exclusively retrieve from the leaf nodes and each upper layer, as well as from different contiguous subsets of the layers. We show findings specific one story in Table revealing that full-tree search, utilizing all layers, outperformed retrieval strategies that focused only on specific layers.
These findings highlight the importance of the full structure in RAPTOR. By providing both the original text and higher-level summaries for retrieval, RAPTOR can effectively handle a wider range of questions, from higher-order thematic queries detail-oriented questions. Detailed results for additional stories and an ablation study on layer contributions can be found in Appendix I.
CONCLUSION
In this paper, we have presented RAPTOR, novel tree-based retrieval system that augments the parametric knowledge of large language models with contextual information at various levels of abstraction. By employing recursive clustering and summarization techniques, RAPTOR creates hierarchical tree structure that is capable of synthesizing information across various sections of the retrieval corpora. During the query phase, RAPTOR leverages this tree structure for more effective retrieval. Our controlled experiments demonstrated that RAPTOR not only outperforms traditional retrieval methods but also sets performance benchmarks several question-answerte tasks.

---

Published as a conference paper at ICLR 2024
6 REPRODUCIBILITY STATEMENT
Language Models for QA and Summarization Four language models are used our RAPTOR experiments: GPT-3 and GPT-4 for QA tasks, and GPT-3.5-turbo for summarization. The gpt-3, gpt-4, and 5 5-turb models can be accessed via API calls (OpenAl API). UnifiedQA, which is used for QA tasks, is publicly available Hugging Face.
Evaluation Datasets The three evaluation datasets used in our QASPER, and NarrativeQA-aun all publicly accessible. These datasets ensure that the retrieval and QA tests conducted in this study can be replicated.
Source Code We have released the code of RAPTOR at https://github. com/
REFERENCES
Charu C Aggarwal, Alexander Hinneburg, and Daniel A Keim. On the Surprising Behavior of Dis- tance Metrics in High Dimensional Space. In Database Theory -CDT 2001: 8th International Conference London, UK, January 4-6, 2001 Proceedings 8, 420-434, Springer, 2001. URL atttps://iin spr springer...... com chapter/10.... - 1007 3- 540-4400040 40..0044503x220 27.
Joshua Ainslie, Tao Lei, Michiel de Jong, Santiago Ontañón, Siddhartha Brahma, Yury Zemlyan- skiy, David Uthus, Mandy Guo, James Lee-Th Yi Tay. et al. CoLT5: Faster long-range transformers with conditional computation. arXiv preprint arXiv:2303.09752, 2023. URL https://arxiv. wttp://.orr/////////ooooooo
09752.
Ekin Akyurek, Tolga Bolukbasi, Frederick Liu, Binbin Xiong, Ian Tenney, Jacob Andreas, and Kelvin Guu. Towards tracing knowledge in language models back the training data. In Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 2429-2446, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Linguistics. doi: 10. 22116633v1/2D/ http:3/w/22222.ninngg-----e findings-emmlp...55 URL. https: /acl laclaaaatooooyyo org.orgg//// 2202..
findings Sindings-ema 1p. 180.
Stefanos Angelidis and Mirella Lapata. Summarizing opinions: Aspect extraction meets sentiment prediction and they are both weakly supervised. arXiv preprint 18Xiv: 08858, 2018. URL
808. 08858.
Manoj Ghuhan Arivazhagan, Lan Liu, Peng Qi, Xinchi Chen, William Yang Wang, and Zhiheng Huang. Hybrid hierarchical retrieval for open-domain question answering. In Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (eds.), Findings of the Association for Computational Linguistics: ACL 2023, pp. 10680-10689, Toronto, Canada, July 2023. Association for Computa- tional Linguisties. doi: 10. (0.188633/11/022 2tn2202...nniingaac...777. URL https: http///aaaaaaotttgogoooooo org/2023. www.ifiinnnninn 679.
Iz Beltagy, Matthew E. Peters, and Arman Cohan. Longformer: The Long-document Transformer, 2020. URL https::///. http//aaivvv/// orgo aabs///////a www.ab//..... 05150. arXiv preprint arXiv:2004.05
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmaann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. International conference on machine learning, pp. 2206-2240. PMLR, 2022. URL https: 04426.
www.iii.org/b///////// 2112.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhari- wal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Rad- ford, Ilya Sutskever, and Dario Amodei. Language Models Few-Shot Leamers. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Advances Neu- ral Information Processing Systems, volume 33, pp. 1877-1901. Curran Associates, Inc.,
10

---

Published as a conference paper at ICLR 2024
2020. URL
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Ka- mar, Peter Tee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, et al. Sparks of Artificial General Intelligence: Early Experiments with GPT-4. arXiv preprint arXiv:2303.22712, 2023. URL https://arxtt org/ 2//.//333..22..
Shuyang Cao and Lu Wang. HIBRIDS: Attention with hierarchical biases for structure long document summarization. Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 786-807, Dublin, Ireland, May 2022. Association for Computational Linguisties. doi: 10. 10.186633//102222aclllomoo // //aclanthology wtwpggy..ggg////// acl- 58.
URL https:
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to Answer Open-Domain Questions. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1870-1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10. 10.18863/11/17-11771 URL https: / //aclanthology wwwphhhooooooooo/1111111111 org/P17-1172
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, al. PaLM: Scaling Language Modeling with Pathways. arXiv preprint arXiv:2204.02311 2022. URL
httts::/arxiv.. org/ abs/ 2204.020220 02311.
Arman Cohan and Nazli Goharian. Contextualizing citations for scientific summarization using word embeddings and domain knowledge. In Proceedings of the 40th International ACM SIGIR Conference on Research and Development Information Retrieval, pp. 1133-1136, 2017. URL https: //dl/ acm, org/ doi abs / 10.10.0 00115/33377333 011777166..887777 3080740.
Zihang Dai, Zhilin Yang, Yiming Yang. Jaime Carbonell, Quoc Le, and Ruslan Salakhutdinov. Transformer-LL. Attentive language models beyond fixed-length context. Proceedings of the 57th Anmual Meeting the Association for Computational Linguistics, pp. 2978 2988, Florence, Italy, July 2019. Association for Computational Linguisties. doi: 10. 10.18653/v1/P199 1285. URL https: / aaacannnholllococcooooco lonttholoo logyy 1ttp//99-1288
Tri Dao, Dan Fu, Stefano Ermon, Atri Rudra, and Christopher Ré. FlashAttention: Fast and memory-efficient exact attention with IO-Awareness. Advances in Neural Information Processing Systems, 35:16344-16335 2022. URL
wwg.aaiiii..iiiaaaaoooooooooooor abs/ /2055 14135.
Pradeep Dasigi, Kyle Lo, Iz Beltagy, Arman Cohan, Noah A. Smith, and Matt Gardner. A Dataset of Information-Seeeking Questions and Answers Anchored in Research Papers. In Proceed- ings of the 2021 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pp. 4599-4610, Online, June 2021. Asso- ciation for Computational Linguistics. doi: 10. 10.18653/v1/2022 http//222..aaaal-a maacl-mmin.366... 365.
URL https:
/ //aclanthology wwt.o/////////.ooo naacl
Mengxing Dong, Bowei Zou, Yanling Li, and Yu Hong. CoLISA: Inner Interaction via Contrastive Learning for Multi-choice Reading Comprehension. In Advances in Information Retrieval: 45th European Conference on Information Retrieval, ECIR 2023, Dublin, Ireland, April 2-6, 2023, Proceedings, Part 1, pp. 264-278. Springer, 2023a. URL chapter/10 /10 1000/00///0 978 3- 3--33-182244721111
com/
Zican Dong, Tianyi Tang, Lunyi Li, and Wayne Xin Zhao. A survey long text modeling with transformers. arXiv preprint arXiv:2302. 14502, 2023b. URL https: http://axxvvvvv org/ / 2302. 14502.
Tianyu Gao, Howard Yen, Jiatong Yu, and Danqi Chen. Enabling large language models generate text with citations. arXiv preprint arXiv:2305. 14627, 2023. URL https: / //aaxixivva. org// abs/2305. 14627.

---

Published as a conference paper at ICLR 2024
Mandy Guo, Joshua Ainslie, David Uthus, Santiago Ontanon, Jianmo Ni, Yun-Hsuan Sung, and Yinfei Yang. LongT5: Efficient text-to-text transformer for long sequences. In Findings of the Association for Computational Linguistics: NAACL 2022, pp. 724-736, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 1 URL wwwph:tpp/..o.. /ttp:p:://aaaaaaie hatpa/caaaattooooog.oooooo wewelogyyoooogooeeeee atg/2022, findi ngs-nnacc htt.22..iniinss-mmeelle
Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. Retrieval Augmented Language Model Pre-Training. In International conference on machine learning, pp. 3929-3938. PMLR, 2020. URL https: /tl::://dooiie org. 48550/ arxiv 2002 08909.
Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models. arXiv preprint arXiv:2203.155500 2022. URL https://arxix org/ aa///bb///// 2203 15556.
Gautier Izacard and Edouard Grave. Distilling Knowledge from Reader to Retriever for Ques- tion Answering, 2022. URL.
04584. arXiv preprint
arXiv:2012.04584.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave. Few-shot learning with re- trieval augmented language models. arXiv preprint arXiv:2208.03299) 2022. URL https: / //arxiv.org/ abs/ 2208 03299.
Zhengbao Jiang, Frank Xu, Jun Araki, and Graham Neubig. How can we know what language models know? Transactions of the Association for Computational Linguistics, 8:423-4333 2020. URL https: //apps:///rxxv...///// org/ abs/ 1ttss/911..1111..1 12543.
Jeff Johnson, Matthijs Douze, and Hervé Jégou. Billion-Scall Similarity Search with GPUs. IEEE Transactions on Big Dato, 7(3):535-5477 547, 2019. URL https:tttaaaeeee org tbbs/102.. 08734.
Nikhil Kandpal, Haikang Deng, Adam Roberts, Eric Wallace, and Colin Raffel. Large Language Models struggle learn Long-Tail Knowledge. In International Conference on Machine Learn- ing. pp. 15696-15707. PMLR, 2023. URL https: http//ppoceee.g..ceceee mlr. kanopa 1 23a 2 http//annpppa22. pdf.. h wtw.://d.gg........ssesss//// www.s//2e2222.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense Passage Retrieval for Open-Domain Question Answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing Online, November 2020. Association for Computational Linguis- URL /ttp///laantttt www.aaccnnnccyyo..aaoaaaaoo or2/2020.
(EMNLP), pp. tics. doi: 10. emrlp ma iiaiinn.. 550.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannanch Hajishirzi. UNIFIEDQA: Crossing format boundaries with single QA system. In Findings of the Association for Computational Linguistics: EMNLP 2020, pp. 1896-1907, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.18653/w12200.. findings-emmp..111. URL https:/ //act//ccc httt:s/://claatthollggg.g.o www.hhollg...oog/gg//oe f indingg emn 171.
Omar Khattab and Matei Zaharia. CoIBERT: Efficient and effective passage search via textualized late interaction over bert. In Proceedings of the 43rd International ACM SIGIR conference research and development in Information Retrieval, pp- 39 48, 2020. URL bittps://arxt wwwoor/gaaao.. abs 12832.
Tomáš Kočiský, Jonathan Schwarz, Phil Blunsom, Chris Dyer, Karl Moritz Hermann, Gábor Melis, and Edward Grefenstette. The NarrativeAA Reading Comprehension Challenge. Transactions of the Association for Computational Linguistics, 6:317-328, 2018. URL https:/ arxxiv. org//bs/17112_0mm 07000.
12

---

Published as a conference paper at ICLR 2024
Patrick Lewis, Ethan Perez. Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, et al. Getrivall-Augennte Gener- ation for Knowledge-Ine Intensive NLP Tasks. Advances in Neural Information Processing Systems, 11:9459-94744 2020. URL https: http::/oo..ooooo
48550/arXiis 2005. 11401.
Jerry Liu. LlamaIndex, 2022. URL
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. arXiv preprint
arXiv:2307.03112, 2023. URL httpp::/arxxi..rrrtt httpg//bb/22000 03172.
Ye Liu, Kazuma Hashimoto, Yingbo Zhou, Semih Yavuz, Caiming Xiong, and Philip Yu. Dense hierarchical retrieval for open-domain question answering. In Marie-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Findings of the Association for Compu- tational Linguisties: EMNLP 2021, pp. 188-200, Punta Cana, Dominican Republic, Novem- ber 2021. Association for Computational Linguistics. doi: 10.18653/v www.o//.nnnniinnnnnn....m URL https::/p//.......t.m aaclaaccaaa antho oogg 2021 find inddings-mmn) 19.
Leland McInnes, John Healy, and James Melville. UMAP: Uniform Manifold Approximation and Projection for Dimension Reduction, 2018. URL
18802
03426. arXiv preprint arXiv:1802.0344
Sewon MIn, Kenton Lee, Ming- Wei Chang, Kristina Toutanova, and Hannaneh Hajishirzi. Joint passage ranking for diverse multi-answer retrieval. In Maric-Francine Moens, Xuanjing Huang, Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 6997-7008, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10. 10118655311/222 emnlp-main.566 URL https:
emnlp main. 560.
Sewon Min, Weijia Shi, Mike Lewis, Xilun Chen, Wen-tau Yih, Hannaneh Hajishirzi, and Luke Zettlemoyer. Nonparametric masked language modeling. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 2097-2118, Toronto, Canada, July 2023. Associ- ation for Computational Linguistics. doi: 10. 10.18653/v1/2023 fn20...n///... 132. URL https: findingsnnansee act.ssacc.132
// //acllantoloogy.oo
Eric Mitchell, Charles Lin, Antoine Bosselut, Christopher D Manning, and Chelsea Finn. Memory-based model editing at scale. In International Conference on Machine Learning, pp. 15817-15831. PMLR, 2022. URL https:/ tttps::/pprceediing mlr, press/v162/ mitche mitchell22a/mitcchaa www.mhhiheeeeaaaaameeeeeeaeeaaaa pdf.
Xiangyang Mou, Mo Yu, Bingsheng Yao, Chenghao Yang, Xiaoxiao Guo, Saloni Potdar, and Hui Su. Frustratingly hard evidence retrieval for QA over books. In Proceedings of the First Joint Workshop on Narrative Understanding, Storylines, and Events, pp. 108-113, Online, July 2020. Association for Computational Linguistics. doi: 10.18653/11/222..msss nuse- 1.13. URL https: / //aclanthology orgg 2220 n.13-1.13.
Inderjeet Nair, Aparna Garimella, Balaji Vasan Srinivasan, Natwar Modani, Niyati Chhaya, Srikr- ishna Karanam, and Sumit Shekhar. A neural CRF-based hierarchical approach for lin- ear text segmentation. In Findings of the Association for Computational Linguistics: EACL 2023, pp. 883-893, Dubrovnik, Croatia, May 2023. Association for Computational Linguis- /aacc//alllaa tics. doi: 10. 10.18653/v1/222, htt.s//////222...ninnneeaaaa findings-eacl.65 URL httpppss///ccllatnellllg/////// org/2023. findings eacl. 65.
Benjamin Newman, Luca Soldaini, Raymond Fok, Arman Cohan, and Kyle Lo. A controllable qa- based framework for decontextuullizatin.. arXiv preprint arXhv:2305.14772 2023. URL https: / //arxiv.
14772.
OpenAl. GPT-4 Technical Report. ArXiv, abs/2303.08774, 2023, URL htlltt//////aiv.orr abs/2303 08774.
Richard Yuanzhe Pang. Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang. Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY:
13

---

Published as a conference paper at ICLR 2024
Question Answering with Long Input Texts, Yes! In Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies. pp. 5336-5358 Seattle. United States, July 2022. Association for Computational Linguistics. URL ht.p:s:::t............p / / acc antho wtw.yo/0rg02222. naacl maa n. 391.
Fabio Petroni, Tim Rocktäschel, Patrick Lewis, Anton Bakhtin, Yuxiang Wu, Alexander Miller, and Sebastian Riedel. Language models knowledge bases? arXiv preprint arXiv: (ee..190900000 2019. URL https: htt///aaxiivvo// hblgg//bb////// 199.. .0....06666 01066.
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katie Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susannah Young, et al. Scaling language models: Methods, Analysis & Insights from Training Gopher. arXiv preprint 1rXiv:2112.11444 2021.
URL wttph:ttttpattteee org/ httg/ab//2112 11446.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhlgay, Amnon Shashua, Kevin Leyton- Brown, and Yoav Shoham. In-comtext retrieval-augmented language models. arXiv preprint arXiv:2302.0000 00083, 2023. URL
abs http://aab//22...oo// 00083.
Nils Reimers and Iryna Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT- networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Lan- guage Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 3982-3992, Hong Kong, China, November 2019. Association for Com- putational Linguistics. doi: 10111855/11/D// 1410. URL https: cettps:/aacaanthee http://aaaaaatttolog..oooo orgg/ D19-18 019-1410.
Adam Roberts, Colin Raffel, and Noam Shazeer. How Much Knowledge Can You Pack Into the Parameters of Language Model? In Proceedings of the 2020 Conference on Empir- ical Methods in Natural Language Processing (EMNLP), pp. 5418 5426, Online, November 2020. Association for Computational Linguistics. doi: 10. 10.11863/1122220ccmm main.437... URL https: http:///aaaaatthhh
org/ 2020. emnlp-main 437.
Stephen Robertson, Hugo Zaragoza, et al. The Probabilistic Relevance Framework: BM25 and Beyond. Foundations and Trends in Information Retrieval, 3(4):333-389, 2009. URL https:
//doi. orooi.orggin 10. 1111/11101100000
Stephen E Robertson, Steve Walker, Susan Jones, Micheline M Hancock-Beaulisn Mike Gatford, et al. Okapi at TREC-3. Nist Special Publication Sp, 109:109, 1995. URL https: //ww. mi microsoft. com/ en us wwwpeeeacwtpppaaaaaaaeeeee
Devendra Singh Sachan, Mike Lewis, Dani Yogatama, Luke Zettlemoyer, Joelle Pineau, and Manzil Zaheer. Questions are all you need to train a dense passage retriever. Transactions of the As- sociation for Computational Linguistics, 11:600-616, 2023, doi: 10.1164/ URL
https: haap::///aalaathhlogg. http:y/..g//////// wttp:///.//0//2..1111.... 2ttp:/21.....a1.....
Gideon Schwarz. Estimating the Dimension of Model. The annals of statistics, pp. 461-464, 1978. URL vol vollum-66/ aos/1176344136 full.
oon aooeeioo-------------oo--------- 1214
Karen Spärek Jones. A Statistical Interpretation of Term Specificity and its Application in Re-
trieval. Journal of documentation, 28(1):11-21) 1972. Utthttps: ort./10.1111011.
eb026526.
Simeng Sun, Kalpesh Krishna, Andrew Mattarella-Mickk. and Mohit lyyer. Do long-range language models actually use long-range context? In Marie-Francine Moens, Xuanjing Huang. Lucia Specia, and Scott Wen-tau Yih (eds.), Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pp. 807-822, Online and Punta Cana, Dominican Republic, November 2021. Association for Computational Linguistics. doi: 10.18653/v120211cmmmm main. 62. URL https / www.ccaannhooooooooooooo org/ 2021 emnnl-mm 1...ain.....
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. decitation-augmeente language models. arXiv preprint arXiv:2210.01296. 2022. URL https: / arxiv. org/ abs/ 221.. 01296.
14

---

Published as a conference paper at ICLR 2024
Alon Talmor, Yanai Elazar, Yoav Goldberg, and Jonathan Berant. oLMpics- what language model pre- training captures. Transactions of the Association for Computational Linguistics. 8: 743-758, 2020. URL https: http://axxiv..oo.. abs
Boxin Wang. Wei Ping, Peng Xu, Lawrence McAfee, Zihan Liu, Mohammad Shoeybi, Yi Dong. Oleksii Kuchaiev, Bo Li, Chaowei Xiao, et al. Shall we pretrain autoregressive language models with retrieval? comprchensive study. arXiv preprint arXiv:2304.06762, 2023. URL https: // //aaaxii.ogggaaaaaea..a wtw.://rggab/////.3ee// 06762.
Jeff Wu, Long Ouyang, Daniel M. Ziegler, Nisan Stiennon, Ryan Lowe, Jan Leike, and Paul Christiano. Recursively Summarizing Books with Human Feedback, 2021. URL https:
Adams Wei Yu. David Dohan, Minh-Thang Thang Luong, Rui Zhao, Kai Chen, Mohammad Norouzi, and Quoc V. Le. QANet: Combining Local Convolution with Global Self-Attention for Read- ing Comprehension, 2018. URL
http://axvvvvvoo/o www.wwwwwwwwwa./////o//r.ar.aree /104. 1804 09541. arXiv preprint
arXiv:1804.0955
Wenhao Yu, Dan Iter. Shuohang Wang. Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large Language Models are strong context generators, 2022. URL https: aaxiv. org /abs//////// 2209. 10063.
Shiyue Zhang, David Wan, and Mohit Bansal. Extractive is not faithful: An investigation of broad unfaithfulness problems in extractive summarization. In Anna Rogers, Jordan Boyd- Graber, and Naoaki Okazaki (eds.), Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 2153-2174, Toronto, Canada, July 2023. Association for Computational Linguistics. doi: 10. 1..18653/1/2223.acl long. 120. URL h
htt.//200..cclooogg 12c.
A SCALABILITY AND COMPUTATIONAL EFFICIENCY OF THE TREE-BUILDING PROCESS
To assess the computational efficiency and cost-effectiveness of RAPTOR's tree-building process, we conducted experiments on consumer- grade laptop. specifically an Apple M1 Mac with 16GB of RAM. These experiments aimed demonstrate the scalability and feasibility of RAPTOR on typical hardware. We varied the context length from 12,500 to 78,000 tokens and measured both the token expenditure and the time required to complete the tree- rre-bbulding process, from initial splitting and embedding to the construction of the final root node.
Narrative OA
Quality
40000
400000
12000
Pronge Complizss 7akra
Pravet Corggania Baree
30000-
30000
Complitinn
420000
8000
20000
300000
10000
10000
25000 38000
30000
100000
200000
3000
5000
8000
Lenght
Figure S: Token cost function of document length for QASPER, NarrativeQA, and QuALITY. RAPTOR tree construction costs scale linearly with document length for each of the datasets.
Token Expenditure We empirically investigated the relationship between the initial document length and the total number of tokens expended during the tree-building process, which includes both the prompt and completion tokens. The document lengths varied significantly across the three
15

---

Published as a conference paper at ICLR 2024
datasets examined: QuALITY, QASPER, and NarrativeQA. Figure 5 illustrates clear linear corre- lation between the initial document length and the total token expenditure, emphasizing that RAP- TOR maintains linear token scaling regardless of document complexity length.
2300
1000
90
20000
50000
GXOO
03030
Figure 6: Build time as a function document length for documents of up to 80,000 tokens. RAP- TOR tree construction time scales linearly with document length for each of the datasets.
Build Time We also empirically observed a consistent linear trend between the document length and the build time, shown in Figure 6. This suggests that RAPTOR scales linearly in terms of time, making it viable solution for efficiently processing large corpora of varying lengths.
Conclusion Overall, our empirical results indicate that RAPTOR scales both in terms of tokens expended and build time. Even as the complexity and volume of the input text grow, the cost of constructing the tree scales predictably and linearly. This demonstrates that RAPTOR is computa- tionally efficient and well-suited for processing large and diverse corpora.
B ABLATION STUDY ON CLUSTERING MECHANISM IN RAPTOR
To assess the effectiveness of the clustering mechanism in our RAPTOR approach, conducted an ablation study on the QuALITY dataset. This study compares RAPTOR's performance with a balanced tree-style encoding and summarization of contiguous chunks, in contrast to standard clustering method.
B. METHODOLOGY
Both configurations in this ablation study utilized SBERT embeddings and UnifiedQA to maintain consistency in retrieval. For RAPTOR, we employed our typical clustering and summarization process. In contrast, the alternative setup involved creating a balanced tree by recursively encoding and summarizing contiguous text chunks. We determined the window size for this setup based on the average cluster size observed in RAPTOR, which is approximately 6.7 nodes. Hence, chose a window size of 7 nodes. The collapsed tree approach was applied for retrieval in both models.
B.2 RESULTS & DISCUSSION
The results of the ablation study are presented in table 9. The results from this ablation study clearly indicate an improvement in accuracy when employing RAPTOR's clustering mechanism over the pecency-based tree approach. This finding substantiates hypothesis that the clustering strategy in RAPTOR is more effective in capturing homogeneous content for summarization, thereby enhancing the overall retrieval performance.
16

---

Published as a conference paper at ICLR 2024
Table Ablation study results comparing RAPTOR with recency-based tree approach
Configuration
Accuracy
RAPTOR SBERT embeddings UnifiedQA Recency- based tree + SBERT embeddings + UnifiedQA
56.6 %
55.8%
C DATASET STATISTICS AND COMPRESSION RATIOS
The average ratio of the summary length the sum of child node lengths across all datasets is 0.28, indicating a 72% compression rate. On average, the summary length is 131 tokens, and the average child node length is 86 tokens. Below are the detailed statisties for all three datasets:
Table 10: Statistics of Average Summary Length and Child Node Length Across Datasets
Dataset
Avg. Summary Length (tokens)
Avg. Child Node Text
Avg. # of Child Nodes Per Parent
Avg.
Compression Ratio
Length (tokens)
All Datasets
131
85.6
.28
QuALITY
124.4
87.9
5.7
.28
NarrativeQA
129.7
85.5
6.8
.27
QASPER
145.9
86.2
5.7
.35
D SUMMARIZATION PROMPT
Table 11 shows the prompt used for summarization.
Table 11: Prompt for Summarization
Role
Content
system
You are a Summarizing Text Portal
Write summary of the following. including many key details possible: [context]:
E HALLUCINATION ANALYSIS
To assess the quality and accuracy of the summarizations within our RAPTOR model, we conducted an analysis focusing on hallucinations in the generated summaries. The summaries were generated by gpt-3.5 10tt-3.5tuuree 3-turbo and subsequently annotated to quantify the rates of hallucinations, to examine whether such inaccuracies propagate to parent nodes, and to evaluate their impact on question- answering (QA) tasks.
E. METHODOLOGY
We randomly sampled 150 nodes across 40 stories and evaluated them for hallucinations. This sampling strategy provides broad view of the model's performance across different contexts. Each node was annotated by hand, and determined if it contained hallucination.
E.2 FINDINGS
Out of the 150 nodes sampled, 4% (6 nodes) contained some form of hallucination. Most commonly, these hallucinations originated from the model adding minor information possibly from its training data that was not present in the text being summarized, or from incorreetly extrapolating some information when creating the summary.
17

---

Published as a conference paper at ICLR 2024
Example:
Text of the child nodes:
"And you will come with me to my people? We may live here among them, and you will be great warrior-oh, when Jor dies you may even be chief, for there is none mighty as my warrior... "But your father will not permit it-Jor, my father, High Chief of the Galus, will not permit it, for like me you are cos-ata-lo. Oh, Co- Tan, we but could!... Bradley noticed that she spoke in English-broke broken English like Co-Tan's 's but equally appealing.
Summary found in the parent of that node:
The protagonist, Bradley, is being asked by Co-Tan to stay with her people and become great warrior, but he refuses and must return to his own country. Tom Billings of Santa Monica arrives and tells them he came to search for a man named Bowen J. Tyler, Jr. Ajor, Co-Tan's 's sister, is excited about the possibility of going to Tom's ' country to see strange and wonderful things...
The hallucination here that the summary states that Jr. Ajor and Co-Tan are sisters, but does not explicitly mention or imply this.
Upon reviewing all parent nodes, we found that hallucinations did not propagate higher layers. Generally, the hallucinations were minor and did not alter the thematic interpretation of the text.
E.3 IMPACT ON QA TASKS
In our findings, hallucinations had no discernible impact on the performance of QA tasks. This sug- gests that hallucination is not major concerns for the summarization component in our RAPTOR architecture.
F PSEUDOCODE FOR RETRIEVAL METHODS
Algorithm Tree Traversal Algorithm
function TRAVERRSETREE(rreS query,
Scurrent <- tree. layer (0)
for layer in range((ree.num_aayr) do
top + []
for node Squrrent do
score + dot product( node)
top X.appendd(nood, score))
end for
Suner f- sorted((oop_k]]
Scurreat Suyer end for
return Sn US, US U...
end function
G QUALITATIVE ANALYSIS
To qualitatively examine RAPTOR's retrieval process, we test it on thematic, multi-hop questions about a 1500-word version of the fairytale Cinderella. We compare the context retrieved by RAP- TOR with the context retrieved by Dense Passage Retrieval (DPR). Figure 4 in the main paper details the retrieval process within RAPTOR's 's tree structure for two questions. The nodes that RAPTOR selects for each question are highlighted, while the leaf nodes that DPR selects for the same question are indicated with arrows. This comparison illustrates the advantage of RAPTOR's tree structure. RAPTOR selects nodes from different layers depending the level of granularity required by the
18

---

Published as a conference paper at ICLR 2024
Algorithm 2 Collapsed Tree Algorithm
function COLLAPSED query, k, max tokens)
tree <- (tree) top_nodes < |
Flatten tree into
for node in tree do
top nodes.append((node dot product(query, node))
end for
top_nodes sorted(top_nodes nodes)
result -
total_tokens
for node in top_nodes do
if total_tokens + node tozensseees < max then
result.append(mode)
end if total_tokens +- total + node.token_size
end for return result end function
Question:
What is the central theme of the story?
RAPTOR
Fairy Godmother helps Cinderella attend a ball by transforming her rags... Cinderella impresses the Prince at the ball... she loses track of time and has to run home alone in the darkness. The Prince is unable to find Cinderella and goes in search of her... She forgave her sisters, and treated them always very kindly, and the Prince had great cause to glad that he had found the glass slipper.
DPR
Two mice were turned into footmen; four grasshoppers into white horses. Next, the Fairy touched Cinderella's 's rags, and they hecame rich satin rohes, trimmed with point lace. Alas! the clock had struck Fleven. She found no coach, only pumpkin, and the rat and mice ran quickly away when they saw her; while all her fine dress turned rags,
to run home alone... They told her a beautiful Princess had been at the ball, with whom the Prince was delighted. They did not know it was Cinderella.
Question: How does Cinderella find happy ending?
RAPTOR
The story of Cinderella involves her godmother, fairy, who transforms a pumpkin into a grand coach with her wand and allows Cindcrella to attend the ball. However, Cindcrella must return home before the clock strikes eleven or her dress will turn back into rags... Cinderella impresses the Prince at the ball but leaves before he find out who she is... The Prince searched for the owner of lost glass slipper and found it belonged Cinderella. She forgave her sisters and the Prince was glad to found her.
DPR
the clock had struck Eleven... The Prince was very much surprised when he missed Cinderella again, and leaving the ball, went in search of her... Fairy touched Cin- derella's 's rags, and they became rich satin robes, trimmed with point lace... Her old shoes became a charming pair of glass slippers, which shone like diamonds. "Now go to the ball, my love," she said, "and enjoy yourself. But remember, you must leave the room before the clock strikes eleven. If you do not your dress will return its original rags."
Table 12: Relevant excerpts from text retrieved by RAPTOR and DPR for the questions on the fairytale Cinderella.
question at hand. Further, the information that would be retrieved by DPR is more often than not included the context retrieved by RAPTOR, either directly as leaf node or indirectly as part of summary from a higher layer.
"The first question we examine is "How does Cinderella find a happy ending?", a multi-hop question best answered by synthesizing information from various text segments. To control for the language model's 's potential familiarity with the Cinderella story, we instructed to rely solely on the retrieved information for its answers. Table 12 shows the text retrieved by both RAPTOR and DPR for this question. RAPTOR's context succinctly describes Cinderella's journey to happiness, while DPR's leaf nodes primarily focus on her initial transformation. The difference in retrieved information
19

---

Published as a conference paper at ICLR 2024
significantly impacts downstream tasks. When GPT-4 is provided with RAPTOR's context, it gen- erates a detailed answer: "Cinderella finds a happy ending when the Prince searches for the owner of the lost glass slipper and discovers it belongs to Cinderella. They eventually marry, transform- ing Cinderella's ' life for the better." In contrast, using DPR's context, GPT-4 states: "Based on the given context, it is not possible to determine how Cinderella finds a happy ending, as the text lacks information about the story's s conclusion."
The second question we examine is "What is the central theme of the story?", a thematic question that requires holistic understanding of the entire text. The text retrieved by RAPTOR and DPR for this question is shown in Table 12. The text retrieved by RAPTOR contains short descriptions of all the major parts of the story, whereas the text retrieved by DPR contains detailed descriptions of a narrow subset of the story. Again, the difference retrieval mechanisms affects the performance of GPT-4 when answering the question. Given DPR's context, it outputs "The central theme of the story is transformation and the power of inner beauty, as Cinderella, kind and humble girl, is magically transformed into beautiful princess, capturing the attention and admiration of the Prince and others at the ball." This answer only takes into account the first portion of the story, up until Cinderella first meets the prince. In contrast, given RAPTOR's 's context, GPT-4 outputs "The central theme of the story is transformation and overcoming adversity, as Cinderella, with the help of her Fairy Godmother, transforms from mistreated and downtrodden girl into beautiful and confident young woman who ultimately finds happiness and love with the Prince." This is a more complete answer, demonstrating a comprehensive understanding of the story.
This qualitative analysis indicates that RAPTOR outperforms prior retrieval mechanisms because the information that it retrieves is more relevant and exhaustive, allowing for better performance on downstream tasks.
We also created 2600-word story along with questions about its narrative and theme. An excerpt from the story is present below and the full PDF of this story is linked here. For questions like "What the central theme of the story?", an upper-level node is retrieved which includes the sentence: "This story is about the power of human connection... inspiring and uplifting each other as they pursued their passions." " This summary, not explicitly present in the original text, almost directly answers the question.
Excerpt from "The Eager Writer"":
"Ethan's passion for writing had always been a part of him. As a child, he would often scribble stories and poems in his notebook, and as he grew older, his love for writing only intensified. His evenings were often spent in the dim light of his room, typing away his laptop. He had recently taken a job as a content writer for online marketing firm to pay the bills, but his heart still longed for the world of storytelling. However, like many aspiring writers, he struggled to find a foothold in the industry. He took job content writer for online marketing firm, but it was growing increasingly evident to him that this was not the path he wanted to pursue. It was during this time that he stumbled upon the Pathways app. The app offered a platform for people in similar professions to connect and share knowledge, and he saw it as opportunity finally connect with others who shared his passion for writing. Ethan saw an opportunity to meet others who shared his passion and could offer guidance and mentorship. He quickly signed up and was surprised by the number of writers he found on the platform, from well establish professionals beginners just starting out in the business."
H NARRATIVEQA EVALUATION SCRIPT
We made several modifications to AllenNLP's evaluation script to better fit our evaluation needs:
Added Smoothing: Smoothing was incorporated to handle cases where BLEU score is zero, due to no n-gram matches occurring in the reference text. A BLEU score of zero skews the results, leading an overly harsh evaluation for rare or novel phrases. By adding
docs all Tennlp.oe patppppooro/dee 8 /mmai n/mon/n httpl////r///oo///ee/eeoo http://////oooss/rrrrtatte
20

---

Published as a conference paper at ICLR 2024
smoothing function, prevent the BLEU scores from dropping to zero, providing more fair evaluation.
Modified BLEU-4 Weighting: The original script applied a weight of the highest order n- D-gram g4-gmm) and 0 to the rest in its BLEU-4 calculation (i.e.. weights=(0,, O, O, 1)). This upproach may overly focus on 4-gram matches while neglecting lower-order matches. To provide more balanced evaluation, we evenly distributed the weight across all n-gram levels, changing the weights for the BLEU-4 calculation to (0.25, 0.25, 0.25, 0.25).
Tokenization before Mapping in METEOR Calculation: The original script utilized a simple split and map method for METEOR calculation. We fixed this by first tokenizing the text and then mapping the tokens. This amendment improves the accuracy of the METEOR calculation by taking into account the correct linguistic boundaries of words.
I ANALYSIS OF DIFFERENT LAYERS ON RAPTOR'S S PERFORMANCE
1.1 HOW DO DIFFERENT LAYERS IMPACT PERFORMANCE ?
In this section, we present a detailed breakdown of RAPTOR's retrieval performance when querying different layers of the hierarchical tree structure for various stories. These tables validate the utility of RAPTOR'S multi-layered structure for diverse query requirements.
NorrativOA
Quality
SEFT
BP3
14
Lagar
Layer
Layer
Figure 7: Histogram showing the percentage of nodes retrieved from different layers of the RAPTOR tree across three datasets (NarrativeQA, Quality, and Qasper) using three retrievers (SBERT, BM25, and DPR). The data indicate that a substantial portion of the nodes contributing the final retrieval comes from non-leaf layers, with notable percentage from the first and second layers, highlighting the importance of RAPTOR's hierarchical summarization in the retrieval process.
Table 13: Performance of RAPTOR when querying different layers of the tree for Story 2.
Layers Queried Start Layer
Layer (Leaf Nodes)
Layer
Layer 2
layer
58.8
41.1 47.1
2 layers
64.7
52,9
3 layers
47.1
Table 14: Performance of RAPTOR when querying different layers of the tree for Story 3.
Layers Queried / Start Layer
Layer (Leaf Nodes)
Layer
Layer
layer
66.6
61.1 66.6
61.1
layers
66.6
layers
83.3
I.2 WHICH LAYERS DO RETRIEVED NODES COME FROM ?
We further conduct an ablation study across all three datasets and across three different retrievers with RAPTOR with the collapsed tree retrieval to examine the layers from which the retrieved nodes
21

---

Published as a conference paper at ICLR 2024
Table 15: Performance of RAPTOR when querying different layers of the for Story 4.
Layers Queried / Start Layer
Layer 0 (Leaf Nodes)
Layer 1
1 layer
94.7
84.2
2 layers
89.4
Table 16: Performance of RAPTOR when querying different layers of the tree for Story 5.
Layers Queried / Start Layer
Layer 0 (Leaf Nodes)
Layer 1
1 layer
57.9
47.3
2 layers
68.4
originate. We observe that between 18.5% 57% of the retrieved nodes come from non-leaf nodes. As illustrated in Figure 7, the retrieval pattern across layers reveals the importance of RAPTOR's multi-layered tree structure. Notably, a significant percentage the nodes retrieved by RAPTOR using the DPR retriever for the NarrativeQA dataset come from the first and second layers of the tree, as opposed to the leaf nodes. This pattern consistent across the other datasets and retrievers, albeit with varying percentages.
Table 17: Percentage of nodes from non-leaf nodes across different datasets and retrievers
Dataset
DPR
SBERT
BM25
NarrativeQA
57.36%
36.78%
34.96%
Quality
32.28%
24.41%
32.36%
Qasper
22.93%
18.49%
22.76%
Table 18: Percentage of nodes from different layers with DPR the retriever
Layer
NarrativeQA
Quality
Qasper
42.64%
67.71%
77.07%
45.00%
29.43%
21.88%
10.57%
2.85%
1.05%
1.78%
0.003%
Table 19: Percentage of nodes from different layers with SBERT as the retriever
Layer
NarrativeQA
Quality
Qasper
0
63.22%
75.59%
81.51%
31.51%
22.78%
17.84%
2
4.85%
1.63%
0.65%
0.42%
Table 20: Percentage of nodes from different layers with BM25 as the retriever
Layer
NarrativeQA
Quality
Qasper
65.04%
77.24% 67.64%
28.79%
28.85%
21.57% 3.51%
2
5.36%
1.19%
0.81% %
22


Reviews:
[{"review_id": "gGHy7AqyAL", "reviewer": "Reviewer_VSBj", "summary": "The authors present an approach to generate recursive summaries for closed-book QA tasks such as NarrativeQA, Qasper, and Quality. The summaries compress multiple documents, which are clustered according to GMM. Then, at test time, the summaries and original passages are retrieved from to perform the task in a RAG-like fashion.", "questions": "Q1: Did you consider using a hierarchical clustering algorithm like agglomerative clustering?\n\nQ2: Why use GMM? In practice, do you assign passages/summaries to more than one cluster?\n\nQ3: Is 4 percent hallucination considered standard? What amount of hallucination have other LLM-based summarization systems experienced?\n\nQ4: Is the example for hallucination in the appendix the full length? I am surprised to see the summary and child node are almost the same size.\n\nQ5: Is the cumulative cosine score used to select the next top-k nodes in beam search approach? If not, then perhaps it makes sense to rename this---the approach does not seem very beam search like.\n\nQ6: When using beam search, will there always be k * d nodes kept? Or is it only d nodes, where you choose the sequence of nodes with the highest cumulative score?\n\nQ7: Why did you need to create your own story? Can you share more details about how the 2600 word story is created?\n\nQ8: Can you provide more details about the difference between collapsed and beam in Figure 3? Does collapsed tend to choose more summaries than beam? Why is top-3 not centered over the 1000 x-ticker? How should we interpret the context length x-axis wrt beam?\n\nPresentation\n\nWhen referencing the appendix, perhaps specify where in the appendix (i.e. which appendix section).\n\nIt is deceptive and not clear to describe RAPTOR as scaling linearly. The actual complexity is closer to O(n * k) where N is the number of tokens and k is the number of layers. It only appears linear because you are using a small k. I suggest to refine the writing to mention it \"scales linearly in practice\" or something along those lines.\n\nCan make it more clear whether using dev or test data. Also, if using dev data to claim state-of-the-art, then perhaps provide some clarification why test is not being used.\n\nRelated Work\n\nSun et al 2021. Do Long-Range Language Models Actually Use Long-Range Context?---This work pre-dates Liu et al and effectively shows transformers do not handle large contexts effectively.\n\nSince your approach is meant to make up for shortcomings of chunked passages, then you may be interested in \"decontextualization\". For example, see: https://arxiv.org/abs/2305.14772\n\nGao et al 2023. Enabling Large Language Models to Generate Text with Citations---This recent work uses summaries in retrieval augmented generation.\n\nMin et al 2021. Joint Passage Ranking for Diverse Multi-Answer Retrieval---This work does tree-based retrieval for open-domain QA.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Strong results compared to multiple baselines, but some choice of baselines are poorly justified and claims of SOTA are not correct.\n\n2. An interesting approach to generate a set of summaries to retrieve from. The nature of the summaries is probably the main value here, especially given that the tree structure of the summaries are mostly ignored. Further, despite other complains for easy scalability, probably this approach will not immediately scale for larger retrieval datasets. Perhaps the focus should be for closed book QA.\n\n3. Results reported on three datasets of varying domains using both closed and open source LMs.", "weaknesses": "1. There is very little analysis of the summarized outputs. The analysis included in the main text, Table 6, is difficult to understand and does not reveal much about the content of the summaries. Based on an in-line example, it seems the benefit of this approach may be the abstractive nature of the summaries, and it would be helpful to verify further whether this is the case or some other property is helping.\n\n2a. The baselines in table 1 and 2 are not well justified. It seems like BM25 and DPR do not use the tree structure, also, SBERT can be applied without tree structure.\n\n2b. There are multiple claims of state of the art that are not accurate. In zeroscrolls, flan-ul2 is reported to get 56.9F1 on Qasper. GPT-4 gets 89.2 on Quality in the same paper.\n\n3. The paper is hard to read at times. Details about beam search are confusing (e.g. do we return all the visited docs, or a single sequence of them?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bsQtzvPDC7", "reviewer": "Reviewer_YAgR", "summary": "In this paper, the authors propose a method called RAPTOR that retrieves chunks and summaries from tree-structured automatically created clusters based on the representation of SBERT. Since the clusters have a hierarchy, RAPTOR can consider high-level and low-level details of texts. To deal with the variable depth of the hierarchy, the cluster size is automatically decided by Gaussian Mixture with Bayesian Information Criterion (BIC) like x-means. The authors summarize each cluster by GPT-3.5-turbo from its elements, chunks, or summaries. When retrieving these chunks and summaries, the authors use two different methods. The first method, beam search, conducts a layer-wise traversal of the clusters beginning at the root nodes. The second method, collapsed tree, considers all nodes in the tree simultaneously. Experimental results show that RAPTOR improves the performance in various QA datasets.", "questions": "- How did you segment texts into chunks? If you use obvious segments in text like paragraphs, please state it clearly.\n- How did you generate summaries by GPT-3.5-turbo? You should show the actually used prompt.\n- What kind of embedding did you use for retrieving chunks and clusters? Did you use the centroids for each cluster or embedding of the summaries? Also, are these calculated by SBERT similar to the clustering step?\n- This is related to the above weakness part. How many retrieved instances are used for baseline models?\n- In the part, \"Comparison to State-of-the-art Systems\", you compared your RAPTOR with GPT-4 to LongT5 that has 3B of parameters and its variant, ColtT5. Considering that the detailed parameter size of GPT-4 is not released and the size of the baselines' parameters, this comparison is not fair. Did you adopt RAPTOR to LongT5 or ColtT5?\n- Similar to the previous question, in Table 5, you should have adopted RAPTOR to the model with almost the same parameter size as that of baselines.\n\nI can update my score based on your response.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The proposed method RAPTOR can indirectly retrieve long texts by tracking tree-structured automatically created clusters.\n- RAPTOR can decide the number of clusters automatically, and thus, it doesn't require manual tuning for creating the tree-structured clusters.\n- RAPTOR can achieve better performances in QA compared with commonly used retrievers.", "weaknesses": "- Considering the information loss by summarization, the benefit of RAPTOR against enumerating possible concatenation of chunks is uncertain.\n- Even if clustering is automatically done, how to segment texts into chunks is still left as a problem.\n- When comparing model performances, the parameter sizes of models should be the same or similar. However, the paper compares models with totally different parameters. This is problematic from the viewpoint of fairness.\n- How many retrieved instances are used for baseline models needs to be described in detail. Thus, judging the fairness of the comparison in the experiment is difficult.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jrUdFpujWB", "reviewer": "Reviewer_PvjZ", "summary": "The paper introduces a new information retrieval mechanism for retrieval augmented generation (RAG). Traditionally, retrieval in RAG is done rather flatly by indexing text-chunks and then retrieving top-k. In the new approach, the authors create a hierarchical encoding. They do this by: \n\n1. Encoding original text-chunks. \n2. Clustering the (encoding,text-chunk) pairs. \n3. Summarizing the clustered text-chunks into a single text-chunk.\n4. Encoding the summary text-chunks to get new (encoding,text-chunk) pairs .\n5. Going to step 2 and repeating the process until the root encoding is created.  \n\nThey also propose two ways to query/retrieve from the hierarchical encoding - one method involving plain top-k retrieval from flattened tree, and another method starts from the root encoding and explores the tree of encodings via beam search. In both cases, information at various levels of granularity can be retrieved.", "questions": "1. It could be good to add a pseudo-code for beam tree-based retrieval and collapsed-tree-based retrieval. \n2. The 20% accuracy boost claim (while technically correct) may be a bit misleading given prior works do not use GPT4 (if I understand correctly). So most of the boost most likely comes from GPT4 rather than RAPTOR.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "1. Overall an elegant, well-motivated, and seemingly novel (to my knowledge) approach. It can set up a new paradigm baseline for RAG that can incite future research on refining the general idea.\n\n2. Decent performance compared to other paradigmatic approaches for retrieval such as DPR.", "weaknesses": "1. One limitation seems to be that the approach requires recursive summarization with an LLM which can add to computational expense (could be also good to share the trade-off).\n\n\n2. While I get the theoretical intuition of clustering (to prevent information loss, by clustering more homogenous content for summarization), it would have been nice to have an empirical demonstration of the effectiveness of clustering. A possible ablation could be: what if we took a balanced tree-style encoding/summarization and simply recursively encoded contiguous chunks (instead of clustering)? \n\n\n3. There is some precedence for using hierarchical retrieval (even though there are crucial differences) [1,2] that could be cited and discussed for better contextualization in the literature.\n\n\n[1] Dense Hierarchical Retrieval for Open-domain Question Answering - Liu et al. EMNLP Findings 2021\n\n[2] Hybrid Hierarchical Retrieval for Open-Domain Question Answering - Arivazhagan et al. ACL Findings 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cqvYolngAJ", "reviewer": "Reviewer_DRXQ", "summary": "In the context of reML, this paper proposes to extend the support documents by generating summaries at different level of granularity (full document to the original text). Summaries are based on a clustered set of chunks of text within a UMAP representation space (which is the novelty in this paper), and the process is recursive. This new set of passages is used as a new dataset, with two strategies for searching: either considering each passage independently, or using a hierarchy to bias the search. The latter strategy is working much worse, and hence the authors focus on the first. The model perform experiments with this strategy, using GPT-3.5/4 to generate the final answer, and comparing to other summarization-based approaches on NarrativeQA, Qasper and QuALITY - and improve the results on the three datasets (especially QuALITY).", "questions": "- Please give examples of generated summaries.\n\n- It would be great to have some insight on the impact of the clustering algorithm, and how the performance degrades with summary quality", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "While the model novelty is low -- instead of using contiguous segments for summarization, the authors propose to segments cluster by similarity -- this simple idea improves results on three different datasets. The different experiments conducted show that the hierarchy is only useful because it produces summaries at different granularity levels (but it does not help in searching for the right passages).", "weaknesses": "It would be good to see the performance using open-domain pre-trained models (e.g. LLAMA or other ones), to ensure a more reproducible research.\n\nThere is no experiment on the impact of the clustering algorithm (what is the difference when not using UMAP? when using more or less neighbors?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)