HippoRAG: Neurobiologically Inspired Long-Term Memory for Large Language Models

Authors: bernal  jiménez gutiérrez
yiheng  shu
yu  gu
michihiro  yasunaga
yu  su
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 20:43:47

Abstract: In order to thrive in hostile and ever-changing natural environments, mammalian brains evolved to store large amounts of knowledge about the world and continually integrate new information while avoiding catastrophic forgetting. Despite the impressive accomplishments, large language models (LLMs), even with retrieval-augmented generation (RAG), still struggle to efficiently and effectively integrate a large amount of new experiences after pre-training. In this work, we introduce HippoRAG, a novel retrieval framework inspired by the hippocampal indexing theory of human long-term memory to enable deeper and more efficient knowledge integration over new experiences. HippoRAG synergistically orchestrates LLMs, knowledge graphs, and the Personalized PageRank algorithm to mimic the different roles of neocortex and hippocampus in human memory. We compare HippoRAG with existing RAG methods on multi-hop question answering (QA) and show that our method outperforms the state-of-the-art methods remarkably, by up to 20%. Single-step retrieval with HippoRAG achieves comparable or better performance than iterative retrieval like IRCoT while being 10-20 times cheaper and 6-13 times faster, and integrating HippoRAG into IRCoT brings further substantial gains. Finally, we show that our method can tackle new types of scenarios that are out of reach of existing methods.

Main paper: 
 introduction: 
 Millions of years of evolution have led mammalian brains to develop the crucial ability to store large amounts of world knowledge and continuously integrate new experiences without losing previous ones. This exceptional long-term memory system eventually allows us humans to keep vast stores of continuously updating knowledge that forms the basis of our reasoning and decision making [19] .Despite the progress of large language models (LLMs) in recent years, such a continuously updating long-term memory is still conspicuously absent from current AI systems. Due in part to its ease of use and the limitations of other techniques such as model editing [46] , retrieval-augmented generation (RAG) has become the de facto solution for long-term memory in LLMs, allowing users to present new knowledge to a static model [36, 42, 66, 87] .However, current RAG methods are still unable to help LLMs perform tasks that require integrating new knowledge across passage boundaries since each new passage is encoded in isolation. Many important real-world tasks, such as scientific literature review, legal case briefing, and medical diagnosis, require knowledge integration across passages or documents. Although less complex, 1 Code and data are available at https://github.com/OSU-NLP-Group/HippoRAG. Tasks that require knowledge integration are particularly challenging for current RAG systems. In the above example, we want to find a Stanford professor that does Alzheimer's research from a pool of passages describing potentially thousands Stanford professors and Alzheimer's researchers. Since current methods encode passages in isolation, they would struggle to identify Prof. Thomas unless a passage mentions both characteristics at once. In contrast, most people familiar with this professor would remember him quickly due to our brain's associative memory capabilities, thought to be driven by the index structure depicted in the C-shaped hippocampus above (in blue). Inspired by this mechanism, HippoRAG allows LLMs to build and leverage a similar graph of associations to tackle knowledge integration tasks.standard multi-hop question answering (QA) also requires integrating information between passages in a retrieval corpus. In order to solve such tasks, current RAG systems resort to using multiple retrieval and LLM generation steps iteratively to join disparate passages [64, 78] . Nevertheless, even perfectly executed multi-step RAG is still oftentimes insufficient to accomplish many scenarios of knowledge integration, as we illustrate in what we call path-finding multi-hop questions in Figure 1 .In contrast, our brains are capable of solving challenging knowledge integration tasks like these with relative ease. The hippocampal memory indexing theory [75] , a well-established theory of human long-term memory, offers one plausible explanation for this remarkable ability. Teyler and Discenna [75] propose that our powerful context-based, continually updating memory relies on interactions between the neocortex, which processes and stores actual memory representations, and the C-shaped hippocampus, which holds the hippocampal index, a set of interconnected indices which point to memory units on the neocortex and stores associations between them [19, 76] .In this work, we propose HippoRAG, a RAG framework that serves as a long-term memory for LLMs by mimicking this model of human memory. Our novel design first models the neocortex's ability to process perceptual input by using an LLM to transform a corpus into a schemaless knowledge graph (KG) as our artificial hippocampal index. Given a new query, HippoRAG identifies the key concepts in the query and runs the Personalized PageRank (PPR) algorithm [30] on the KG, using the query concepts as the seeds, to integrate information across passages for retrieval. PPR enables HippoRAG to explore KG paths and identify relevant subgraphs, essentially performing multi-hop reasoning in a single retrieval step.This capacity for single-step multi-hop retrieval yields strong performance improvements of around 3 and 20 points over current RAG methods [10, 35, 53, 70, 71] on two popular multi-hop QA benchmarks, MuSiQue [77] and 2WikiMultiHopQA [33] . Additionally, HippoRAG's online retrieval process is 10 to 30 times cheaper and 6 to 13 times faster than current iterative retrieval methods like IRCoT [78] , while still achieving comparable performance. Furthermore, our approach can be combined with IRCoT to provide complementary gains of up to 4% and 20% on the same datasets and even obtain improvements on HotpotQA, a less challenging multi-hop QA dataset. Finally, we provide a case study illustrating the limitations of current methods as well as our method's potential on the previously discussed path-finding multi-hop QA setting. (!, ) ! ! " # $ % ! " # $ % ( , !)Figure 2 : Detailed HippoRAG Methodology. We model the three components of human long-term memory to mimic its pattern separation and completion functions. For offline indexing (Middle), we use an LLM to process passages into open KG triples, which are then added to our artificial hippocampal index, while our synthetic parahippocampal regions (PHR) detect synonymy. In the example above, triples involving Professor Thomas are extracted and integrated into the KG. For online retrieval (Bottom), our LLM neocortex extracts named entities from a query while our parahippocampal retrieval encoders link them to our hippocampal index. We then leverage the Personalized PageRank algorithm to enable context-based retrieval and extract Professor Thomas. 4 set of query nodes, just as the hippocampus extracts associated signals from specific partial cues. 2Finally, as is done when the hippocampal signal is sent upstream, we aggregate the output PPR node probability over the previously indexed passages and use that to rank them for retrieval.Luís Miguel Assunção Joaquim (born 5 March 1979 in Vila Franca de Xira, Lisbon), known as Alhandra, is a Portuguese retired footballer who played mainly as a left back -he could also appear as a midfielder.Chirakkalkulam is a small residential area near Kannur town of Kannur District, Kerala state, South India. Chirakkalkulam is located between Thayatheru and Kannur City. Chirakkalkulam's significance arises from the birth of the historic Arakkal Kingdom. 
detailed methodology: 
 Offline Indexing. Our indexing process involves processing a set of passages P using an instructiontuned LLM L and a retrieval encoder M . As seen in Figure 2 we first use L to extract a set of noun phrase nodes N and relation edges E from each passage in P via OpenIE. This process is done via 1-shot prompting of the LLM with the prompts shown in Appendix I. Specifically, we first extract a set of named entities from each passage. We then add the named entities to the OpenIE prompt to extract the final triples, which also contain concepts (noun phrases) beyond named entities. We find that this two-step prompt configuration leads to an appropriate balance between generality and bias towards named entities. Finally, we use M to add the extra set of synonymy relations E ′ discussed above when the cosine similarity between two entity representations in N is above a threshold τ . As stated above, this introduces more edges to our hippocampal index and allows for more effective pattern completion. This indexing process defines a |N | × |P | matrix P, which contains the number of times each noun phrase in the KG appears in each original passage.Online Retrieval. During the retrieval process, we prompt L using a 1-shot prompt to extract a set of named entities from a query q, our previously defined query named entities C q = {c 1 , ..., c n } (Stanford and Alzheimer's in our Figure 2 example ). These named entities C q from the query are then encoded by the same retrieval encoder M . Then, the previously defined query nodes are chosen as the set of nodes in N with the highest cosine similarity to the query named entities C q . More formally, query nodes are defined as R q = {r 1 , ..., r n } such that r i = e k where k = arg max j cosine_similarity(M (c i ), M (e j )), represented as the Stanford logo and the Alzheimer's purple ribbon symbol in Figure 2 .After the query nodes R q are found, we run the PPR algorithm over the hippocampal index, i.e., a KG with |N | nodes and |E| + |E ′ | edges (triple-based and synonymy-based), using a personalized probability distribution #» n defined over N , in which each query node has equal probability and all other nodes have a probability of zero. This allows probability mass to be distributed to nodes that are primarily in the (joint) neighborhood of the query nodes, such as Professor Thomas, and contribute to eventual retrieval. After running the PPR algorithm, we obtain an updated probability distribution #» n ′ over N . Finally, in order to obtain passage scores, we multiply #» n ′ with the previously defined P matrix to obtain #» p , a ranking score for each passage, which we use for retrieval.Node Specificity. We introduce node specificity as a neurobiologically plausible way to further improve retrieval. It is well known that global signals for word importance, like inverse document frequency (IDF), can improve information retrieval. However, in order for our brain to leverage IDF for retrieval, the number of total "passages" encoded would need to be aggregated with all node activations before memory retrieval is complete. While simple for normal computers, this process would require activating connections between an aggregator neuron and all nodes in the hippocampal index every time retrieval occurs, likely introducing prohibitive computational overhead. Given these constraints, we propose node specificity as an alternative IDF signal which requires only local signals and is thus more neurobiologically plausible. We define the node specificity of node i ass i = |P i | -1, where P i is the set of passages in P from which node i was extracted, information that is already available at each node. Node specificity is used in retrieval by multiplying each query node probability #» n with s i before PPR; this allows us to modulate each of their neighborhood's probability as well as their own. We illustrate node specificity in Figure 2 through relative symbol size: the Stanford logo grows larger than the Alzheimer's symbol since it appears in fewer documents.Vila Franca de Xira is a municipality in the Lisbon District in Portugal. The population in 2011 was 136,886, in an area of 318.19 km². Situated on both banks of the Tagus River, 32 km north-east of the Portuguese capital Lisbon, settlement in the area dates back to neolithic times, as evidenced by findings in the Cave of Pedra Furada. Vila Franca de Xira is said to have been founded by French followers of Portugal's first king, Afonso Henriques, around 1200.The Frank T. and Polly Lewis House is located in Lodi, Wisconsin, United States. It was added to the National Register of Historic Places in 2009. The house is located within the Portage Street Historic District. 
datasets: 
 We evaluate our method's retrieval capabilities primarily on two challenging multi-hop QA benchmarks, MuSiQue (answerable) [77] and 2WikiMultiHopQA [33] . For completeness, we also include the HotpotQA [89] dataset even though it has been found to be a much weaker test for multi-hop reasoning due to many spurious signals [77] , as we also show in Appendix B. To limit the experimental cost, we extract 1,000 questions from each validation set as done in previous work [63, 78] . In order to create a more realistic retrieval setting, we follow IRCoT [78] and collect all candidate passages (including supporting and distractor passages) from our selected questions and form a retrieval corpus for each dataset. The details of these datasets are shown in Table 1 . We compare against several strong and widely used retrieval methods: BM25 [69] , Contriever [35] , GTR [53] and ColBERTv2 [70] . Additionally, we compare against two recent LLM-augmented baselines: Propositionizer [10] , which rewrites passages into propositions, and RAPTOR [71] , which constructs summary nodes to ease retrieval from long documents. In addition to the single-step retrieval methods above, we also include the multi-step retrieval method IRCoT [78] as a baseline.We report retrieval and QA performance on the datasets above using recall@2 and recall@5 (R@2 and R@5 below) for retrieval and exact match (EM) and F1 scores for QA performance. HotpotQA Average R@2 R@5 R@2 R@5 R@2 R@5 R@2 R@5 HotpotQA Average R@2 R@5 R@2 R@5 R@2 R@5 R@2 R@5IRCoT + BM25 (Default) By default, we use GPT-3.5-turbo-1106 [55] with temperature of 0 as our LLM L and Contriever [35] or ColBERTv2 [70] as our retriever M . We use 100 examples from MuSiQue's training data to tune HippoRAG's two hyperparameters: the synonymy threshold τ at 0.8 and the PPR damping factor at 0.5, which determines the probability that PPR will restart a random walk from the query nodes instead of continuing to explore the graph. Generally, we find that HippoRAG's performance is rather robust to its hyperparameters. More implementation details can be found in Appendix H.In the U.S., the issuance of birth certificates is a function of the Vital Records Office of the states, capital district, territories and former territories …Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes] Justification: All of the code and data used in this study as well as the necessary documentation to run it has been included in this submission and will be released upon acceptance. Guidelines:• The answer NA means that paper does not include experiments requiring code. Justification: We discuss the local computing resources we utilize in Appendix H and detail the time and costs of using APIs in Appendix G.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 
results: 
 We present our retrieval and QA experimental results below. Given that our method indirectly affects QA performance, we report QA results on our best-performing retrieval backbone ColBERTv2 [70] . However, we report retrieval results for several strong single-step and multi-step retrieval techniques.Single-Step Retrieval Results. As seen in Table 2 , HippoRAG outperforms all other methods, including recent LLM-augmented baselines such as Propositionizer and RAPTOR, on our main datasets, MuSiQue and 2WikiMultiHopQA, while achieving competitive performance on HotpotQA. We notice an impressive improvement of 11 and 20% for R@2 and R@5 on 2WikiMultiHopQA and around 3% on MuSiQue. This difference can be partially explained by 2WikiMultiHopQA's entity-centric design, which is particularly well-suited for HippoRAG. Our lower performance on HotpotQA is mainly due to its lower knowledge integration requirements, as explained in Appendix B, as well as a due to a concept-context tradeoff which we alleviate with an ensembling technique described in Appendix F.2.Step Retrieval Results. For multi-step or iterative retrieval, our experiments in Table 3 demonstrate that IRCoT [78] and HippoRAG are complementary. Using HippoRAG as the retriever for IRCoT continues to bring R@5 improvements of around 4% for MuSiQue, 18% for 2WikiMultiHopQA and an additional 1% on HotpotQA. Question Answering Results. We report QA results for HippoRAG, the strongest retrieval baselines, ColBERTv2 and IRCoT, as well as IRCoT using HippoRAG as a retriever in Table 4 . As expected, improved retrieval performance in both single and multi-step settings leads to strong overall improvements of up to 3%, 17% and 1% F1 scores on MuSiQue, 2WikiMultiHopQA and HotpotQA respectively using the same QA reader. Notably, single-step HippoRAG is on par or outperforms IRCoT while being 10-30 times cheaper and 6-13 times faster during online retrieval (Appendix G). 
what makes hipporag work?: 
 OpenIE Alternatives. To determine if using a closed model like GPT-3.5 is essential to retain our performance improvements, we replace it with an end-to-end OpenIE model REBEL [34] as well as the 8B and 70B instruction-tuned versions of Llama-3.1, a class of strong open-weight LLMs [1] . As shown in Table 5 row 2, building our KG using REBEL results in large performance drops, underscoring the importance of LLM flexibility. Specifically, GPT-3.5 produces twice as many triples as REBEL, indicating its bias against producing triples with general concepts and leaving many useful associations behind.In terms of open-weight LLMs, Table 5 (rows 3-4) shows that the performance of Llama-3.1-8B is competitive with GPT-3.5 in all datasets except for 2Wiki, where performance drops substantially. Nevertheless, the stronger 70B counterpart outperforms GPT-3.5 in two out of three datasets and is still competitive in 2Wiki. The strong performance of Llama-3.1-70B and the comparable performance of even the 8B model is encouraging since it offers a cheaper alternative for indexing over large corpora. The graph statistics for these OpenIE alternatives can be found in Appendix C.To understand the relationship between OpenIE and retrieval performance more deeply, we extract 239 gold triples from 20 examples from the MuSiQue training set. We then perform a small-scale intrinsic evaluation using the CaRB [6] framework for OpenIE. We find that both Llama-3.1-Instruct Table 5 : Dissecting HippoRAG. To understand what makes it work well, we replace its OpenIE module and PPR with plausible alternatives and ablate node specificity and synonymy-based edges.HotpotQA Average R@2 R@5 R@2 R@5 R@2 R@5 R@2 R@5 HippoRAG 40.9 51.9 70. 7 models underperform GPT-3.5 slightly on this intrinsic evaluation but all LLMs vastly outperform REBEL. More details about this evaluation experiments can be found in Appendix D.PPR Alternatives. As shown in Table 5 (rows 5-6), to examine how much of our results are due to the strength of PPR, we replace the PPR output with the query node probability #» n multiplied by node specificity values (row 5) and a version of this that also distributes a small amount of probability to the direct neighbors of each query node (row 6). First, we find that PPR is a much more effective method for including associations for retrieval on all three datasets compared to both simple baselines. It is interesting to note that adding the neighborhood of R q nodes without PPR leads to worse performance than only using the query nodes themselves.Ablations. As seen in Table 5 (rows 7-8), node specificity obtains considerable improvements on MuSiQue and HotpotQA and yields almost no change in 2WikiMultiHopQA. This is likely because 2WikiMultiHopQA relies on named entities with little differences in terms of term weighting. In contrast, synonymy edges have the largest effect on 2WikiMultiHopQA, suggesting that noisy entity standardization is useful when most relevant concepts are named entities, and improvements to synonymy detection could lead to stronger performance in other datasets.A major advantage of HippoRAG over conventional RAG methods in multi-hop QA is its ability to perform multi-hop retrieval in a single step. We demonstrate this by measuring the percentage of queries where all the supporting passages are retrieved successfully, a feat that can only be accomplished through successful multi-hop reasoning. Table 6 below shows that the gap between our method and ColBERTv2, using the top-5 passages, increases even more from 3% to 6% on MuSiQue and from 20% to 38% on 2WikiMultiHopQA, suggesting that large improvements come from obtaining all supporting documents rather than achieving partially retrieval on more questions.Table 6 : All-Recall metric. We measure the percentage of queries for which all supporting passages are successfully retrieved (all-recall, denoted as AR@2 or AR@5) and find even larger performance improvements for HippoRAG.HotpotQA Average AR@2 AR@5 AR@2 AR@5 AR@2 AR@5 AR@2 AR@5ColBERTv2 We further illustrate HippoRAG's unique single-step multi-hop retrieval ability through the first example in Table 7 . In this example, even though Alhandra was not mentioned in Vila de Xira's passage, HippoRAG can directly leverage Vila de Xira's connection to Alhandra as his place of birth to determine its importance, something that standard RAG methods would be unable to do directly. Additionally, even though IRCoT can also solve this multi-hop retrieval problem, as shown in Appendix G, it is 10-30 times more expensive and 6-13 times slower than ours in terms of online retrieval, arguably the most important factor when it comes to serving end users.Table 7 : Multi-hop question types. We show example results for different approaches on path-finding vs. path-following multi-hop questions.In which district was Alhandra born?The second example in Table 7 , also present in Figure 1 , shows a type of questions that is trivial for informed humans but out of reach for current retrievers without further training. This type of questions, which we call path-finding multi-hop questions, requires identifying one path between a set of entities when many paths exist to explore instead of following a specific path, as in standard multi-hop questions. 5More specifically, a simple iterative process can retrieve the appropriate passages for the first question by following the one path set by Alhandra's one place of birth, as seen by IRCoT's perfect performance. However, an iterative process would struggle to answer the second question given the many possible paths to explore-either through professors at Stanford University or professors working on the neuroscience of Alzheimer's. It is only by associating disparate information about Thomas Südhof that someone who knows about this professor would be able to answer this question easily. As seen in Table 7 , both ColBERTv2 and IRCoT fail to extract the necessary passages since they cannot access these associations. On the other hand, HippoRAG leverages its web of associations in its hippocampal index and graph search algorithm to determine that Professor Thomas is relevant to this query and retrieves his passages appropriately. More examples of these path-finding multi-hop questions can be found in our case study in Appendix E.6 Related Work [58, 82, 32] , no methodology has yet to emerge as a robust solution for continual learning in LLMs [26, 46, 97] .RAG as Long-Term Memory. On the other hand, using RAG methods as a long-term memory system offers a simple way to update knowledge over time [36, 42, 66, 73] . More sophisticated RAG methods, which perform multiple steps of retrieval and generation from an LLM, are even able to integrate information across new or updated knowledge elements [38, 64, 72, 78, 88, 90, 92] , another crucial aspect of long-term memory systems. As discussed above, however, this type of online information integration is unable to solve the more complex knowledge integration tasks that we illustrate with our path-finding multi-hop QA examples.Some other methods, such as RAPTOR [71] , MemWalker [9] and GraphRAG [18] , integrate information during the offline indexing phase similarly to HippoRAG and might be able to handle these more complex tasks. However, these methods integrate information by summarizing knowledge elements, which means that the summarization process must be repeated any time new data is added. In contrast, HippoRAG can continuously integrate new knowledge by simply adding edges to its KG.Long Context as Long-Term Memory. Context lengths for both open and closed source LLMs have increased dramatically in the past year [11, 17, 22, 61, 68] . This scaling trend seems to indicate that future LLMs could perform long-term memory storage within massive context windows. However, the viability of this future remains largely uncertain given the many engineering hurdles involved and the apparent limitations of long-context LLMs, even within current context lengths [41, 45, 96, 21] . 
multi-hop qa & graphs: 
 Many previous works have also tackled multi-hop QA using graph structures. These efforts can be broadly divided in two major categories: 1) graph-augmented reading comprehension, where a graph is extracted from retrieved documents and used to improve a model's reasoning process and 2) graph-augmented retrieval, where models find relevant documents by traversing a graph structure.Graph-Augmented Reading Comprehension. Earlier works in this category are mainly supervised methods which mix signal from a hyperlink or co-occurrence graph with a language model through a graph neural network (GNN) [20, 67, 65] . More recent works use LLMs and introduce knowledge graph triples directly into the LLM prompt [57, 43, 47] . Although these works share HippoRAG's use of graphs for multi-hop QA, their generation-based improvements are fully complementary to HippoRAG's, which are solely based on improved retrieval.Graph-Augmented Retrieval. In this second category, previous work trains a re-ranking module which can traverse a graph made using Wikipedia hyperlinks [16, 100, 54, 14, 4, 44] . HippoRAG, in contrast, builds a KG from scratch using LLMs and performs multi-hop retrieval without any supervision, making it much more adaptable.Combining the strengths of language models and knowledge graphs has been an active research direction for many years, both for augmenting LLMs with a KG in different ways [48, 80, 84] or augmenting KGs by either distilling knowledge from an LLM's parametric knowledge [7, 85] or using them to parse text directly [8, 29, 94] . In an exceptionally comprehensive survey, Pan et al. [56] present a roadmap for this research direction and highlight the importance of work which synergizes these two important technologies [37, 74, 27, 91, 99] . Like these works, HippoRAG shows the potential for synergy between these two technologies, combining the knowledge graph construction abilities of LLMs with the retrieval advantages of structured knowledge for more effective RAG. 
conclusions & limitations: 
 Our proposed neurobiologically principled methodology, although simple, already shows promise for overcoming the inherent limitations of standard RAG systems while retaining their advantages over parametric memory. HippoRAG's knowledge integration capabilities, demonstrated by its strong results on path-following multi-hop QA and promise on path-finding multi-hop QA, as well as its dramatic efficiency improvements and continuously updating nature, makes it a powerful middleground framework between standard RAG methods and parametric memory and offers a compelling solution for long-term memory in LLMs.Nevertheless, several limitations can be addressed in future work to enable HippoRAG to achieve this goal better. First, we note that all components of HippoRAG are currently used off-the-shelf without any extra training. There is therefore much room to improve our method's practical viability by performing specific component fine-tuning. This is evident in the error analysis discussed in Appendix F, which shows most errors made by our system are due to NER and OpenIE and thus could benefit from direct fine-tuning. Given that the rest of the errors are graph search errors, also in Appendix F, we note that several avenues for improvements over simple PPR exist, such as allowing relations to guide graph traversal directly. Additionally, as shown in Appendix F.4, more work must be done to improve the consistency of OpenIE in longer compared to shorter documents. Finally, and perhaps most importantly, HippoRAG's scalability still calls for further validation. Although we show that Llama-3.1 could obtain similar performance to closed-source models and thus reduce costs considerably, we are yet to empirically prove the efficiency and efficacy of our synthetic hippocampal index as its size grows way beyond current benchmarks.Question In which district was Alhandra born? Answer Lisbon 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: We do not anticipate our work to have any meaningful positive or negative societal impacts.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: We release no models and the data we release is either already publicly available or purely the output of an LLM doing OpenIE on such data. We believe that this paper poses no such risks.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes] Justification: We credit the owners of all code, models and data used in this work. Much of this information can be found in §3.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL.• Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper. • We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution. • For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review. 
 
 Appendix: Distractor Passages (Excerpts) To better demonstrate how our HippoRAG pipeline works, we use the path-following example from the MuSiQue dataset shown in Table 7 . We use HippoRAG's indexing and retrieval processes to follow this question and a subset of the associated corpus. The question, its answer, and its supporting and distractor passages are as shown in Figure 3 . The indexing stage is shown in Figure 4 , showing both the OpenIE procedure as well as the relevant subgraph of our KG. Finally, we illustrate the retrieval stage in Figure 5 , including query NER, query node retrieval, how the PPR algorithm changes node probabilities, and how the top retrieval results are calculated.To analyze the differences between the three datasets we use, we pay special attention to the quality of the distractor passages, i.e., whether they can be effectively confounded with the supporting passages. We use Contriever [35] to calculate the match score between questions and candidate passages and show their densities in Figure 6 . In an ideal case, the distribution of distractor scores should be close to the mean of the support passage scores. However, it can be seen that the distribution of the distractor scores in HotpotQA is much closer to the lower bound of the support passage scores compared to the other two datasets. The similarity score of HotpotQA distractors is not substantially larger than that of the least similar supporting passages, meaning that these distractors are not very effective.We use GPT-3.5 Turbo, REBEL [34] and Llama-3.1 (8B and 70B) [1] for OpenIE ablation experiments.As shown in Table 8 , compared to both GPT-3.5 Turbo and both Llama models, REBEL generates around half the number of nodes and edges. This illustrates REBEL's lack of flexibility in open information extraction when compared to using both open and closed-source LLMs. Meanwhile, both Llama-3.1 versions produce a similar amount of OpenIE triples than GPT-3.5 Turbo. In order to better understand how OpenIE and retrieval interact, we extracted gold triples from 20 documents from the MuSiQue training dataset. In total, we extracted 239 gold triples. From the results in Table 9 , we first note that there is a massive difference between end-to-end information extraction systems like REBEL and LLMs. Additionally, we note that there is some correlation better OpenIE and retrieval performance, given that the 8B Llama-3.1-Instruct version performs worse that its 70B counterpart in both retrieval and intrinsic metrics. More specifically, we see that this larger model only provides intrinsic improvements in the recall metric, which seems specially important in improving retrieval performance. Finally, we note that this evaluation is not perfectly correlated with retrieval performance, since GPT-3.5's intrinsic performance is much stronger than Llama-3.1-70B-Instruct while its retrieval score is only slightly higher.Table 9 : Intrinsic OpenIE evaluation using the CaRB [6] framework on 20 annotated passages.GPT-3.5 Turbo (1106) [ E Case Study on Path-Finding Multi-Hop QAAs discussed above, path-finding multi-hop questions across passages are exceedingly challenging for single-step and multi-step RAG methods such as ColBERTv2 and IRCoT. These questions require integrating information across multiple passages to find relevant entities among many possible candidates, such as finding all Stanford professors who work on the neuroscience of Alzheimer's.These questions and the curated corpora around them were built through the following procedure. The first two questions follow a slightly separate process as the third one as well as the motivating example in the main paper. For the first two, we first identify a book or movie and then found the book's author or the movie's director. We would then find 1) a trait for either the book/movie and 2) another trait for the author/director. These two traits would then be used to extract distractors from Wikipedia for each question.For the third question and our motivating example, we first choose a professor or a drug at random as the answer for each question. We then obtain the university the professor works at or the disease the drug treats as well as one other trait for the professor or drug (in these questions research topic and mechanism of action were chosen). In these questions, distractors were extracted from Wikipedia using the University or disease on the one hand and the research topic or mechanism of action on the other. This process, although quite tedious, allowed us to curate these challenging but realistic path-finding multi-hop questions.In Table 10 , we show three more examples from three different domains that illustrate HippoRAG's potential for solving retrieval tasks that require such cross-passage knowledge integration.In the first question of Table 10 , we want to find a book published in 2012 by an English author who won a specific award. In contrast to HippoRAG, ColBERTv2 and IRCoT are unable to identify Mark Haddon as such an author. ColBERTv2 focuses on passages related to awards while IRCoT mistakenly decides that Kate Atkinson is the answer to such question since she won the same award for a book published in 1995. For the second question, we wanted to find a war film based on a non-fiction book directed by someone famous for sci-fi and crime movies. HippoRAG is able to find our answer Black Hawk Down by Ridley Scott within the first four passages, while ColBERTv2 misses the answer completely and retrieves other films and film collections. In this instance, even though IRCoT is able to retrieve Ridley Scott, it does so mainly through parametric knowledge. The chain-of-thought output discusses his and Denis Villeneuve fame as well as their sci-fi and crime experience. Given the three-step iteration restriction used here and the need to explore two directors, the specific war film Black Hawk Down was not identified. Although a bit convoluted, people often ask these first two questions to remember a specific movie or book they watched or heard about from only a handful of disjointed details.Finally, the third question is more similar to the motivating example in the main paper and shows the importance of this type of question in real-world domains. In this question, we ask for a drug used to treat lymphocytic leukemia through a specific mechanism (cytosolic p53 interaction). While HippoRAG is able to leverage the associations within the supporting passages to identify the Chlorambucil passage as the most important, ColBERTv2 and IRCoT are only able to extract passages associated with lymphocytic leukemia. Interestingly enough, IRCoT uses its parametric knowledge to guess that Venetoclax, which also treats leukemia, would do so through the relevant mechanism even though no passage in the curated dataset explicitly stated this. In this section, we provide a detailed error analysis of 100 errors made by HippoRAG on the MuSiQue dataset. As shown in Table 11 , these errors can be categorized into three main types: NER, OpenIE and PPR.The main error type, with nearly half of all error examples, is due to limitations of our NER based design. As further discussed in §F.2, our NER design does not extract enough information from the query for retrieval. For example, in the question "When was one internet browser's version of Windows 8 made accessible?", only the phrase "Windows 8" is extracted, leaving any signal about "browsers" or "accessibility" behind for the subsequent graph search. OpenIE errors, the second most common, are discussed in more detail in §F.3.We define the third error category as cases where both NER and OpenIE are functioning properly but the PPR algorithm is still unable to identify relevant subgraphs, often due to confounding signals.For instance, consider the query "How many refugees emigrated to the European country where Huguenots felt a kinship for emigration?". Despite the term "Huguenots" being accurately extracted from both the question and the supporting passages, and the PPR algorithm initiating with the nodes labeled "European" and "Huguenots", the PPR algorithm struggles to find the appropriate subgraphs around them that define the most related passage. This occurs when multiple passages exist in the corpus that discuss very similar topics since the PPR algorithm is not able to leverage query context directly. Given our method's entity-centric nature in extraction and indexing, it has a strong bias towards concepts that leaves many contextual signals unused. This design enables single-step multi-hop retrieval while also enabling contextual cues to avoid distracting from more salient entities. As seen in the first example in Table 12 , ColBERTv2 uses the context to retrieve passages that are related to famous Spanish navigators but not "Sergio Villanueva", who is a boxer. In contrast, HippoRAG is able to hone in on "Sergio" and retrieve one relevant passage.Unfortunately, this design is also one of our method's greatest limitations since ignoring contextual cues accounts for around 48% of errors in our small-scale error analysis. This problem is more apparent in the second example since the concepts are general, making the context more important. Since the only concept tagged by HippoRAG is "protons", it extracts passages related to "Uranium" and "nuclear weapons" while ColBERTv2 uses the context to extract more relevant passages associated with the discovery of atomic numbers. Table 13 : Single-step retrieval performance. HippoRAG performs substantially better on MuSiQue and 2WikiMultiHopQA than all baselines and achieves comparable performance on the less challenging HotpotQA dataset. To get a better trade-off between concepts and context, we introduce an ensembling setting where HippoRAG scores are ensembled with dense retrievers when our parahippocampal region shows uncertainty regarding the link between query and KG entities. This process represents instances when no hippocampal index was fully activated by the upstream parahippocampal signal and thus the neocortex must be relied on more strongly. We only use uncertainty ensembling if one of the query-KG entity scores cosine_similarity(M (c i ), M (e j )) is lower than a threshold θ, for example, if there was no Stanford node in the KG and the closest node in the KG is something that has a cosine similarity lower than θ such as Stanford Medical Center. The final passage score for uncertainty Model Retriever MuSiQue 2Wiki HotpotQA Average R@2 R@5 R@2 R@5 R@2 R@5 R@2 R@One of HippoRAG's main advantages against iterative retrieval methods is the dramatic online retrieval efficiency gains brought on by its single-step multi-hop retrieval ability in terms of both cost and time. Specifically, as seen in Table 17 , retrieval costs for IRCoT are 10 to 30 times higher than HippoRAG since it only requires extracting relevant named entities from the query instead of processing all of the retrieved documents. In systems with extremely high usage, a cost difference of an order of magnitude such as this one could be extremely important. The difference with IRCoT in terms of latency is also substantial, although more challenging to measure exactly. Also as seen in Table 17 , HippoRAG can be 6 to 13 times faster than IRCoT, depending on the number of retrieval rounds that need to be executed (2-4 in our experiments). 6 Table 17 : Average cost and efficiency measurements for online retrieval using GPT-3.5 Turbo on 1,000 queries.Apart from the details included in §3.4, we use implementations based on PyTorch [59] and Hug-gingFace [86] for both Contriever [35] and ColBERTv2 [70] . We use the python-igraph [13] implementation of the PPR algorithm. For BM25, we employ Elastic Search [24] . For multi-step retrieval, we use the same prompt implementation as IRCoT [78] and retrieve the top-10 passages at each step. We set the maximum number of reasoning steps to 2 for HotpotQA and 2WikiMultiHopQA and 4 for MuSiQue due to their maximum reasoning chain length. We combine IRCoT with different retrievers by replacing its base retriever BM25 with each retrieval method, including HippoRAG, noted as "IRCoT + HippoRAG" below. 8 For the QA reader, we use top-5 retrieved passages as the context and 1-shot QA demonstration with CoT prompting strategy [78] .In terms of compute requirements, most of our compute requirements are unfortunately not disclosed by the OpenAI. We run ColBERTv2 and Contriever for indexing and retrieval we use 4 NVIDIA RTX A6000 GPUs with 48GB of memory. For indexing with Llama-3.1 models, we use 4 NVIDIA H100 GPUs with 80GB of memory. Finally, we used 2 AMD EPYC 7513 32-Core Processors to run the Personalized PageRank algorithm.The prompts we used for indexing and query NER are shown in Figure 7 and Figure 8 , while the OpenIE prompt is shown in Figure 9 .Your task is to extract named entities from the given paragraph.Respond with a JSON list of entities.Paragraph: ``R adio City Radio City is India's first private FM radio station and was started on 3 July 2001. It plays Hindi, English and regional songs. Radio City recently forayed into New Media in May 2008 with the launch of a music portal -PlanetRadiocity.com that offers music related news, videos, songs, and other music-related features. ``{ "named_entities": ["Radio City", "India", "3 July 2001", "Hindi","English", "May 2008", "PlanetRadiocity.com"]}Paragraph: ``P ASSAGE TO INDEX ```P assage NER (Indexing) Your task is to construct an RDF (Resource Description Framework) graph from the given passages and named entity lists. Respond with a JSON list of triples, with each triple representing a relationship in the RDF graph. Pay attention to the following requirements: -Each triple should contain at least one, but preferably two, of the named entities in the list for each passage.-Clearly resolve pronouns to their specific names to maintain clarity.Convert the paragraph into a JSON dict, it has a named entity list and a triple list.Paragraph: ``R adio City Radio City is India's first private FM radio station and was started on 3 July 2001. It plays Hindi, English and regional songs. Radio City recently forayed into New Media in May 2008 with the launch of a music portal -PlanetRadiocity.com that offers music related news, videos, songs, and other music-related features. ``{ "named_entities": ["Radio City", "India", "3 July 2001", "Hindi","English", "May 2008", "PlanetRadiocity.com"]} NeurIPS Paper Checklist Guidelines:• The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.Justification: We reviewed the NeurIPS Code of Ethics and made sure that our paper conforms to it in every respect.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Intriguingly, some work in cognitive science has also found a correlation between human word recall and the output of the PageRank algorithm[25].4 Many details around the hippocampal memory indexing theory are omitted from this study for simplicity. We encourage interested reader to follow the references in §2.1 for more.Lisa GiocomoPath-finding questions require knowledge integration when search entities like Stanford and Alzheimer's do not happen to appear together in a passage, a condition which is often satisfied for new information.Since the original IRCoT does not provide a score for each retrieved passage, we employ beam search for the iterative retrieval process. Each candidate passage maintains the highest historical score during beam search. 


Decision: This paper presents HippoRAG, a new "retrieval framework inspired by the hippocampal indexing theory of human long-term memory to enable deeper and more efficient knowledge integration over new experiences". HippoRAG utilises LLMs, knowledge graphs, and PageRank to mimic the different roles of neocortex and hippocampus in human memory. The authors experimentally demonstrate the effectiveness of their approach on multi-hop question answering and show substantial improvements over the state of the art. The authors have addressed the reviewers' concerns in the rebuttal and have included additional experiments as needed. The paper is well-written,  technically solid, with thorough and sound evaluation experiments. The idea is new and will serve as a solid foundation towards long-term memory solutions in LLMs.

Reviews: review: paper_summary: paper_summary: The paper introduces HippoRAG, a retrieval framework inspired by hippocampal indexing theory to enhance large language models (LLMs) in integrating new information. The algorithm is a combination of LLMs, knowledge graphs (KGs), and the Personalized PageRank algorithm. HippoRAG outperforms existing retrieval-augmented generation (RAG) methods in multi-hop question answering (QA) by a significant margin. It achieves comparable or better performance than iterative retrieval methods like IRCoT while being significantly cheaper and faster.
strength_weakness: strength_weakness: - Some baselines from KG-LLM for multi-hop QA literature are missing. These could be beneficial to replace Page Rank for ablation study or comparing with KGQA on open-source knowledge graph.

- Some citation are missing (see questions)
questions: questions: - Can we evaluate the performance of constructed knowledge graph alone?
    
- Some citations are missing:
    
    - Park et al. Graph Elicitation for Guiding Multi-Step Reasoning in Large Language Models
        
    - Jin et al. Improving embedded knowledge graph multi-hop question answering by introducing relational chain reasoning
limitations: limitations: Yes
paper_summary: paper_summary: This paper presents Hippo-RAG, which enables knowledge integration across retrieval results and supports long-term memory with a mechanism that resembles the hippocampal memory indexing theory. Hippo-RAG includes two steps: offline indexing to extract, encode, and index the passages to KG, and online retrieval to extract entities from the query and retrieve the results from the KG. Hippo-RAG outperforms various existing methods on the multi-hop QA benchmarks and is able to perform multi-hop retrieval in a single step.
strength_weakness: strength_weakness: 1. The mechanism is great, but the generalization of the method is not good enough. Intuitively, the mentioned mechanism should work for several scenarios that require long-term memory. Still, the proposed method narrows it down and limits the experiments to multi-hop reasoning. 
2. The framework's performance depends on the NER ability of LLM. However, [1] demonstrates a large performance gap between prompting LLMs for NER and fine-tuned NER models. The error analysis also indicates that most errors in the system are from NER.
3. The comparison of baselines may not be fair since the parameter sizes of Hippo-RAG are significantly larger than other methods. Contriever and ColBERTv2 achieve comparable results on MuSiQue and HotpotQA, but when combined with HippoRAG, the improvement is insignificant. ColBERTv2 is the second-best model, and the improvement from Hippo-RAG seems incremental on the two datasets.



[1] https://arxiv.org/abs/2304.10428
questions: questions: 1. Can Hippo-RAG work for other tasks that require long-term memory?
2. Why does Hippo-RAG perform significantly better on 2Wiki?
limitations: limitations: The authors adequately discussed the limitations and potential negative societal impact of their work.
paper_summary: paper_summary: This paper proposes a retrieval-augmented generation (RAG) method that is inspired by the hippocampal indexing theory of human memory to enable longer knowledge storage and efficient knowledge integration over new experiences.
strength_weakness: strength_weakness: 1. Extracting triplets from the passage strongly depends on the power of the triplets extracting model, which is a language model in the paper's implementation. I'm concerned that it may lose information when the passage becomes longer. Is this a common strategy in other retrieval methods? 
2. Missing citations. Since the paper talks about long-term memory in related work, I believe the following papers may need to be cited.  
[1] Memoria: Resolving Fateful Forgetting Problem through Human-Inspired Memory Architecture.   
[2] MEMORYLLM: Towards Self-Updatable Large Language Models.   
[3] CAMELoT: Towards Large Language Models with Training-Free Consolidated Associative Memory.
questions: questions: see weaknesses
limitations: limitations: yes, the authors have addressed the limitations.

Comments: title: Response to the rebuttal
Thank you!
Rebuttal by Authors (Continued)
Rebuttal by Authors (Continued)
comment: Thank the authors for the responses. I don't have more concerns.
Thank you again for your helpful suggestions and for considering our response.
Thank you for considering our responses and kindly updating your score. Please feel free to let us know if you have any further questions or concerns.

We appreciate your suggestion to include more details about the LLM-augmented baselines. In order to ensure our experiments appropriately represent these baselines, we used the embedding methods which obtain the strongest performance in their respective experiments. Both of these were Sentence Transformer models, `sentence-transformers/multi-qa-mpnet-base-cos-v1` for RAPTOR and GTR (`sentence-transformers/gtr-t5-base`) for the Proposition-izer.

In order to provide a direct comparison with ColBERTv2 as the reviewer suggests, we ran these two baselines using ColBERTv2 as their retrieval component.

| Method                    | MuSiQue  | MuSiQue  | 2Wiki   | 2Wiki   | HotpotQA | HotpotQA |
|---------------------------|----------|----------|---------|---------|----------|----------|
|                           | R@2      | R@5      | R@2     | R@5     | R@2      | R@5      |
| ColBERTv2                 | 37.9     | 49.2     | 59.2    | 68.2    | **64.7**     | **79.3**     |
| RAPTOR                    | 35.7     | 45.3     | 46.3    | 53.8    | 58.1     | 71.2     |
| RAPTOR (ColBERTv2)        | 36.9     | 46.5     | 57.3    | 64.7    | 63.1     | 75.6     |
| Proposition               | 37.6     | 49.3     | 56.4    | 63.1    | 58.7     | 71.1     |
| Proposition (ColBERTv2)   | 37.8     | 50.1     | 55.9    | 64.9    | 63.9     | 78.1     |
| HippoRAG (ColBERTv2)      | **40.9** | **51.9** | **70.7**| **89.1**| 60.5 | 77.7 |

We find that both baselines obtain stronger performance using ColBERTv2 than their original models. However, they still underperform ColBERTv2 itself in all results except for MuSiQue (R@5), where only the Proposition-izer model outperforms ColBERTv2 and gets 0.8% closer to HippoRAG. 

We point out that RAPTOR's poor performance compared to ColBERTv2 demonstrates that their "cluster and summarize" methodology is mostly ineffective for the comprehensive knowledge integration required in these datasets. Finally, we note that the Proposition-izer's  unstable performance, especially its poor performance in 2Wiki, illustrates that separating passages based on propositions can sharply diminish a retriever's knowledge integration capabilities.

We will include these results and discussion in the revised version.
Thank you for the analyses provided. I raise the rating from 4 to 5 for they address most of my concerns.

My remaining concerns are about W3 experimental comparison. 
The settings for RAPTOR are not stated in the paper (which retriever (SBERT/DPR/BM25) is used for its implementation). In Table 2, ColBERTv2 outperforms the two LLM-augmented methods, and it is not discussed in the paper. These are my concerns, but they are not the weakness of the paper. I would appreciate it if the authors could include them in later versions.
Thank the authors for their detailed rebuttal. My concerns are addressed adequately and I will keep my score.
### References

[1] Zhou et al. (2024). UniversalNER: Targeted Distillation from Large Language Models for Open Named Entity Recognition.\
[2] Trivedi et al. (2023). Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions.\
[3] Press et al. (2023). Measuring and Narrowing the Compositionality Gap in Language Models.\
[4] Shao et al. (2023). Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy.\
[5] Yoran et al. (2023). Answering Questions by Meta-Reasoning over Multiple Chains of Thought.\
[6] Trivedi et al. (2022). ♫ MuSiQue: Multihop Questions via Single-hop Question Composition.\
[7] Sarthi et al. (2024). RAPTOR: Recursive Abstractive Processing for Tree-Organized Retrieval.\
[8] Karpukhin et al. (2020). Dense Passage Retrieval for Open-Domain Question Answering.
- **W3: The comparison of baselines may not be fair since the parameter sizes of Hippo-RAG are significantly larger than other methods. Contriever and ColBERTv2 achieve comparable results on MuSiQue and HotpotQA, but when combined with HippoRAG, the improvement is insignificant. ColBERTv2 is the second-best model, and the improvement from Hippo-RAG seems incremental on the two datasets.**

In the response below, we present evidence and arguments that we hope will convince the reviewer that our performance improvements are substantial and our experimental setting is sound and fair.

#### **Improvements on MuSiQue**

We first highlight that our method “achieves significant performance improvement from various challenging multi-hop QA benchmarks”, as pointed out by the reviewer when listing our strengths.
HippoRAG improves R@5 on MuSiQue by 5.5% and 2.7% over Contriever and ColBERTv2 respectively, as well as 3.4% F1 score on QA compared to ColBERTv2. Many well-received and highly cited works such as IRCoT [2], Self-Ask [3], ITER-RETGEN [4] and MCR [5] highlight improvements similar to ours.

#### **HotpotQA’s Weaknesses as a Knowledge-Integration Benchmark**

As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, our lower performance on HotpotQA is mainly due to its lower need for knowledge integration due to existing shortcut signals. This HotpotQA limitation is referenced when constructing MuSiQue [6], but also explored more deeply in Appendix B of our paper. In bridge multi-hop questions, the query and second supporting passage should be linked only through a bridge entity. However, as shown in Figure 6, HotpotQA queries are on average as similar to the second supporting passage as to their distractors, instead of less similar like in the other datasets, making that second supporting document easier to detect without knowledge integration.

#### **Measuring the Impact of Improved Knowledge-Integration Directly**

Additionally, when we measure the impact of improved knowledge integration in MuSiQue and 2WikiMultiHopQA, our improvements are even larger than our overall results. In Table 8, we show that HippoRAG’s ability to find all supporting documents is key to its strong performance, obtaining a 6.3% and 38.6% improvement over ColBERTv2 in All-Recall@5, compared to a 2.7% and 21.3% improvement in standard Recall@5.

#### **Our Experimental Setting is Sound and Fair**

Finally, we refer to the reviewer’s concern that our comparison with Contriever and ColBERTv2 might not be fair given their small size compared to LLM. We understand the reviewer’s apprehension, however, we argue that improving the performance of a system using an LLM and comparing it to the original system is a well-accepted paradigm in AI research. As a specific example, many well-received QA works such as IRCoT [2], Self-Ask [3] and RAPTOR [7] leverage a large LLM in different ways and compare with the original pipeline; which in these papers contain various retrieval methods like BM25, DPR [8] and ColBERTv2. We note that HippoRAG follows this same research paradigm, which is related to the large and growing body of recent work that leverages LLMs to challenge prior state of the art based on smaller models.
As a final remark, we also note that our method outperforms RAPTOR and is comparable with IRCoT (and considerably more efficient). Since both of these methods augment the original retrieval process with outputs from an LLM, they can be seen as HippoRAG’s most direct baselines in that sense.

- **Q2: Why does Hippo-RAG perform significantly better on 2Wiki?**

As discussed in subsection “Single-Step Retrieval Results” (Section 4) of our paper, 2WikiMultiHopQA’s construction is more entity-centric than the other two datasets, making it particularly well-suited for HippoRAG’s design. 

To provide some extra context, 2WikiMultiHopQA was created by leveraging the Wikidata KG to determine which entities and relations could be found in Wikipedia passages to create compositional, comparison, inference or bridge-comparison multi-hop questions. Due to this construction process, queries in this dataset include at least one named entity, a characteristic that our methodology can leverage given its previously mentioned reliance on NER to link queries to the KG.