SiReRAG: Indexing Similar and Related Information for Multihop Reasoning
Authors: Nan Zhang
Prafulla Kumar Choubey
Alexander Fabbri
Gabriel Bernadett-Shapiro
Rui Zhang
Prasenjit Mitra
Caiming Xiong
Chien-Sheng Wu
Venue: ICLR 2025 Poster (2025)
Keywords: Retrieval-augmented generation (RAG)
RAG indexing
Multi-hop question answering
Primary area: applications to computer vision, audio, language, and other modalities
TL;DR: We introduce an innovative RAG indexing approach that organizes data by considering both similarity and relatedness, achieving strong performance.

Abstract: Indexing is an important step towards strong performance in retrieval-augmented generation (RAG) systems. However, existing methods organize data based on either semantic similarity (similarity) or related information (relatedness), but do not cover both perspectives comprehensively. Our analysis reveals that modeling only one perspective results in insufficient knowledge synthesis, leading to suboptimal performance on complex tasks requiring multihop reasoning. In this paper, we propose SiReRAG, a novel RAG indexing approach that explicitly considers both similar and related information. On the similarity side, we follow existing work and explore some variances to construct a similarity tree based on recursive summarization. On the relatedness side, SiReRAG extracts propositions and entities from texts, groups propositions via shared entities, and generates recursive summaries to construct a relatedness tree. We index and flatten both similarity and relatedness trees into a unified retrieval pool. Our experiments demonstrate that SiReRAG consistently outperforms state-of-the-art indexing methods on three multihop datasets (MuSiQue, 2WikiMultiHopQA, and HotpotQA), with an average 1.9% improvement in F1 scores. As a reasonably efficient solution, SiReRAG enhances existing reranking methods significantly, with up to 7.8% improvement in average F1 scores. Our code is available at https://github.com/SalesforceAIResearch/SiReRAG.

Published as a conference paper at ICLR 2025
SIRERAG: INDEXING SIMILAR AND RELATED INFOR- MATION FOR MULTIHOP REASONING
Nan Zhang Prafulla Kumar Choubey
Alexander Fabbri Gabriel Bernadett-Shapiro ?
Rui Zhang* Prasenjit Mitra The Pennsylvania State University {njz5124, rmz5227, pmi pmitra teaitra @psu.. edu (pchoubey, afbbrri, wwwenndeet cxiong, wu. jason htt.://eo.ooooo.///////oa/ooooor csaeesorecolss
Caiming Xiong
Chien-Sheng Wu
9 Salesforce AI Research
ABSTRACT
Indexing is an important step towards strong performance in retrieval-augmented generation (RAG) systems. However, existing methods organize data based on either semantic similarity (similarity) related information (relatedness), but do not cover both perspectives comprehensively. Our analysis reveals that model- ing only perspective results in insufficient knowledge synthesis, leading suboptimal performance on complex tasks requiring multihop reasoning. In this paper, we propose SIRERAG, novel RAG indexing approach that explicitly considers both similar and related information. On the similarity side, we fol- low existing work and explore some variances construct a similarity tree based on recursive summarization. On the relatedness side, SIRERAG extracts propo- sitions and entities from texts, groups propositions via shared entities, and gen- erates recursive summaries to construct relatedness tree. We index and flatten both similarity and relatedness trees into unified retrieval pool. Our experi- ments demonstrate that SIRERAG consistently outperforms state-of-the-an in- dexing methods on three multihop datasets (MuSiQue, 2WikiMultiHopQA, and HotpotQA), with an average 1.9% improvement in Fl scores. As reasonably efficient solution, SIRERAG enhances existing reranking methods significantly, with up to 7.8% improvement in average F1 scores. Our code is available at
INTRODUCTION
generation (RAG) has shown strong potential in augmenting large language models (LLMs)) with highly specialized and constantly updated knowledge (Lewis et al., 2020 Gao et al., 2024). Getting rid of fine-tuning LLMs, it is an efficient method for handling users' queries that require domain knowledge. A typical RAG pipeline may involve chunking, embedding. index- ing, retrieval with queries, reranking, and LLM response generation (Wang et al. (204)).
The indexing step a prerequisite. focuses on organizing a large amount of data and serves as an upstream step of retrieval. For example, RAPTOR (Sarhhi et al., 2024) shows significant perfor- mance improvement by adding recursive summaries to text chunks of a dataset, which demonstrates the potential of adding synthesized information for retrieval. The added recursive summaries com- bine semantically similar information within dataset. GraphRAG (Edge et al.) (202)), on the other hand, indices an entire corpus via entity- entity-guided knowledge graph. It then constructs summaries from closely-receee related entities and their mentions, synthesizing the connections and relatedness among different pieces of information.
However, none of the existing methods address the importance of indexing from both similarity and relatedness sides, which limits holistic understanding of the provided dataset. We define similarity as the semantic distance of text pieces and relatedness as the degree of connection of texts based on signals such entities and propositions. Indexing similar and related information facilitates more comprehensive knowledge integration than indexing individual kind of information. As shown in
Work done while interning Salesforce AT Research.

---

Published as a conference paper at ICLR 2025
Multihop Reasoing
Indexing Chunks
E
Enity 2
Ently
Q: the father the artist who painted Head
Ensly Chunk
Entity 2 Chunk
Enilyy Chunk
Nicholas Bacon
for semantic distance
Entity 1 Chunk
Entity 2 Chunk Francis Bacon painted a landscape figure in 1945, based on a photograph of Eric Hall dozing in Hyde Park.
Entity 3 Chunk Francis Bacon was the son of Sir Nicholas Bacon, Lord Keeper of the Great Seal, and Anne (Cooke) Bacon.
Head is small oil and tempera
Similarity
hardboard painting by Francis Bacon, completed in 1948.
(close semantic
distance)
Relatedness
(shared topic: Introduction of Francis Bacon)
Entity 2: Francis Bacon :
Enlity +2 Nicholas Bacon
Figure 1: Challenges of existing RAG indexing methods for multihop reasoning. Entity 1 and 2 chunks contain similar information while entity 2 and 3 chunks contain related contents. Since synthesizing information only based on entity and (or entity 2 and 3) will lead to a higher probability of wrong answer, indexing method that considers both similarity and relatedness is needed to maximize retrieving relevant knowledge for multihop questions.
Figure I complex question that involves two hops of reasoning requires the retrieval and synthesis of relevant entity chunks. For example, synthesizing entity 1 and 2 chunks would encourage LLMs to generate "Francis Bacon", which a common mistake. Entity and 2 chunks have close semantic distance. On the other hand, synthesizing entity 2 and 3 chunks would not maximize the probability of the correct answer even when entity 1 chunk is retrieved, since LLMs may struggle to reason about the painter of "Head I" in long-context environment (Liu et al., 2024). In this case, entity 2 and 3 chunks are related due to a shared topic. Therefore, it is important to synthesize both similar and related information in order to maximize the chance of retrieving relevant knowledge, in particular, for multihop reasoning questions. We verify the bottleneck solely modeling similarity or relatedness through quantitative methods in Section and demonstrate that neither perspective yields the optimal performance.
In this paper, propose SIRERAG, which stands for RAG indexing of similarity and relatedness as shown in Figure On the similarity side, SIRERAG follows RAPTOR (Sarthi et al. 2024) to build recursive tree based on chunk similarity. We adopt shallow tree with levels in total. On the re- latedness side, SIRERAG first extracts entities (e.g.. "Sonnet 110" and "William Shakespeare") and fine-gran ame-grained propositions (e.g.. "Sonnet 110 is one of 154 sonnets written by William Shakespeare." ") from each text chunk/document using LLMs. We group these propositions into aggregated ones via entities, simply concatenating them with the original order in chunk/document, These proposition aggregates contain related information, because they mention shared entities. Then, recursive sum- maries with soft clustering are built on top of those aggregated propositions. Finally, we index both trees by flattening nodes in each tree for retrieval.
We show that SIRERAG is effective on a variety of multihop question answering (QA) datasets in- cluding MuSiQue (Trivedi al.) 2022), 2WikiMultiHopQA (Ho et al. 2020), and HotpotQA (Yang jet al. 2018). We find SIRERAG consistently outperforms the strongest RAG indexing methods, achieving average Fl improvement of least 1.9%. We conduct an ablation study several components in STRER AG and observe that adding proposition aggregates to the similar information within dataset yields the most improvement, which echoes our motivation. SIRERAG is also reasonably efficient model, as it does not introduce many lengthy or redundant retrieval candidates.
2 RELATED WORK
RAG RAG is framework that integrates retrieval mechanisms into generative models to enhance text generation by leveraging external knowledge. This concept has evolved from earlier retrieval- based methods such DrQA (Chen et al., 2017) and DPR (Karpukhin separating retrieval and generation phases, researchers also showed the potential of tightly coupling retrieval and generation into an end-to-end framework (Lewis et al. 2020).
al. 2020). Instead of
2

---

Published as a conference paper at ICLR 2025
Similarity Tree
Relatedness Tree
Summary
Summary
:
Summmary
Summary
Summary
Summary
Summary
Proposition
Proposition
Preposition
Chunk
Chunk
Chunk
Chunk
Aggregate
Aggregate 2
Aggregate
entity
Proposition
Proposition
Proposition 3
Proposition M
Unified Retrieval Pool:
Chunk
Summmary
Propostion Aggregate
Summary
Figure 2: SIRERAG Tree. We adopt RAPTOR (Sarthi et al, 2024) to construct the similarity tree
(left). On the right, construct the relatedness tree by clustering the propositions based on their entities to get proposition aggregates and having recursive summaries on top. Note that propositions included in the relatedness tree, their connections proposition aggregates marked with dashed lines.
Recent advances retrieval mechanisms include leveraging LLMs as retrievers (Yu et al. Sun al., 2023) and exploring retrieval granularity such as proposition (Chen et al. 2023) Here each proposition is an atomic expression that contains factoid presented in natural language, which similar to the propositions used by SIRERAG. Inspired by the idea of retrieval granularity, we combine several different text granularities (entity, propositions, text chunk, and summary) to in- dex data. As a representative work on text segmentation, researchers proposed the frst supervised approach to generate hierarchical segmentation structures (Nair et al. (023).
2023
RAG Indexing An earlier work (Lewis et al. 2020) shows the benefits of document index on the overall retrieval performance. Most recent works on RAG indexing include RAPTOR that builds a tree with recursive summaries (Sarthi et al. 2024). HippoRAG that leverages the hippocampal indexing theory of human long-term memory for deep knowledge integration (Gutiérrez et al.) 2024), and GraphRAG that constructs an entity-guided knowledge graph (Edge et al., 2024). However, all these works overlooked the importance of considering both similarity and rel relatedness during indexing. Specifically, RAPTOR integrated knowledge only hased on similarity. and the other two only considered relatedness to synthesize information. Although these three approaches achieved competitive performance on different kinds of datasets and HippoRAG has the same goal as ours (multihop reasoning benchmarks). SIRERAG is fundamentally different from them in terms of the explicit incorporation of both similar and related knowledge.
3 BOTTLENECK OF SOLELY MODELING SIMILARITY OR RELATEDNESS
To verify our hypothesis of insufficient knowledge integration when solely modeling similarity or relatedness. we perform different kinds of clustering philosophies on retrieval corpus of MuSiQue (Trivedi et al., 2022). Using the same retrieval corpus HippoRAG (Gutiérrez et al.) 2024], we obtain 1000 questions from the validation set of MuSiQue along with their candidate passage clusters (each cluster includes supporting and distractor passages). We include distractor passages for more realistic setting, since they are semantically close and/or related to the sup- porting candidates. Treating these clusters the gold labels for different queries, clustering on all passages based on either similarity or relatedness.
our own
Following RAPTOR (Sarthi al. 2024),
Gaussian Mixture Models (GMMs) perform soft clustering, assuming that a candidate passage can belong to multiple clusters. For similarity, we run GMMs on the deep representations of all passages find semantically similar groups. For related-
3

---

Published as a conference paper at ICLR 2025
Table 1: Coverage percentage between different clusters. MuSiQue clusters include supporting and distractor passages, The "all" setting treats both as gold, while "supporting only" uses only supporting passages as gold. We first show the coverage of supporting or all passages under two clustering philosophies. We then report the overlapping ratio between "supporting only" similarity and relatedness to motivate our work of combining both philosophies.
Similarity Coverage
Relatedness Coverage
Overlapping Ratio
Supporting Only
Supporting Only
All
Overlap Similarry Overlap Relatedness
Coverage
19.14
10.70%
13.94%
8.51%
50.15%
68 68.85%
ness, we first extract the topic of each passage using OpenAl GPT-4o and then cluster passages based on the representations of their topics, because we assume that related passages share similar topics. To obtain deep representations of either passages or topics, we use OpenAl text-embedding-Be5 -smllll for balance of performance and efficiency. To evaluate the overlapping ratio among clusters, convert every cluster into pairwise connections. For instance, given two clusters [1, 2, 3] and
[3, 5], the resulting pairwise connections as follows: "1-2", "1-3", "2-3", and "3-5". By computing the number of shared pairwise connections between gold clusters and predicted clusters, aim to see how many pairwise connections from gold labels are covered by our two clustering philosophies, and we believe this coverage is a key indicator of knowledge integration for RAG in- dexing. We also report the overlapping ratio between "supporting only" similarity and relatedness to make point of combining both clustering philosophies.
Table shows the coverage of supporting only or all passages as gold labels. Higher coverage is obtaned when we only use supporting passages as gold, which indicates the commonality of supporting ones with respect to distractors. Both philosophies are able to capture this commonality. However, all these percentage scores are low, with the highest one being around one-fifth of the total coverage (19.14 14%). This indicates a significant insufficiency in knowledge synthesis when we model similarity or relatedness solely. As result, the chance of retrieving relevant knowledge for multihop reasoning questions could be suboptimal.
Taking further step in the supporting-oni setting, we find that only 50.15% of the correct simi- larity connections overlap with the correct relatedness connections. Correct similarity connections those gold connections covered by similarity, while correct relatedness connections those covered by relatedness. Conversely, 68.85% of the correct relatedness connections overlap with the correct similarity connections. In other words, predictions based on similarity relatedness not identical, and we can potentially leverage both to improve retrieval performance and facilitate more comprehensive knowledge integration process. Although more customized clustering algo- rithms of each clustering philosophy can be proposed, combining both similarity and relatedness offers an effective and straightforward solution.
METHODOLOGY
We propose SIRERAG, RAG indexing framework guided by similarity and relatedness. As shown in Figure 2 its left tree integrates information based on similarity while its right tree integrates in- formation based on relatedness. As a first step, we study an alternative tree design to determine whether we can develop generalized tree structure for similarity and relatedness trees, and be- yond (Section [.1]. After the construction of the similarity tree, we extract propositions and their entities from our multihop reasoning dataset and perform clustering based on entities to synthesize related information (Section (4.2). Indexing separate similarity and relatedness trees (Section 4.3 SIRERAG explicitly models both kinds of information within dataset.
4.1 EXPLORING A HIERARCHICAL STRUCTURE OF TREES
For efficiency, we stick to tree structure to organize texts and explore whether more structured tree design would offer performance improvement. RAPTOR placed all text chunks at the bottom level and recursive summaries upper levels, but this design does not closely follow the commonsense
4

---

Published as a conference paper at ICLR 2025
Table 2: QA performance of having hierarchical text chunks on the validation of QuALITY. Due to randomization of clustering and generation temperature of LLMs, we run each indexing method for 5 times and compute their average and standard deviation of accuracies.
Average Accuracy Standard Deviation
RAPTOR Hierarchical Text Chunks
78.88
0.005
78.76
0.004
definition of a tree (Zhang et al. 2002), where multiple levels of data abstraction are provided. The reason is that different text chunks of document may showcase different levels of abstraction, which serves as the assumption of many document discourse trees (Maekawa et al. 2024 Liu al.)
2021). For example, text chunks of the introduction section of research paper are more likely to be more abstractive than those from the methodology section. Therefore, we first explore placing different text chunks different levels of a tree based on their levels of abstractiveness.
For this analysis, we choose QuALITY dataset (Pang et al. 2022) instead of multihop reasoning ones, because the documents of QuALITY are longer, resulting in more text chunks than using the other datasets. We prompt GPT-4o to identify a two-level hierarchy for all candidate chunks: low (text chunks describing fine-grained details about topic) and high (text chunks giving overview of topic and summarizing fine-grained details) abstractiveness. Summary nodes will still start from the second level (the level above the bottom level), so the second level will contain both summary nodes (of the low-abstractive chunks) and high-abstractive chunks. As in RAPTOR, nodes of the third level and above are recursive summaries of their children. Using the same text chunks across two methods, we compare RAPTOR against our hierarchical design. Since we simply retrieve the top 10 relevant nodes via an embedding model sext--ebeddinng3-
when a query arrives, the idea of hierarchical structure of chunks would affect summary nodes due to the differences of their children.
As shown in Table we do not observe clear trend of improvement from using hierarchical text chunks, even averaged over different runs. The RAPTOR tree is relatively more efficient, as iden- tifying hierarchical text chunks and document discourse trees would require additional computation. Thus, we conclude that a more structured tree design would not significantly improve performance, as long as the correct information is indexed to answer question. Since RAPTOR uses Gaussian Mixture Models and representations of text chunks to perform clustering, it sets a nice example of integrating knowledge based on similarity. We adopt it to construct our similarity tree shown in the left part of Figure 2
4.2 SYNTHESIZING INFORMATION BASED ON RELATEDNESS
For relatedness, we need to synthesize information based on a different philosophy than RAPTOR. Because related information pieces always share some degree of connection (e.g. overlapping sub- jects), we assume that two text pieces are related if they mention the same entity (e.g., person, location, product, etc). For example, entity 2 and 3 chunks in Figure are unlikely be clustered in the same group based on similarity, but since they both mention Francis Bacon, we are able to connect them together.
Modeling Relatedness with Entity-Specific Propositions: To effectively use entities for organiz- ing related content, we first need to determine the appropriate granularity for text pieces. There are three main limitations with directly connecting entities to standard text chunks. First, chunk often contains information beyond the scope of specific entity, making it challenging to localize informa- tion about one entity, potentially adding noise. Second, aggregating all chunks in an indexing corpus for each entity can result in hundreds of thousands of tokens for each entity, which may lead to long context performance issues, such as losing critical information in the middle Liu et al. (2024) or experiencing low coverage and citation performance Laban et al. (2024). Third, linking with chunks will introduce redundancy as each chunk may be part of multiple entity clusters. Therefore, in- spired by recent works retrieval granularity (Liu al. 2023; Chen et al.) 2023), we propose to use short entity- "propositions" to represent iine-grained knowledge about entities and build relatedness tree.
5

---

Published as a conference paper at ICLR 2025
Table 3: Key Statistics for extracted propositions and entities from MuSiQue, 2Wiki, and HotpotQA datasets. We show the number of chunks, propositions, entities, proposition aggregates, and the average, maximum, and minimum number of propositions per entity across the three datasets.
#Props per Entity
Dataset
#Chunks
#Props.
#Ents
#Pops Aggs.
Avg.
Max.
Min.
MuSiQue
11,656
54,605
50,926
20,788
2.74
168
2Wiki
6,119
27.697
29,490
11,108
2.49
195
HotpoiQA 9,221
47,153
46,856
18,278
2.66
165
Propoxition Drug sales are reaching record highs as new therapies are developed and approved.
Entities: []
Proposition 2: Drug sales for Eli Lilly's s Mounjaro and Novo Nordisk's semaglutide are reaching record highs as therapies developed and approved. Entilies: [Eli Lilly, Mounjaro, Novo Nordisk, Semaglutide]
Figure 3; Examples of propositions with and without associated entities.
Extracting Propositions and Entities from Documents: We define proposition "a factual statement describing important information (preferably about some entities) from a paragraph". We extract entities and propositions using the Distill-SynthKG pipeline (Choubey et al. its SynthKG workflow. First, we rewrite chunks of 10K documents from the BAAT/ AAA/nnduuryyyrpuu resolve entity references. using
2024), adapting
2024) with the rewriting prompt shown in Figure 6 Next, we prompt the same LLM to extract entities from these rewritten chunks (prompt is shown in Figure 7 After obtaining these entities, we again prompt the LLM to identify all relevant propositions and their associated entities (prompt is shown Figure S We then consolidate the resulting propositions and entities to fine-tune -estra----Instructee
(AI@ Metal
I (Jiang et al. 2023). This smaller fine-tuned model is subsequently used to extract propositions and their associated entities from our multihop datasets.
Our prompt for extracting propositions and entities does not require every proposition to have sociated entitics. When we prompt LLMs to generate entities for each proposition, they sometimes produce common nouns entities for those propositions that lack actually associated entities. This can lead to the clustering of unrelated propositions based on common nouns, potentially introducing noise into the relatedness tree. For example, as illustrated in Figure 3] we prefer to avoid LLMs generating Drug as an entity for proposition 1 due to its ambiguity. Subsequently, we exclude propositions without associated entities when constructing the relatedness tree, ensuring that only high-quality. lntity-linked propositions are utilized. The Table 3 provides a detailed breakdown of key statistics for extracted propositions and entities from the MuSiQue, 2Wiki, and HotpotQA datasets.
From Entity-specific propositions to Relatedness Tree: We concatenate related propositions that share the same entity using exact match to form proposition aggregates. We ensure that all propositions from the same document are grouped together and maintain their original order. By treating these proposition aggregates pseudo-documents, we apply the same clustering pipeline in RAPTOR to obtain recursive summaries at levels above them and build the relatedness tree. Given that most propositions involve multiple entities, each proposition is associated with several entity clusters, offering two key advantages. First, it effectively mimics soft clustering as a single proposition may belong multiple aggregates. Secondly, when constructing recursive summaries within the RAPTOR framework, the shared propositions across different aggregates result in high embedding similarity, ensuring that these aggregates remain clustered together even at higher levels in the tree. This relatedness tree forms the right part of STRERAC AG as shown in Figure 2 Note that we only index aggregated propositions instead of individual propositions for better inference efficiency.
Instruct

---

Published as a conference paper at ICLR 2025
4.3 INDEXING SIMILARITY AND RELATEDNESS TREES
We propose to construct similarity and relatedness trees independently. This approach ensures that summary nodes in one do not access the clusters of the other, leading to simpler design. There is another slightly complex design in which we allow summary nodes from one tree to access clusters from the other tree. This interaction may enable summary nodes in both trees to inform and enhance each other, improving their informativeness and consequently performance. However, this approach sacrifices the distinction between similarity and relatedness. Additionally, allowing cross-tree interaction leads to more nodes to cluster at each level as well as requires summarization based on a greater number of nodes per cluster, all of which increases the overall complexity of the system. We experimented with both settings and did not observe performance improvement as shown in Appendix A Therefore, we opted for the simpler first implementation in our evaluation.
Flattening all tree nodes, we place them into unified retrieval pool. In other words, regardless of node's s origin (e.g. bottom or upper levels, similarity or relatedness trees), it is added single list containing all nodes.
5
EXPERIMENT SETUP
5.1 DATASETS
To demonstrate the effectiveness of SIRERAG, we select three representative multihop QA datasets: MuSiQue (Trivedi al., 2022), 2WikiMultiHoPQA (Ho et al. 2020), and HotpotQA (Yang al., 2018). Using the same corpus as HippoRAG (Gutiérrez et al. 2024), we obtain 1000 questions from each validation of these three datasets.
5.2 BASELINES
We select RAPTOR, HippoRAG, and GraphRAG state-of-the-ane retrieval baselines. As discussed above, RAPTOR integrates knowledge based on similarity while the other two approaches focus on relatedness. Specifically, HippoRAG has both indexing and retrieval components, and we use Col- BERTv2 (Santhanam al. 2022) as the retriever of HippoRAG due to its strongest QA performance reported. Although GraphRACI has a different goal (global questions directed at entire dataset) than ours, we include it to show its performance on multihop QA datasets. Since the queries in our datasets ask fine-gratd siuengrinedd details, search. Additional details of GraphRAG are specified Appendix F
the local scarch function of GraphRAG instead of its global
5.3 EVALUATION METRICS
We use exact match (EM) and Fl scores measure the QA performance of different models. Both metrics evaluate how accurate generated answer is with respect to the ground truth. Like RAP- TOR (Sarthi al.) 2024). we do not assess retrieval performance directly. The reason is that both SIRERAG and RAPTOR create new candidates (e.g., summary and proposition aggregate) in the retrieval pool, so it would be unfair to compare methods in terms of retrieval scores across different pools. Instead, QA performance is the best indicator of the overall capability of both RAG pipelines.
We use the average time per query (TPQ) and the time-pool efficiency ratio (TPER) to measure efficiency of SIRERAG and RAPTOR, as both methods share a significant portion of their retrieval candidates. Average TPQ measures the average time (in seconds) taken to answer a query, and represents the inference time of method. For TPER, it computes the growth of total inference time with respect to the growth of the retrieval pool size between two methods:
TPER Inference-Time A A/Inference- Time B
(I)
Pool-Size A/Pool-Size B
Setting SIRERAG as method A and a baseline as method B, we aim to ensure that the growth of inference time does not scale proportionally with the increase in the retrieval pool size. The reason behind is that there are many efficiency considerations (e.g., length and redundancy of retrieval can- didates) beyond just the sheer number of retrieval candidates. Parallelization could also be designed

---

Published as a conference paper at ICLR 2025
to retrieve candidates simultaneously from similarity and relatedness trees, thereby minimizing the effect of retrieval pool size. A TPER value less than I indicates reasonable efficiency, whereas a TPER value greater than 1 signifies low efficiency.
5.4 IMPLEMENTATION DETAILS
To generate final answer, use GPT-4o and the same prompt ( ("answer this question in as fewer number of words as possible.") to answer queries for methods, since we aim to control the cntruuction-oollowinn capabilities across all methods. We either GPT-3.5-Turbo
GPT-4o as the choice of L.M if any methods involve LLM calls. We use OpenAI's text-eexeeeeeeee -mweeding----mmmiieie the embedding model for all methods. During retrieval, we select top 20 candidates that match the provided query for all methods, because there is a large number of text chunks in our datasets and SIRERAG expected to perform better when retrieving more due to the incorporation of proposi- tion aggregates and their recursive summaries.
RESULTS AND ANALYSIS
Our results and analysis aim to answer the following research questions:
RQ 1: How does SIRERAG compare against other state-of-the-an art baselines (sec 6.1
RQ 2: As important contribution of SIRERAG, is considering both similarity and relatedness an effective method (s 6.22)
RQ 3: What is the effect of each component in SIRERAG(see 6.2
RQ 4: What is the applicability of SIRER (sec 6.3
RQ 5: With the addition of relatedness tree, SIRERAG an efficient method (sec (sec 6.4
Table QA performance SIRERAG and baselines. As elaborated in Section 5.4
GPT-4o is used
to handle QA for all models, and we use two different LLMs (specified in the parentheses) to build indexing structures. We highlight the best scores using either L.LM for indexing in green color.
MuSiQue
2Wiki
HotpotQA
Average
Model
EM
F1
EM
F1
EM
F1
EM
F1
HG (GPT-3.5-Turbo))
32.60
43.78
66.40
74.01
59.90
74.29
52.97
64.03
RAPTOR (GPT-3.5-Turbo)
35.30
47.47
54.90
61.20
58.10
72.48
49.43
60.38
GraphRAG (GPT-4o)
12.10
20.22
22.50
27.49
31.70
42.74
22.10
30.15
RAPTOR (GPT-4o)
36.40
49.09
53.80
61.45
58.00
73.08
49.40
61.21
SIRERAG (GPT-3.5-Turbo)
38,90
52.08
60.40
68.20
62.50
77.36
53.93
65.88
SIRERAG (GPT-4o)
40.50
53.08
59.60
67.94
61.70
76.48
53.93
65.83
6.1 OVERALL RESULTS
Our overall results are presented in Table We show results on more datasets (single-hop QA, other multihop QA, and ambiguous questions) Appendix B show the generality of SIRERAG across various complex reasoning tasks. Besides quantitative scores, we also conduct our qualitative analysis in Appendix D
Improvement over baselines SIRERAG delivers consistent improvement over RAPTOR, Hip- poRAG, and GraphRAG. With an exception on 2Wiki when comparing against HippoRAG, SIR- ERAG achieves significantly higher performance than indexing baselines (e.g., approximately 5% higher than RAPTOR on average F1, up to 8.3% improvement of on MuSiQue than HippoRAG, and more than 20% higher than GraphRAG on average EM and F1). This demonstrates the ad- vantage of SIRERAG on multihop QA and modeling both similarity and relatedness. Specifically, SIRERAG outperforms RAPTOR due to the incorporation of relatedness tree, and it has better overall performance than HippoRAG, because we explicitly model similarity while HippoRAG pri- oritizes relatedness signals such as nodes with the most edges. We see that HippoRAG is particularly strong on 2Wiki benchmark, which is also reported in its original paper. Thus, we believe 2Wiki the best fit of HippoRACe AG, but has lower performance scores than SIRER AG other datasets.
8

---

Published as a conference paper at ICLR 2025
Table 5: Ablation study of SIRERAG.
MuSiQue
2Wiki
Ho
Average
Variants
EM
EM
F1
EM
F1
EM
F1
SIRERAG
40.50
53.08
59.60
67.94
61.70
76.48
53,93
65.83
(A): SIRERAG Re. Summary
37.50
50.38
57.70
65.75
61.20
75.99
52.13
64.04
(B): SIRERAG + Proposition
39.10
51.80
58.10
65.53
60.80
75.26
52.67
64.20
(C): (B) Aggregate
34.70
47.82
53.20
60.22
58.90
73.38
48.93
60.47
(D): (C) Re. Summary
33.90
46.19
53.00
59.13
57.00
71.09
47.97
58.80
(E): Dual Clustering Chunks
34.80
47.32
53.50
59.93
56.60
71.84
48.30
59.70
One potential reason is that the entity-centric design of 2Wiki may be well-suited for HippoRAG, as noted in the HippoRAG paper.
As for GraphRAG, it considers relatedness solely, and it delivers the worst performance scores on our datasets. After manual verification, we find that GraphRAG often provides "I don't know" answers, suggesting that it prefers not to give a concrete answer. Since GraphRAG designed to handle query-focused summarization of entire corpus, it is not the most competitive approach in terms of accuracy for existing multihop QA tasks.
Effect of LLM choice When comparing the performance of SIRERAG using GPT-4o the LLM against itself using GPT-3.5-Ture Turbbo, we find the QA performance not significantly affected. This phenomenon also holds RAPTOR. Since the choice of LLM for SIRERAG and RAPTOR only affects summarization results, we believe GPT-3.5-Turbe is sufficiently good option for both meth- ods. This allows researchers to pursue a more cost-effective solution with SIRERAG for indexing.
6.2 ABLATION STUDY
To dissect SIRERAG, we perform a comprehensive ablation analysis as shown in Table There are several variances, including (A) remove the recursive summary on the relatedness tree; (B) add all the propositions into the retrieval pool, and keep all aggregated propositions and recursive summary on the relatedness tree; (C) same as (B) but remove aggregated propositions; (D) same as (C) but further remove the recursive summary design on the relatedness tree.
Entity clustering (E), we do not maintain a separate relatedness tree and add an additional clus- tering philosophy to the similarity tree. Specifically, each text chunk in the similarity tree is simpli- fied to, "This chunk mentions entity 1 and entity 2,' if both entities are extracted by our LLM. We then run GMMs (the same clustering method as RAPTOR) on these simplified chunks. Once the clustering decisions are obtained, we group the original chunks as additional clusters and append these clusters to the similarity tree, allowing higher levels of the tree incorporate both clustering philosophies. Since entities primarily determine the outcome of this additional clustering approach, apply entity clustering model relatedness the similarity tree. This allows to eliminate proposition aggregates in order to examine their utility.
Findings Overall, we observe performance drops across all variations, highlighting the effectiveness of our design for SIRERAG. First, the recursive summary on the relatedness tree proves beneficial, as seen in both (A) and (D). Interestingly, adding more propositions retrieval negatively impacts performance, as shown in (B). This indicates adding redundant information into the retrieval pool hurts the QA performance, since we keep the aggregated propositions in SIRERAG. From (C), we also find that aggregated propositions are essential, with their removal resulting in significant per- formance decline. This an important indicator that adding grouped knowledge about relatedness to the similarity tree would offer improvements, which echoes the bottlenecks described in Section
Both (A) and (B) have better performance than RAPTOR (GPT-4o) from Table which indicates the advantage of proposition aggregates. In contrast, although (E) also models both similarity and re- latedness, it exhibits a notable decline comparing against SIRERAG. This finding demonstrates the necessity of proposition aggregates of modeling relatedness. Because proposition aggregates reduce noise and information redundancy more effectively than text chunks as described in Section 4.2 they serve effective carrier of related dataset contents.

---

Published as a conference paper at ICLR 2025
Table 6: Applicability of SIRERAG when specific retrieval method is selected. We feed indexing models with the retrieval pool of SIRERAG and see whether QA performance improves.
MuSiQue
2Wiki
HotpotQA
Average
Variants
EM
F1
EM
F1
EM
EM
F1
B.255
25.90
35.88
53.00
58.58
57.70
71.32
45.53
55.26
RAPTOR BM25
27.00
38.91
50.60
57.00
56.90
70.88
44.83
55.60
SIRERAG + BM25
35,00
47.66
58,20
65.72
61.70
75,88
51.63
63.09
ColBERTv2
34.00
46.80
52.90
59.48
59.00
73.42
48.63
59.90
RAPTOR ColBERTv2
34.20
47.17
50.30
57.51
57.90
72.31
47.47
59.00
SIRERAG + ColBERTv2
38.10
51.32
56.70
64.74
60.90
75.72
51.90
63.93
Table 7: Efficiency of SIRERAG and RAPTOR. Since both methods share significant portion of retrieval candidates, we designate SIRERAG Method A and RAPTOR as Method in the TPER columns, as defined in Equation [
MuSiQue
2Wiki
HotpotQA
Average
Model TPQ4
TPER
TPQ4
TPER|
TPQ4
TPER
TPQ4
TPER
RAPTOR 1.560
1,437
1.502
1,500
SIRERAG 2.653
0.600
1.974
0.499
2.319
0.517
2.315
0.539
6.3 APPLICABILITY SIRERAG
We analyze how applicable SIRERAG is when specific retrieval method is chosen. Therefore, we select BM25 (Robertson Walker 1994) and ColBERTv2 (Santhanam al., 2022) as additional beranking-based options. We run them on the retrieval pool of STRERAG to demonstrate its utility. We show how SIRERAG can complement these non-indexing options in Table Results show that having SIRERAG benefits both BM25 and ColBERTv2 significantly, which demonstrates the advantage of our solution as the upstream step of these methods. On the other hand, RAPTOR only improves the QA performance MiSuQue while showing performance degradation other datasets. Thus, the utility of SIRERAG surpasses that of RAPTOR in the context of multihop rea- soning. We also apply SIRERAG on an iterative retrieval method called self-ask (Press et al. 2023) and obtain significant performance improvement as shown in Appendix Our method showcases wide applicability on multihop QA across various retrieval methods.
6.4 EFFICIENCY OF SIRERAG
As shown in Table we compare the efficiency of SIRERAG and RAPTOR using the metrics described in Section 5.3 All the values listed involve the time taken to retrieve the top 20 candidates and prompt GPT-4o to answer the query.
RAPTOR requires less inference time than SIRERAG on average, which is expected due to SIR- BRAG's larger retrieval pool. However, with slightly longer inference time, SIRERAG has much better performance
discussed previously. To evaluate whether SIRERAG remains reasonably efficient method, we compute its TPER values to measure its growth of total inference time relative to its growth of retrieval pool size. Since all its TPER values are well below SIRERAG demon- strates reasonable efficiency without introducing many lengthy or redundant retrieval candidates.
7 CONCLUSION
In this paper, we identify the bottleneck of solely modeling similarity relatedness when need to index a multihop reasoning dataset for knowledge integration. To address it, we introduce SIR- ERAG, innovative RAG indexing approach that considers both similarity and relatedness. SIR- ERAG delivers a consistent improvement over state-of-th of-t-e-rttttt indexing baselines across several multihop QA benchmarks.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
A Meta. Llama 3 model card. 2024. URL http:t/s//gggghh...mm/mmttttiiae wwt.://..coom/mmaaaaaaaaa hotp:///eetttl11mmmmmmmmmm//ool
main/MODEL md
Danqi Chen, Adam Fisch, Jason Weston, and Antoine Bordes. Reading Wikipedia to answer open- domain questions. In Regina Barzilay and Min-Yen Kan (eds.), Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp. 1870- 1879, Vancouver, Canada, July 2017. Association for Computational Linguistics. doi: 10.18653/ v1/P17-1171. URL
Tong Chen, Hongwei Wang, Sihao Chen, Wenhao Yu, Kaixin Ma, Xinran Zhao, Dong Yu, and Hongming Zhang. Dense x retrieval: What retrieval granularity should we use? arXiv preprint arXiv:2312.06664, 2023.
Prafulla Kumar Choubey, Xin Su, Man Luo, Xiangyu Peng, Caiming Xiong, Tiep Le, Shachar Rosenman, Vasudev Lal, Phil Mui, Ricky Ho. Phillip Howard, and Chien-Sheng Wu. Distill- synthkg: Distilling knowledge graph synthesis workflow for improved coverage and efficiency, 2024.
www.://////b///////// 16597
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: graph rag approach to query-focused summariza- tion. arXiv preprint arXiv:2404. 16130, 2024.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrievallauggmente generation for large language models: A survey, 2024. URL
10997
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neurobiologically inspired long-term memory for large language models. arXiv preprint arXiv:2405.148311 2024.
Xanh Ho. Anh- Ahoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. Constructing a multi- hop QA dataset for comprehensive evaluation of reasoning steps. In Donia Scott, Nuria Bel, and Chengqing Zong (eds.). Proceedings of the 28th International Conference on Computational Linguistics. pp. 6629,6625, Barcelona, Spain (Online), December 2020. International Com- mittee Computational Linguistics. doi: 12166///1/22200..inngmmaaaa
URL https:
//aclanthology, grg/ 2020. cowlng--maaaaeeeee 580
Shayekh Bin Islam, Md Asib Rahman, K S M Tozammel Hossain, Enamul Hoque, Shafiq Joty, and Md Rizwan Parvez. Open-RAG: Enhanced retrieval augmented reasoning with open-source large language models. In Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (eds.), Find- ings of the Association for Computational Linguisties: EMNLP 2024, pp. 14231-14244, Miami, Florida, USA, November 2024. Association for Computational Linguistics. doi: 10.18653/v1/ 2024. 1924.indings-emnlp...
org http///22fffffnnnneeee findings-emnlp 831
Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chap- lot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https: //arxiv.org/ wtt.org//abb////aaee 06825
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Bonnie Webber, Trevor Cohn, Yulan He, and Yang Liu (eds.), Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pp. 7769-6781. Online, Novem- ber 2020. Association for Computational Linguistics. doi: 10. 10.18653/v1/2020..mm URL httpp://acclatthol//
org/ 2wwpg//222...0 wtt..r/g//2....mmmmmaaa..... emnlp-main......
Philippe Laban, Alexander R Fabbri, Caiming Xiong, and Chien-Sheng Wu. Summary of a haystack: A challenge to long-context llms and rag systems. arXiv preprint 20070:240..01370, 2024.
11

---

Published as a conference paper at ICLR 2025
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Na- man Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. Retrieval-augmented generation for knowledge-intente tasks. In H. Larochelle, M. Ranzato, R. Hadsell, M.F. Balcan, and H. Lin (eds.), Ad- vances in Neural Information Processing Systems, volume 33, pp. 9459- Curran Asso- ciates, Inc., 2020, (D493230206f78800
nip
neur httppp/.ccccpppp....ee///peeeeee
Nelson F Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models use long contexts. Transactions of the Association for Computational Linguistics, 12:157-173. 2024.
Yixin Liu, Alexander R Fabbri, Yilun Zhao, Pengfei Liu, Shafiq Joty, Chien-Sheng Wu, Caiming Xiong, and Dragomir Radev. Towards interpretable and efficient automatic reference-based sum- marization evaluation. arXiv preprint arXiv:2303.033600 2023.
Zhengyuan Liu, Ke Shi, and Nancy F Chen. Dmrst: A joint framework for document-level multilin- gual rst discourse segmentation and parsing. arXiv preprint arXiv:2110.05518. 2021.
Aru Mackawa, Tsutomu Hirao, Hidetaka Kamigaito, and Manabu Okumura. Can obtain sig- nificant success in RST discourse parsing by using large language models? In Yvette Gra- ham and Matthew Purver (eds.), Proceedings of the 18th Conference of the European Chap- ter of the Association for Computational Linguistics (Volume 1: Long Papers), pp. 2803-2815, St. Julian's, s, Malta, March 2024. Association for Computational Linguistics. URL https: 7/aclanthology. org/ 2024. eacl pacl-long.11 171.
Inderject Nair, Aparna Garimella, Balaji Vasan Srinivasan, Natwar Modani, Niyati Chhaya, Srikr- ishna Karanam, and Sumit Shekhar. A neural CRF-based hierarchical approach for linear text segmentation. In Andreas Vlachos and Isabelle Augenstein (eds.), Findings of the As- sociation for Computational Linguistics: EACL 2023, pp. 883-893, Dubrovnik, Croatia, May 2023, Association for Computational Linguistics. doi: 10. 118653/1120223 findings-eacl.66 URL https://aclanthollog/ wtg/ findings-eacl.
Richard Yuanzhe Pang, Alicia Parrish, Nitish Joshi, Nikita Nangia, Jason Phang, Angelica Chen, Vishakh Padmakumar, Johnny Ma, Jana Thompson, He He, and Samuel Bowman. QuALITY: Question answering with long input texts, yes! In Marine Carpuat, MaarieCCtheriie de Marneffe, and Ivan Vladimir Meza Ruiz (eds.). Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pp- 5336-5358, Seattle, United States, July 2022. Association for Computational Linguistics. doi: 10. 18653/v haal:: m91. URL https:://claatthoogy/ www.p//g......./ee 2022. www.l-maccnannnmm 391
Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah Smith, and Mike Lewis. Mcasuring and narrowing the compositionality gap in language models. In Houda Bouamor, Juan Pino, and Kalika Bali (eds.), Findings of the Association for Computational Linguistics: EMNLP 2023, pp. 5687-5711, Singapore, December 2023. Association for Computational Linguss- htttchology.rrg/20000 org/2023.
tics. doi: 10. 10.18653/v1/20222 findings-emee wwwpigs--enll..33 378
120.6//5111122....innneeemmmamm findinng---emml..7.... URL
Stephen E Robertson and Steve Walker. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SIGIR 94: Proceedings of the Seventeenth Annual International ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pp. 232 241. Springer, 1994.
Keshav Santhanam, Omar Khattab, Jon Saad-Falcon Christopher Potts, and Matei Zaharia. Col- BERTv2: Effective and efficient retrieval via lightweight late interaction. In Marine Carpuat, Marie-Catherine de Marneffe, and Ivan Vladimir Meza Ruiz (eds.), Proceedings of the 2022 Conference of the North American Chapter of the Association for Computational Linguistics:
Human Language Technologies, pp. 7715-3734, Seattle, United States, July 2022. Associa- tion for Computational Linguistics. doi: 11118663111222..... www.o222nacllaiin.... main.272. 7/ w/acllnnthology.org//00 wwt.g///2...aaammmaaeee. naacl-main. 272
URL https:
12

---

Published as a conference paper at ICLR 2025
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Man- ning. Raptor: Recursive abstractive processing for tree-organized retrieval. arXiv preprint TrXiv:2401.18059 2024.
Ivan Stelmakh, Luan, Bhuwan Dhingra, and Ming-War Wei Chang. ASQA: Factoid questions meet long-form answers. In Yoav Goldberg, Zornitsa Kozareva, and Yue Zhang (eds.). Proceed- ings of the 2022 Conference on Empirical Methods in Natural Language Processing, pp. 8273- 8288, Abu Dhabi, United Arab Emirates, December 2022. Association for Computational Lin- guistics. doi: 10. 2ttp33////222.emmmmmamm.. emnlp-main. 566
URL https::/aclantthlolgg wtwp:/ogggg.ggg//222222//
Zhiqing Sun, Xuezhi Wang, Yi Tay, Yiming Yang, and Denny Zhou. Recitation-augmentte language models, 2023. URL https://arxiv. wttp://wrrggbb//////// 01296
Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking hop queries, 2024. URL htLLhhttss//arvi..or////b/ 153311
generation for multi-
Harsh Trivedi, Niranjan Balasubramanian. Tushar Khot, and Ashish Sabharwal. Musique: Multihop
questions via single-hop question composition, 2022. URL https://arxiv.orrrabb//// 005573
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang. Shizheng Li, Qi Qian, Ruicheng Yin, Changze Lv, Xiaoqing Zheng, and Xuanjing Huang. Searching for best practices in retrieval-augmeeteed generation, 2024. URL
http:///axxi...rg/abbo 2eolab/s///.... 0ttes/2477.11222 012199
Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Ellen Riloff, David Chiang, Julia Hockenmaier, and Jun' ichi Tsuji (eds.). Proceed- ings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp- 2369- 2380, Brussels, Belgium, Novembrr 2018. Association for Computational Linguistics. doi: 10.18653/v1/D1.. 1259.
www.looyo.oogggoggo org/ 1259
Wenhao Yu, Dan Iter, Shuohang Wang, Yichong Xu, Mingxuan Ju, Soumya Sanyal, Chenguang Zhu, Michael Zeng, and Meng Jiang. Generate rather than retrieve: Large language models are strong context generators, 2023. https:/t:///r org hwrgg/abs/2200 10063.
Jun Zhang, Adrian Silvescu, and Vasant Honavar. Ontology-driven induction of decision trees at multiple levels of abstraction. In Abstraction, Reformulation, and Approximation: 5th Interna- tional Symposium, SARA 2002 Kananaskis, Alberta, Canada August 2-4, 2002 Proceedings 5, pp. 316-323. Springer, 2002.
A AN ALTERNATIVE DESIGN OF ALLOWING CROSS- INTERACTION
We discuss an alternative design of combining similarity and relatedness trees. Specifically, this de- sign combines nodes from both sides in the same pool for finding additional clusters and performing summarization every tree level. In other words, find additional clusters by concatenating the nodes of both trees, which considers cross-tree interaction instead of keeping them separate.
As shown in Table g the performance of considering cross-tree interaction slightly lower than SIRERAG. Therefore. it is more efficient to keep trees separate in order to reduce the overall com- plexity of the system discussed in Section 4.3
B ADDITIONAL EXPERIMENT ON OTHER NON-INDEXING METHODS
Although there are many existing methods that work on multihop reasoning tasks, SIRERAG is about indexing corpus data under RAG setup. In other words, instead of being baselines, other non-indexing (orks(Press et al. 2023 Islam et al, 2024) focus on other dimensions of improving performance complex reasoning tasks.
13

---

Published as a conference paper at ICLR 2025
Table 8: QA performance of designs: separating similarity and relatedness trees (SIRERAG)
and allowing cross-tree interaction.
MuSiQue
2Wiki
HotpotQA
Average
Variants
EM
F1
EM
F1
EM
F1
EM
F1
SIRERAG
40.50
53.08
59.60
67.94
61.70
76.48
53.93
65.83
Cross-tree interaction
40.20
53.06
58.30
65.25
60.30
75.71
52.93
64.67
Table 9: QA performance of closed-book and self-ask. We feed self-ast with the unified retrieval pool of SIRERAG and see whether performance benifits from that.
MuSiQue
2Wiki
HotpotQA
Average
Model
EM
F1
EM
F1
EM
F1
EM
F1
Closed-book 31.20
10.0
22.0
19.0
34.0
29.0
44.0
19.3
33.3
Self-ask
44.35
55.00
61.99
57.10
71.11
47.77
59.15
SIRERAG + self-ask
36.50
49.12
57.20
65.13
59.70
74.07
51.13
62.77
To further demonstrate the applicability of SIRERAG, we have run two additional sets of experi- ments: (I) the closed-book setting, and (2) an iterative retrieval method (Press et al. (023) called self-ask specifically designed for multihop reasoning. The closed-book setting means to directly get the final answer from GPT-4o without any retrieval. For self-ask, prompt GPT-4o in two iterations without using a search engine. In the first iteration, the model is prompted to propose follow-up questions and provide answers to them. In the second iteration, GPT-4o is instructed to answer the final question by incorporating the follow-up thought process. We feed the model with a one-shot example and 10 retrieved candidates that match the final question both iterations. For the self-ask retrieval pool, we use either all text chunks or SIRERAG.
Similar Table 6 our experiment in Table on self-ask shows that SIRERAG can complement existing methods for optimal performance. We see that the closed-boot setting yields the worst performance, which indicates that LLMs' parametric knowledge alone does not offer decent per- formance our datasets. Then, we see SIRERAG successfully improves the scores of self-ask, demonstrating its wide applicability. By leveraging STRERAG's retrieval pool, we view our method augmentation to other non-indexing methods for multihop reasoning, rather than competi- tor.
Efficiency-wise; we show TPQ of using both SIRERAG and self-ask in Table 10 By having SIR- ERAG, the TPQ of self-ask increases by approximately 1.2 seconds over the three datasets. Since self-ask requires two iterations of LLM prompting in implementation, the increase in TPQ is relatively small compared to the significant performance improvement brought by SIRERAG.
C RETRIEVAL POOL SIZE
The retrieval pool sizes of SIRERAG on MuSiQue, 2Wiki, and HotpotQA are 35070, 19100, and 29934 respectively. The retrieval pool sizes of RAPTOR MuSiQue, 2Wiki, and HotpotQA 12371, 6939, and 10031 respectively. SIRERAG's retrieval pool size is slightly less than three times the size of RAPTOR's. Considering the discussion TPER in Section 6.4 we believe method is reasonably efficient.
D AN EXAMPLE SIRERAG TREE
Using the question "who is the father of the artist who painted Head I?" as an example, we focus on the relevant part of the SIRERAG tree in Figure to conduct qualitative analysis.
For the multihop question in Figure ( answer: Nicholas Bacon), the MuSiQue corpus con- (correct tains one relevant paragraph stating, "Francis Bacon was born 22 January 1561 at York House
14

---

Published as a conference paper at ICLR 2025
Simiarity Tree
The provided cortaina blog aetiapiical ditals vartuss
matorioal figares fram thowet
courties inclucing
Relatedness Tree
ri bary ofluars por bollriars, wrilers, and other rotabde inciiduarr deralis Ernert Bamnet...
Franda Bacor: Eorn o Janaay 1861, son Sir Nicotta Bacon.
gerandogial
hutoiial fgunns
infomtttonn about aoveral indiidarra from difiooen lami los and
paricc..
Sie Robbrt Park, tel Baronet...
Chartes Barnn wax the eon of Heney sangham Btaiing, who
15. Frondis Bacoet Born or 22 onuary 1661, son ofr Casll for Bamn nu Buighey
sir Fransis Bariig nst Baronet
(Cooee Raacne making Willien Decil, hot fawns nlughhey, Nis
Bacen parnee landscape figar...
Fremci thace here iemary 1551 York Lionze Strand Lander,
Francia Bezon
Jaruary 1551
York House noar Skann in Loaton
Nichokaa Stcce (Land Keeper the Sead
Frandi flacon was
die Nithotas flaner,
recudd wite, fome (Cocke Busns for thed an the nead hummiia Authaay Cooks muchec' nommeed
Lord Kaeper of the Great Seal and Arme (Cooka)
threghay.
Heedi small and tampera on hardboard
panntng by
Figure 4: Relevant part of the SIRERAG tree for the question: "who is the father of the artist who painted Head 1?"
Table 10: Efficiency of using self-ask with SIRERAG. We show TPQ on MuSiQue, 2Wiki, and HotpotQA datasets.
Model
MuSiQue TPQ
2Wiki TPQ
HotpotQA TPQ
Average TPQ
Self-ask
2.72 4.53
2.21 3.07
2.29
2.41 3.63
SIRERAG self-ask
3.30
near the Strand in London, the of Sir Nicholas Bacon...". This paragraph is a leaf node of the similarity tree shown in Figure 4 The similarity tree for the entire MuSiQue corpus has two more mentions (both mentions in summary nodes) of Nicholas Bacon, of which reads: "r.nFran- cis Bacon: Born on 22 January 1561, son of Sir Nicholas Bacon and Anne Cooke Bacon..." The addition of relatedness tree adds aggregate (" FrannFsaaccis Bacon was the son of Sir Nicholas Bacon, Lord Keeper of the Great Seal, and Anne (Cooke) Bacon... Head is a small oil and tempera on hardboard painting by Francis Bacon, completed in 1948..."), and the other one is summary node ("The text provides detailed genealog- ical and biographical information about several individuals from different families and historical periods... Francis Bacon was the son of Sir Nicholas Bacon and Anne (Cooke) Bacon, making William Cecil, 1st Baron Burghley, his uncle..."). Thus, our similarity tree (RAPTOR tree) has three mentions of Nicholas Bacon, but none of them contains Head I information. SIRERAG has five mentions of Nicholas Bacon, and one of them (the proposition aggregate) contains Head informa- tion. This proposition aggregate groups several propositions together via the entity "Francis Bacon" Because this node the only retrieval candidate that fully matches the question, retrieving would maximize the chance of generating the correct answer. If the RAPTOR tree only, will
more mentions of Nicholas Bacon: one is in a proposition
15

---

Published as a conference paper at ICLR 2025
not have this retrieval candidate. We believe this is excellent example of how comprehensive knowledge integration process can enhance the performance of RAG in multihop reasoning.
E PERFORMANCE ON SINGLE-HOP QA, MULTIHOP-RAG, AND AMBIGUOUS QUESTIONS
To showcase the generality of SIRERAG more datasets of complex reasoning tasks, we run the comparison between SIRERAG and RAPTOR on single-hop questions and MultiHop-RAG dataset (Tang & Yang 2024). We also try ASQA dataset (Stelmakh et al.] 2022) that contains ambiguous factoid questions. For single-hop questions, we use MuSiQue dataset and collect all the decomposed questions of multihop queries. We filter out some decomposed questions if they are still multihop or are based on another question. As a result, we end up with 502 single-hop questions from MuSiQue. As for MultiHop-RAG, it is more recent dataset. We filter all unanswerable questions and randomly select 350 "comparison" queries and 350 "inference" queries, which forms a pool of 700 queries in total.
Moreover, the primary difference between the multihop QA datasets and ASQA is that ASQA re- quires LLMs to reason across multiple perspectives (e.g., disambiguated questions) of an ambiguous question and organize their generation into a coherent and detailed answer. We report scores on ASQA based on all 948 ambiguous questions of its development set.
Table 11 shows the performance scores of RAPTOR and our method. On the single-hop questions, SIRERAG still outperforms RAPTOR, but the lead narrows compared to the scores in Table Since all queries in MuSiQue involve at least two hops, we observe that an increased number of reasoning hops positively impacts SIRERAG'S performance. This because single-hop questions may not require comprehensive knowledge synthesis, as they only involve retrieving the relevant chunks for the single hop. However, with more hops, we not only need to retrieve relevant chunks but also MultiHop-RAG,
synthesize them comprehensively. SIRERAG also delivers better performance which echoes our main experiment.
Table II also displays STR-EM (string exact match), Disambig-Fi, and Disambiguation met- rics for ASQA dataset. Specifically, STR-EM and Disambig-Fi dissect RAG answers to ambiguous questions into multiple perspectíves and measure their accuracy with respect to each disambiguated question. Disambiguation Rouge serves as an overall statistic that incorporates both ROUGE (with respect the references) and accuracy scores. We SIRER AG delivers consistent improvement over RAPTOR, which again demonstrates the benefit of adopting SIRERAG on ambiguous queries.
Our comprehensive selection of datasets demonstrates the generality and contribution of SIRERAG across various complex reasoning tasks, with the most significant improvement observed in multihop QA.
F ADDITIONAL GRAPHRAG DETAILS
We adhere GraphRAG documentation construct its indexing structure and handle QA. Fol- lowing the recommendation, we use its "auto tuning" to generate domain adapted prompts for the creation of its knowledge graph. We use the default values for most hyperparameters, except that we set the response type to "a few words" and do not include covariates during indexing.
G LLM PROMPTS
The prompt we use to perform summarization on a cluster of nodes is "summarize the provided text, including as many key details needed". This prompt the same RAPTOR. In Section 3 use 'identify the high-level topic of this paragraph as concise as possible" to extract the topic of each passage. As mentioned in Section 4.1l the prompt used for identifying a two-level hierarchy for all chunks is shown in Figure 5S] As mentioned in Section 4.2] the LLM prompt for rewriting chunks shown in Figure 6, the prompt for extracting named entities from rewritten chunks is shown in Figure 7 and the prompt for extracting propositions is shown in Figure g
hhttps://iicrosoft..o wwt.tttttgttuuu..... io/graphrag/
16

---

Published as a conference paper at ICLR 2025
Table 11: QA performance on single-hop questions, MultiHop-RAGG and ASQA.
Single-Hop
MultiHop-RAG
ASQA
Model
EM
Fl
EM
F1
STR-EM
Disambig-F1
Disambiguation-Ruge
RAPTOR
73.90
80.67
85,43
86.37
51.47
40.30
42.72
AIRERAG
74.10
82.64
87.57
88.41
52.81
41.82
43.47
Instructions
1. Group sentences based on their tepic and abstractiveness.
2. For each sentence group define metadata consisting of topic and the abstractiveness level.
3. There are two abstractiveness levels:
a. low: sentences describe fine-grained details about the topic
high: sentences gives overview of topic, summarizing fine-grained details.
4. Ensure that each group consists of contiguous sentences.
5.
Respond using the below json format.
"group_1":
{
"sentences": [list sentence ids],
"topic": "topic of the group", "abstractiveness": "high"
"group_2":
"sentences": Ilist of sentence idsl, "topic": "topic of the group",
"abstractiveness": "Low"
"group_3":
"sentences": [list of sentence idsl, "topic": "topic of the group",
"abstractiveness": "high"
Figure 5: Prompt of identifying a two-level hierarchy for all candidate chunks.
17

---

Published as a conference paper at ICLR 2025
Previous paragraph from Document:
that leaves most visitors tongue-tied, is new list of the 50 best places to live in the United States. Men's Journal magazine describes Gualala as an öutpost of adventure lifestylein its latest edition, which goes on sale today. The magazine describes Gualala (pronounced wa-LA-la by locals) as one of the below-the-radat places to make a move on before the word gets There were five such citics. The others were Homer, Alaska; Newport, Vt.; Logan, Utah; and Walla Walla, Wash. Rolling Stone magazine's Jann Wenner publishes Men's Journal, which a paid circulation about 620,000. Gualala joined three other California communities on magazine's 's list: Santa Cruz, Mammoth Lakes and Bishop. We were looking for places that combined affordability, proximity to outdoor adventure and generally undiscovered quality of life,šaid Erica Kestenbaum, spokeswoman for Men's Journal. Instruction:
Gualala, the isolated Mendocino Coast with
Rewrite the below paragraph by resolving all entity coreferences with the preceding paragraph from document.
Resolve all inter-sentence pronoun references.
Make sure that all pronouns sentence refers some named entity with in the same sentence. Explicitly mention entity names wherever necessary to remove ambiguity from a sentence. Remember to make each sentence clear and unambiguous.
For each entity, use only the one most informative name.
Do generate anything except the rewritten paragraph.
Paragraph:
She said isolation played a factor. in Northern California, it's particularly difficult to find a beautiful coastal setting that isn't entirely overrun,šhe said. Gualala residents Monday were largely unaware of the magazine listing the attention could bring the old logging town turned tourist center. A few coastal residents chuckled about any notion of affordability, given influx of newcomers who't driven median housing price to $580.000 compared to median family income $47,778. Others recalled an era when the Gualala region was better known for the logging of ancient redwoods. marijuana growing and boisterous beer drinking at the historic Gualala Hotel. Still there was certain pride to the magazine's 's designation. Yvette White, 25-ycar resident who works the Gualala Sport; Tackle shop, said she's proud her made it on the list.
Output:
Erica Kestenbaum said isolation played a factor. Northern California, it's particularly difficult to find beautiful coastal setting that isn't entirely overrun,Erica Kestenbaum said. Gualala residents Monday were largely unaware of the Men's Journal magazine listing the attention could bring to old logging town turned tourist center.
coastal residents of Gualala chuckled about any notion of affordability. given an influx of newcomers who've driven the Gualala's median housing price to $580,000 compared to the median family income of $47.778. Other Gualala residents recalled an era when the Gualala region better known for the logging of ancient redwoods, marijuana growing and boisterous beer drinking the historic Gualala Hotel. Still there was certain pride the Men's Journal magazine's 's designation. Yvette White, a 25-ycar Gualala resident who works at the Gualala Sport; Tackle shop, said she's proud her town made it on the list. Previous paragraph from Document: [previous paragraph]
Instruction:
Rewrite the below paragraph by resolving all entity coreferences with the preceding paragraph from document.
Resolve all inter-sentence pronoun references.
named entity with in the sentence. Explicitly mention entity names wherever necessary remove ambiguity from sentence. Remember to make each sentence clear and unambiguous.
Make sure that all pronouns in sentence refers
For each entity, use only the one most informative name.
Do not generate anything except the rewritten paragraph.
Paragraph: [paragraph] Output:
Figure 6: Prompt for rewriting a paragraph (e.g., document chunk) by resolving entity corefer- ences.
18

---

Published as a conference paper at ICLR 2025
Extraci sumed entiiies from documeni, Also genernse the type for each eniity. Instructions
Generste only the most informative same for each nomed entity, Example: if John P. Parker, John Parker are coveferential, only generate John Parker. your best understanding best domain pangrraph decide appropriate entlly types.
Respond using joon formut provided below.
"nl":("naxo": name", "type", "n2":(>
Below examll reference. Paragraph: Tucked
Cevergy-AAHH-in phase study tirzegatide metabolic dystonction-assiisttt steatoheyiitis (MASH). adolts nonslosholic steatoheputhis (NASH), Ouiput:
earnings report, compony rovealed
"n1": ("name": "Ell Lilly", "type": "Organization"..
"n2": ("name": "Synergy-UsS"" "type": "Ciinical Trial"),
("nans": "tárzopatide" "type": "Drug").
"n5":
("name": "nonalcoholic steatohepatitis" "type": "Disease"),
"n6": ("hame": "estabolic systunetion-ssoccatee steatahepatstis", "type": "Disease"),
"n7": ("nase": "year-end
report", "Docunent"))
Figure 7: Prompt for extracting entities from document (e.g., rewritten chunk).
19

---

Published as a conference paper at ICLR 2025
Extrict all fiess from the document. For each fact, also generate all semuntic triplets.
Instructions
Consistenily
entlly
triplets,
Avoid pronoums ambiguns refermors in facts triplets. Instead, directly include all relevant named cmiities facts.
-nnaree that each semanic tiplet contains head esrny, predicate and tall estit.
Eneurv that leas (preferably buth) enclty each semantic triplet is preseni in the given entikees list. provided below:
Reqood using
"E1"S(
"fact:: "A factuel staterent describing important information (preferably about some entities) from the paragraph",, "triplets: CC'entity 1", "yredicaate", "entity 2"), ('entity 1", "predicate", "entity 3*]]
reference.
Paragraph:
semaglutide franchise, Lilly' tirzepatide beginning come its sales and anid anempts to show the dual GIPYGLP-1 agonist can strike out beyond dibbetes and dbesity. As Mounjaro, lir lizzepatide won its lina FDW nod typo 2 dibeses back in May 2022. As obesity approval followed last November, with that formulation tizepanide adopting the commerciaa maniker increase
buttle
regurds
Zepoound. 2023's fourth Yearco
Mounjaro generated whupping $22 billion sales, around 55.2 billion
$279 million pulled down camings release Tuesday, Zepbound.
drug brought for its part, generated $175.8 million during its first quarter on the market. Overall, Lilly recled in around $9.4 bullion in fourth fourth-quanee quaster sales, growing 28% over the 57,3 billion made for the quaster 2022
during
Entiies: Eli Lilly, Novo Nordisk, Tireppaide. Semaglatide, GLP-I, 1, GIP FDA. Mounjoro, Zepbound Ourput:
"M1"s [
"fact": "ELL lilly's tirzepatide is corpeting with Neve Norcisk's seraglutide franchise.
"triplats": [['E11 Lilly", "corpating with", "Nova Nordisk"), " ('Tirzapatid"", "is compoting with", "Somaglutide"]]
"P?":
"fact': "EIs Lilly is trying to show tirzesatide, the dual GIP/CLP-1 agonist, can strike out beyand diabates and abosity.", "triplats": [L'EL1 Lilly", "is trying to show", "Tiraepacide"), j, L'Tirzepatide", "is a", "eual GIP/CLP-1 agonist"), beyons", "Diabetes"], ("Tiezapatiid"" "can beyand", "Obexity"]1
[TTrzegatide"
"33% [
"Fact"; "Tirzepatide, under the brand name Mounjaro, recelved its First FDA appreval For Type diabates in May 2072.", "triplots' [["Trreeatid"", "branded "Hounjaro"), L"Hounjaro", "won", "FDA ["FAA approval", "for", "Type diabetes"), DFFDA approval", "was In", "Nay 2822"]]
1r4t [
"fatt: "Tirzepatide,
brand Zephaund, received obesity approval "triplat": [["iizenatide" branded as", "7ighound"), 1 ["7aybound", "received", "Obesiit approval"] "],
2022."
L"Obesity approval", "was in", "Novenber 2822*]]
"ist:
"ract": Jaoun generated $2.2
sales in the rourth quarter of 2823, an eight-rold increase
during the same perfod in 2822.
"triplats: [['Hounjaro, "2823's Fourth quarter sales", *$2.2 billion sales"), million'1]
["Mounjaro", "2822's
quarter sales",
"ftt: {
"fact": "Hounjaro brought in around 55, 2 billion in year-to-de to--tte 2823, Lilly said earnings release Tuesday", "rippets": (L'Mounjar"", "2023 sales year-to-date", "$5.7 bil1
"E7": {
Factt: "Tephound generated $175.8 millioon in sales in its first quarter on the narket.", 'triplets': (["Zepbound", "First quarter sales", "$175.0 million'1]
"fat: {
'fact": "Eli Lilly's fourth-qurteer sales
$9.4 billion, 28% increase the $7.3 billion during the "triplata": [["F1 iilly, *2833 fourth-qurter sales", "39.4 billiion ['tli Lilly", "2622 fourth-querter sales", "57.3 billion, "J]
period
Figure 8: Prompt for extracting propositions (e.g., facts) and their corresponding entities from document.
20


Reviews:
[{"review_id": "8ocvaV158c", "reviewer": "Reviewer_ymw3", "summary": "The paper introduces SiReRAG, which proposes to consider both similar and relatedness information when creating retrieval indices that address queries with multi-hop reasoning. The paper gives motivations to demonstrate the bottleneck of solely modelling relatedness or similar information only. For similar information, the paper constructs a similarity tree based on recursive summarization, while for relatedness, SiReRAG extracts propositions and entities from text, and groups them via shared entities to construct a relatedness tree. Experimental results show SiReRAG considerably improves over other baselines like HippoRAG, GraphRAG and RAPTOR when evaluated on various multi-hop QA datasets.", "questions": "1)\tThe motivation example in Figure 1 is a bit misleading. The authors suggest that the question “Who is the father of the artist who painted Head I?” has three hops of reasoning, whereas it only requires two hops, i.e. finding the artist in 1st hop and then identifying the author in the 2nd hop. This example can be a bit confusing to the reader on what the overall motivation of the approach is.\n\n2)\tIt is very surprising to see almost similar performance for models of highly different capabilities such as GPT-3.5-turbo and GPT-4o. The authors should provide more insight/analysis on why this is happening? Also, what is the performance when the LLM used is an open-source model such Llama-3 8B.\n\n3)\tIt would be interesting if the authors could show performance separately based on the number of reasoning hops present in the question. Also, does the approach show any benefits over RAPTOR for single hop/simple queries?\n\n4)\tThe authors should also evaluate more recent multi-hop QA datasets, such as FanOutQA or FreshQA. All the datasets considered are pre-2022, raising concerns about leakage into LLM pretraining data. \n\n5)\tThe authors should also consider adding a few qualitative analysis examples that demonstrate how and “why” (i.e. which part of the method helps) SiReRAG improved over RAPTOR due to incorporating the relatedness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1)\tThe approach shows considerable performance improvements over RAPTOR on a variety of multi-hop QA datasets.\n\n2)\tThe experimental ablation settings are thorough and show the benefit of different design choices made by the authors for the SiReRAG approach.", "weaknesses": "1)\tThe paper does read like incremental work over RAPTOR, and it is hard to be convinced that the paper has enough novelty for acceptance at ICLR.\n\n2)\tSome important baselines such as the closed book approach, i.e. directly getting the final answer from the LLM without any retrieval, or iterative retrieval, such as Self-Ask [1] or DSPy [2] are missing. Also, the authors should include these baselines when doing the inference latency comparison.\n\n3)\tThe writing needs to be improved in Sections 1 and 3 of the paper since it’s not easy to grasp the main intuitions or motivations of SiReRAG. \n\n[1] Measuring and Narrowing the Compositionality Gap in Language Models\n\n[2] DSPY: COMPILING DECLARATIVE LANGUAGE MODEL CALLS INTO SELF-IMPROVING PIPELINES", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "O7GMvJRFYj", "reviewer": "Reviewer_gA7K", "summary": "This paper presents SiReRAG: a new indexing and retrieval technique that takes into account both \"semantic similarity\" and \"entity relatedness\" for answering complex multi-hop queries.\n\nThe suggested technique is built mostly on top of another technique called RAPTOR (which indexes text chunks on similarity only), but augments it by: first, extracting entity-level facts or propositions s (which are single facts/statements about entities), using LLMs like chatGPT or LLama-instruct, then applying the RAPTOR pipeline to extracted propositions to create relatedness trees akin to the original RAPTOR similarity trees. The final method merges the output from the two trees (one from original RAPTOR built on text chunks, and another from RAPTOR built on extracted and aggregated propositions).\n\nThe paper includes comprehensive evaluations and shows significant improvement over baseline methods that use only similarity or relatedness features.", "questions": "- The reasoning behind why HippoRAG does significantly better for 2Wiki dataset is not very clear in my opinion: I think what might be happening here is the following: when the RAPTOR text clustering pipeline is applied to the extracted entity-relatedness propositions, it treats the full propositions as text, completely ignoring the underlying structure of the proposition/fact (which is usually of the form Subject/Predicate/Object). That destroys certain information from the triple. My guess is that if we modify the RAPTOR encoder to encoder these features separately (e.g. with SPO markers tokens or with different encoders altogether), then we could see that we're able to encode the triple structure better, and we might be able to recover that difference.\n\n- It would be very informative to share more information about the average number of candidates considered by the method compared to the baseline methods per query. This would show the computational increase compared to the baseline.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "Originality: The work presented in SiReRAG is original in the sense it combines ideas from two existing methods/philosophies about indexing, similarity and entity-relatedness into one solution.\n\nQuality: The thought process and reasoning of the paper is mostly intuitive and simple (merge two ideas that have been shown to improve performance), the experimental section supports this reasoning and is somewhat comprehensive with high quality and well-studied datasets. \n\nClarity: The paper is mostly well-written and easy to follow, with certain exceptions are included in the Weaknesses sections. The claims of the authors about coverage of relatedness or similarity only in table 1 serves as a good benchmark and motivation for the work.\n\nSignificance: I consider this method to be an incremental addition on top of RAPTOR, The main metrics reported show significant improvement over the baseline methods that use only one signal.", "weaknesses": "- The paper is not very clear regarding how to merge the result from the two trees (similarity and relatedness trees), but I might have missed this.\n\n- The metric \"time per retrieval pool size\" which is used to quantify the time of the SiReRAG vs other baselines method seems very contrived and not at all relevant, since it seems to say more about the number of candidates (the denominator) than about the time spent answering a query. Which means that the number of candidates generated by this method can be an order of magnitude more than the baseline methods in some cases, which can be prohibitive in many use-cases.\n\n- The main body of the paper has only two examples, without much detail: I recommend adding a full simple example for how the relatedness tree would look for a simple paragraph, as it's more difficult to visualize the entity-related tree as opposed to the similarity/summary between text chunks.\n\n- The suggested method ends up performing badly on the more structured datasets (2Wiki): this indicates that the way it's encoding the relatedness is not representative enough to capture the structure of the graph/triples/facts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jv49SRSWXg", "reviewer": "Reviewer_gpao", "summary": "The paper addresses the retrieval component of the retrieval-augmented generation. It argues that existing RAG methods tend to focus exclusively on semantic similarity or relational links, leading to suboptimal performance in complex, multi-hop reasoning tasks. The paper motivates this with a coverage study to show that similarity or relatedness can only return a few correct entity connections, and the return results are half overlapped. The paper proposes SIRERAG, which is designed to optimize information retrieval by indexing data based on similarity and relatedness. The index is based on a similarity and relatedness tree using a a method similar to RAPTOR's.\n\nExperiments with three datasets (MuSiQue, 2WikiMultiHopQA, and HotpotQA) showed improvements in EM and F1 scores compared to previous indexing methods, which include similarity or relatedness. The gain in performance had a slight negative impact on the inference time compared to a similarity-only method due to a larger retrieval pool size. Further analyses showed that the relatedness tree was indeed helpful.", "questions": "1. It is unclear whether you re-ran the baseline or obtained the results from the original papers. I could not find the exact numbers reported elsewhere in Table 4.\n2. What was the retrieval size of the baselines?\n3. Overall, I found that TPRS was not meaningful. Wasn't it simply a result of whether the documents were short or long?\n\n------\nAfter the discussion, the score was raised from 6 to 8.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper's main idea is well-motivated and supported by its preliminary experiments (Table 1). I found that its arguments were well-articulated and supported by existing literature, yet the hard evidence provided in Table 1 was a helpful addition.\n2. The paper acknowledged an alternative approach in some steps and some results to reject the alternatives (Sections 4.1 and 4.3).\n3. The method proposed in the paper consistently improved over three multi-hop reasoning datasets (Tables 4 and 6). In addition, ablation analysis also justified the need for the relatedness tree (Table 5).\n4. The paper included an inference time experiment, showing that inference time increased with pool size.", "weaknesses": "1. Although not related directly to the paper, the paper should still acknowledge the state-of-the-art multi-hop reasoning method such as [Open-RAG](https://openragmoe.github.io/).\n2. Section 4.3, which described an alternative method rather than the main method, did not explain flattened indexing well. Figure 2 showed an \"+\" sign but was not technical enough to confirm what was being done.\n3. A few prompts were not provided, such as the summary prompt (maybe similar to RAPTOR?), the topic extraction prompt (Section 3), and the hierarchy prompt (Section 4.1).\n4. Some missing details of the baseline might be crucial to interpret the results (see Questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Fo3cCCKzqb", "reviewer": "Reviewer_F1mV", "summary": "Indexing is critical for IR or RAG systems. In this paper, the authors propose a RAG indexing method which considers both semantic similarity and relatedness for better retrieval, named SIRERAG. Specifically, for similarity-based indexing this paper constructs a similarity tree via recursive summarization, and for relatedness this paper constructs a relatedness tree via entity and proposition extraction and grouping. Experimental results show some performance improvement on multi-hop QA datasets over previous similarity-based baselines and relatedness-based baselines.", "questions": "1. The experimental results on other multi-hop reasoning tasks.\n2. The performance using other relatedness-based trees (e.g., entity-based, or entity pair-based)", "limitations": "", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "In general, I think it is helpful to leverage multiple types of relevance for better retrieval results, and the experimental results also verify its effectiveness. It is also helpful to investigate and identify that similarity-only is not enough for complex QA tasks (although this is not a new discovery as many previous studies in traditional IR studies).", "weaknesses": "1. Because the similarity-based indexing component just follows RAPTOR (Sarthi et al., 2024), I think the main contribution of this paper maybe the relatedness-based indexing component.  Unfortunately, I found the relatedness-based indexing algorithm is ad hoc. Firstly, the relatedness are modeled using entity-specific propositions, which I think is over-specialized to multi-hop QA tasks(where the hop is just entity-entity associations and this is why the proposed method can achieve performance improvements). However, I think entity-specific propositions may not a good decision for many other complex reasoning tasks, and the authors should explain and verify the effectiveness and generality in more tasks. Secondly, I found there are many heuristic decisions, such as how to filter proportions, how to resolve entity references, etc.\n2. The similarity and relatedness trees are constructed and used independently, which I think is  straightforward, it is important to consider the interaction between different relevance scores.\n3. There are many multi-hop QA baselines, such as iterative RAG-based, agent-based, etc. The authors should compare them for more convincing experimental results.\n4. The writings of this paper should be improved.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)