GeAR: Graph-enhanced Agent for Retrieval-augmented Generation
Authors: Zhili Shen, Chenxin Diao, Pavlos Vougiouklis, Pascual Merita, Shriram Piramanayagam, Damien Graux, Dandan Tu, Zeren Jiang, Ruofei Lai, Yang Ren, Jeff Z. Pan
Accepted at: acl25

GEAR: Graph-enhanced Agent for Retrieval-augmented Generation

Zhili Shen† Chenxin Diao† Pavlos Vougiouklis† Pascual Merita†
Shriram Piramanayagam Enting Chen Damien Graux
Andre Melo Ruofei Lai Zeren Jiang Zhongyang Li
Ye Qi Yang Ren Dandan Tu Jeff Z. Pan
Huawei Technologies Co., Ltd.
Edinburgh, United Kingdom
{zhilishen, chenxindiao, pavlos.vougiouklis, pascual.merita}@huawei.com
{renyang1, jeff.pan}@huawei.com

Abstract

Retrieval-augmented Generation (RAG) relies on effective retrieval capabilities, yet traditional sparse and dense retrievers inherently struggle with multi-hop retrieval scenarios. In this paper, we introduce GEAR, a system that advances RAG performance through two key innovations: (i) an efficient graph expansion mechanism that augments any conventional base retriever, such as BM25, and (ii) an agent framework that incorporates the resulting graph-based retrieval into a multi-step retrieval framework. Our evaluation demonstrates GEAR’s superior retrieval capabilities across three multi-hop question answering datasets. Notably, our system achieves state-of-the-art results with improvements exceeding 10% on the challenging MuSiQue dataset, while consuming fewer tokens and requiring fewer iterations than existing multi-step retrieval systems. The project page is available at https://gear-rag.github.io.

1 Introduction

Retrieval-augmented Generation (RAG) has enhanced the performance of Large Language Models (LLMs) (OpenAI, 2023) in Question Answering (QA) tasks (Lewis et al., 2020). While effective for simple queries, multi-hop QA presents a more complex challenge, requiring reasoning across several passages or documents. Consider the example in Table 1, where finding the correct answer requires building a 3-hop reasoning chain starting from the question’s main entity (i.e. “Stephen Curry”). A base retriever cannot, by design, retrieve all necessary information in a single step.

To address these complex reasoning requirements, researchers have increasingly turned to graph-based approaches (Fang et al., 2024; Li et al., 2024; Edge et al., 2024; Gutierrez et al., 2024; Liang et al., 2024). By extracting entities, atomic facts, or semantic triples (Li et al., 2024; Fang et al., 2024; Gutierrez et al., 2024), these graphs can establish more direct pathways for multi-hop reasoning. For instance, HippoRAG extracts triples from passages to form a knowledge graphs and employs a variant of PageRank for passage retrieval (Gutierrez et al., 2024). GraphReader uses an LLM agent with graph-navigating operations to explore the resulting graph structure (Li et al., 2024). TRACE relies on an LLM to iteratively select triples for constructing reasoning chains, which then either ground answer generation, or filter irrelevant documents from initially retrieved results (Fang et al., 2024). However, recursively prompting LLMs to traverse graphs remains computationally expensive, particularly as search spaces expand.

In this paper, we present GEAR, a Graph-enhanced Agent for Retrieval-augmented generation. During the offline stage, we align passages with their extracted triples to create interconnected indices. This alignment allows passages to be connected through graphs of triples. GEAR features a graph-based passage retrieval component

<table>
  <tr>
    <th colspan="2">In what year did Stephen Curry’s father join the team from which he started his college basketball career?</th>
  </tr>
  <tr>
    <td>Base Retriever Results</td>
    <td>works with<br>Eric Collins<br>Dell Curry<br>college team<br>Virginia Tech<br>1982</td>
  </tr>
  <tr>
    <td>son of<br>Stephen Curry<br>Dell Curry</td>
    <td>expansion<br>Dell Curry<br>born in<br>Stephen Curry<br>1988</td>
  </tr>
</table>

Table 1: A motivating example of a multi-hop question where a base retriever cannot, by design, retrieve all necessary information in a single step. Graph expansion (see §4.2), which we incorporate within GEAR, enables retrieval of subsequent hops and guides the system toward the correct answer without using an LLM.
called SyncGE. Unlike previous approaches that require expensive LLM calls for graph exploration, our method uses an LLM only to locate initial nodes (triples) and then employs a generic semantic model to expand the triple sub-graph by exploring diverse triple beams. Additionally, GEAR uses multi-hop context retrieved by SyncGE to construct a memory that summarises information for multi-step retrieval.

Our work refines the neurobiology-inspired paradigm proposed by Gutierrez et al. by modeling the communication between the hippocampus and neocortex during episodic memory formation. In our design, an array of proximal triples functions as a memory gist learned through the hippocampus within one or a few shots (iterations), which is then projected back to the neocortex for later recall stages (Hanslmayr et al., 2016; Griffiths et al., 2019). We highlight the complementary potential between our graph retrieval approach and an LLM, which, within our system, emulates the synergy between the hippocampus and neocortex, offering insights from a biomimetic perspective.

We evaluate the retrieval performance of GEAR on three multi-hop QA benchmarks: MuSiQue, HotpotQA, and 2WikiMultihopQA. GEAR pushes the state of the art, achieving significant improvements in both single- and multi-step retrieval settings, with gains exceeding 10% on the most challenging MuSiQue dataset. Furthermore, we demonstrate that our framework can address multi-hop questions in fewer iterations while consuming significantly fewer LLM tokens. Even with a single iteration, GEAR offers a more efficient alternative to other iterative retrieval methods, such as HippoRAG w/ IRCoT. Our contributions can be summarised as follows:

• We introduce a novel graph-based retriever, SyncGE, which leverages an LLM for locating initial nodes for graph exploration and subsequently expands them by diversifying beams of triples that link multi-hop passages.

• We incorporate this graph retrieval method within an LLM-based agent framework, materialising GEAR, achieving state-of-the-art retrieval performance across three datasets.

• We conduct comprehensive experiments showcasing the synergetic effects between our proposed graph-based retriever and the LLM within the GEAR framework.

2 Related Work

Our work draws inspiration from two branches of research: (i) retrieval-augmented models for QA and (ii) multi-hop QA using combinations of LLMs with graphical structures.

2.1 Retrieval-augmented Models for QA

Lewis et al. first showcased the benefits of augmenting language models’ input context with relevant passages. Recent work by Wang et al. and Shen et al. explores query expansion approaches, generating pseudo-documents from language models to expand the content of original queries. Subsequent frameworks, beginning with IRCot, have investigated the interleaving of retrieval and prompting steps, allowing each step to iteratively guide and refine the other (Trivedi et al., 2023; Jiang et al., 2023; Su et al., 2024).

2.2 Multi-hop QA with LLMs and Graphs

Several architectures have introduced an offline indexing phase to form hierarchical passage summaries (Chen et al., 2023; Sarthi et al., 2024; Edge et al., 2024). However, summarisation must be repeated when adding new data, making knowledge base updates computationally expensive. Recent approaches have leveraged structured knowledge to address multi-hop QA challenges with LLMs (Park et al., 2023; Shen et al., 2024b; Li et al., 2024; Gutierrez et al., 2024; Liang et al., 2024; Wang et al., 2024). GraphReader, TRACE and HippoRAG propose offline methods for extracting entities and atomic facts or semantic triples from passages (Li et al., 2024; Fang et al., 2024; Gutierrez et al., 2024). This allows chunks containing the same or neighbouring entities to construct a graph of indexed passages. TRACE relies on an LLM to iteratively select triples for reasoning chains, which ground answer generation or filter retrieved results. However, its search space is limited by pre-filtered candidate lists for each query. Li et al. employ an LLM agent that selects from a set of predefined actions to traverse knowledge graph nodes in real time given an input question. Liang et al. later introduced additional graph standardisation, including instance-to-concept linking and semantic relation completion. However, this approach heavily depends on associating triples with pre-defined concepts for logical form-based retrieval.

HippoRAG leverages an alignment of passages
and extracted triples to retrieve passages based on the Personalized PageRank algorithm (Gutierrez et al., 2024). While achieving improvements for single- and multi-step retrieval (when coupled with IRCoT (Trivedi et al., 2023)), it remains agnostic to the semantic relationships of extracted triples. In this paper, we leverage a similar alignment of passages and extracted triples but introduce a new graph-based retrieval framework that uses a small semantic model for exploring multi-hop relationships. Our framework considers the contributions of all triple elements participating in reasoning chains, offering a more robust solution for associating questions with triple reasoning chains.

3 Preliminaries

Let \( \mathbf{C} = \{c_1, c_2, \ldots, c_C\} \) be an index of passages and \( \mathbf{T} = \{t_1, t_2, \ldots, t_T : t_j = (s_j, p_j, o_j)\} \) be another index representing a set of triples associated with the passages in \( \mathbf{C} \) s.t. \( \forall t_j \in \mathbf{T} \exists! c_i \in \mathbf{C} \), where \( s_j, p_j \) and \( o_j \) the respective subject, predicate and object of the \( j \)-th triple. In this setup, each triple is uniquely linked to exactly one passage, and a passage can potentially be associated with multiple triples.

Given an input query \( q \) and an index of interest \( \mathbf{R} = \{r_1, \ldots, r_R\} \), retrieving items from \( \mathbf{R} \) relevant to \( q \) can be achieved by using a base retrieval function \( h_{base}^k(q, \mathbf{R}) \subseteq \mathbf{R} \) that returns a ranked list of \( k \) items from \( \mathbf{R} \) in descending order, according to a retrieval score. BM25 or a conventional dense retriever can serve as a base retrieval function, without requiring any multi-hop capabilities.

Our goal is to retrieve relevant passages from \( \mathbf{C} \) that enable a retrieval-augmented model to answer multi-hop queries (Lewis et al., 2020). To this end, we introduce GEAR, which is a graph-enhanced framework of retrieval agent (see Figure 1).

4 Retrieval with Synchronised Graph Expansion

Given an input query \( q \), let \( C'_q = h_{base}^k(q, \mathbf{C}) \) be a list of passages returned by the base retriever. Given this list, \( C'_q \), our goal is to derive relevant multi-hop contexts (passages) by retrieving a sub-graph of triples that interconnect their source passages. Two challenges arise in materialising such sub-graph retrieval: (i) locating initial triples (i.e. starting nodes) \( T_q \), and (ii) expanding the graph from these initial triples while reducing the search space. The following sections address these challenges within GEAR.

![System Architecture diagram showing the retrieval and multi-step extension processes](page_768_120_670_495.png)

Figure 1: System Architecture. The left section (blue) depicts SyncGE (detailed in §4), while the the right section (orange), completes the GEAR system (described in §5). A visual step-by-step walk-through of the system is presented in Appendix C.

4.1 Knowledge Synchronisation

We describe a knowledge Synchronisation (Sync) process to locate initial nodes for graph expansion. We first employ an LLM to read \( C'_q \) (see Appendix K.2) and summarise knowledge triples that can support answering the current query \( q \):

\[
T'_q = \text{read}\left(C'_q, q\right).
\]

\( T'_q \) is a collection of triples to which we refer as proximal triples. Initial nodes \( T_q \) for graph expansion are identified by linking each triple in \( T'_q \) to a triple in \( \mathbf{T} \), using the tripleLink function:

\[
T_q = \{t_i | t_i = \text{tripleLink}(t'_i) \forall t'_i \in T'_q\}.
\]

The implementation of tripleLink can vary. However, in this paper we consider it to be simply retrieving the most similar triple from \( \mathbf{T} \) using \( h_{base}^1(t'_i, \mathbf{T}) \forall t'_i \in T'_q \).

4.2 Diverse Triple Beam Search

We borrow the idea of constructing reasoning triple chains (Fang et al., 2024) for expanding the graph,
Algorithm 1 Diverse Triple Beam Search
Input: q: query
    b: beam size
    l: maximum length
    score(·, ·): scoring function
    {t_1, t_2, ..., t_n}: initial triples
    γ: hyperparameter for diversity
1: \( B_0 \leftarrow [] \)
2: for \( t \in \{t_1, t_2, ..., t_n\} \) do
3:     s ← score(q, [t])
4:     \( B_0 \).add((s, [t]))
5: \( B_0 \leftarrow \text{top}(B_0, b) \)
6: for \( i \in \{1, \ldots, l - 1\} \) do
7:     \( B \leftarrow [] \)
8:     for \( (s, T) \in B_{i-1} \) do
9:         V ← []
10:        for \( t \in \text{get_neighbours}(T.\text{last}()) \) do
11:           if exists(t, B_{i-1}) then
12:              continue
13:              s' ← s + score(q, T ∘ t) # concat
14:              V.add((s', T ∘ t))
15:         sort(V, descending)
16:         for \( n \in \{0, \ldots, V.\text{length}() - 1\} \) do
17:             \( (s', T \circ t) \leftarrow V[n] \)
18:             s' ← s' × e^{-\frac{\min(n, \gamma)}{\gamma}}
19:             \( B_i \).add((s', T ∘ t))
20:     \( B_i \leftarrow \text{top}(B, b) \)
21: return \( B_l \)

and present a retrieval algorithm: *Diverse Triple Beam Search* (see Alg. 1).

We maintain top-\( b \) sequences (beams) of triples and the scores at each step are determined by a scoring function. In this paper, we focus on leveraging a dense embedding model to compute the cosine similarity between embeddings of the query and a candidate sequence of triples, leaving other implementations of the scoring function for future work (see Section 9).

Considering all possible triple extensions at each step, in a Viterbi decoding fashion, would be intractable due to the size of \( \mathbf{T} \). Consequently, we define the neighbourhood of a triple as the set of triples with shared head or tail entities (i.e. get_neighbours in Alg. 1). During each expansion step, we only consider neighbours of the last triple in the sequence, and avoid selecting previously visited triples (i.e. exists in Alg. 1) to further reduce the search space.

While regular beam search can reduce the search space, it is prone to producing high-likelihood sequences that differ only slightly from one another (Ippolito et al., 2019; Vijayakumar et al., 2018). Our algorithm increases the diversity across beams to improve the recall for retrieval. In detail, for each beam, we sort candidate sequences extended from that beam in descending order, and weight their scores based on their relative positions. Candidate sequences that are ranked lower, within a beam, will receive smaller weights. Consequently, the resulting top-\( b \) beams at each step are less likely to share the same starting sequence.

The top-\( b \) returned sequences are flattened in a breadth-first order. Each triple in the resulting list is then mapped to its source passage. This alignment between triples and passages is described in more detail in Section 3. Let \( \tilde{C}_q \) be the list of unique passages after alignment. The output of our graph expansion is then given by the Reciprocal Rank Fusion (RRF) (Cormack et al., 2009) of \( \tilde{C}_q \) and the initial \( C_q' \) list of passages:

\[
C_q = \text{RRF}\left(\tilde{C}_q, C_q'\right).
\]

We refer to this method for retrieving passages as **Synchronised Graph Expansion** (**SyncGE**).

5 Multi-step Extension

We further present an agentic framework that models a human-like information-seeking process through multi-turn interactions with the graph-enhanced retriever. The resulting agent is referred to as GEAR. We focus on:

• maintaining a gist memory of proximal knowledge obtained throughout the different steps

• incorporating a similar synchronisation process that summarises retrieved passages in proximal triples to be stored in this multi-turn gist memory

• determining if additional steps are needed for answering the original input question

Within this multi-turn setting, the original input question \( q \) is iteratively decomposed into simpler queries: \( q^{(1)}, \ldots, q^{(n)} \), where \( q^{(1)} = q \) and \( n \in \mathbb{N} \) represents the number of the current step. For each query \( q^{(n)} \), we use the graph retrieval method introduced in Section 4 in order to retrieve relevant passages \( C_{q^{(n)}} \).
5.1 Gist Memory Constructor
To facilitate the multi-step capabilities of our agent, we introduce a gist memory, \( \mathcal{G}^{(n)} \), which is used for storing knowledge as an array of proximal triples. At the beginning of the first iteration, the gist memory is empty. During the \( n \)-th iteration, similar to the knowledge synchronisation module explained in Section 4.1, we employ an LLM to read a collection of retrieved paragraphs \( C_{q^{(n)}} \) and summarise their content with proximal triples:

\[
\mathbf{T}_{q^{(n)}}^{\mathcal{G}} = \begin{cases}
\text{read}\left(C_{q^{(n)}}, q\right), & \text{if } n = 1 \\
\text{read}\left(C_{q^{(n)}}, q, \mathcal{G}^{(n-1)}\right), & \text{if } n \geq 2
\end{cases}
\]

Apart from the first iteration where Eq. 1 and 4 are identical, the inclusion of the memory in the read operation differentiates the construction of proximal triples produced at the subsequent steps compared to the ones from Eq. 1. \( \mathcal{G}^{(n)} \) maintains the aggregated content of proximal triples s.t.

\[
\mathcal{G}^{(n)} = \left[ \mathbf{T}_{q^{(1)}}^{\mathcal{G}} \circ \cdots \circ \mathbf{T}_{q^{(n)}}^{\mathcal{G}} \right],
\]

where \( \circ \) defines the concatenation operation. The triple memory serves as a concise representation of all the accumulated evidence, up to the \( n \)-th step.

We believe the process introduced by the read step along with the information storage paradigm served by the gist memory, aligns well with the communication between the hippocampus and neocortex. The combination of the two establishes the synergetic behaviour between our graph retriever and the LLM that we seek to achieve within GEAR.

5.2 Reasoning for Termination
After updating \( \mathcal{G}^{(n)} \), we assess whether it contains sufficient evidence to answer the original question through an LLM reasoning step:

\[
a^{(n)}, r^{(n)} = \text{reason}(\mathcal{G}^{(n)}, q),
\]

where \( a^{(n)} \) denotes the query’s answerability given the evidence in \( \mathcal{G}^{(n)} \), and \( r^{(n)} \) represents the relevant reasoning. When the query is deemed answerable, the system concludes its iterative process.

5.3 Query Re-writing
The query re-writing process leverages an LLM that incorporates three key inputs: the original query \( q \), the accumulated memory, and crucially, the reasoning output \( r^{(n)} \) from the previous step. This process can be formally expressed as:

\[
q^{(n+1)} = \text{rewrite}\left(\mathcal{G}^{(n)}, q, r^{(n)}\right),
\]

where \( q^{(n+1)} \) represents the updated query, which serves as input for the retriever in the next iteration.

5.4 After Termination
GEAR aims to return a single ranked list of passages. Given the final gist memory \( \mathcal{G}^{(n)} \) upon termination, we link each proximal triple in \( \mathcal{G}^{(n)} \) to a list of passages as follows:

\[
C_{t_j} = \text{passageLink}\left(t_j, k\right),
\]

where \( j \in \{1, \ldots, |\mathcal{G}^{(n)}|\} \). Similar to tripleLink, passageLink is implemented by retrieving passages with a triple as the query (see Appendix B.2). The final list of passages returned by GEAR is the RRF of the resulting linked passages and passages retrieved across iterations:

\[
C_q^{(n)} = \text{RRF}(C_{t_1}, \ldots, C_{t_{|\mathcal{G}^{(n)}|}}, C_{q^{(1)}}, \ldots, C_{q^{(n)}}).
\]

All relevant prompts for the read, reason and rewrite steps are provided in Appendix K.2.

6 Experimental Setup
We evaluate our framework on three open-domain multi-hop QA datasets: MuSiQue (Trivedi et al., 2022), HotpotQA (Yang et al., 2018), and 2Wiki-MultiHopQA (2Wiki) (Ho et al., 2020). For MuSiQue and 2Wiki, we use the data provided in the IRCoT paper (Trivedi et al., 2023) which includes the full corpus, while for HotpotQA, we follow the same setting as HippoRAG (Gutierrez et al., 2024) to limit experimental costs. More details are provided in Appendix A.

We measure both retrieval and QA performance, with our primary contributions focused on the retrieval component. For retrieval evaluation, we use Recall@k (R@k) for \( k \in \{5, 10, 15\} \), showing the percentage of questions where the correct entries are found within the top-\( k \) retrieved passages. We include an analysis about the selected recall ranks in Appendix D. Following standard practices, QA performance is evaluated with Exact Match (EM) and F1 scores (Trivedi et al., 2023).
<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th colspan="4">MuSiQue</th>
    <th colspan="4">2Wiki</th>
    <th colspan="4">HotpotQA</th>
  </tr>
  <tr>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@15</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@15</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@15</th>
  </tr>
  <tr>
    <td>ColBERTv2</td>
    <td>39.4</td>
    <td>44.8</td>
    <td>47.7</td>
    <td>59.1</td>
    <td>64.3</td>
    <td>66.2</td>
    <td>79.3</td>
    <td>87.1</td>
    <td>90.1</td>
  </tr>
  <tr>
    <td>HippoRAG</td>
    <td>41.0</td>
    <td>47.0</td>
    <td>51.4</td>
    <td><b>75.1</b></td>
    <td><b>83.2</b></td>
    <td><b>86.4</b></td>
    <td>79.8</td>
    <td>89.0</td>
    <td>92.4</td>
  </tr>
  <tr>
    <td>BM25</td>
    <td>33.8</td>
    <td>38.5</td>
    <td>41.3</td>
    <td>59.5</td>
    <td>62.7</td>
    <td>64.1</td>
    <td>74.2</td>
    <td>83.6</td>
    <td>86.3</td>
  </tr>
  <tr>
    <td>+ NaiveGE</td>
    <td>37.5</td>
    <td>45.5</td>
    <td>48.4</td>
    <td>65.0</td>
    <td>70.7</td>
    <td>71.8</td>
    <td>79.1</td>
    <td>89.1</td>
    <td>91.9</td>
  </tr>
  <tr>
    <td>+ SyncGE</td>
    <td>44.7</td>
    <td>52.6</td>
    <td>57.4</td>
    <td>70.5</td>
    <td>76.1</td>
    <td>79.3</td>
    <td>87.4</td>
    <td>93.0</td>
    <td>94.0</td>
  </tr>
  <tr>
    <th rowspan="5">Single-step Retrieval</th>
    <td>SBERT</td>
    <td>31.1</td>
    <td>37.9</td>
    <td>41.6</td>
    <td>41.2</td>
    <td>48.1</td>
    <td>51.5</td>
    <td>72.1</td>
    <td>79.3</td>
    <td>84.0</td>
  </tr>
  <tr>
    <td>+ NaiveGE</td>
    <td>32.2</td>
    <td>41.4</td>
    <td>45.4</td>
    <td>45.1</td>
    <td>54.0</td>
    <td>57.3</td>
    <td>76.1</td>
    <td>84.7</td>
    <td>88.8</td>
  </tr>
  <tr>
    <td>+ SyncGE</td>
    <td>41.6</td>
    <td>51.3</td>
    <td>54.2</td>
    <td>54.8</td>
    <td>64.9</td>
    <td>70.7</td>
    <td>84.1</td>
    <td>89.6</td>
    <td>92.8</td>
  </tr>
  <tr>
    <td>Hybrid</td>
    <td>39.9</td>
    <td>46.3</td>
    <td>49.1</td>
    <td>60.0</td>
    <td>65.8</td>
    <td>66.6</td>
    <td>77.8</td>
    <td>85.8</td>
    <td>89.7</td>
  </tr>
  <tr>
    <td>+ NaiveGE</td>
    <td>41.8</td>
    <td>49.4</td>
    <td>53.0</td>
    <td>63.0</td>
    <td>70.8</td>
    <td>72.6</td>
    <td>80.6</td>
    <td>89.4</td>
    <td>92.7</td>
  </tr>
  <tr>
    <td>+ SyncGE</td>
    <td><b>48.7</b></td>
    <td><b>57.7</b></td>
    <td><b>61.2</b></td>
    <td><b>72.6</b></td>
    <td><b>80.9</b></td>
    <td><b>82.4</b></td>
    <td><b>87.4</b></td>
    <td><b>93.3</b></td>
    <td><b>95.2</b></td>
  </tr>
  <tr>
    <th rowspan="4">Multi-step Retrieval</th>
    <td>IRCoT (BM25)</td>
    <td>46.1</td>
    <td>54.9</td>
    <td>57.9</td>
    <td>67.9</td>
    <td>75.5</td>
    <td>76.1</td>
    <td>87.0</td>
    <td>92.6</td>
    <td>92.9</td>
  </tr>
  <tr>
    <td>IRCoT (ColBERTv2)</td>
    <td>47.9</td>
    <td>54.3</td>
    <td>56.4</td>
    <td>60.3</td>
    <td>86.6</td>
    <td>69.7</td>
    <td>86.9</td>
    <td>92.5</td>
    <td>92.8</td>
  </tr>
  <tr>
    <td>HippoRAG w/ IRCoT</td>
    <td>48.8</td>
    <td>54.5</td>
    <td>58.9</td>
    <td>82.9</td>
    <td>90.6</td>
    <td>93.0</td>
    <td>90.1</td>
    <td>94.7</td>
    <td>95.9</td>
  </tr>
  <tr>
    <td>GEAR</td>
    <td><b>58.4</b></td>
    <td><b>67.6</b></td>
    <td><b>71.5</b></td>
    <td><b>89.1</b></td>
    <td><b>95.3</b></td>
    <td><b>95.9</b></td>
    <td><b>93.4</b></td>
    <td><b>96.8</b></td>
    <td><b>97.3</b></td>
  </tr>
</table>

Table 2: Retrieval performance for single- and multi-step retrievers on MuSiQue, 2Wiki, and HotpotQA. Results are reported using Recall@k (R@k) metrics for \( k \in \{5, 10, 15\} \).

6.1 Baselines

We evaluate GEAR against strong, multi-step baselines, including IRCot (Trivedi et al., 2023) and HippoRAG w/ IRCoT (Gutierrez et al., 2024), which, similar to our framework, combines graph retrieval and a multi-step agent. To demonstrate our graph retriever’s (i.e. SyncGE) benefits, we evaluate it against several stand-alone, single-step retrievers: (i) BM25, (ii) Sentence-BERT (SBERT), (iii) a hybrid approach combining BM25 and SBERT through RRF and (iv) HippoRAG. Following Gutierrez et al., we refer to the single-step setup when multiple LLM iterations are not supported.

6.2 Implementation Details

We reproduce HippoRAG and IRCoT using the code provided by Gutierrez et al.. To ensure fair comparisons, we employ GPT-4o mini (gpt-4o-mini-2024-07-18) for all methods that require an LLM as well as their corresponding triple extraction. The temperature is set to 0. Our triple extraction prompt (in Appendix K.1) is inspired by Gutierrez et al.. We run QA experiments using the prompts provided in Appendix K.3.

In addition to our proposed SyncGE, we consider a more naive implementation of GE (i.e. NaiveGE) to evaluate the performance when no LLM is involved and further demonstrate the effectiveness of synchronisation. In NaiveGE, we input all triples associated with \( C_q' \) (see Section 4) for diverse triple beam search. Comprehensive implementation details are provided in Appendices B–D. Additionally, Appendix F provides more experiments evaluating GEAR with varying configurations.

Figure 2: R@15 over 4 iterations on MuSiQue. Recall is computed at each iteration using the cumulative set of retrieved documents, with prior recall values carried forward for questions that terminated in earlier iterations. The horizontal line indicates the single-step performance of Hybrid + SyncGE.

7 Results

GEAR demonstrates state-of-the-art performance in multi-step retrieval The multi-step results in Table 2 show that our agent-based approach to multi-step retrieval is highly effective, achieving state-of-the-art results across all datasets. While we see significant improvements on saturated datasets like 2Wiki and HotpotQA, GEAR especially excels on MuSiQue, delivering performance gains of over 10% compared to competitors.

SyncGE contributes to state-of-the-art performance in single-step retrieval As shown in the
single-step section of Table 2, our proposed Hybrid + SyncGE method achieves state-of-the-art single-step retrieval performance on both MuSiQue and HotpotQA datasets. We observe consistent improvements using NaiveGE and SyncGE, outperforming HippoRAG in many setups regardless of the base retriever (i.e. sparse, dense or hybrid). Most notably, Hybrid + SyncGE surpasses HippoRAG by up to 9.8% at R@15 on MuSiQue.

Higher recall leads to higher QA performance
Our analysis shows a positive correlation between recall and QA performance, aligning with the results of prior works (Gutierrez et al., 2024). As shown in Table 3, GEAR achieves the highest EM and F1 scores. A closer examination reveals interesting insights. Taking MuSiQue as an example, GeAR shows a 21% relative improvement in R@15 compared to HippoRAG w/ IRCot, while achieving a 37% relative improvement in both EM and F1 scores. Similarly to Table 2, SyncGE outperforms HippoRAG on MuSiQue and HotpotQA.

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th colspan="2">MuSiQue</th>
    <th colspan="2">2Wiki</th>
    <th colspan="2">HotpotQA</th>
  </tr>
  <tr>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
    <th>EM</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>No Passages</td>
    <td>2.6</td>
    <td>12.5</td>
    <td>17.2</td>
    <td>27.9</td>
    <td>19.5</td>
    <td>34.3</td>
  </tr>
  <tr>
    <td>Gold Passages</td>
    <td>36.6</td>
    <td>59.2</td>
    <td>54.4</td>
    <td>70.3</td>
    <td>55.0</td>
    <td>75.9</td>
  </tr>
  <tr>
    <td>Hybrid + SyncGE</td>
    <td>14.0</td>
    <td>27.1</td>
    <td>38.0</td>
    <td>50.2</td>
    <td>45.0</td>
    <td>63.4</td>
  </tr>
  <tr>
    <td>HippoRAG</td>
    <td>8.2</td>
    <td>18.2</td>
    <td>39.8</td>
    <td>51.8</td>
    <td>40.1</td>
    <td>57.6</td>
  </tr>
  <tr>
    <td>IRCoT (BM25)</td>
    <td>7.6</td>
    <td>15.9</td>
    <td>28.8</td>
    <td>38.5</td>
    <td>34.3</td>
    <td>50.8</td>
  </tr>
  <tr>
    <td>IRCoT (ColBERTv2)</td>
    <td>12.2</td>
    <td>24.1</td>
    <td>32.4</td>
    <td>43.6</td>
    <td>45.2</td>
    <td>63.7</td>
  </tr>
  <tr>
    <td>HippoRAG w/ IRCot</td>
    <td>14.2</td>
    <td>25.9</td>
    <td>45.6</td>
    <td>59.0</td>
    <td>49.2</td>
    <td>67.9</td>
  </tr>
  <tr>
    <td>GEAR</td>
    <td>19.0</td>
    <td>35.6</td>
    <td>47.4</td>
    <td>62.3</td>
    <td>50.4</td>
    <td>69.4</td>
  </tr>
</table>

Table 3: End-to-end QA performance using the top-5 retrieved passages. The best model is in bold and second best is underlined. The top part shows the lower and upper bounds of QA performance, while the middle and bottom sections display scores for single-step and multi-step retrievers, respectively.

<table>
  <tr>
    <th>Metric</th>
    <th>Dataset</th>
    <th>w/ Diversity</th>
    <th>w/o Diversity</th>
  </tr>
  <tr>
    <td></td>
    <td>MuSiQue</td>
    <td>48.7</td>
    <td>47.0</td>
  </tr>
  <tr>
    <td>R@5</td>
    <td>2Wiki</td>
    <td>72.6</td>
    <td>68.2</td>
  </tr>
  <tr>
    <td></td>
    <td>HotpotQA</td>
    <td>87.4</td>
    <td>85.0</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSiQue</td>
    <td>57.7</td>
    <td>53.9</td>
  </tr>
  <tr>
    <td>R@10</td>
    <td>2Wiki</td>
    <td>80.9</td>
    <td>76.0</td>
  </tr>
  <tr>
    <td></td>
    <td>HotpotQA</td>
    <td>93.3</td>
    <td>92.2</td>
  </tr>
  <tr>
    <td></td>
    <td>MuSiQue</td>
    <td>61.2</td>
    <td>58.4</td>
  </tr>
  <tr>
    <td>R@15</td>
    <td>2Wiki</td>
    <td>82.4</td>
    <td>77.4</td>
  </tr>
  <tr>
    <td></td>
    <td>HotpotQA</td>
    <td>95.2</td>
    <td>94.3</td>
  </tr>
</table>

Table 4: Effects of beam search diversity on Hybrid + SyncGE across MuSiQue, 2Wiki and HotpotQA.

Diverse Triple Beam Search improves performance
As shown in Table 4, our diverse triple beam search consistently outperforms standard beam search across all datasets and recall ranks. By incorporating diversity weights into beam search, we align a language modelling-oriented solution with information retrieval objectives that involve satisfying multiple information needs underlying multi-hop queries (Drosou and Pitoura, 2010).

GEAR mostly nails it the first time
While GEAR supports multiple iterations, Figure 2 shows that GEAR achieves strong retrieval performance in a single iteration on MuSiQue. This differentiates it from IRCot-oriented setups that require at least 2 iterations to reach maximum performance. This can be attributed to the fact that GEAR reads (Eq. 4) multi-hop contexts and associates proximal triples in gist memory with passages, establishing synergy between our graph retriever and the LLM. We believe this mirrors the hippocampal process of forming and resolving sparse representations, where gist memories are learned in a one or few-shot manner (Hanslmayr et al., 2016). The 10% performance gap between Hybrid + SyncGE and GEAR at \( n = 1 \) indicates that the LLM reading and linking processes effectively approximate the hippocampus’s role within our framework.

8 Discussion

8.1 What makes GEAR work?

NaiveGE vs SyncGE
As shown in Table 2, both graph expansion variants enhance every base retriever’s performance across all datasets. The superior performance of SyncGE indicates the effectiveness of using LLMs for locating initial nodes. Notably, it surpasses HippoRAG w/ IRCot’s on MuSiQue without multiple iterations.

8.2 How robust is GEAR?

GEAR excels at questions of low-to-moderate complexity
Figure 3 presents a detailed breakdown of retrieval performance across different hop types, including path-finding and path-following questions categorized by Gutierrez et al.. For 2-hop questions, while GEAR and HippoRAG w/ IRCot achieve similar interquartile ranges, GEAR shows a higher mean recall, indicating superior performance on low-complexity questions. This
Figure 3: Analysis of R@15 performance divided by hop types on MuSiQue. The hop categorisation follows the MuSiQue documentation. Mean recall values are indicated by grey dots for each hop type.

advantage becomes more pronounced with 3-hop questions, where GeAR’s entire interquartile range exceeds HippoRAG w/ IRCot’s median performance across both hop subdivisions. This demonstrates GeAR’s enhanced capability in handling moderately complex questions. In addition to MuSiQue, 2Wiki and HotpotQA, we test GeAR against the hand-picked case study data provided by Gutierrez et al.. These include four path-finding questions across four different domains. Our findings (Appendix I.3) indicate that GeAR’s performance is on par with HippoRAG w/ IRCot, out-performing the competition in three out of the four cases, in terms of recall.

GeAR’s performance remains consistent across chunks with varying numbers of triples Using MuSiQue, we group questions based on the average number of triples (i.e. triple density, \( \rho_t \)) associated with their golden passages, and evaluate R@15 across four ranges: (i) \( \rho_t < 9 \), (ii) \( 9 \leq \rho_t < 11 \), (iii) \( 11 \leq \rho_t < 13 \) and (iv) \( 13 \leq \rho_t \). Across all these ranges, the recall performance of both SyncGE and GeAR exhibits lower variation, with significantly smaller standard deviations of \( 1.18 \ll 2.04 \) and \( 2.08 \ll 5.59 \), respectively, compared to NaiveGE and HippoRAG w/ IRCot. Further details are provided in Appendix H.

8.3 Is GeAR efficient?
As observed in Figure 2, GeAR requires fewer iterations than the competition to reach its maximum recall performance. Furthermore, Figure 4 shows that GeAR can act as a more efficient alternative with respect to LLM token utilisation. We note that even for a single iteration, GeAR uses fewer tokens than HippoRAG w/ IRCot. In contrast to ours, this trend exacerbates for the competition as the number of iterations increases. These findings also reiterate the value of SyncGE, which outperforms a significantly more LLM-heavy solution on MuSiQue, using almost 2.9 million fewer tokens. Even in the case that HippoRAG w/ IRCot runs for a single iteration it would require more than 0.7 million tokens that Hybrid + SyncGE, with a substantially lower R@15 of 51.7.

Figure 4: Progressive accumulation of input and output LLM tokens across agent iterations on MuSiQue. The Hybrid + SyncGE method appears only to the left of Iteration 1 as it is a single-step approach.

9 Conclusion
We propose GeAR, a novel framework that incorporates a graph-based retriever within a multi-step retrieval agent to model the information-seeking process for multi-hop question answering.

We showcase the synergy between our proposed graph retriever (i.e. SyncGE) and the LLM within the GeAR framework. SyncGE leverages the LLM to synchronise information from passages with triples and expands the graph by exploring diverse
beams of triples that link multi-hop contexts. Our experiments reveal that this strategy improves over more naive implementations, demonstrating the LLM’s capability to guide the exploration of initial nodes for graph expansion. Furthermore, GeAR utilises multi-hop contexts returned by SyncGE and constructs a gist memory which is used for effectively summarising information across iterations. GeAR achieves superior performance compared to other multi-step retrieval methods while requiring fewer iterations and LLM tokens.

Limitations

The scope of this paper is limited to retrieval using a graph of triples that bridge corresponding passages. While we demonstrate the effectiveness of our graph expansion approach and GeAR, several components could be further refined. More sophisticated graph construction methods addressing challenges such as entity disambiguation (Dredze et al., 2010) and knowledge graph completion (Lin et al., 2015) may yield further improvements, as discussed in Appendix E. Similarly, our choice of a dense embedding model for the scoring function in the diverse triple beam search could be replaced by alternatives, such as formulating this as a natural language inference task (Wang et al., 2021).

Beyond the core graph and scoring mechanisms, our design choices for other agent components—such as the gist memory, reasoner, and query rewriter—adopt commonly validated practices from prior research (Trivedi et al., 2022; Li et al., 2024; Fang et al., 2024). While effective, exploring more sophisticated designs specifically tailored for these functions presents an opportunity for future performance gains.

References

Howard Chen, Ramakanth Pasunuru, Jason Weston, and Asli Celikyilmaz. 2023. Walking down the memory maze: Beyond context limit through interactive reading. Preprint, arXiv:2310.05029.

Gordon V. Cormack, Charles L A Clarke, and Stefan Buettcher. 2009. Reciprocal rank fusion outperforms condorcet and individual rank learning methods. In Proceedings of the 32nd International ACM SIGIR Conference on Research and Development in Information Retrieval, SIGIR ’09, page 758–759, New York, NY, USA. Association for Computing Machinery.

Yiran Ding, Li Lyna Zhang, Chengrudong Zhang, Yuanyuan Xu, Ning Shang, Jiahang Xu, Fan Yang, and Mao Yang. 2024. LongRoPE: Extending LLM context window beyond 2 million tokens. In Proceedings of the 41st International Conference on Machine Learning, volume 235 of Proceedings of Machine Learning Research, pages 11091–11104. PMLR.

Mark Dredze, Paul McNamee, Delip Rao, Adam Gerber, and Tim Finin. 2010. Entity disambiguation for knowledge base population. In Proceedings of the 23rd International Conference on Computational Linguistics (Coling 2010), pages 277–285, Beijing, China. Coling 2010 Organizing Committee.

Marina Drosou and Evaggelia Pitoura. 2010. Search result diversification. SIGMOD Rec., 39(1):41–47.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph RAG approach to query-focused summarization.

Jinyuan Fang, Zaiqiao Meng, and Craig MacDonald. 2024. TRACE the evidence: Constructing knowledge-grounded reasoning chains for retrieval-augmented generation. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8472–8494, Miami, Florida, USA. Association for Computational Linguistics.

Benjamin J. Griffiths, George Parish, Frederic Roux, Sebastian Michelmann, Mircea van der Plas, Luca D. Kolibuis, Ramesh Chelvarajah, David T. Rollings, Vijay Sawlani, Hajo Hamer, Stephanie Gollwitzer, Gernot Kreiselmeier, Bernhard Staresina, Maria Wimber, and Simon Hanslmayr. 2019. Directional coupling of slow and fast hippocampal gamma with neocortical alpha/beta oscillations in human episodic memory. Proceedings of the National Academy of Sciences, 116(43):21834–21842.

Bernal Jimenez Gutierrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. 2024. HippoRAG: Neuropsychologically inspired long-term memory for large language models. In The Thirty-eighth Annual Conference on Neural Information Processing Systems.

Simon Hanslmayr, Bernhard P. Staresina, and Howard Bowman. 2016. Oscillations and episodic memory: Addressing the synchronization/desynchronization conundrum. Trends in Neurosciences, 39(1):16–25.

Xanh Ho, Anh-Khoa Duong Nguyen, Saku Sugawara, and Akiko Aizawa. 2020. Constructing a multi-hop QA dataset for comprehensive evaluation of reasoning steps. In Proceedings of the 28th International Conference on Computational Linguistics, pages 6609–6625, Barcelona, Spain (Online). International Committee on Computational Linguistics.

Pere-Lluís Huguet Cabot and Roberto Navigli. 2021. REBEL: Relation extraction by end-to-end language generation. In Findings of the Association for Computational Linguistics: EMNLP 2021, pages 2370–2381, Punta Cana, Dominican Republic. Association for Computational Linguistics.
Daphne Ippolito, Reno Kriz, João Sedoc, Maria Kustikova, and Chris Callison-Burch. 2019. Comparison of diverse decoding methods from conditional language models. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 3752–3762, Florence, Italy. Association for Computational Linguistics.

Zhengbao Jiang, Frank Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 7969–7992, Singapore. Association for Computational Linguistics.

Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, Sebastian Riedel, and Douwe Kiela. 2020. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS ’20, Red Hook, NY, USA. Curran Associates Inc.

Shilong Li, Yancheng He, Hangyu Guo, Xingyuan Bu, Ge Bai, Jie Liu, Jiaheng Liu, Xingwei Qu, Yangguang Li, Wanli Ouyang, Wenbo Su, and Bo Zheng. 2024. GraphReader: Building graph-based agent to enhance long-context abilities of large language models. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 12758–12786, Miami, Florida, USA. Association for Computational Linguistics.

Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu, Peilong Zhao, Zhongpu Bo, Jin Yang, Huaidong Xiong, Lin Yuan, Jun Xu, Zaoyang Wang, Zhiqiang Zhang, Wen Zhang, Huaqun Chen, Wenguang Chen, and Jun Zhou. 2024. KAG: Boosting llms in professional domains via knowledge augmented generation. Preprint, arXiv:2409.13731.

Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. 2015. Learning entity and relation embeddings for knowledge graph completion. Proceedings of the AAAI Conference on Artificial Intelligence, 29(1).

OpenAI. 2023. GPT-4 technical report. Preprint, arXiv:2303.08774.

Jinyoung Park, Ameen Patel, Omar Zia Khan, Hyunwoo J. Kim, and Joo-Kyung Kim. 2023. Graph-guided reasoning for multi-hop question answering in large language models. CoRR, abs/2311.09762.

Jinyoung Park, Ameen Patel, Omar Zia Khan, Hyunwoo J. Kim, and Joo-Kyung Kim. 2024. Graph elicitation for guiding multi-step reasoning in large language models. Preprint, arXiv:2311.09762.

Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher D Manning. 2024. RAPTOR: Recursive abstractive processing for tree-organized retrieval. In The Twelfth International Conference on Learning Representations.

Tao Shen, Guodong Long, Xiubo Geng, Chongyang Tao, Yibin Lei, Tianyi Zhou, Michael Blumenstein, and Daxin Jiang. 2024a. Retrieval-augmented retrieval: Large language models are strong zero-shot retriever. In Findings of the Association for Computational Linguistics: ACL 2024, pages 15933–15946, Bangkok, Thailand. Association for Computational Linguistics.

Zhili Shen, Pavlos Vougiouklis, Chenxin Diao, Kaustubh Vyas, Yuanyi Ji, and Jeff Z. Pan. 2024b. Improving retrieval-augmented text-to-SQL with AST-based ranking and schema pruning. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 7865–7879, Miami, Florida, USA. Association for Computational Linguistics.

Weihang Su, Yichen Tang, Qingyao Ai, Zhijing Wu, and Yiqun Liu. 2024. DRAGIN: Dynamic retrieval augmented generation based on the real-time information needs of large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 12991–13013, Bangkok, Thailand. Association for Computational Linguistics.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2022. MuSiQue: Multi-hop questions via single-hop question composition. Transactions of the Association for Computational Linguistics, 10:539–554.

Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, and Ashish Sabharwal. 2023. Interleaving retrieval with chain-of-thought reasoning for knowledge-intensive multi-step questions. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 10014–10037, Toronto, Canada. Association for Computational Linguistics.

Ashwin K Vijayakumar, Michael Cogswell, Ramprasad R. Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. Diverse beam search: Decoding diverse solutions from neural sequence models. Preprint, arXiv:1610.02424.

Liang Wang, Nan Yang, and Furu Wei. 2023. Query2doc: Query expansion with large language models. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pages 9414–9423, Singapore. Association for Computational Linguistics.

Sinong Wang, Han Fang, Madian Khabsa, Hanzi Mao, and Hao Ma. 2021. Entailment as few-shot learner. Preprint, arXiv:2104.14690.

Yu Wang, Nedim Lipka, Ryan A. Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. 2024. Knowledge
graph prompting for multi-document question answering. Proceedings of the AAAI Conference on Artificial Intelligence, 38(17):19206–19214.

Zhilin Yang, Peng Qi, Saizheng Zhang, Yoshua Bengio, William Cohen, Ruslan Salakhutdinov, and Christopher D. Manning. 2018. HotpotQA: A dataset for diverse, explainable multi-hop question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2369–2380, Brussels, Belgium. Association for Computational Linguistics.
A  Dataset Choices and Statistics

<table>
  <tr>
    <th></th>
    <th>MuSiQue</th>
    <th>2Wiki</th>
    <th>HotpotQA</th>
  </tr>
  <tr>
    <th>Spli t Source</th>
    <td>IRCoT</td>
    <td>IRCoT</td>
    <td>HippoRAG</td>
  </tr>
  <tr>
    <th># Hops</th>
    <td>2 – 4</td>
    <td>2</td>
    <td>2</td>
  </tr>
  <tr>
    <th># Documents</th>
    <td>139,416</td>
    <td>430,225</td>
    <td>9,221</td>
  </tr>
  <tr>
    <th># Test Queries</th>
    <td>500</td>
    <td>500</td>
    <td>1,000</td>
  </tr>
  <tr>
    <th># Chunks (C)</th>
    <td>148,793</td>
    <td>490,454</td>
    <td>10,293</td>
  </tr>
  <tr>
    <th># Triples (T)</th>
    <td>1,521,136</td>
    <td>4,993,637</td>
    <td>122,492</td>
  </tr>
  <tr>
    <th>Av. # T/C</th>
    <td>10.2</td>
    <td>10.2</td>
    <td>11.9</td>
  </tr>
</table>

Table 5: Dataset characteristics and preprocessing statistics, where triples are extracted from chunks, and Av. # T/C represents the average number of triples per chunk.

Table 5 serves as a summary of various facts and statistics related to the employed datasets and the chunking and triple extraction process introduced in Section 3. Please note for all the evaluated datasets, we use their open-domain setting and answerable subset if applicable.

Reasoning behind dataset split choices For MuSiQue and 2Wiki, we use the data provided by Trivedi et al., including the full corpus and sub-sampled test cases for each dataset. To limit the experimental cost for HotpotQA, we follow the setting by Gutierrez et al. where both the corpus and test split are smaller than IRCoT’s counterpart.

B  More Implementation Details

B.1 Baselines Details

We implement all proposed approaches using Elasticsearch1. For SBERT, we employ the all-mpnet-base-v2 model with approximate k-nearest neighbours and cosine similarity for vector comparisons. In IRCoT experiments, we evaluate both ColBERTv2 and BM25 retrievers — ColBERTv2 for alignment with HippoRAG’s baselines, and BM25 for consistency with the original IRCoT implementation.

For all multi-step approaches, including ours, we follow Gutierrez et al. with respect to the maximum number of retrieval iterations, which vary based on the hop requirements of each dataset. Thus, we use a maximum of 4 iterations for MuSiQue and 2 iterations for HotpotQA and 2Wiki.

B.2 GEAR Details

GEAR involves several hyperparameters, such as the beam size inside graph expansion. We randomly sampled 500 questions from the MuSiQue development set, which we ensure not to overlap with the relevant test set. We select our hyperparameters based on this sample without performing a grid search across all possible configurations. Our goal is to demonstrate that our method is able to achieve state-of-the-art results without extensive parameter tuning.

The initial retrieval phase utilises the chunks index C as the information source, while leaving the triple index T unused. Our graph expansion component implements beam search with length 2, width 10, and 100 neighbours per beam. The hyperparameter \( \gamma \) employed in diverse triple beam search is set to twice the beam search width. For the scoring function, we use the cosine similarity score and the SBERT embedding model. In Appendix F, we test the performance of GEAR across different beam search length values and maximum numbers of agent iterations.

For the single-step configurations (i.e. any base retriever with NaiveGE or SyncGE), we set the base retriever’s maximum number of returned chunks to match our evaluation recall threshold. With the multi-step setup, we maintain a consistent maximum of 10 retrieved chunks before knowledge synchronisation for the purpose of matching IRCoT’s implementation. While this 10-chunk limitation applies to individual retrieval rounds, please note that the total number of accessible chunks can exceed this threshold through graph expansion and multiple GEAR iterations.

passageLink Details  We use passageLink to link each triple \( t_j \in \mathcal{G}^{(n)} \) to its corresponding passages in C by running a retrieval step as follows:

\[
h_{base}^k(\mathbf{q}, \mathbf{C} \cup \mathbf{T}) = \mathrm{RRF}\left(h_{base}^k(t_j, \mathbf{C}), h_{base}^k(t_j, \mathbf{T})\right),
\]

where \( j \in \{1, \ldots, |\mathcal{G}^{(n)}|\} \) and \( h_{base}^k(t_j, \mathbf{C} \cup \mathbf{T}) \) is the RRF of passages returned by both T and C when queried with \( t_j \).

C  Walkthrough Example

Table 6 provides a visual step-by-step example of our framework. The example demonstrates how each component contributes to the final retrieved passages, which we hope offers useful context for understanding our design decisions.

1https://www.elastic.co
<table>
  <tr>
    <th>Module</th>
    <th>Intermediate Output</th>
    <th>Fig.1 Sections</th>
  </tr>
  <tr>
    <td colspan="3"><b>Offline Index Building Stage</b></td>
  </tr>
  <tr>
    <td>For each passage \( c_i \in C = \{c_1, c_2, \ldots, c_L\} \), an LLM extracts a triple set, such that each triple is uniquely linked to one single passage.</td>
    <td>Title: Bremen Cathedral<br>Triple Extractor<br>dedicated to<br>Bremen Cathedral → St. Peter</td>
    <td></td>
  </tr>
  <tr>
    <td colspan="3"><i>Input Query</i> from MuSiQue (Trivedi et al. 2022) <b>(Answer: 1929)</b><br>When did the location of the basilica which is named for the same saint that the Bremen Cathedral is named for become a country?</td>
  </tr>
  <tr>
    <td colspan="3"><b>Online Retrieval Stage</b></td>
  </tr>
  <tr>
    <td><b>1. Base Retrieval</b> (see §4)<br>For a query \( q \), \( C_q = h_{base}(q, C) \) is a list of passages given by the retriever, implemented as BM25, SBERT, or a mix of both.</td>
    <td>\( P_1 \) Bremen Cathedral, \( P_2 \) Münster Cathedral, \( P_3 \) Basilica of the Sacred Heart<br>\( P_4 \) Saint Justin’s Church, Frankfurt-Höchst, \( P_5 \) Alatri Cathedral</td>
    <td></td>
  </tr>
  <tr>
    <td><b>2. Reader</b> (see §4.1)<br>An LLM reads \( C_q \) and summarises knowledge triples, outputting a collection \( T_q^* \) of triples: the <i>proximal triples</i>.</td>
    <td>\( T_1^* \) (Bremen Cathedral, dedicated to, St. Peter)<br>\( T_2^* \) (Alatri Cathedral, dedicated to, Saint Paul)<br>\( T_3^* \) (Alatri Cathedral, co-cathedral of, Diocese Anagni-Alatri)<br>\( T_4^* \) (Bremen, is located in, Germany)</td>
    <td></td>
  </tr>
  <tr>
    <td><b>2. tripleLink</b> (see §4.1)<br>Initial nodes \( T_q \) for graph expansion are identified by linking each triple in \( T_q^* \) to a triple in \( T \), using the tripleLink function.</td>
    <td>\( T_1 \) (Bremen Cathedral, dedicated to, St. Peter)<br>\( T_2 \) (Alatri Cathedral, dedicated to, Saint Paul)<br>\( T_3 \) (Diocese of Macerata-Tolentino-Recanati-Cingoli-Treia, type, co-cathedral)<br>\( T_4 \) (Bremen, part of, Germany)</td>
    <td></td>
  </tr>
  <tr>
    <td><b>4. Graph Expansion</b> (see §4.2)<br>The primary component of graph expansion is <i>Diverse Triple Beam Search</i>. Here, we explore neighbourhood of a triple (defined as other triples with shared head or tail entities) and maintain top-b sequences (beams) of triples.</td>
    <td><img src="page_670_1012_388_246.png" alt="Graph Expansion example"></td>
    <td></td>
  </tr>
  <tr>
    <td><b>5. Gist Memory</b> (see §5.1)<br>Similar to the Reader, an LLM reads a collection of retrieved paragraphs \( C_q^{(n)} \) and extracts an array of proximal triples \( T_q^{(n)*} \), which are stored in the Gist Memory \( G^{(n)} \).</td>
    <td>\( T_1 \) (Bremen Cathedral, dedicated to, St. Peter)<br>\( T_2 \) (Alatri Cathedral, dedicated to, Saint Paul)<br>\( T_3 \) (Lund Cathedral, dedicated to, Saint Lawrence)<br>\( T_4 \) (Bremen, part of, Germany)</td>
    <td></td>
  </tr>
  <tr>
    <td><b>6. Reasoner</b> (see §5.2)<br>After updating \( G^{(n)} \), we assess whether it contains sufficient evidence to answer the original question via an LLM reasoning step.</td>
    <td>Answerable: False<br>Answer or reason: The provided facts do not contain information about the location of the basilica named for St. Peter, nor do they provide any details about when it became a country. The facts only mention the dedication of other cathedrals to different saints.</td>
    <td></td>
  </tr>
  <tr>
    <td><b>7. Rewriter</b> (see §5.3)<br>Given the original \( q \), the accumulated memory \( G^{(n)} \), and the reasoning output \( r^{(n)} \), an LLM is used to re-write the query. We return to step 1 and repeat.</td>
    <td>Next query: What is the location of the basilica dedicated to St. Peter, and when did that location become a country?</td>
    <td></td>
  </tr>
</table>

Table 6: <b>Visual walk-through example</b> of the modules involved in offline index construction and online retrieval in GEAR. After query-rewriting, steps 1–7 are repeated until termination by the Reasoner, or reaching the maximum number of iterations.
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">MuSiQue</th>
    <th colspan="3">2Wiki</th>
    <th colspan="3">HotpotQA</th>
  </tr>
  <tr>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
  </tr>
  <tr>
    <td>HippoRAG<br>original prompt</td>
    <td>41.9</td><td>46.9</td><td>51.1</td>
    <td>75.4</td><td>83.5</td><td>86.9</td>
    <td>79.7</td><td>88.4</td><td>91.4</td>
  </tr>
  <tr>
    <td>HippoRAG<br>our prompt</td>
    <td>41.0</td><td>47.0</td><td>51.4</td>
    <td>75.1</td><td>83.2</td><td>86.4</td>
    <td>79.8</td><td>89.0</td><td>92.4</td>
  </tr>
  <tr>
    <td>HippoRAG w/ IRCoT<br>original prompt</td>
    <td>49.9</td><td>56.4</td><td>59.3</td>
    <td>81.5</td><td>90.2</td><td>92.3</td>
    <td>90.2</td><td>94.7</td><td>95.8</td>
  </tr>
  <tr>
    <td>HippoRAG w/ IRCoT<br>our prompt</td>
    <td>48.8</td><td>54.5</td><td>58.9</td>
    <td>82.9</td><td>90.6</td><td>93.0</td>
    <td>90.1</td><td>94.7</td><td>95.9</td>
  </tr>
</table>

Table 7: Retrieval performance comparison between HippoRAG’s sequential triple extraction method and our joint extraction approach across three datasets.

<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th rowspan="2">Retriever</th>
    <th colspan="3">MuSiQue</th>
    <th colspan="3">2Wiki</th>
    <th colspan="3">HotpotQA</th>
  </tr>
  <tr>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
  </tr>
  <tr>
    <td rowspan="3">GPT-3.5 turbo</td>
    <td>Hybrid + SyncGE</td>
    <td>48.3</td><td>55.0</td><td>58.1</td>
    <td>70.7</td><td>78.4</td><td>79.7</td>
    <td>86.1</td><td>92.0</td><td>94.3</td>
  </tr>
  <tr>
    <td>HippoRAG w/ IRCoT</td>
    <td>45.5</td><td>51.0</td><td>54.8</td>
    <td>80.0</td><td>87.9</td><td>89.8</td>
    <td>87.4</td><td>92.6</td><td>94.6</td>
  </tr>
  <tr>
    <td>GeAR</td>
    <td>52.9</td><td>62.1</td><td>64.4</td>
    <td>84.2</td><td>90.0</td><td>90.3</td>
    <td>91.0</td><td>95.6</td><td>96.1</td>
  </tr>
</table>

Table 8: Retrieval performance for our proposed retrievers and HippoRAG w/ IRCoT across various datasets. We use gpt-3.5-turbo-1106 (temperature = 0) as the underlying LLM, to replicate HippoRAG’s experimental setup.

D Ensuring Fair Comparisons

Although related studies often use common datasets, their experimental settings are frequently inconsistent. For instance, Gutierrez et al. used heavily sub-sampled corpora, drastically reducing the number of documents (e.g., from over 5M to 9k for HotpotQA) compared to the full datasets originally established for IRCoT (Trivedi et al., 2022). We believe such reductions significantly simplify the retrieval task. Therefore, in our paper, we reproduced HippoRAG on MuSiQue and 2Wiki using the same dataset settings (i.e., full corpus and identical evaluation split) as defined in the original IRCoT paper (Trivedi et al., 2022). On HotpotQA, however, processing the full corpus established by Trivedi et al. (2022) (over 5 million passages) was computationally prohibitive, so we follow the same setting as HippoRAG to limit the experimental cost.

To ensure fairness in our comparisons, we ran all baselines using a consistent experimental setup. Additionally, for areas of potential discrepancy, and where possible, we report the retrieval performance of baselines in their original configuration to confirm that the used experimental setup does not adversely affect performance. Below, we address any potential discrepancies in the following areas: (i) triple extraction methodology, (ii) retrieval metrics, and (iii) LLMs.

Choice of triple extraction methodology HippoRAG employs a sequential approach to triple extraction: it first identifies named entities from a text chunk, and then uses these entities to guide triple extraction in a second step. In contrast, our method extracts both entities and triples simultaneously. Table 7 shows that both approaches achieve comparable retrieval performance across all datasets, with each method excelling in different scenarios. These results validate that joint entity and triple extraction can match the effectiveness of sequential extraction while reducing the number of required processing steps.

Reasoning behind retrieval metrics The MuSiQue dataset contains 2-hop, 3-hop and 4-hop questions, where a k-hop question is defined as one that requires k pieces of evidence to reach the correct answer (Trivedi et al., 2022). This means 3-hop and 4-hop questions require more than 2 pieces of evidence. If we used Recall@2 to evaluate them, as previous works such as (Gutierrez et al., 2024) do, we would be misjudging these questions, since it assumes only two pieces of evidence are enough for perfect recall. Additionally, given modern LLMs’ expanding context length capabilities (Ding et al., 2024), examining recall beyond R@5 (HippoRAG’s highest evaluated rank) provides valuable insights. Following IRCoT’s approach, we measure up to R@15 and include R@10 as an intermediate point, offering a comprehensive view of model performance across retrieval depths. Therefore, our evaluation employs recall at ranks 5, 10, and 15 (R@5, R@10, R@15).

Choice of LLM Gutierrez et al. use gpt-3.5-turbo-1106 for their experiments, whereas in this paper we reproduce it with GPT-4o
mini. GPT-4o mini was selected as a more capable alternative to GPT-3.5 Turbo (please refer to: https://openai.com/index/gpt-4-mini-advancing-cost-efficient-intelligence). In order to alleviate any concerns regarding discrepancies with respect to the selected LLM, we also run experiments using gpt-3.5-turbo-1106. Table 8 shows the retrieval results of our proposed methods against HippoRAG w/ IRCoT. We observe a similar trend to that in Table 2—GEAR surpasses the performance of HippoRAG w/ IRCoT.

E Why this graph construction method?

We adopt an LLM-based triple extraction methodology, following the approach outlined in HippoRAG (Gutierrez et al., 2024). In their study, they evaluated the performance of various LLMs in OpenIE and compared these results with those of the end-to-end REBEL model (Huguet Cabot and Navigli, 2021). They reported substantial improvements in triple extraction when using LLMs in domains that deviate from conventional ClosedIE or OpenIE settings, which are respectively overly constrained and unconstrained in terms of named entities and pre-defined relations. Similar concerns about the generalisability and scalability of conventional KG construction approaches in open-domain scenarios are recognised by Wang et al., who sought to construct their graphs without relying on pre-existing ontologies, or KGs for named entity disambiguation.

These findings resonate with the growing interest in recent literature towards applying such methodologies for automatic, schema-free knowledge graph construction (Li et al., 2024; Fang et al., 2024; Gutierrez et al., 2024; Park et al., 2024). As our primary focus is retrieval rather than graph construction, we adopt the triple extraction methodology from HippoRAG and refer readers to their paper for a more detailed analysis.

Our work presents a novel framework for advancing the performance of RAG systems in the context of texts associated with schema-free triples. We follow HippoRAG’s graph construction approach, as exploring graph construction methods falls outside the scope of this paper. However, this does not imply that our proposed method relies on this specific approach, and we believe that further improvements in graph construction could lead to additional gains.

F GEAR across Different Configurations

Table 9 illustrates the performance of GEAR across varying hyperparameter configurations, including beam search length—applied during graph expansion—and the maximum number of agent iterations. As the maximum number of iterations increases, GEAR achieves better retrieval performance. However, consistently with the trends shown in Figure 2, this improvement levels off when setting the maximum number of iterations at \( n \geq 3 \). In contrast, increasing beam sffsearch length above 2 slightly reduces performance. Despite this, GEAR maintains highly competitive results and significantly outperforms alternative methods shown in Table 2.

G Compatibility with Open-weight Models

GEAR Results As shown in Table 10, we evaluate GEAR using popular 7-8B parameter open-weight models, comparing them against a closed-source alternative. On HotpotQA, Llama-3.1-7B surpasses the closed-source alternative, achieving higher recall rates at R@10 and R@15. For MuSiQue and 2Wiki, while the closed-source model maintains a slight superior edge in performance, the margin is narrow. Importantly, all tested open-weight models consistently outperform the previous state-of-the-art, HippoRAG w/IRCoT. This decouples GEAR from the need to use closed-source models, suggesting that state-of-the-art multi-step retrieval can be achieved using more accessible models.

Diverse Beam Search Results Expanding upon Table 4, Table 11 demonstrates that diverse beam search consistently improves retrieval performance across both closed-source and open-weight models when using our proposed Hybrid + SyncGE setup. This further confirms the broader applicability of this approach.

H Robustness Studies

We assess the robustness of our framework in retrieving passages when triple extraction produces either limited or excessive triple content. Using the MuSiQue dataset, we group questions based on the average number of triples (i.e. triple density, \( p_t \)) associated with their golden passages and evaluate R@15 performance across these ranges. Table 12 presents the results for both the single- and multi-step retrieval settings. The passage length remains
<table>
  <tr>
    <th rowspan="2"> </th>
    <th colspan="3">n = 1</th>
    <th colspan="3">n = 2</th>
    <th colspan="3">n = 3</th>
    <th colspan="3">n = 4</th>
  </tr>
  <tr>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
  </tr>
  <tr>
    <th>Beam Search Length</th>
    <td>b = 2</td><td>58.1</td><td>66.0</td><td>69.5</td><td>59.2</td><td>68.9</td><td>71.3</td><td>57.9</td><td>68.0</td><td>71.5</td><td>58.4</td><td>67.6</td><td>71.5</td>
  </tr>
  <tr>
    <td></td><td>b = 3</td><td>55.9</td><td>64.6</td><td>67.9</td><td>57.2</td><td>66.6</td><td>70.2</td><td>58.1</td><td>67.8</td><td>71.0</td><td>56.7</td><td>66.1</td><td>70.4</td>
  </tr>
  <tr>
    <td></td><td>b = 4</td><td>54.9</td><td>62.9</td><td>67.3</td><td>56.6</td><td>66.3</td><td>69.3</td><td>58.1</td><td>67.9</td><td>71.0</td><td>56.1</td><td>66.1</td><td>69.9</td>
  </tr>
</table>

Table 9: Gear’s retrieval performance across different hyper-parameters in terms of maximum number of agent iterations (n) and graph expansion’s beam search length (b). Results are reported using Recall@k (R@k) for \( k \in \{5, 10, 15\} \) for the MuSiQue dataset.

consistent across passages; hence, triple density serves as a proxy for the quality of triple extraction.

The results showcase that SyncGE and Gear are more robust than the competition at retrieving suitable passages. NaiveGE’s performance tends to decline when the average number of triples associated with the gold passages either falls below or exceeds a certain threshold (for MuSiQue, the average number of triples extracted from the gold passages is 11.71). A similar trend is observed for HippoRAG w/ IRCoT in the case of golden passages associated with more than 11 triples. We believe that this trend can be partially attributed to the Personalised PageRank machinery that makes HippoRAG agnostic to the semantic relationships of the extracted triples. In contrast, SyncGE and Gear are able to maintain consistent performance across both dense and sparse triple extraction outcomes.

I Qualitative Analysis

I.1 Positive Instances in MuSiQue

Table 13 showcases some query instances where Gear achieves perfect recall in a single iteration, while HippoRAG w/ IRCoT achieves lower recall and consumes all available iterations. The presented examples illustrate how Gear’s Gist Memory \( \mathcal{G}^{(n)} \) precisely captures the essential information needed to answer MuSiQue’s queries, maintaining the appropriate level of granularity without including superfluous details. In contrast, HippoRAG w/ IRCoT struggles to retrieve crucial information—whether due to limitations in its triple extraction step or retriever functionality—such as the exact population of Venice, which is necessary for accurate responses. Furthermore, the verbose nature of IRCoT’s thought process component contrasts with Gear’s streamlined approach. The lack of such verbose component in our approach contributes to the fact that Gear requires fewer LLM tokens than the competition, as explained in subsection 8.3.

I.2 Negative Instances in MuSiQue

We manually assess 20 problematic cases in MuSiQue where Gear did not achieve full recall performance, and we identify the specific error types responsible for each. The findings, presented in Table 14 indicate that the majority of the errors are due to hallucinations of the LLM read steps, and only a very limited number of cases can be attributed to triple extraction.

I.3 Beyond MuSiQuE, 2Wiki and HotpotQA

We evaluate our framework on three open-domain multi-hop QA datasets: MuSiQue (Trivedi et al., 2022), HotpotQA (Yang et al., 2018), and 2Wiki-MultiHopQA (2Wiki) (Ho et al., 2020). Our dataset choices closely align with the multi-hop QA tasks, and are consistent with related studies in this space (Li et al., 2024; Fang et al., 2024; Gutierrez et al., 2024; Park et al., 2024).

In order to explore the generalisability of Gear in additional scenarios, we use the hand-picked case study data2 provided by Gutierrez et al.. These include four path-finding questions across four different domains: books, movies, universities and biomedicine. We test Gear against HippoRAG w/ IRCoT on these cases. Table 15 displays the results. In three out of the four cases, Gear outperforms the competition in recall, successfully identifying more relevant passages, and misses the relevant passages in only one case.

J Increasing the Number of Agent Iterations

Figure 5 expands upon the analysis shown in Figure 2 by evaluating retrieval performance over 20 iterations, rather than the initial 4 iterations. The results

2https://github.com/OSU-NLP-Group/HippoRAG/tree/main/data
<table>
  <tr>
    <th rowspan="2">LLM</th>
    <th colspan="4">MuSiQue</th>
    <th colspan="4">2Wiki</th>
    <th colspan="4">HotpotQA</th>
  </tr>
  <tr>
    <th>R@5</th><th>R@10</th><th>R@15</th><th>R@5</th><th>R@10</th><th>R@15</th><th>R@5</th><th>R@10</th><th>R@15</th>
  </tr>
  <tr>
    <td>Closed-source</td>
    <td>GPT-4o mini</td>
    <td>58.4</td><td>67.6</td><td>71.5</td><td>89.1</td><td>95.3</td><td>95.9</td><td>93.4</td><td>96.8</td><td>97.3</td>
  </tr>
  <tr>
    <td rowspan="2">Open-weight</td>
    <td>Llama-3.1-8B</td>
    <td>52.4</td><td>62.3</td><td>66.7</td><td>81.6</td><td>91.0</td><td>93.7</td><td>92.2</td><td>97.4</td><td>98.1</td>
  </tr>
  <tr>
    <td>Qwen-2.5-8B</td>
    <td>53.7</td><td>63.7</td><td>66.7</td><td>85.9</td><td>91.6</td><td>93.0</td><td>91.7</td><td>96.2</td><td>96.9</td>
  </tr>
</table>

Table 10: Retrieval performance of GEAR across different closed-source and open-weight models on MuSiQue, 2Wiki and HotpotQA. Results are reported using Recall@k (R@k) for \( k \in \{5, 10, 15\} \), showing the percentage of questions for which the correct entries are found within the top-\( k \) retrieved passages.

<table>
  <tr>
    <th rowspan="2"></th>
    <th colspan="3">MuSiQue</th>
    <th colspan="3">2Wiki</th>
    <th colspan="3">HotpotQA</th>
  </tr>
  <tr>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
    <th>R@5</th><th>R@10</th><th>R@15</th>
  </tr>
  <tr>
    <td>GPT-4o mini</td>
    <td>w/ diversity</td><td>48.7</td><td>57.7</td><td>61.2</td><td>72.6</td><td>80.9</td><td>82.4</td><td>87.4</td><td>93.3</td><td>95.2</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o diversity</td><td>47.0</td><td>53.9</td><td>58.4</td><td>68.2</td><td>76.0</td><td>77.4</td><td>85.0</td><td>92.2</td><td>94.3</td>
  </tr>
  <tr>
    <td>Llama-3.1-8B-Instruct</td>
    <td>w/ diversity</td><td>46.2</td><td>54.3</td><td>57.4</td><td>69.1</td><td>78.1</td><td>81.6</td><td>87.3</td><td>92.8</td><td>95.1</td>
  </tr>
  <tr>
    <td></td>
    <td>w/o diversity</td><td>44.9</td><td>52.7</td><td>55.0</td><td>66.9</td><td>75.9</td><td>78.2</td><td>85.0</td><td>91.7</td><td>94.4</td>
  </tr>
</table>

Table 11: Retrieval performance of the Hybrid + SyncGE method with different LLMs for the read step (see Eq. 1) w/ and w/o diversity for triple beam search. Results are reported using Recall@k (R@k) for \( k \in \{5, 10, 15\} \), showing the percentage of questions for which the correct entries are found within the top-\( k \) retrieved passages.

<table>
  <tr>
    <th rowspan="2">Retriever</th>
    <th colspan="4">\( \rho_t < 9 \)</th>
    <th colspan="4">\( 9 \leq \rho_t < 11 \)</th>
    <th colspan="4">\( 11 \leq \rho_t < 13 \)</th>
    <th colspan="4">\( 13 \leq \rho_t \)</th>
  </tr>
  <tr>
    <th></th><th></th><th></th><th></th>
    <th></th><th></th><th></th><th></th>
    <th></th><th></th><th></th><th></th>
    <th></th><th></th><th></th><th></th>
  </tr>
  <tr>
    <td>Single-step Retrieval</td>
    <td>Hybrid + NaiveGE</td><td>50.6</td><td>54.6</td><td>54.1</td><td>50.0</td>
  </tr>
  <tr>
    <td></td>
    <td>Hybrid + SyncGE</td><td>62.8 (\( \uparrow 12.2\% \))</td><td>61.2 (\( \uparrow 6.6\% \))</td><td>59.8 (\( \uparrow 5.7\% \))</td><td>60.1 (\( \uparrow 10.1\% \))</td>
  </tr>
  <tr>
    <td>Multi-step Retrieval</td>
    <td>HippoRAG w/ IRCot</td><td>64.4</td><td>65.5</td><td>55.0</td><td>52.8</td>
  </tr>
  <tr>
    <td></td>
    <td>GEAR</td><td>73.6 (\( \uparrow 9.2\% \))</td><td>73.7 (\( \uparrow 8.2\% \))</td><td>69.3 (\( \uparrow 14.3\% \))</td><td>69.7 (\( \uparrow 16.9\% \))</td>
  </tr>
</table>

Table 12: Retrieval performance for single- and multi-step retrievers across different triple density measurements in MuSiQue. Results are reported using R@15. Triple densities (\( \rho_t \)) are calculated as the average number of triples associated with the gold documents for the questions within the MuSiQue’s test set.

![Line plot showing evolution of R@15 over 20 iterations on MuSiQue, with multiple retrieval methods compared.](page_1012_1342_1012_388.png)

Figure 5: Evolution of R@15 over 20 iterations on MuSiQue. Recall is computed at each iteration using the cumulative set of retrieved documents, with prior recall values carried forward for questions that terminated in earlier iterations. The horizontal line indicates the single-step performance of Hybrid + SyncGE.
<table>
  <tr>
    <th>Query</th>
    <th>GEAR</th>
    <th>HippoRAG w/ IRCoT</th>
  </tr>
  <tr>
    <td>What team is the player who led the league in sacks in 2017 a member of?</td>
    <td>(Chandler Jones, team, Arizona Cardinals)<br>(Chandler Jones, led the league in sacks in, 2017)</td>
    <td>To answer the question about the player who led the league in sacks in 2017, I need to identify the player and their team from external sources, as the provided documents do not contain that specific information</td>
  </tr>
  <tr>
    <td>Who is the CEO of FleetBoston Financial’s purchaser?</td>
    <td>(FleetBoston Financial, purchaser, Bank of America)<br>(Bank of America, CEO, Brian Moynihan)</td>
    <td>The purchaser of FleetBoston Financial is Bank of America. To find the CEO of Bank of America, I need to look for information about the current leadership of the bank. Next, I will check for the current CEO of Bank of America to answer the question accurately.</td>
  </tr>
  <tr>
    <td>What UK label was purchased by the company that, along with the distributor of The Cube, and ABC, is the other major New York-based broadcaster?</td>
    <td>(CBS, purchased, Oriole Records)<br>(CBS, is a major broadcaster based in, New York)<br>(NBC, is a major broadcaster based in, New York)<br>(The Cube, was distributed by, NBC)</td>
    <td>The original network of Undercovers is NBC, which is one of the major broadcasters based in New York. Now, I need to find out which UK label was bought by NBC ...</td>
  </tr>
  <tr>
    <td>What was the 2018 population of the Italian city that’s underwater?</td>
    <td>(Venice, population in 2018, 260 897)</td>
    <td>The Italian city that is underwater is Krag, British Columbia, which is a ghost town...</td>
  </tr>
</table>

Table 13: Comparison of MuSiQue queries where GEAR achieves 100% recall at R@15 in a single iteration, while HippoRAG w/ IRCoT shows lower performance despite using all four available iterations. Cell colors indicate recall performance: green for 100% recall, red for 0% recall, and yellow for any intermediate value. Cell values in GEAR represent the proximal triples stored in the Gist Triple Memory. Cell values in HippoRAG w/ IRCoT represent IRCoT’s thought process.

demonstrate a consistent pattern across all methods: retrieval performance stabilises after approximately 4 iterations, with no substantial improvements or degradation in subsequent iterations. While some minor fluctuations occur beyond this point, they are negligible.

This performance plateau can be attributed to two key factors. First, the query re-writing mechanisms in all investigated approaches struggle to generate effective subsequent queries. Second, our analysis has identified several cases of unanswerable queries within MuSiQue’s answerable subset. A representative example is provided in Table 16.
<table>
  <tr>
    <th>Error Category</th>
    <th>Count</th>
    <th>Example</th>
  </tr>
  <tr>
    <td rowspan="2">Base Retriever Limitations</td>
    <td rowspan="2">2/20 (10%)</td>
    <td><b>Question:</b> Who owns the record label where the singer of All Right records?<br><b>Failure Explanation:</b> Instead of retrieving passages about the song 'All Right,' the base retriever returns passages about the songs 'All Right Now' and 'Who Owns My Heart,' which are unrelated to the gold passage.</td>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Reasoner Hallucinations</td>
    <td rowspan="2">2/20 (10%)</td>
    <td><b>Question:</b> What is the name of the castle in the city where the headquarters of the production company of A Cosmic Christmas is located?<br><b>Failure Explanation:</b> The LLM Reasoner concluded the question was answerable, even though key information related to 'A Cosmic Christmas' was still missing.</td>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Reader Hallucinations</td>
    <td rowspan="2">7/20 (35%)</td>
    <td><b>Question:</b> Who played the character in Willy Wonka and the Chocolate Factory that the performer of Victrola was named after?<br><b>Failure Explanation:</b> The triple {Victrola, named after, Violet Beauregardel} is hallucinated to complete the hop needed to reach the answer.</td>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Dataset Issues</td>
    <td rowspan="2">5/20 (25%)</td>
    <td><b>Question:</b> What other recognition did the Oscar winner for Best Actor in 2006 receive?<br><b>Failure Explanation:</b> The corpus has no mention that the 'Oscars in 2006' were also called the '78th academy awards', preventing the system from finding the relevant information.</td>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Missing Details during Triple Extraction</td>
    <td rowspan="2">1/20 (5%)</td>
    <td><b>Question:</b> What was the form of the language that the last name Sylvester comes from, used in the era of the man crowned Roman Emperor in AD 800, later known as?<br><b>Failure Explanation:</b> No triple extracted about the last sentence in the gold paragraph (titled 'Middle Ages'): 'By the reign of Charlemagne, the language had so diverged from the classical that it was later called Medieval Latin.'</td>
  </tr>
  <tr></tr>
  <tr>
    <td rowspan="2">Quality of the Constructed Graph (Triple Index)</td>
    <td rowspan="2">3/20 (15%)</td>
    <td><b>Question:</b> What is the largest medical school in the nation where, along with the country of citizenship of the mother of Marie Antoinette, many expelled French Jews relocated?<br><b>Failure Explanation:</b> The disambiguation between Marie Antoinette and Duchess Marie Antoinette is not correctly handled.</td>
  </tr>
  <tr></tr>
</table>

Table 14: Analysis of 20 problematic cases in the MuSiQue dataset, divided into various error categories.
<table>
  <tr>
    <th rowspan="2">Query</th>
    <th colspan="4">GEAR</th>
    <th colspan="4">HippoRAG w/ IRCoT</th>
  </tr>
  <tr>
    <th>Passage Titles</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@15</th>
    <th>Passage Titles</th>
    <th>R@5</th>
    <th>R@10</th>
    <th>R@15</th>
  </tr>
  <tr>
    <td>Which book was published in 2012 by an English author who is a Whitbread Award winner?</td>
    <td>\( \mathcal{P}_1 \) A Stitch in Time<br>\( \mathcal{P}_2 \) Stevie Parle<br>\( \mathcal{P}_3 \) The Red House<br>\( \mathcal{P}_4 \) Whitbread Awards<br>\( \cdots \)<br>\( \mathcal{P}_{10} \) Mark Haddon</td>
    <td>50</td>
    <td>100</td>
    <td>100</td>
    <td>\( \mathcal{P}_1 \) Oranges Are Not<br>\( \cdots \)<br>\( \mathcal{P}_2 \) Mark Haddon<br>\( \mathcal{P}_3 \) William Trevor . . .<br>\( \mathcal{P}_4 \) The Curious . . .<br>\( \cdots \)<br>\( \mathcal{P}_{11} \) The Red House</td>
    <td>50</td>
    <td>50</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Which war film based on a non fiction book was directed by someone famous in the science fiction and crime genres?</td>
    <td>\( \mathcal{P}_1 \) And the Band . . .<br>\( \mathcal{P}_2 \) Band of Brothers<br>\( \mathcal{P}_3 \) Aircraft in Fiction<br>\( \mathcal{P}_4 \) Unchained<br>\( \cdots \)</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
    <td>\( \mathcal{P}_1 \) Shangai Patrol<br>\( \mathcal{P}_2 \) Black Hawk Down<br>\( \mathcal{P}_3 \) Ridley Scott<br>\( \mathcal{P}_4 \) Outline of science<br>\( \cdots \)</td>
    <td>100</td>
    <td>100</td>
    <td>100</td>
  </tr>
  <tr>
    <td>Which Stanford professor works on the neuroscience of Alzheimer’s?</td>
    <td>\( \mathcal{P}_1 \) Thomas C. Sudhof<br>\( \mathcal{P}_2 \) Thomas C. Sudhof<br>\( \mathcal{P}_3 \) Judes Poirier<br>\( \mathcal{P}_4 \) Thomas C. Sudhof<br>\( \cdots \)<br>\( \mathcal{P}_{10} \) Robert Malenka<br>\( \mathcal{P}_{13} \) Robert Malenka</td>
    <td>50</td>
    <td>75</td>
    <td>100</td>
    <td>\( \mathcal{P}_1 \) Thomas C. Sudhof<br>\( \mathcal{P}_2 \) Thomas C. Sudhof<br>\( \mathcal{P}_3 \) Manolis Kellis<br>\( \mathcal{P}_4 \) Giovanna Malluci<br>\( \mathcal{P}_5 \) Dena Dubal</td>
    <td>50</td>
    <td>50</td>
    <td>50</td>
  </tr>
  <tr>
    <td>What drug is used to treat chronic lymphocytic leukemia by interacting with cytosolic p53?</td>
    <td>\( \mathcal{P}_1 \) P53 Regulation<br>\( \mathcal{P}_2 \) Venetoclax<br>\( \mathcal{P}_3 \) Chlorambucil<br>\( \mathcal{P}_4 \) Chronic Lymphocytic Leukemia</td>
    <td>50</td>
    <td>50</td>
    <td>50</td>
    <td>\( \mathcal{P}_1 \) P53<br>\( \mathcal{P}_2 \) Cimtuzumab<br>\( \mathcal{P}_3 \) MDC1 Function<br>\( \mathcal{P}_4 \) Chronic Lymphocytic Leukemia</td>
    <td>0</td>
    <td>0</td>
    <td>0</td>
  </tr>
</table>

Table 15: Retrieval performance comparison between GeAR and HippoRAG w/ IRCoT. Both models are configured with a maximum number of 4 iterations. The example questions are taken from Gutierrez et al. and showcase multi-hop path-finding queries across different domains: books, movies, universities and biomedicine. Cell colours indicate recall performance: green for 100% recall, red for 0% recall, and yellow for any intermediate value. Retrieved passage titles are listed in the 'Passage Titles' columns, with bold text indicating gold passages and \( \mathcal{P}_n \) indicating their position in the retrieved list.

<table>
  <tr>
    <th>Question</th>
    <th>Who did the producer of Big Jim McLain play in True Grit?</th>
  </tr>
  <tr>
    <th rowspan="2">Gold Passages</th>
    <td>1. <b>Big Jim McLain:</b> <i>Big Jim McLain</i> is a 1952 political thriller film starring John Wayne and James Arness as HUAC investigators.</td>
  </tr>
  <tr>
    <td>2. <b>True Grit</b> is a 1969 American western film. It is the first film adaptation of Charles Portis’ 1968 novel of the same name. The screenplay was written by Marguerite Roberts. The film was directed by Henry Hathaway and starred Kim Darby as Mattie Ross and John Wayne as U.S. Marshal Rooster Cogburn. Wayne won his only Academy Award for his performance in this film and reprised his role for the 1975 sequel Rooster Cogburn.</td>
  </tr>
  <tr>
    <th>Comment</th>
    <th>No information about who was the producer of Big Jim McLain is provided in the gold passages</th>
  </tr>
</table>

Table 16: Example of a query from MuSiQue that is not answerable solely based on the provided gold passages.
K  Prompts

K.1  Offline Prompts

Triple Extraction

# Instruction

Your task is to construct an RDF (Resource Description Framework) graph from the given passages and named entity lists.
Respond with a JSON list of triples, with each triple representing a relationship in the RDF graph.
Pay attention to the following requirements:
- Each triple should contain at least one, but preferably two, of the named entities in the list for each passage.
- Clearly resolve pronouns to their specific names to maintain clarity.

Convert the paragraph into a JSON dict containing a named entity list and a triple list.

# Demonstration #1

Paragraph:
...
Magic Johnson

After winning a national championship with Michigan State in 1979, Johnson was selected first overall in the 1979 NBA draft by the Lakers, leading the team to five NBA championships during their "Showtime" era.
...
{
  "named_entities": ["Michigan State", "national championship", "1979", "Magic Johnson", "National Basketball Association", "Los Angeles Lakers", "NBA Championship"]
}
{
  "triples": [
    ("Magic Johnson", "member of sports team", "Michigan State"),
    ("Michigan State", "award", "national championship"),
    ("Michigan State", "award date", "1979"),
    ("Magic Johnson", "draft pick number", "1"),
    ("Magic Johnson", "drafted in", "1979"),
    ("Magic Johnson", "drafted by", "Los Angeles Lakers"),
    ("Magic Johnson", "member of sports team", "Los Angeles Lakers"),
    ("Magic Johnson", "league", "National Basketball Association"),
    ("Los Angeles Lakers", "league", "National Basketball Association"),
    ("Los Angeles Lakers", "award received", "NBA Championship")
  ]
}
...

# Demonstration #2

Paragraph:
...
Elden Ring

Elden Ring is a 2022 action role-playing game developed by FromSoftware. It was directed by Hidetaka Miyazaki with worldbuilding provided by American fantasy writer George R. R. Martin.
...
{
  "named_entities": ["Elden Ring", "2022", "Role-playing video game", "FromSoftware", "Hidetaka Miyazaki", "United States of America", "fantasy", "George R. R. Martin"]
}
{
  "triples": [
    ("Elden Ring", "publication", "2022"),
    ("Elden Ring", "genre", "action role-playing game"),
    ("Elden Ring", "publisher", "FromSoftware"),
    ("Elden Ring", "director", "Hidetaka Miyazaki"),
    ("Elden Ring", "screenwriter", "George R. R. Martin"),
    ("George R. R. Martin", "country of citizenship", "United States of America"),
    ("George R. R. Martin", "genre", "fantasy")
  ]
}
...

# Input
Convert the paragraph into a JSON dict, it has a named entity list and a triple list.

Paragraph:
...
{wiki_title}
{passage}

K.2  Online Retrieval Prompts

The blue-highlighted portions of the Reader prompt below indicate additional text that is only required when the Gist Memory \( \mathcal{G}^{(n)} \) is active. When Gist Memory is inactive, these blue sections should be omitted, and the {triples} parameter should be left empty.

<table>
  <tr>
    <th>Reader with and without Gist Memory</th>
    <th>(§4.1 and §5.1)</th>
  </tr>
  <tr>
    <td colspan="2">
      Your task is to find facts that help answer an input question.<br><br>
      You should present these facts as knowledge triples, which are structured as ("subject", "predicate", "object").<br>
      Example:<br>
      Question: When was Neville A. Stanton's employer founded?<br>
      Facts: ("Neville A. Stanton", "employer", "University of Southampton"), ("University of Southampton", "founded in", "1862")<br><br>
      Now you are given some documents:<br>
      {docs}<br><br>
      Based on these documents <a href="https://www.example.com">and some preliminary facts provided below.</a><br>
      find additional supporting fact(s) that may help answer the following question.<br><br>
      Note: if the information you are given is insufficient, output only the relevant facts you can find.<br><br>
      Question: {query}<br>
      Facts: {triples}
    </td>
  </tr>
</table>

<table>
  <tr>
    <th>Reasoning for Termination</th>
    <th>(§5.2)</th>
  </tr>
  <tr>
    <td colspan="2">
      # Task Description:<br>
      You are given an input question and a set of known facts:<br>
      Question: {query}<br>
      Facts: {triples}<br><br>
      Your reply must follow the required format:<br>
      1. If the provided facts contain the answer to the question, your should reply as follows:<br>
      Answerable: Yes<br>
      Answer: ...<br><br>
      2. If not, you should explain why and reply as follows:<br>
      Answerable: No<br>
      Why: ...<br><br>
      # Your reply:
    </td>
  </tr>
</table>

<table>
  <tr>
    <th>Query Re-writing</th>
    <th>(§5.3)</th>
  </tr>
  <tr>
    <td colspan="2">
      # Task Description:<br>
      You will be presented with an input question and a set of known facts.<br>
      These facts might be insufficient for answering the question for some reason.<br>
      Your task is to analyze the question given the provided facts and determine what else information is needed for the next step.<br><br>
      # Example:
    </td>
  </tr>
</table>
Question: What region of the state where Guy Shepherdson was born, contains SMA Negeri 68?
Facts: ("Guy Shepherdson", "born in", "Jakarta")
Reason: The provided facts only indicate that Guy Shepherdson was born in Jakarta, but they do not provide any information about the region of the state that contains SMA Negeri 68.
Next Question: What region of Jakarta contains SMA Negeri 68?

# Your Task:
Question: {query}
Facts: {triples}
Reason: {reason}
Next Question:

K.3 Online Question Answering Prompts

The following prompt with retrieved passages combines the QA generation prompts from Gutierrez et al. and Wang et al.. For the variation without retrieved passages, we omit the preamble and only include the instruction, highlighted in purple .

<table>
  <tr>
    <th colspan="2">Retrieved Passages with In-context Example</th>
    <th>(Table 3)</th>
  </tr>
  <tr>
    <td colspan="3">
      As an advanced reading comprehension assistant, your task is to analyze text passages and corresponding questions meticulously, with the aim of providing the correct answer.<br>
      ===============<br>
      For example:<br>
      ===============<br>
      Wikipedia Title: Edward L. Cahn<br>
      Edward L. Cahn (February 12, 1899 – August 25, 1963) was an American film director.<br><br>
      Wikipedia Title: Laughter in Hell<br>
      Laughter in Hell is a 1933 American Pre-Code drama film directed by Edward L. Cahn and starring Pat O'Brien. The film's title was typical of the sensationalistic titles of many Pre-Code films. Adapted from the 1932 novel of the same name buy Jim Tully, the film was inspired in part by "I Am a Fugitive from a Chain Gang" and was part of a series of films depicting men in chain gangs following the success of that film. O'Brien plays a railroad engineer who kills his wife and her lover in a jealous rage and is sent to prison. The movie received a mixed review in "The New York Times" upon its release. Although long considered lost, the film was recently preserved and was screened at the American Cinematheque in Hollywood, CA in October 2012. The dead man's brother ends up being the warden of the prison and subjects O'Brien's character to significant abuse. O'Brien and several other characters revolt, killing the warden and escaping from the prison. The film drew controversy for its lynching scene where several black men were hanged. Contrary to reports, only blacks were hung in this scene, though the actual executions occurred off-camera (we see instead reaction shots of the guards and other prisoners). The "New Age" (an African American weekly newspaper) film critic praised the scene for being courageous enough to depict the atrocities that were occurring in some southern states.<br><br>
      Wikipedia Title: Theodred II (Bishop of Elmham)<br>
      Theodred II was a medieval Bishop of Elmham. The date of Theodred’s consecration unknown, but the date of his death was sometime between 995 and 997.<br><br>
      Wikipedia Title: Etan Boritzer<br>
      Etan Boritzer (born 1950) is an American writer of children ’s literature who is best known for his book" What is God?" first published in 1989. His best selling" What is?" illustrated children’s book series on character education and difficult subjects for children is a popular teaching guide for parents, teachers and child- life professionals. Boritzer gained national critical acclaim after "What is God?" was published in 1989 although the book has caused controversy from religious fundamentalists for its universalist views. The other current books in the "What is?" series include What is Love?, What is Death?, What is Beautiful?, What is Funny?, What is Right?, What is Peace?, What is Money?, What is Dreaming?, What is a Friend?, What is True?, What is a Family?, What is a Feeling?" The series is now also translated into 15 languages. Boritzer was first published in 1963 at the age of 13 when he wrote an essay in his English class at Wade Junior High School in the Bronx, New York on the assassination of John F. Kennedy. His essay was included in a special anthology by New York City public school children compiled and published by the New York City Department of Education.<br><br>
      Wikipedia Title: Peter Levin<br>
      Peter Levin is an American director of film, television and theatre.<br><br>
      Question: When did the director of film Laughter In Hell die?<br>
      Answer: August 25, 1963.<br>
      ===============
      Given the following text passages and questions, please present a concise, definitive answer, devoid of additional elaborations, and of maximum length of 6 words.
      ===============
    </td>
  </tr>
</table>
Wikipedia Title : {title} {text} for each retrieved passage ...
Question: {question}
Answer:

<table>
  <tr>
    <th colspan="2">No Retrieved Passages</th>
    <th>(Table 3)</th>
  </tr>
  <tr>
    <td colspan="3">
      <span style="color:#ff0066;">Given the following question, please present a concise, definitive answer, devoid of additional elaborations, and of maximum length of 6 words.</span>
    </td>
  </tr>
  <tr>
    <td colspan="3">Question: {question}</td>
  </tr>
  <tr>
    <td colspan="3">Answer:</td>
  </tr>
</table>

Reviewer paper summaries: This article presents a novel system that significantly improves retrieval performance in multi-hop question answering tasks through graph expansion and an agent framework. The system showcases strengths in introducing innovative retrieval methods, achieving state-of-the-art results, and offering a more efficient approach to multi-hop question answering.
Summary of strengths: 1. GEAR enhances retrieval performance in multi-hop question answering showing better results on challenging datasets like MuSiQue.
2. Graph expansion and an agent framework improves the efficiency of retrieval-augmented generation systems.
3. Introduces SyncGE, a novel graph-based retriever that leverages an LLM for graph exploration, leading to state-of-the-art retrieval performance.
4. Shows the ability to answer multi-hop questions in fewer iterations with fewer  tokens.
Summary of weaknesses: 1. The scalability of the system or its performance under different conditions are not sufficiently discussed.
2. The paper could benefit from a more detailed comparison with existing retrieval-augmented models for Q.
3. Evaluation of GEAR focuses primarily on retrieval performance and efficiency, while ignoring other aspects like robustness to noise.
4. Article lacks a thorough discussion on the generalizability of GEAR beyond the datasets mentioned.
Comments, suggestions and typos: "GEAR archives superior performance compared to other multi-step retrieval methods..."
Correction: Replace "archives" with "achieves.
Overall assessment scores: 3