$\text{M}^3\text{GQA}$: A Multi-Entity Multi-Hop Multi-Setting Graph Question Answering Benchmark
Authors: Boci Peng, Yongchao Liu, Xiaohe Bo, Jiaxin Guo, Yun Zhu, Xuanbo Fan, Chuntao Hong, Yan Zhang
Accepted at: acl25

M^3GQA: A Multi-Entity Multi-Hop Multi-Setting Graph Question Answering Benchmark

Boci Peng1  Yongchao Liu2*  Xiaohe Bo3  Jiaxin Guo1
Yun Zhu4  Xuanbo Fan1  Chuntao Hong2  Yan Zhang1*
1School of Intelligence Science and Technology, Peking University
2Ant Group
3Gaoling School of Artificial Intelligence, Renmin University of China
4College of Computer Science and Technology, Zhejiang University
{bcpeng,guojiaxin,xuanbo.fan}@stu.pku.edu.cn, {yongchao.ly,chuntao.hct}@antgroup.com,
xiaoh@ruc.edu.cn, zhuyun_dcd@zju.edu.cn, zhyzh@pku.edu.cn

Abstract

Recently, GraphRAG systems have achieved remarkable progress in enhancing the performance and reliability of large language models (LLMs). However, most previous benchmarks are template-based and primarily focus on few-entity queries, which are monotypic and simplistic, failing to offer comprehensive and robust assessments. Besides, the lack of ground-truth reasoning paths also hinders the assessments of different components in GraphRAG systems. To address these limitations, we propose M^3GQA, a complex, diverse, and high-quality GraphRAG benchmark focusing on multi-entity queries, with six distinct settings for comprehensive evaluation. In order to construct diverse data with semantically correct ground-truth reasoning paths, we introduce a novel reasoning-driven four-step data construction method, including tree sampling, reasoning path backtracking, query creation, and multi-stage refinement and filtering. Extensive experiments demonstrate that M^3GQA effectively reflects the capabilities of GraphRAG methods, offering valuable insights into the model performance and reliability. By pushing the boundaries of current methods, M^3GQA establishes a comprehensive, robust, and reliable benchmark for advancing GraphRAG research. Our code and dataset are available at https://github.com/pengboci/M3GQA.

1 Introduction

In recent years, Retrieval-Augmented Generation (RAG) (Fan et al., 2024) has demonstrated significant potential in enhancing the performance and mitigating hallucinations in LLMs by retrieving relevant external knowledge. However, traditional RAG methods often struggle to answer global-level or complex questions effectively (Edge et al., 2024), as they fail to capture essential relational and contextual knowledge. To address these limitations,

![Comparison of examples between multi-entity queries and few-entity queries.](page_784_654_496_246.png)

Figure 1: Comparison of examples between multi-entity queries and few-entity queries.

GraphRAG methods (Edge et al., 2024; Guo et al., 2024b; Li et al., 2024; Peng et al., 2024b) are proposed to enhance LLMs by incorporating relational and contextual knowledge between key entities, integrating relevant structural information to improve response accuracy and comprehension.

As the field of GraphRAG continues to advance rapidly, effectively evaluating the capabilities of GraphRAG methods has become essential for driving further research. However, previous benchmarks, as listed in Table 1, are monotypic and simplistic, failing to offer comprehensive and robust assessments: (1) They lack complex, multi-entity queries, which are common in real-world scenarios and challenging to models. Although CWQ (Talmor and Berant, 2018), GraILQA (Gu et al., 2021), and LC-QUAD 2.0 (Dubey et al., 2019) include few multi-entity queries, they mainly focus on cases with only three entities, and the quantity is quite limited. (2) The reliance on templates or manually annotated QA pairs in these benchmarks leads to high labor costs and restricted query formats, which may cause the model overfitting to specific query structures, limiting the models’ robustness and adaptability. (3) These benchmarks lack ground-truth reasoning paths, which impedes a comprehensive evaluation of various components in GraphRAG systems. While earlier
<table>
  <tr>
    <th></th>
    <th>Multi Hop</th>
    <th>Multi Entity</th>
    <th>Multi Setting</th>
    <th>Non-Template or Automatic</th>
    <th>Reasoning Paths</th>
  </tr>
  <tr>
    <td>WebQ (Berant et al., 2013)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>SimpliQ (Bordes et al., 2015)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>WebQSP (Yih et al., 2016)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>MetalQA (Zhang et al., 2018)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>GrauQA (Gu et al., 2021)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>LC-QUAD2.0 (Dubey et al., 2019)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>QALD10-en (Perevalov et al., 2022)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>CWQ (Talmor and Berant, 2018)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>TextGraphs2024 (Sakhovskiy et al., 2024)</td>
    <td>✓</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
    <td>x</td>
  </tr>
  <tr>
    <td>M^3GQA (ours)</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
    <td>✓</td>
  </tr>
</table>

Table 1: Comparisons between M^3GQA and previous datasets. In “Multi Entity” column, “✓” indicates the data contains less than 10% multi-entity queries or only contains queries with no more than three entities. “Reasoning Paths” indicates whether the dataset provides the ground-truth reasoning paths, and “✓” denotes that using heuristic rules to generate reasoning paths.

studies (Sakhovskiy et al., 2024) use shortest paths between query and answer entities as the ground-truth, these approaches have inherent limitations: First, reasoning paths for a given query may not necessarily be the shortest. Second, multiple shortest paths with different semantics may exist, making it difficult to ensure the semantic correctness.

To solve these challenges, we propose M^3GQA, a complex, diverse, and high-quality GraphRAG benchmark, with six distinct settings to comprehensively evaluate GraphRAG systems. Unlike previous query-first data construction methods, we propose a novel reasoning-first approach. Here, we first sample two sets of nodes with certain relations, serving as query and answer entities. Then we construct the ground-truth reasoning paths with a backtracking algorithm. After that, an LLM is utilized to generate corresponding queries based on previous information. Due to the flexibility of sampling, this method generates more complex and diverse QA data, while ensuring the semantic correctness of reasoning paths. Moreover, our method is independent of specific graph schemas or expert knowledge, making it easily adaptable to new domains. During the query generation process, since the factual information, such as the reasoning path, query entity, and answer entity, is derived directly from knowledge graphs, issues such as hallucination and bias in LLMs are effectively mitigated. Nevertheless, to further improve the data quality, we introduce a multi-stage refinement and filtering process, and manually curate the validation and test sets. In this way, M^3GQA offers a comprehensive and reliable evaluation for GraphRAG systems.

Extensive experimental results across six settings, focusing on two key aspects: the effectiveness and efficiency of graph retrieval, and the quality of answer generation, demonstrate that our benchmark accurately reflects the capabilities of GraphRAG methods, providing valuable insights into the model performance and reliability, thereby advancing the GraphRAG research.

Our contributions can be summarized as follows:

• To the best of our knowledge, M^3GQA is the first GraphRAG benchmark focusing on multi-entity queries, a highly practical yet challenging aspect in GraphRAG systems.

• To automatically generate complex, diverse, high-quality multi-entity query data with semantically correct reasoning paths, we propose an innovative four-step reasoning-first data construction method. The method is also easily adaptable to other domains.

• Extensive experiments show that M^3GQA accurately reflects the capabilities of GraphRAG methods, providing valuable insights into the model performance and reliability, thereby advancing GraphRAG research.

2 Related Work

GraphRAG Methods. GraphRAG is an emerging technique that enhances Vallina RAG by incorporating graph data, with three main steps: graph-based indexing, graph-guided retrieval, and graph-enhanced generation (Peng et al., 2024b). The indexing phase focuses on constructing and indexing knowledge graphs from text corpus (Edge et al., 2024; Guo et al., 2024b). This paper primarily focuses on the latter two stages, i.e. the retrieval of relevant graph information and the answer generation. Early methods for retrieval and generation mainly rely on GNNs (Lin et al., 2019; Yasunaga et al., 2021; Dong et al., 2023; Taunk et al., 2023) or small language models (Sun et al., 2019; Zhang et al., 2022; Jiang et al., 2023b; Huang et al., 2023; Li et al., 2023; Peng et al., 2024a) (e.g., BERT (Devlin et al., 2019), RoBERTa (Liu et al., 2019)). Recently, due to the outstanding reasoning and text generation capabilities of LLMs (OpenAI, 2024; Zhao et al., 2024), various LLM-based GraphRAG methods have been proposed. Some methods (Jiang et al., 2023a; Sun et al., 2024; Luo et al., 2024; Jin et al., 2024; Jiang et al., 2024a) utilize LLMs for both graph retrieval and
answer generation, while others leverage graph algorithms (Salnikov et al., 2023; He et al., 2024; Wang et al., 2024; Jiang et al., 2024b; Wen et al., 2024) or smaller models (Wu et al., 2023; Guo et al., 2024a; Hu et al., 2024; Mavromatis and Karypis, 2024) as graph retrievers, considering the time costs and expenses. Previous methods have achieved remarkable success on traditional datasets and a more challenging benchmark is needed for further development of GraphRAG.

GraphRAG Datasets. Various benchmarks (Berant et al., 2013; Bordes et al., 2015; Yih et al., 2016; Mihaylov et al., 2018; Talmor and Berant, 2018; Zhang et al., 2018; Talmor et al., 2019; Dubey et al., 2019; Gu et al., 2021; Perevalov et al., 2022; Cao et al., 2022; Sen et al., 2022; Mallen et al., 2023; Sakhovskiy et al., 2024) have been introduced to evaluate the performance of GraphRAG systems. Table 1 lists several widely used datasets along with their features, showing that current datasets mainly focus on few-entity queries. Although CWQ (Talmor and Berant, 2018), GrailQA (Gu et al., 2021), and LC-QUAD 2.0 (Dubey et al., 2019) contain several multi-entity queries, the number is relatively small and primarily concentrated in the case of three entities. Furthermore, previous datasets are created based on templates or manually annotated QA pairs, which may cause restricted query formats and model overfit. Also, reasoning paths corresponding to the QA pairs are not available in previous datasets. Therefore, we propose a novel reasoning-first data construction method and create a multi-entity multi-hop multi-setting benchmark named M^3GQA.

3 Task Formulation

Graph Question Answering (GQA) requires models to answer natural language queries according to a given graph, which is an important application of GraphRAG. The graph can be formally defined as \( G = (V, E, \{x_v\}, \{x_e\}) \), where \( V \) and \( E \) are node and edge sets, \( \{x_v\} \) and \( \{x_e\} \) represent their corresponding test attributes. In this paper, for the convenience of data construction, we use Knowledge Graphs to provide structured information, where attributes of nodes and edges are their names. Given a query \( q \) and a graph \( G \), models need to first locate the nodes \( V_q \subset V \) in the query (query entities), and then determine a node or a node set \( V_a \subset V \) (answer entities) in the graph \( G \) as the answer.

In M^3GQA, we focus primarily on multi-entity queries, which are defined here as queries that involve at least three nodes from the graph, i.e. \( |V_q| \geq 3 \). We first introduce 4 general settings with different complexities, which are as follows:

Single-hop Setting: Queries in the single-hop setting can be answered with a single reasoning step. We ensure there is only one answer node, meaning that in the reasoning path, the answer node \( v_a \) is at a distance of 1 from all query nodes. Therefore, the reasoning path can be shaped like: \( \{v_q \xrightarrow{e} v_a | v_q \in V_q\} \). This setting is theoretically the simplest, designed to assess the model’s basic ability to handle multi-entity queries.

Multi-hop Setting: Queries in this setting require starting from multiple query entities and reaching a single answer node using multi-hop reasoning. The number of reasoning steps from different query entities may vary, so the reasoning path can be defined as: \( \{v_q \xrightarrow{e_1} v_m \xrightarrow{e_2} \cdots \xrightarrow{e_k} v_a | v_q \in V_q\} \), where \( k > 1 \) represents the hop count, which may vary with different query entities. The answering process can be intuitively understood as starting from the child nodes at different levels of a tree (query entities) and searching for the root node (answer entity). Due to the highly variable number and distribution of query entities, the reasoning process can be significantly more complex than few-entity multi-hop questions.

Set Setting: The main difference between the set setting and the two settings mentioned above is that the answer to each query is a set of nodes, that is \( |V_a| > 1 \). The primary focus of the setting is to assess the model’s ability to obtain a complete answer without any omissions.

Aggregation Setting: Building on the set setting, we introduce additional constraints (e.g., maximum value, minimum value, etc.) to the set of answer nodes, so that the final answer is a single node from the node set satisfying the given conditions. In this setting, the model first needs to identify the original answer set, and then, by incorporating information from neighbor nodes, compare and reason across multiple similar entities. Therefore, the reasoning path in the aggregation setting can be denoted as: \( \{v_q \xrightarrow{e_1} v_m \xrightarrow{e_2} \cdots \xrightarrow{e_k} v_a \xleftarrow{e} v_n | v_q \in V_q, v_n \in V_n\} \), where \( k \geq 1 \) and \( V_n \) is a subset of \( v_a \)'s neighbor nodes.

Beyond the general settings above, we innovatively propose other two special settings:

Editing Setting: Considering that many queries can be answered by LLMs solely based on their parameterized knowledge, we introduce the editing
setting. Specifically, based on the single-hop setting and multi-hop setting, we modify the answer node \( v_a \) in the original graph, replacing it with a new node \( v'_a \) generated by LLMs that have different textual attributes. The model needs to respond with the new answer node \( v'_a \), rather than the original one. The performance under this setting highly depends on the retrieval quality. It also enables us to investigate the potential applications of GraphRAG in the context of knowledge updating for LLMs.

Answerability Setting: In the answerability setting, we first construct the data following single and multi-hop settings, and then randomly select a subset of the data, remove the answer nodes and their neighbors from \( G \) to create a new graph \( G' \). Given a query \( q \) and graph \( G' \), the model is required to determine whether \( G' \) contains the answer to the query. This setting accesses the model’s ability to perform graph retrieval and evaluate its sensitivity to the information in the retrieved subgraphs.

4 Data Construction

In this section, we will introduce our reasoning-first data construction method. Our ultimate goal is to obtain diverse queries with multiple entities, their corresponding answers, and reasoning paths. To achieve this, we first obtain the graphical representation of the query, i.e., reasoning paths, and then map it to the textual representation, i.e., QA pairs of natural language. Specifically, we first propose a tree sampling algorithm to sample two sets of nodes with certain relations, serving as query and answer nodes respectively. Since directly generating queries from query and answer nodes is challenging, we then utilize a backtracking algorithm to obtain the original reasoning paths. Afterward, an LLM is leveraged to create queries based on query nodes, answer nodes, and corresponding reasoning paths. Finally, we refine reasoning paths and filter low-quality data with a multi-stage strategy. Figure 2 shows the overview of our method.

4.1 Tree Sampling

To sample query nodes and answer nodes\footnote{For simplicity, we first consider the case where there is only a single answer node.} with certain relationships, we propose a novel tree sampling method. Specifically, we randomly select a node \( v_a \in V \) as the starting point and perform a constrained breadth-first search (C-BFS), where both the search width and depth are limited. All search paths form a tree with the selected node \( v_a \) as the root node, which then serves as the answer node. The process can be formulated as:

\[
G_{T_{v_a}} = \text{C-BFS}(v_a, G, D_{\max}, W_{\max}),
\]

where \( D_{\max} \) is the maximum search depth and \( W_{\max} \) is the maximum search width. \( G_{T_{v_a}} = (V_{T_{v_a}}, E_{T_{v_a}}) \) denotes the search tree with \( v_a \) as the root node, which is a subgraph of \( G \).

Next, we randomly sample \( n \ (n \geq 3) \) nodes from the non-root nodes of the tree, serving as the query nodes. Afterwards, we can obtain multiple \( (v_a, V_q) \) pairs, where \( V_q = \{v_{q_i}\}_{i=1}^n \) denotes the query node set. This process can be further formulated as \( \{(v_a, V_q)|v_a \in V, V_q \subset V_{T_{v_a}} \setminus \{v_a\}\} \). Considering that each setting in M^3GQA has its unique characteristics and constraints, we then sequentially explain how to adapt the tree sampling algorithm to these different settings. A detailed introduction with strict definitions and mathematical notation can be found in the Appendix D.

Single-hop Setting: The main characteristic of the single-hop setting is that the hop count is single, that is \( D_{\max} = 1 \). Besides, to ensure the uniqueness of the answer node, we examine all other common neighboring nodes of query nodes to ensure that no other node is linked to query nodes with the same relations.

Multi-hop Setting: In the multi-hop setting, the search depth must be at least 2, i.e. \( D_{\max} > 1 \). Additionally, to ensure the answer’s uniqueness, we examine the first level ancestor nodes (the root node at level 0) of each query node, using the same detection method as the single-hop setting.

Set Setting: The data construction method in the set setting is more complex. Since the answer is a node set, we need to consider the initialization of the answer nodes and the process for conducting the search. For answer node selection, we first adopt the same searching approach as the single-hop setting. Here, we additionally restrict that the relation remains consistent during the search process. Then we treat the query node in the single-hop setting as the answer nodes in the set setting, ensuring that answer nodes share some similar properties. As for the search process, a straightforward approach is to use the answer nodes as the starting points and apply multi-source BFS. However, this may result in a disconnected search tree, which makes it hard to form a query. To address this issue, we first detect all the shared
Figure 2: The overview of our four-step data construction method: tree sampling, reasoning path backtracking, query creation, and multi-stage refinement and filtering.

neighboring nodes of the answer nodes and then start the multi-source BFS from these nodes.

Aggregation Setting: Based on the set setting, the aggregation setting imposes further constraints on answer nodes. Specifically, we examine all common neighbors of answer nodes in the reverse direction of the BFS process and construct the data using two strategies. The first strategy is to identify a common relation shared by all answer nodes. Then neighboring nodes connected by this relation are included in the search tree for subsequent processes. The second strategy is to obtain the relation that is unique to one of the answer nodes. In this case, we add the edge to the search tree and treat that special answer node as the final answer.

Editing Setting: To construct data in the editing setting, we first modify the graph using the approach mentioned in Section 3 and then apply the tree sampling algorithm using the same method as the single-hop and multi-hop settings.

Answerability Setting: In contrast to the steps in the editing setting, here we first apply the tree sampling algorithm and then modify the graph. In this way, we can determine whether the query constructed based on the reasoning path can be answered using the given graph.

4.2 Reasoning Path Backtracking

After obtaining the query and answer nodes, we construct possible reasoning paths based on the search tree. Considering that the search tree may contain a lot of irrelevant information, we propose a backtracking method to prune it. Specifically, we take each query node \( v_q \) as the starting point and trace back along the search tree until we reach each answer node \( v_a \). This process can be denoted as:

\[
p_{v_q, v_a} = \text{BackTrack}(v_q, v_a, G_{T_{v_a}}).
\]

The resulting reasoning paths can be represented as \( \mathcal{P} = \{ p_{v_q, v_a} | v_q \in V_q, v_a \in V_a \} \). To provide an intuitive understanding, we include examples of reasoning paths in six settings in Appendix A.3.

4.3 Query Creation

In this section, we aim to create the corresponding queries according to query nodes \( V_q \), answer nodes \( V_a \), and reasoning paths \( \mathcal{P} \). Thanks to the powerful text generation capabilities of LLMs, we can generate diverse, and fluent queries with carefully designed prompts. Besides, since the factual information, such as the reasoning path, query entity, and answer entity, is derived directly from the knowledge graph, issues such as hallucination and bias in LLMs are effectively mitigated during the question generation process. The query creation process can be formalized as:

\[
q = \text{LLM}_c(V_q, V_a, \mathcal{P}).
\]

Queries in majority settings of M^3GQA can be directly generated using Equation 3, with the only exception of the aggregation setting. This is because queries in this setting not only need to represent the relationship between the query and answer
nodes but also impose additional constraints to ensure that the final answer is one of the nodes with similar attributes, which is too complex for LLMs. Therefore, we employ a two-step method to generate queries in the aggregation setting. We first create queries whose answer is a node set, ignoring the additional constraints. Subsequently, we incorporate the constraints into prompts and call LLMs again to rewrite the queries in the previous step.

Additionally, to make created queries more natural and fluent, and better resemble the questions humans would ask in real-world scenarios, we add an additional LLM call for query refinement, which can be formulated as:

\[
q = \text{LLM}_r(q, V_q).
\]

We provide all prompts in Appendix F.

4.4 Multi-Stage Refinement and Filtering

Considering that LLMs may not incorporate all sampled query nodes when generating queries, we also need to refine the initial reasoning paths, aligning them with actual query nodes included in the final queries. Specifically, we first employ text matching to identify the included query nodes, and then apply the same backtracking algorithm outlined in Section 4.2 to refine the reasoning paths.

To further ensure the data quality, we implement diverse filtering methods, including rule-based filtering, LLM-based filtering, and manual filtering, which are carried out in sequence.

Rule-based Filtering. We design the following rules to remove low-quality data: (1) The number of nodes in each query must be no less than 3. (2) The queries must not contain answer nodes. (3) The query must be a standalone problem. (4) The reasoning paths must include the answer nodes. (5) To prevent data leakage, for each setting, we use random deduplication to avoid multiple entries sharing the same answer.

LLM-based Filtering. We employ LLMs to verify the correctness of the generated data. Specifically, we feed the query, reasoning paths, and the answer into LLMs, and ask LLMs to determine whether the answer is correct. We retain the data deemed correct by LLMs. The prompt used in this stage is shown in Appendix F.

Manual Filtering. We manually review and correct the data in validation and test sets, focusing on the clarity of query expressions, the correctness of answers, and the accuracy of reasoning paths.

Appendix B demonstrates more details of the filtering stage, including detailed guidelines for manual filtering and additional data statistics.

<table>
  <tr>
    <th>Entry Count</th>
    <th>Avg. Token</th>
    <th>Avg. Entities</th>
    <th>Avg. Hops</th>
    <th>Avg. Triplets</th>
    <th>Avg. Nodes</th>
  </tr>
  <tr>
    <td>Single-hop</td>
    <td>1,542</td>
    <td>29.68</td>
    <td>4.00</td>
    <td>1.00</td>
    <td>3.92</td>
    <td>4.92</td>
  </tr>
  <tr>
    <td>Multi-hop</td>
    <td>1,430</td>
    <td>39.93</td>
    <td>3.43</td>
    <td>2.66</td>
    <td>5.23</td>
    <td>6.23</td>
  </tr>
  <tr>
    <td>Set</td>
    <td>1,335</td>
    <td>41.59</td>
    <td>3.36</td>
    <td>2.83</td>
    <td>10.17</td>
    <td>9.53</td>
  </tr>
  <tr>
    <td>Aggregation</td>
    <td>1,136</td>
    <td>37.22</td>
    <td>3.65</td>
    <td>2.12</td>
    <td>11.07</td>
    <td>9.16</td>
  </tr>
  <tr>
    <td>Editing</td>
    <td>926</td>
    <td>39.98</td>
    <td>3.45</td>
    <td>2.70</td>
    <td>5.31</td>
    <td>6.31</td>
  </tr>
  <tr>
    <td>Answerability</td>
    <td>1,997</td>
    <td>40.40</td>
    <td>3.35</td>
    <td>-</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <th>Total</th>
    <th>8,366</th>
    <th>38.06</th>
    <th>3.54</th>
    <th>2.20</th>
    <th>7.00</th>
    <th>7.14</th>
  </tr>
</table>

Table 2: Data analysis across six settings.

4.5 Data Analysis

Our M^3GQA dataset consists of 8,366 instances, with an average of 3.54 entities in each query. We conduct a statistical analysis of the data quantity in six settings and examine the average number of query tokens, entities, reasoning hops, as well as the average number of triplets and nodes in the reasoning paths for each setting. Relevant results are presented in Table 2. We additionally visualize the number of query entities and the inference hops of queries in the dataset, as shown in Figure 4.

5 Experiments

5.1 Experimental Settings

Benchmark Settings. We construct M^3GQA based on Freebase (Bollacker et al., 2008) and randomly split the data into train/validation/test sets by 6/1/3 for each setting.

Baselines. We evaluate baselines on M^3GQA, which can be divided into 3 classes:
LLM-only Methods: We adopt GPT-4 (model: gpt-4-turbo), ChatGPT (model: gpt-3.5-turbo), and Claude-3.5 (model: claude-3.5-sonnet) to answer queries directly.
Traditional Methods: We first retrieve all shortest paths between query nodes as a rule-based baseline (abbrev. SP). Besides, we leverage sparse retriever (BM25 (Robertson and Zaragoza, 2009)) and dense retriever (paraphrase-multilingual-MiniLM-L12-v2 (Wang et al., 2021)) to retrieve relevant triplets, and feed the triplets with similarity scores from high to low into GPT-4 to generate answers.
GraphRAG Methods: We reproduce several state-of-the-art GraphRAG methods on M^3GQA, includ-
<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Single-hop Setting</th>
    <th colspan="4">Multi-hop Setting</th>
    <th colspan="4">Set Setting</th>
    <th colspan="4">Aggregation Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
  </tr>
  <tr>
    <td colspan="16"><b>LLM-Only Methods</b></td>
  </tr>
  <tr>
    <td>GPT4</td><td>33.41</td><td>34.10</td><td>35.85</td><td>33.26</td><td>11.87</td><td>12.20</td><td>12.82</td><td>11.66</td><td>21.92</td><td>24.79</td><td>44.75</td><td>28.00</td><td>25.65</td><td>26.98</td><td>26.03</td><td>24.93</td>
  </tr>
  <tr>
    <td>ChatGPT</td><td>29.64</td><td>30.35</td><td>32.18</td><td>28.94</td><td>7.73</td><td>7.90</td><td>8.39</td><td>6.99</td><td>16.72</td><td>19.02</td><td>34.75</td><td>19.00</td><td>21.01</td><td>21.59</td><td>23.46</td><td>21.99</td>
  </tr>
  <tr>
    <td>Claude3.5</td><td>30.71</td><td>31.78</td><td>34.77</td><td>31.32</td><td>10.14</td><td>10.76</td><td>12.12</td><td>8.86</td><td>17.24</td><td>19.40</td><td>40.25</td><td>21.50</td><td>24.75</td><td>24.05</td><td>27.57</td><td>25.51</td>
  </tr>
  <tr>
    <td colspan="16"><b>Traditional Methods</b></td>
  </tr>
  <tr>
    <td>SP</td><td><b>82.48</b></td><td><b>83.96</b></td><td><b>87.47</b></td><td><b>80.13</b></td><td>10.68</td><td>10.83</td><td>11.19</td><td>10.26</td><td>26.49</td><td>28.22</td><td>43.00</td><td>33.00</td><td>43.19</td><td>44.21</td><td>47.21</td><td>41.64</td>
  </tr>
  <tr>
    <td>Sparse</td><td>65.06</td><td>66.28</td><td>69.33</td><td>64.15</td><td>14.82</td><td>15.48</td><td>16.78</td><td>14.22</td><td>34.57</td><td>37.51</td><td>54.50</td><td>37.75</td><td>36.68</td><td>37.70</td><td>40.47</td><td>35.78</td>
  </tr>
  <tr>
    <td>Dense</td><td>69.04</td><td>70.54</td><td>74.30</td><td>65.66</td><td><b>25.20</b></td><td><b>28.26</b></td><td><b>35.20</b></td><td><b>22.61</b></td><td><b>43.30</b></td><td><b>47.16</b></td><td><b>70.00</b></td><td>45.00</td><td><b>49.01</b></td><td><b>50.57</b></td><td><b>55.13</b></td><td><b>44.57</b></td>
  </tr>
  <tr>
    <td colspan="16"><b>GraphRAG Methods</b></td>
  </tr>
  <tr>
    <td>ToG</td><td>71.33</td><td>71.94</td><td>74.00</td><td>68.00</td><td>19.00</td><td>19.31</td><td>20.00</td><td>20.00</td><td>23.13</td><td>25.42</td><td>44.00</td><td>30.00</td><td>28.67</td><td>28.97</td><td>30.00</td><td>26.00</td>
  </tr>
  <tr>
    <td>RoG</td><td>57.05</td><td>58.15</td><td>61.12</td><td>56.59</td><td>16.78</td><td>17.59</td><td>19.58</td><td>15.38</td><td>27.94</td><td>31.21</td><td>50.75</td><td>34.50</td><td>35.72</td><td>36.75</td><td>39.59</td><td>33.72</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>68.54</td><td>69.74</td><td>73.00</td><td>67.60</td><td>17.10</td><td>18.29</td><td>20.75</td><td>16.32</td><td>31.14</td><td>34.81</td><td>58.00</td><td>37.00</td><td>34.01</td><td>34.66</td><td>36.36</td><td>32.26</td>
  </tr>
  <tr>
    <td>G-Retriever</td><td>55.97</td><td>56.92</td><td>59.40</td><td>55.08</td><td>21.74</td><td>23.05</td><td>25.87</td><td>21.21</td><td>36.97</td><td>41.45</td><td>69.75</td><td><b>47.50</b></td><td>38.78</td><td>39.59</td><td>41.64</td><td>38.71</td>
  </tr>
</table>

Table 3: Performance of different methods across four general settings of M^3GQA. We randomly sample 50 entries from each setting for the evaluation of ToG considering the time and expenses.

ing: G-Retriever (He et al., 2024), ToG (Sun et al., 2024), RoG (Luo et al., 2024). We additionally replace the retriever of RoG with Qwen2.5-7B-Instruct, which is trained with SFT on the training set of M^3GQA (abbrev. RoG-SFT), as a baseline.

The introduction of these methods and details of their reproduction are presented in Appendix C.1.

Evaluation Metrics. We evaluate the performance of baselines from two aspects. For graph retrieval, we utilize two metrics to measure the effectiveness, including: recall of triplets in reasoning paths (Recall) and whether all answer nodes have been retrieved (Acc). In addition, we also report the total number of triplets contained in the subgraphs retrieved (Num) by different baselines. For answer generation, we utilize Macro-F1 (abbrev. MacF1), Micro-F1 (abbrev. MicF1), Hit, and Hit@1 to measure the quality of predictions. For the answerability setting, since it is essentially a binary classification task, we use Accuracy, Precision, Recall, and F1 as evaluation metrics.

5.2 Overall Performance

We evaluate the baselines across six settings from answer prediction and graph retrieval aspects, which are shown in Table 3, 4 and 5, we summarize our findings about M^3GQA as follows:

Comparison between different settings Although M^3GQA overall presents a significant challenge, there are still noticeable differences in the difficulty of six settings. From the results, the single-hop setting appears to be the simplest, while the multi-hop setting is the toughest, possibly due to the complex reasoning involved in multi-entity multi-hop queries. Meanwhile, the set and aggregation settings also demonstrate a considerable degree of difficulty. Considering there are multiple answer entities in the set setting, we additionally introduce the Exact Match (EM) metric to reflect the model’s ability to provide complete answers, which is shown in Table 10 in Appendix C.3. It appears that the current model’s performance in this regard is not satisfying.

Comparison between different Baselines Since GraphRAG requires the model to select entities from the graph as answers, the performance of LLM-only methods is not ideal. Compared to the current GraphRAG approaches, traditional retrieval methods still show strong competitiveness, likely due to the more comprehensive content retrieved by these methods. In addition, current state-of-the-art GraphRAG models perform suboptimally on M^3GQA, indicating that our dataset is relatively complex and presents challenges to the retrieval and answering strategies of previous methods. A detailed analysis of the performance of state-of-the-art methods can be found in Appendix C.2.

Correlation between retrieval and generation By analyzing the results of graph retrieval and answer prediction simultaneously, we can see a clear

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Editing Setting</th>
    <th colspan="4">Answerability Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>Acc</th><th>Precision</th><th>Recall</th><th>F1</th>
  </tr>
  <tr>
    <td colspan="8"><b>Traditional Methods</b></td>
  </tr>
  <tr>
    <td>SP</td><td>5.90</td><td>5.98</td><td>6.12</td><td>5.76</td><td>47.25</td><td>60.59</td><td>43.70</td><td>50.78</td>
  </tr>
  <tr>
    <td>Sparse</td><td>8.92</td><td>9.50</td><td>10.79</td><td>8.63</td><td>50.58</td><td>65.98</td><td>46.23</td><td>51.79</td>
  </tr>
  <tr>
    <td>Dense</td><td>17.95</td><td>19.94</td><td><b>24.46</b></td><td>17.63</td><td>52.92</td><td>62.96</td><td>59.25</td><td>61.05</td>
  </tr>
  <tr>
    <td colspan="8"><b>GraphRAG Methods</b></td>
  </tr>
  <tr>
    <td>ToG</td><td><b>20.00</b></td><td><b>20.00</b></td><td><b>20.00</b></td><td><b>20.00</b></td><td>48.00</td><td><b>68.42</b></td><td>39.39</td><td>50.00</td>
  </tr>
  <tr>
    <td>RoG</td><td>8.19</td><td>9.25</td><td>11.51</td><td>7.19</td><td>46.41</td><td>61.40</td><td>37.53</td><td>46.59</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>8.82</td><td>10.13</td><td>12.59</td><td>8.27</td><td>48.75</td><td>63.10</td><td>42.63</td><td>50.88</td>
  </tr>
  <tr>
    <td>G-Retriever</td><td>11.82</td><td>12.80</td><td>15.47</td><td>12.59</td><td>54.26</td><td>62.28</td><td>57.29</td><td>64.69</td>
  </tr>
</table>

Table 4: Performance of different methods across two special settings of M^3GQA.
<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Single-hop Setting</th>
    <th colspan="3">Multi-hop Setting</th>
    <th colspan="3">Set Setting</th>
    <th colspan="3">Aggregation Setting</th>
    <th colspan="3">Editing Setting</th>
  </tr>
  <tr>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
  </tr>
  <tr>
    <td colspan="15"><b>Traditional Methods</b></td>
  </tr>
  <tr>
    <td>SP</td><td>92.00</td><td>95.03</td><td>51.31</td><td>61.86</td><td>10.72</td><td>34.26</td><td>40.78</td><td>15.25</td><td>43.95</td><td>45.53</td><td>55.13</td><td>52.60</td><td>60.59</td><td>8.63</td><td>32.28</td>
  </tr>
  <tr>
    <td>Sparse</td><td>41.60</td><td>79.27</td><td>100.00</td><td>32.68</td><td>24.01</td><td>100.00</td><td>26.90</td><td>39.00</td><td>100.00</td><td>23.92</td><td>59.24</td><td>100.00</td><td>31.84</td><td>19.42</td><td>100.00</td>
  </tr>
  <tr>
    <td>Dense</td><td>67.73</td><td>88.55</td><td>100.00</td><td>51.34</td><td>66.43</td><td>100.00</td><td>46.50</td><td>53.75</td><td>100.00</td><td>45.53</td><td>82.99</td><td>100.00</td><td>51.35</td><td>59.71</td><td>100.00</td>
  </tr>
  <tr>
    <td colspan="15"><b>GraphRAG Methods</b></td>
  </tr>
  <tr>
    <td>ToG</td><td>41.87</td><td>74.00</td><td>4.89</td><td>32.82</td><td>16.00</td><td>5.64</td><td>12.60</td><td>4.00</td><td>5.98</td><td>8.81</td><td>38.00</td><td>5.76</td><td>34.92</td><td>24.00</td><td>5.60</td>
  </tr>
  <tr>
    <td>RoG</td><td>38.17</td><td>62.42</td><td>58.06</td><td>13.36</td><td>24.01</td><td>41.33</td><td>12.31</td><td>30.75</td><td>77.47</td><td>12.07</td><td>56.89</td><td>76.50</td><td>14.56</td><td>23.38</td><td>41.71</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>43.83</td><td>70.41</td><td>19.26</td><td>12.11</td><td>25.87</td><td>48.10</td><td>13.10</td><td>34.50</td><td>124.64</td><td>9.56</td><td>48.68</td><td>58.32</td><td>11.07</td><td>31.65</td><td>45.66</td>
  </tr>
  <tr>
    <td>G-Retriever</td><td>50.15</td><td>82.94</td><td>208.53</td><td>41.72</td><td>75.76</td><td>236.10</td><td>29.44</td><td>60.50</td><td>207.25</td><td>31.88</td><td>79.47</td><td>199.60</td><td>40.43</td><td>67.99</td><td>226.91</td>
  </tr>
</table>

Table 5: Evaluation of graph retrieval in different models across five settings. Since a considerable portion of queries in the answerability setting could not retrieve subgraphs, we do not report the results in that setting.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Simple Single-hop</th>
    <th colspan="3">Simple Multi-hop</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
  </tr>
  <tr>
    <td colspan="6"><b>LLM-Only Methods</b></td>
  </tr>
  <tr>
    <td>GPT4</td><td>34.83</td><td>35.65</td><td>37.98</td><td>35.19</td><td>23.84</td><td>24.61</td><td>26.65</td><td>22.98</td>
  </tr>
  <tr>
    <td>ChatGPT</td><td>30.55</td><td>31.25</td><td>32.83</td><td>30.81</td><td>19.99</td><td>20.68</td><td>22.25</td><td>19.56</td>
  </tr>
  <tr>
    <td>Claude3.5</td><td>30.15</td><td>31.45</td><td>31.46</td><td>31.33</td><td>20.10</td><td>20.95</td><td>23.23</td><td>20.05</td>
  </tr>
  <tr>
    <td colspan="6"><b>Traditional Methods</b></td>
  </tr>
  <tr>
    <td>Sparse</td><td>62.44</td><td>62.95</td><td>64.16</td><td>62.88</td><td>35.72</td><td>36.49</td><td>38.14</td><td>34.47</td>
  </tr>
  <tr>
    <td>Dense</td><td>84.50</td><td>85.30</td><td>87.34</td><td>84.12</td><td>50.95</td><td>53.25</td><td>58.19</td><td>48.17</td>
  </tr>
  <tr>
    <td colspan="6"><b>GraphRAG Methods</b></td>
  </tr>
  <tr>
    <td>ToG</td><td>81.47</td><td>82.21</td><td>84.00</td><td>82.00</td><td>54.00</td><td>54.00</td><td>54.00</td><td>54.00</td>
  </tr>
  <tr>
    <td>RoG</td><td>58.69</td><td>59.34</td><td>60.73</td><td>59.01</td><td>32.04</td><td>33.55</td><td>36.67</td><td>31.05</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>69.38</td><td>69.83</td><td>70.82</td><td>68.88</td><td>35.15</td><td>36.27</td><td>38.63</td><td>33.99</td>
  </tr>
  <tr>
    <td>G-Retriever</td><td>67.50</td><td>68.33</td><td>70.39</td><td>68.24</td><td>43.40</td><td>44.68</td><td>47.43</td><td>41.56</td>
  </tr>
</table>

Table 6: Difficulty comparison of M^3GQA against other two simple settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Simple Single-hop</th>
    <th colspan="3">Simple Multi-hop</th>
  </tr>
  <tr>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
  </tr>
  <tr>
    <td colspan="6"><b>Traditional Methods</b></td>
  </tr>
  <tr>
    <td>Sparse</td><td>29.94</td><td>64.81</td><td>100.00</td><td>24.63</td><td>36.92</td><td>100.00</td>
  </tr>
  <tr>
    <td>Dense</td><td>88.84</td><td>96.35</td><td>100.00</td><td>61.53</td><td>70.17</td><td>100.00</td>
  </tr>
  <tr>
    <td colspan="6"><b>GraphRAG Methods</b></td>
  </tr>
  <tr>
    <td>ToG</td><td>83.00</td><td>90.00</td><td>2.88</td><td>45.20</td><td>36.00</td><td>4.90</td>
  </tr>
  <tr>
    <td>RoG</td><td>33.80</td><td>48.07</td><td>20.90</td><td>23.96</td><td>14.42</td><td>20.44</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>43.78</td><td>56.22</td><td>5.45</td><td>24.69</td><td>14.67</td><td>14.99</td>
  </tr>
  <tr>
    <td>G-Retriever</td><td>62.12</td><td>88.63</td><td>217.99</td><td>43.26</td><td>71.15</td><td>213.55</td>
  </tr>
</table>

Table 7: Evaluation of the graph retrieval process of different models on two simple settings.

correlation between them, indicating that improving the performance of the retrievers is crucial to the overall performance. In M^3GQA, we provide ground-truth reasoning paths, which are believed to be helpful in enhancing the retriever’s capabilities. We also identify some special cases from the results. For example, in some settings (e.g., set setting), the retrieval accuracy (Acc) of RoG is lower than the Hit value in answer prediction. This suggests that some queries might be correctly answered relying solely on the knowledge in LLMs. Moreover, although G-retriever shows high retrieval accuracy, the answer prediction performance is relatively average. We believe this may be due to incomplete retrieval paths, as the recall score of the retriever is relatively low. Furthermore, we believe that besides recall, accuracy, and triplet number, the order in which the retrieved triplets are fed into the LLMs can also affect the model’s answer prediction performance. We validate this idea through experiments in Appendix C.5.

5.3 Difficulty of M^3GQA

We further construct two additional settings: simple single-hop and simple multi-hop settings, by reducing the number of entities in queries. We evaluate the models’ performance from both perspectives of answer prediction and graph retrieval to show the difficulty of M^3GQA, with results in Table 6 and 7. We do not report the results of SP because it is not applicable to queries that involve only a single entity. By comparing these results with the original single-hop and multi-hop settings in Table 3 and 5, we can observe that increasing the number of entities can significantly improve the difficulty of queries, which further highlights the challenges and importance of multi-entity queries. We also compare M^3GQA with classical datasets such as WebQSP and CWQ. Due to the lack of ground-truth reasoning paths in WebQSP and CWQ, we only evaluate the model’s ability to predict answers, which are shown in Table 8. We can see that compared to CWQ and WebQSP, the models’ performance on M^3GQA shows a significant decline, indicating that our dataset presents a considerable challenge to current models.

6 Conclusion

In this paper, we introduce the challenge of multi-entities queries and construct a Multi-Entity Multi-Hop Multi-Setting Graph Question Answering Benchmark, named M^3GQA, to evaluate the ca-
<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="2">WebQSP</th>
    <th colspan="2">CWQ</th>
    <th colspan="2">M^3GQA</th>
  </tr>
  <tr>
    <th>MacF1</th>
    <th>Hit</th>
    <th>MacF1</th>
    <th>Hit</th>
    <th>MacF1</th>
    <th>Hit</th>
  </tr>
  <tr>
    <td>ToG</td>
    <td>-</td>
    <td>82.60</td>
    <td>-</td>
    <td>67.60</td>
    <td>37.72</td>
    <td>43.72</td>
  </tr>
  <tr>
    <td>RoG</td>
    <td>70.26</td>
    <td>86.67</td>
    <td>54.63</td>
    <td>61.94</td>
    <td>38.81</td>
    <td>48.34</td>
  </tr>
  <tr>
    <td>G-Retriever</td>
    <td>53.41</td>
    <td>73.46</td>
    <td>-</td>
    <td>-</td>
    <td>38.73</td>
    <td>49.68</td>
  </tr>
</table>

Table 8: Difficulty comparison of M^3GQA against WebQSP and CWQ.

pabilities of GraphRAG methods. To ensure the quality and diversity of the data, we introduce a non-template approach for data construction and classify the data into six settings according to query types and complexity. We reproduce the state-of-the-art methods and assess their performance focusing on both graph retrieval and answer prediction. Experimental results show that M^3GQA is highly challenging and holds significant implications for the future development of the field.

Ethical Consideration

In this paper, we construct M^3GQA based on Freebase, with carefully designed methods to minimize the ethical risks. As Freebase contains publicly available knowledge, no private or sensitive individual data is included, ensuring compliance with data privacy standards. M^3GQA is intended strictly for academic research and educational purposes, with clear guidelines to prevent misuse in harmful applications. Furthermore, the dataset adheres to Freebase’s licensing terms, with proper attribution required. In summary, M^3GQA poses low ethical risk while enabling advancements of both retrieval and reasoning processes in GraphRAG systems.

Limitations

In this paper, we propose the first GraphRAG benchmark named M^3GQA consisting of multi-entity queries. To facilitate the construction of the dataset, we utilize the knowledge graph (Freebase) as the primary source of graph-structured information. We believe that extending this approach to more general graph data, multilingual data (Zhao and Zhang, 2024), and other vertical domains (e.g., biomedical (Jiang et al., 2024b), academic (Jin et al., 2024), and financial (Arslan and Cruz, 2024)) represents a promising direction for future improvements. Besides, due to time and expense constraints, we only reproduce a subset of the state-of-the-art methods. Expanding the reproduction to include more baselines would undoubtedly contribute to further advancements in the field. Additionally, this work primarily focuses on graph question answering scenarios, which is currently the most widely applied task for GraphRAG methods. Designing benchmarks for other GraphRAG tasks (e.g., long context summary) is also a direction worth exploring.

Acknowledgments

This work is supported by the National Key Research and Development Plan of China (2023YFB4502305), and Ant Group through Ant Research Intern Program. This work is also supported in part by Ucap Cloud and the State Key Laboratory of General Artificial Intelligence.

References

Muhammad Arslan and Christophe Cruz. 2024. Business-rag: Information extraction for business insights. In Proceedings of the 21st International Conference on Smart Business Technologies, ICSBT 2024, Dijon, France, July 9-11, 2024, pages 88–94.

Jonathan Berant, Andrew Chou, Roy Frostig, and Percy Liang. 2013. Semantic parsing on freebase from question-answer pairs. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, EMNLP 2013, 18-21 October 2013, Grand Hyatt Seattle, Seattle, Washington, USA, A meeting of SIGDAT, a Special Interest Group of the ACL, pages 1533–1544.

Kurt D. Bollacker, Colin Evans, Praveen K. Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collaboratively created graph database for structuring human knowledge. In Proceedings of the ACM SIGMOD International Conference on Management of Data, SIGMOD 2008, Vancouver, BC, Canada, June 10-12, 2008, pages 1247–1250.

Antoine Bordes, Nicolas Usunier, Sumit Chopra, and Jason Weston. 2015. Large-scale simple question answering with memory networks.

Shulin Cao, Jiaxin Shi, Liangming Pan, Lunyiu Nie, Yutong Xiang, Lei Hou, Juanzi Li, Bin He, and Hanwang Zhang. 2022. KQA pro: A dataset with explicit compositional programs for complex question answering over knowledge base. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 6101–6119.

Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. 2019. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the
North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pages 4171–4186.

Junnan Dong, Qinggang Zhang, Xiao Huang, Keyu Duan, Qiaoyu Tan, and Zhiming Jiang. 2023. Hierarchy-aware multi-hop question answering over knowledge graphs. In Proceedings of the ACM Web Conference 2023, WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages 2519–2527. ACM.

Mohsin Dubey, Debayan Banerjee, Abdelrahman Abdellkawi, and Jens Lehmann. 2019. Lc-quad 2.0: A large dataset for complex question answering over wikidata and dbpedia. In The Semantic Web - ISWC 2019 - 18th International Semantic Web Conference, Auckland, New Zealand, October 26-30, 2019, Proceedings, Part II, volume 11779 of Lecture Notes in Computer Science, pages 69–78.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024. A survey on RAG meeting llms: Towards retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, KDD 2024, Barcelona, Spain, August 25-29, 2024, pages 6491–6501.

Yu Gu, Sue Kase, Michelle Vanni, Brian M. Sadler, Percy Liang, Xifeng Yan, and Yu Su. 2021. Beyond I.I.D.: three levels of generalization for question answering on knowledge bases. In WWW ’21: The Web Conference 2021, Virtual Event / Ljubljana, Slovenia, April 19-23, 2021, pages 3477–3488.

Tiezhen Guo, Qingwen Yang, Chen Wang, Yanyi Liu, Pan Li, Jiawei Tang, Dapeng Li, and Yingyu Wen. 2024a. Knowledgenavigator: Leveraging large language models for enhanced reasoning over knowledge graph.

Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. 2024b. Lightrag: Simple and fast retrieval-augmented generation.

Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V. Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering.

Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. 2024. Grag: Graph retrieval-augmented generation.

Yongfeng Huang, Yanyang Li, Yichong Xu, Lin Zhang, Ruyi Gan, Jiaxing Zhang, and Liwei Wang. 2023. Mvp-tuning: Multi-view knowledge retrieval with prompt tuning for commonsense reasoning. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 13417–13432.

Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Xin Zhao, and Ji-Rong Wen. 2023a. Structgpt: A general framework for large language model to reason over structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, EMNLP 2023, Singapore, December 6-10, 2023, pages 9237–9251.

Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong Wen. 2024a. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowledge graph.

Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. 2023b. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In The Eleventh International Conference on Learning Representations, ICLR 2023, Kigali, Rwanda, May 1-5, 2023.

Xinke Jiang, Ruizhe Zhang, Yongxin Xu, Rihong Qiu, Yue Fang, Zhiyuan Wang, Jinyi Tang, Hongxin Ding, Xu Chu, Junfeng Zhao, and Yasha Wang. 2024b. Hykge: A hypothesis knowledge graph enhanced framework for accurate and reliable medical llms responses.

Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Zheng Li, Ruinri Li, Xianfeng Tang, Suhang Wang, Yu Meng, and Jiawei Han. 2024. Graph chain-of-thought: Augmenting large language models by reasoning on graphs. In Findings of the Association for Computational Linguistics, ACL 2024, Bangkok, Thailand and virtual meeting, August 11-16, 2024, pages 163–184.

Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. 2023. Graph reasoning for question answering with triplet retrieval. In Findings of the Association for Computational Linguistics: ACL 2023, Toronto, Canada, July 9-14, 2023, pages 3366–3375.

Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. 2024. Structrag: Boosting knowledge intensive reasoning of llms via inference-time hybrid information structurization.

Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2829–2839.

Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis,
Luke Zettlemoyer, and Veselin Stoyanov. 2019. Roberta: A robustly optimized bert pretraining approach.

Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2024. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.

Alex Mallen, Akari Asai, Victor Zhong, Rajarshi Das, Daniel Khashabi, and Hannaneh Hajishirzi. 2023. When not to trust language models: Investigating effectiveness of parametric and non-parametric memories. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2023, Toronto, Canada, July 9-14, 2023, pages 9802–9822.

Costas Mavromatis and George Karypis. 2024. Gnnrag: Graph neural retrieval for large language model reasoning.

Todor Mihaev, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? A new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, Brussels, Belgium, October 31 - November 4, 2018, pages 2381–2391.

OpenAI. 2024. Gpt-4 technical report.

Boci Peng, Yongchao Liu, Xiaohe Bo, Sheng Tian, Baokun Wang, Chuntao Hong, and Yan Zhang. 2024a. Subgraph retrieval enhanced by graph-text alignment for commonsense question answering. In Machine Learning and Knowledge Discovery in Databases. Research Track - European Conference, ECML PKDD 2024, Vilnius, Lithuania, September 9-13, 2024, Proceedings, Part VI, pages 39–56.

Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. 2024b. Graph retrieval-augmented generation: A survey.

Aleksandr Perevalov, Dennis Diefenbach, Ricardo Usbeck, and Andreas Both. 2022. Qald-9-plus: A multilingual dataset for question answering over dbpedia and wikidata translated by native speakers. In 16th IEEE International Conference on Semantic Computing, ICSC 2022, Laguna Hills, CA, USA, January 26-28, 2022, pages 229–234.

Stephen E. Robertson and Hugo Zaragoza. 2009. The probabilistic relevance framework: BM25 and beyond. Found. Trends Inf. Retr., 3(4):333–389.

Andrey Sakhovskiy, Mikhail Salnikov, Irina Nikishina, Aida Usmanova, Angelie Kraft, Cedric Möller, Debayan Banerjee, Junbo Huang, Longquan Jiang, Rana Abdullah, Xi Yan, Dmitry Ustalov, Elena Tutubalina, Ricardo Usbeck, and Alexander Panchenko. 2024. TextGraphs 2024 shared task on text-graph representations for knowledge graph question answering. In Proceedings of TextGraphs-17: Graph-based Methods for Natural Language Processing, pages 116–125.

Mikhail Salnikov, Hai Le, Prateek Rajput, Irina Nikishina, Pavel Braslavski, Valentin Malykh, and Alexander Panchenko. 2023. Large language models meet knowledge graphs to answer factoid questions. In Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation, PACLIC 2023, The Hong Kong Polytechnic University, Hong Kong, SAR, China, 2-4 December 2023, pages 635–644.

Priyanka Sen, Alham Fikri Aji, and Amir Saffari. 2022. Mintaka: A complex, natural, and multilingual dataset for end-to-end question answering. In Proceedings of the 29th International Conference on Computational Linguistics, COLING 2022, Gyeongju, Republic of Korea, October 12-17, 2022, pages 1604–1619.

Haitian Sun, Tania Bedrax-Weiss, and William W. Cohen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2380–2390.

Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhao Wang, Chen Lin, Yeyun Gong, Lionel M. Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In The Twelfth International Conference on Learning Representations, ICLR 2024, Vienna, Austria, May 7-11, 2024.

Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2018, New Orleans, Louisiana, USA, June 1-6, 2018, Volume 1 (Long Papers), pages 641–651.

Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. 2019. Commonsenseqa: A question answering challenge targeting commonsense knowledge. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2019, Minneapolis, MN, USA, June 2-7, 2019, Volume 1 (Long and Short Papers), pages 4149–4158.

Dhaval Taunk, Lakshya Khanna, Siri Venkata Pavan Kumar Kandru, Vasudeva Varma, Charu Sharma, and Makarand Tapaswi. 2023. Grapeqa: Graph augmentation and pruning to enhance question-answering. In Companion Proceedings of the ACM Web Conference
2023. WWW 2023, Austin, TX, USA, 30 April 2023 - 4 May 2023, pages 1138–1144.

Wenhui Wang, Hangbo Bao, Shaohan Huang, Li Dong, and Furu Wei. 2021. Minilmv2: Multi-head self-attention relation distillation for compressing pre-trained transformers. In Findings of the Association for Computational Linguistics: ACL/IJCNLP 2021, Online Event, August 1-6, 2021, pages 2140–2151.

Yuqi Wang, Boran Jiang, Yi Luo, Dawei He, Peng Cheng, and Liangcai Gao. 2024. Reasoning on efficient knowledge paths: Knowledge graph guides large language model for domain question answering.

Yilin Wen, Zifeng Wang, and Jimeng Sun. 2024. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2024, Bangkok, Thailand, August 11-16, 2024, pages 10370–10388.

Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, Anhuan Xie, and Wei Song. 2023. Retrieve-rewrite-answer: A kg-to-text enhanced lms framework for knowledge graph question answering.

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 535–546.

Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 2: Short Papers.

Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), ACL 2022, Dublin, Ireland, May 22-27, 2022, pages 5773–5784.

Yuyu Zhang, Hanjun Dai, Zornitsa Kozareva, Alexander J. Smola, and Le Song. 2018. Variational reasoning for question answering with knowledge graph. In Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the 8th AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, February 2-7, 2018, pages 6069–6076.

Jinman Zhao and Xueyan Zhang. 2024. Large language model is not a (multilingual) compositional reasoner. In First Conference on Language Modeling.

Wayne Xin Zhao, Kun Zhou, Junyi Li, Tianyi Tang, Xiaolei Wang, Yupeng Hou, Yingqian Min, Beichen Zhang, Junjie Zhang, Zican Dong, Yifan Du, Chen Yang, Yushuo Chen, Zhipeng Chen, Jinbao Jiang, Ruiyang Ren, Yifan Li, Xinyu Tang, Zikang Liu, Peiyu Liu, Jian-Yun Nie, and Ji-Rong Wen. 2024. A survey of large language models.
A  Data Analysis of M^3GQA

A.1  Statistics of WebQSP and CWQ

We calculate the number of multi-entity queries in WebQSP and CWQ respectively. The results are presented in Figure 3 below.

![Bar chart showing the number of query entities in the WebQSP and CWQ datasets.](page_246_370_529_246.png)

Figure 3: Statistics on the number of query entities in the WebQSP and CWQ datasets.

A.2  Visualization of the Data

We visualize the number of query entities and the number of reasoning hops for each query under six settings, and present the results in Figure 4.

A.3  Pattern of Reasoning Paths

To provide a clearer understanding of the various settings discussed in this paper, we present examples of reasoning path patterns for each setting, which are presented in Figure 5.

B  Details of Filtering

B.1  Guidelines of Manual Filtering

We assign samples from the validation and test sets of M^3GQA randomly to five volunteers for manual inspection. Each volunteer identify any samples they consider unreasonable, and then all five volunteers judge these samples individually, with a majority vote determining whether each sample should be retained or discarded. The guidelines for determining whether a sample is reasonable are as follows:
• Clarity of the question: Check if the query includes any extraneous query entities, if the question is ambiguous, if it is likely to have a clear answer, and exclude subjective questions.
• Manual reasoning path and answer validation: For each question, we manually track the reasoning path and compare the answer with the ground-truth provided in the dataset.

B.2  Data Statistics

We show the proportions of data filtered out in each filtering stage in Table 9.

From the results, we can observe that (1) Each filtering stage is capable of eliminating a portion of low-quality data, which validates the necessity of every filtering stage. (2) The entire data construction pipeline is reliable because the proportion of manually filtered out unreasonable data is relatively small.

Furthermore, to ensure the reliability of the manual filtering process, we calculate the inter-annotator agreement using Fleiss’ Kappa, which measures the level of agreement among multiple annotators. The Fleiss’ Kappa score for our annotators is 0.74, indicating substantial agreement.

C  Additional Experiments

C.1  Implementation Details

For traditional methods, we retrieve the top 100 triplets according to similarity scores and feed them into GPT-4 as additional inputs for question answering. For GraphRAG methods, we leverage the open-source codes and first reproduce them on the WebQSP dataset, ensuring that the reproduction results are consistent with those reported in original papers. Then we evaluate them on M^3GQA and adjust hyper-parameters according to the performance on the validation set. For fair comparison, we use GPT-4 as the base generator.

C.2  Introduction and Analysis of Baselines

In this section, we will provide a detailed introduction to the current state-of-the-art GraphRAG methods and thoroughly analyze the potential issues that may arise when applying them to M^3GQA.

ToG  This method\(^2\) (Sun et al., 2024) proposes utilizing LLMs to perform graph retrieval starting from query entities, iteratively selecting edges and neighboring nodes related to the current node until the model can answer the question. Applying the method to M^3GQA may face the following challenges:
• Error Propagation in Iterative Search: Since the method employs a BFS-like iterative retrieval approach, any errors occurring at a particular search layer will render subsequent searches invalid.
• Complexity in Handling Nodes with High Connectivity: If a node in the search process is con-

\( ^2 \)https://github.com/IDEA-FinAI/ToG
(a) Number of query entities across six settings.        (b) Number of reasoning hops across five settings.

Figure 4: Visualization of query entities and reasoning hops in M^3GQA.

(a) Single-hop Setting        (b) Multi-hop Setting        (c) Set Setting

(d1) Aggregation Setting        (d2) Aggregation Setting

Answer Node   Intermediate Answer Node   Query Node   Start Node   Other Node   Reasoning Path

Figure 5: Pattern of reasoning paths under single-hop setting, multi-hop setting, set setting, and aggregation setting.

<table>
  <tr>
    <th></th>
    <th>Single-hop</th>
    <th>Multi-hop</th>
    <th>Set</th>
    <th>Aggregation</th>
    <th>Editing</th>
    <th>Answerability</th>
  </tr>
  <tr>
    <td>Rule-Based</td>
    <td>4.75%</td>
    <td>7.41%</td>
    <td>8.44%</td>
    <td>8.57%</td>
    <td>7.96%</td>
    <td>6.47%</td>
  </tr>
  <tr>
    <td>LLM-Based</td>
    <td>5.11%</td>
    <td>15.88%</td>
    <td>19.09%</td>
    <td>23.40%</td>
    <td>17.47%</td>
    <td>9.72%</td>
  </tr>
  <tr>
    <td>Manual (test)</td>
    <td>2.94%</td>
    <td>4.46%</td>
    <td>5.44%</td>
    <td>7.59%</td>
    <td>3.14%</td>
    <td>3.54%</td>
  </tr>
  <tr>
    <td>Manual (val)</td>
    <td>2.57%</td>
    <td>4.19%</td>
    <td>5.83%</td>
    <td>7.21%</td>
    <td>3.19%</td>
    <td>3.41%</td>
  </tr>
</table>

Table 9: Data statistics of each filtering stage.

connected to many or highly complex relationships, the model may struggle to correctly select the relevant edges and neighboring nodes.

• Over-confidence of Large Language Models: Due to the tendency of large language models to be overly confident, this approach might lead to the model answering the question prematurely before reaching the correct answer entity node, which would degrade answer prediction performance.

RoG This method\(^3\) (Luo et al., 2024) proposes to train an LLM to generate several relation chains as reasoning plans, and then parsing out reasoning paths from the graph that match the relation chains. During the reasoning phase, since the trained language model has not seen the relations in the graph from the test set in advance, it becomes difficult to parse out reasoning paths from the graph that match the reasoning plan generated by the LLM.

\(^3\)https://github.com/RManLuo/reasoning-on-graphs
Figure 6: Number of API Calls of ToG on six settings.

G-Retriever This method4 (He et al., 2024) proposes using the Prize-Collected Steiner Tree (PCST) algorithm to retrieve subgraphs relevant to the query. However, the method’s performance on M^3GQA is not ideal, possibly due to the following reasons:
• The subgraphs retrieved using the PCST algorithm typically have low recall values, indicating that the retrieved reasoning paths may not be comprehensive enough, posing a high requirement for the answer reasoning ability of LLMs.
• The subgraphs retrieved using this algorithm are relatively large, which poses a challenge to the context comprehension ability of LLMs for subsequent answer prediction.
• Since this retrieval algorithm cannot obtain the relevance scores for the corresponding triplets, it is difficult to determine an appropriate triplet order as input for the answer generator, which could also impact the model’s performance.

C.3 EM Results on Set Setting

We additionally introduce the EM metric to evaluate the models’ ability to answer the queries without any omission. By analyzing the results in Table 10, we can find that the model’s ability in this regard still requires improvement.

C.4 Statistics of API Calls in ToG

ToG proposes using LLMs to retrieve relevant information from the graph, which introduces significant API expenses, especially for multi-entity queries. We calculate the average number of API calls per query for ToG across six settings, and the results are shown in Figure 6.

4https://github.com/XiaoxinHe/G-Retriever

C.5 Retrieval Factors Affecting the Answer Performance

In the previous section, we evaluate the retrieval quality of the retriever using Recall, Acc, and Num, and briefly analyze the relationship between the retrieval quality and the final answer prediction performance. Here, we conduct an in-depth analysis of the impact of the number and order of retrieved triplets on the prediction results. Relevant results analysis are as follows.

C.5.1 Order of Retrieved Triplets

In this paper, we further investigate the impact of the concatenation order of triplets retrieved by the dense retriever on the model’s answer prediction performance. Specifically, we randomly shuffle the retrieved triplets and compare them with the original Dense method. Table 11 and 12 show a comparison of the results from the two methods across six settings. Through analysis, we can observe that the order of the triplets does have an impact on the model’s prediction performance.

C.5.2 Number of Retrieved Triplets

To facilitate control over the number of retrieved triplets, we use the dense retriever-based approach as the backbone model for GraphRAG. We explore the model’s task performance under six different settings when the number of retrieved triplets is 50, 100, 200, and 300. We evaluate the model performance using the same metrics described in the main text, focusing both on the retrieval and answer prediction process. Corresponding results can be found in Figure 7, 8 and 9. Through the experiments, we can find a rough positive correlation between the answer prediction performance and the number of retrieved triplets.

C.6 Performance of Hybrid Baselines

We additionally incorporate a hybrid baseline that combines the top 100 results based on the dense retrieval with the shortest paths between query nodes. The results are shown in Table 13, 14, and 15.

From the results above, we can observe that SP performs relatively well in simpler scenarios (such as single-hop setting), but they perform poorly in more complex scenarios (such as multi-hop setting). Furthermore, the hybrid method effectively combines the advantages of the SP method and dense retrieval, achieving good results in scenarios that involve both single-hop and multi-hop reasoning.
<table>
  <tr>
    <th>Set Setting</th>
    <th>GPT4</th>
    <th>ChatGPT</th>
    <th>Claude3.5</th>
    <th>MiniLM</th>
    <th>BM25</th>
    <th>ToG</th>
    <th>RoG</th>
    <th>RoG-SFT</th>
    <th>G-Retriever</th>
  </tr>
  <tr>
    <th>EM</th>
    <td>6.25</td>
    <td>4.00</td>
    <td>3.75</td>
    <td>22.25</td>
    <td>17.00</td>
    <td>6.00</td>
    <td>9.00</td>
    <td>9.50</td>
    <td>10.25</td>
  </tr>
</table>

Table 10: EM results of different models on the set setting.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Single-hop</th>
    <th colspan="4">Multi-hop</th>
    <th colspan="4">Set</th>
    <th colspan="4">Aggregation</th>
  </tr>
  <tr>
    <th>Macro-F1</th>
    <th>Micro-F1</th>
    <th>Hit</th>
    <th>Hit@1</th>
    <th>Macro-F1</th>
    <th>Micro-F1</th>
    <th>Hit</th>
    <th>Hit@1</th>
    <th>Macro-F1</th>
    <th>Micro-F1</th>
    <th>Hit</th>
    <th>Hit@1</th>
    <th>Macro-F1</th>
    <th>Micro-F1</th>
    <th>Hit</th>
    <th>Hit@1</th>
  </tr>
  <tr>
    <td>Dense</td>
    <td>69.04</td>
    <td>70.54</td>
    <td>74.30</td>
    <td>65.66</td>
    <td>25.20</td>
    <td>28.26</td>
    <td>35.20</td>
    <td>22.61</td>
    <td>43.30</td>
    <td>47.16</td>
    <td>70.00</td>
    <td>45.00</td>
    <td>49.01</td>
    <td>50.57</td>
    <td>55.13</td>
    <td>44.57</td>
  </tr>
  <tr>
    <td>Dense +Shuffle</td>
    <td>69.96</td>
    <td>71.34</td>
    <td>74.95</td>
    <td>67.39</td>
    <td>23.28</td>
    <td>25.75</td>
    <td>31.24</td>
    <td>22.14</td>
    <td>43.57</td>
    <td>47.49</td>
    <td>71.00</td>
    <td>48.75</td>
    <td>45.55</td>
    <td>46.91</td>
    <td>50.73</td>
    <td>42.23</td>
  </tr>
</table>

Table 11: Effect of triplets order for answer prediction on four general settings.

(a) Single-hop Setting. (b) Multi-hop Setting. (c) Set Setting. (d) Aggregation Setting.
Figure 7: The answer prediction performance of dense retriever-based methods varies with the number of retrieved triplets across four settings.

(a) Single-hop Setting. (b) Multi-hop Setting. (c) Set Setting. (d) Aggregation Setting.
Figure 8: The graph retrieval performance of dense retriever-based methods varies with the number of retrieved triplets across four settings.

(a) Editing Setting. (b) Answerability Setting. (c) Editing Setting.
Figure 9: The performance of dense retriever-based methods varies with the number of retrieved triplets across two special settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Editing</th>
    <th colspan="4">Answerability</th>
  </tr>
  <tr>
    <th>Macro-F1</th>
    <th>Micro-F1</th>
    <th>Hit</th>
    <th>Hit@1</th>
    <th>Acc</th>
    <th>Precision</th>
    <th>Recall</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Dense</td>
    <td>17.95</td>
    <td>19.94</td>
    <td>24.46</td>
    <td>17.63</td>
    <td>52.92</td>
    <td>62.96</td>
    <td>59.25</td>
    <td>61.05</td>
  </tr>
  <tr>
    <td>Dense +Shuffle</td>
    <td>17.81</td>
    <td>19.92</td>
    <td>25.18</td>
    <td>16.19</td>
    <td>54.76</td>
    <td>64.01</td>
    <td>62.47</td>
    <td>63.23</td>
  </tr>
</table>

Table 12: Effect of triplets order for answer prediction on two special settings.

Additionally, the retrieval performance further supports the correlation between retrieval quality and generation quality.

C.7 Generalization of Query Structures

To investigate the generalization of different query structures, we train RoG using training data from both single-hop and multi-hop settings and evaluate it across all settings (abbrev. RoG-subset). Furthermore, for a fair comparison, we randomly sample an equivalent amount of training data as the former from all settings (abbrev. RoG-general). We show results in Table 16, 17, and 18.

From the results, we observe that RoG-subset
<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Single-hop Setting</th>
    <th colspan="4">Multi-hop Setting</th>
    <th colspan="4">Set Setting</th>
    <th colspan="4">Aggregation Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>SP</td><td>82.48</td><td>83.96</td><td>87.47</td><td>80.13</td>
    <td>10.68</td><td>10.83</td><td>11.19</td><td>10.26</td>
    <td>26.49</td><td>28.22</td><td>43.00</td><td>33.00</td>
    <td>43.19</td><td>44.21</td><td>47.21</td><td>41.64</td>
  </tr>
  <tr>
    <td>Dense</td><td>69.04</td><td>70.54</td><td>74.30</td><td>65.66</td>
    <td>25.20</td><td>28.26</td><td>35.20</td><td>22.61</td>
    <td>43.30</td><td>47.16</td><td>70.00</td><td>45.00</td>
    <td>49.01</td><td>50.57</td><td>55.13</td><td>44.57</td>
  </tr>
  <tr>
    <td>Hybrid</td><td>79.07</td><td>80.67</td><td>84.67</td><td>77.11</td>
    <td>24.81</td><td>27.43</td><td>33.10</td><td>22.61</td>
    <td>45.85</td><td>50.04</td><td>71.50</td><td>47.75</td>
    <td>52.39</td><td>53.89</td><td>58.06</td><td>49.85</td>
  </tr>
</table>

Table 13: Performance of the hybrid baseline across four general settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="4">Editing Setting</th>
    <th colspan="4">Answerability Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
    <th>Ace</th><th>Precision</th><th>Recall</th><th>F1</th>
  </tr>
  <tr>
    <td>SP</td><td>5.90</td><td>5.98</td><td>6.12</td><td>5.76</td>
    <td>47.25</td><td>60.59</td><td>43.70</td><td>50.78</td>
  </tr>
  <tr>
    <td>Dense</td><td>17.95</td><td>19.94</td><td>24.46</td><td>17.63</td>
    <td>52.92</td><td>62.96</td><td>59.25</td><td>61.05</td>
  </tr>
  <tr>
    <td>Hybrid</td><td>17.20</td><td>19.26</td><td>23.38</td><td>16.19</td>
    <td>55.26</td><td>62.90</td><td>68.63</td><td>65.64</td>
  </tr>
</table>

Table 14: Performance of the hybrid baseline across two special settings.

and RoG-general outperform vanilla RoG in the vast majority of scenarios. This is attributed to the fact that vanilla RoG is trained on WebQSP and CWQ, which have a significantly different data distribution from that of M^3GQA. Furthermore, we find that RoG-subset surpasses RoG-general in the single-hop and multi-hop settings; however, the opposite is true for other settings. This indicates that there are substantial differences in query structures across various settings.

C.8 Ablation Studies

We further conduct ablation studies on reasoning path lengths, query entities count, and reasoning path construction.

Reasoning Path Lengths We first conduct an ablation study on reasoning path lengths, selecting Dense, Sparse, and G-Retriever as baselines for experiments across three general settings (the single-hop setting, having only one hop by definition, is excluded from this analysis). The path lengths are categorized into four groups (1-4 hops), with results as shown in Table 19, Table 20, and Table 21. We can observe that: (1) In the vast majority of cases, the longer the reasoning path, the more difficult the query becomes, and the lower the performance of the baselines. The set setting is an exception when the reasoning paths reach 4, as the relatively small number of test samples leads to significant fluctuations in the results. (2) Compared to the semantic matching-based retrieval methods of Dense and Sparse, G-Retriever performs relatively poorly in single-hop scenarios but demonstrates better performance in multi-hop queries. This highlights G-Retriever’s advantage in retrieving sub-graphs for capturing higher-order information.

Query Entities Count We conduct an ablation study on the number of entities in the query, dividing them into three groups: 3, 4, and \( \geq 5 \). The results are as shown in Table 22, Table 23, and Table 24. We can observe that: Except for the single-hop setting, the model’s performance decreases as the number of entities increases in all other settings. This indicates that a higher number of entities leads to greater problem difficulty. In the single-hop setting, the results across the three groups are comparable. A possible reason is that the model tends to retrieve one-hop neighbors, preventing a decline in performance. However, it is worth noting that this observation does not contradict the conclusions in Section 5.3, because when the number of entities is 1 or 2, the task only involves examining neighbors of a single entity or paths of length 2 between two entities. Thus, when the number of entities \( \geq 3 \), the difficulty increases significantly.

Reasoning Path Construction We implement an approach in the multi-hop setting where all shortest paths between the answer entities and query entities are used as reasoning paths. We find that only 66.43% of the samples had identical reasoning paths between the tree sampling method and the shortest-path method. We then compute the average recall of the shortest-path method (treating tree sampling reasoning paths as ground truth), which is 83.56%. This indicates that the shortest-path method fails to retrieve all reasoning paths—meaning some reasoning paths are not the shortest paths between the two sets of entities. Additionally, we measure the average precision, which is 32.54%, suggesting that the reasoning paths obtained via the shortest-path method contain substantial redundant information.

D Algorithm and Formalization

D.1 Algorithm

We show the algorithm pseudo code of the data construction pipeline in Algorithm 1.
<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Single-hop Setting</th>
    <th colspan="3">Multi-hop Setting</th>
    <th colspan="3">Set Setting</th>
    <th colspan="3">Aggregation Setting</th>
    <th colspan="3">Editing Setting</th>
  </tr>
  <tr>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
  </tr>
  <tr>
    <td>SP</td><td>92.00</td><td>95.03</td><td>51.31</td><td>61.86</td><td>10.72</td><td>34.26</td><td>40.78</td><td>15.25</td><td>43.95</td><td>45.53</td><td>55.13</td><td>52.60</td><td>60.59</td><td>8.63</td><td>32.28</td>
  </tr>
  <tr>
    <td>Dense</td><td>67.73</td><td>88.55</td><td>100.00</td><td>51.34</td><td>66.43</td><td>100.00</td><td>46.50</td><td>53.75</td><td>100.00</td><td>45.53</td><td>82.99</td><td>100.00</td><td>51.35</td><td>59.71</td><td>100.00</td>
  </tr>
  <tr>
    <td>Hybrid</td><td>97.93</td><td>99.78</td><td>132.94</td><td>79.00</td><td>69.00</td><td>124.17</td><td>66.78</td><td>60.25</td><td>130.80</td><td>67.18</td><td>90.62</td><td>137.89</td><td>77.87</td><td>61.51</td><td>121.49</td>
  </tr>
</table>

Table 15: Graph retrieval performance of the hybrid baseline in different models across five settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Single-hop Setting</th>
    <th colspan="3">Multi-hop Setting</th>
    <th colspan="3">Set Setting</th>
    <th colspan="3">Aggregation Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
  </tr>
  <tr>
    <td>RoG</td><td>57.05</td><td>58.15</td><td>61.12</td><td>56.59</td><td>16.78</td><td>17.59</td><td>19.58</td><td>15.38</td><td>27.94</td><td>31.21</td><td>50.75</td><td>34.50</td><td>35.72</td><td>36.75</td><td>39.59</td><td>33.72</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>68.54</td><td>69.74</td><td>73.00</td><td>67.60</td><td>17.10</td><td>18.29</td><td>20.75</td><td>16.32</td><td>31.14</td><td>34.81</td><td>58.00</td><td>37.00</td><td>34.01</td><td>34.66</td><td>36.36</td><td>32.26</td>
  </tr>
  <tr>
    <td>RoG-subset</td><td>58.74</td><td>60.05</td><td>62.63</td><td>57.67</td><td>16.45</td><td>17.35</td><td>19.11</td><td>15.62</td><td>28.02</td><td>31.05</td><td>51.25</td><td>34.75</td><td>33.50</td><td>34.85</td><td>35.19</td><td>31.67</td>
  </tr>
  <tr>
    <td>RoG-general</td><td>57.34</td><td>58.10</td><td>61.99</td><td>56.80</td><td>15.97</td><td>16.81</td><td>18.64</td><td>15.15</td><td>29.12</td><td>32.85</td><td>55.75</td><td>35.78</td><td>33.92</td><td>34.96</td><td>36.07</td><td>31.96</td>
  </tr>
</table>

Table 16: Performance of different RoG versions across four general settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="6">Editing Setting</th>
    <th colspan="6">Answerability Setting</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>Arc</th>
    <th>Precision</th><th>Recall</th><th>F1</th>
  </tr>
  <tr>
    <td>RoG</td><td>8.19</td><td>9.25</td><td>11.51</td><td>7.19</td><td>46.41</td><td>61.40</td><td>37.53</td><td>46.59</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>8.82</td><td>10.13</td><td>12.59</td><td>8.27</td><td>48.75</td><td>63.10</td><td>42.63</td><td>50.88</td>
  </tr>
  <tr>
    <td>RoG-subset</td><td>8.32</td><td>9.53</td><td>11.87</td><td>7.91</td><td>46.74</td><td>62.26</td><td>41.23</td><td>49.61</td>
  </tr>
  <tr>
    <td>RoG-general</td><td>8.67</td><td>9.75</td><td>12.23</td><td>7.91</td><td>47.25</td><td>62.90</td><td>42.76</td><td>50.91</td>
  </tr>
</table>

Table 17: Performance of different RoG versions across two special settings.

<table>
  <tr>
    <th rowspan="2">Baselines</th>
    <th colspan="3">Single-hop Setting</th>
    <th colspan="3">Multi-hop Setting</th>
    <th colspan="3">Set Setting</th>
    <th colspan="3">Aggregation Setting</th>
    <th colspan="3">Editing Setting</th>
  </tr>
  <tr>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
    <th>Recall</th><th>Acc</th><th>Num</th>
  </tr>
  <tr>
    <td>RoG</td><td>38.17</td><td>62.42</td><td>58.06</td><td>13.36</td><td>24.01</td><td>41.33</td><td>12.31</td><td>30.75</td><td>77.47</td><td>12.07</td><td>56.89</td><td>76.50</td><td>14.56</td><td>23.38</td><td>41.71</td>
  </tr>
  <tr>
    <td>RoG-SFT</td><td>43.83</td><td>70.41</td><td>19.26</td><td>12.11</td><td>25.87</td><td>48.10</td><td>13.10</td><td>34.50</td><td>124.64</td><td>9.56</td><td>48.68</td><td>58.32</td><td>11.07</td><td>31.65</td><td>45.66</td>
  </tr>
  <tr>
    <td>RoG-subset</td><td>42.05</td><td>65.87</td><td>20.90</td><td>12.47</td><td>24.24</td><td>43.96</td><td>12.27</td><td>32.75</td><td>101.94</td><td>9.23</td><td>47.50</td><td>59.94</td><td>10.45</td><td>26.25</td><td>42.37</td>
  </tr>
  <tr>
    <td>RoG-general</td><td>41.21</td><td>64.57</td><td>20.15</td><td>11.88</td><td>22.84</td><td>46.15</td><td>13.08</td><td>33.50</td><td>111.40</td><td>9.41</td><td>48.09</td><td>63.21</td><td>10.88</td><td>26.98</td><td>48.74</td>
  </tr>
</table>

Table 18: Evaluation of graph retrieval in different RoG versions across five settings.

<table>
  <tr>
    <th rowspan="2">Path Lengths</th>
    <th colspan="3">1</th>
    <th colspan="3">2</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
  </tr>
  <tr>
    <td>Multi-hop</td><td>-</td><td>-</td><td>-</td><td>28.78</td><td>32.38</td><td>41.41</td><td>24.67</td><td>23.47</td><td>30.83</td><td>22.50</td><td>17.23</td><td>19.35</td><td>24.05</td><td>16.46</td>
  </tr>
  <tr>
    <td>Set</td><td>79.96</td><td>82.39</td><td>92.00</td><td>38.95</td><td>42.38</td><td>49.70</td><td>42.42</td><td>41.04</td><td>44.88</td><td>67.96</td><td>41.32</td><td>40.90</td><td>45.78</td><td>87.50</td><td>62.50</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>69.61</td><td>70.47</td><td>73.53</td><td>64.71</td><td>47.68</td><td>49.42</td><td>54.47</td><td>42.28</td><td>42.90</td><td>44.02</td><td>47.54</td><td>42.62</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 19: Ablation study of dense retriever on reasoning path lengths.

<table>
  <tr>
    <th rowspan="2">Path Lengths</th>
    <th colspan="3">1</th>
    <th colspan="3">2</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
  </tr>
  <tr>
    <td>Multi-hop</td><td>-</td><td>-</td><td>-</td><td>15.07</td><td>15.81</td><td>17.18</td><td>14.54</td><td>15.45</td><td>16.17</td><td>17.50</td><td>15.00</td><td>12.45</td><td>12.78</td><td>13.92</td><td>11.39</td>
  </tr>
  <tr>
    <td>Set</td><td>80.13</td><td>81.61</td><td>88.00</td><td>33.60</td><td>37.13</td><td>57.58</td><td>36.36</td><td>30.92</td><td>33.77</td><td>51.20</td><td>34.43</td><td>48.53</td><td>53.94</td><td>75.00</td><td>37.50</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>66.18</td><td>67.99</td><td>73.53</td><td>61.76</td><td>33.57</td><td>34.63</td><td>37.40</td><td>33.33</td><td>32.79</td><td>33.15</td><td>34.43</td><td>31.15</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 20: Ablation study of sparse retriever on reasoning path lengths.

<table>
  <tr>
    <th rowspan="2">Path Lengths</th>
    <th colspan="3">1</th>
    <th colspan="3">2</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
    <th>MacF1</th><th>MicF1</th><th>Hit</th>
  </tr>
  <tr>
    <td>Multi-hop</td><td>-</td><td>-</td><td>-</td><td>22.08</td><td>23.46</td><td>26.43</td><td>21.59</td><td>25.16</td><td>26.51</td><td>29.17</td><td>25.00</td><td>15.14</td><td>16.15</td><td>18.99</td><td>13.92</td>
  </tr>
  <tr>
    <td>Set</td><td>79.43</td><td>82.11</td><td>96.00</td><td>32.04</td><td>38.08</td><td>66.67</td><td>27.27</td><td>34.85</td><td>39.19</td><td>68.86</td><td>46.41</td><td>13.11</td><td>16.76</td><td>37.50</td><td>25.00</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>57.84</td><td>58.68</td><td>61.76</td><td>55.88</td><td>39.60</td><td>40.50</td><td>42.68</td><td>40.24</td><td>24.86</td><td>25.24</td><td>26.23</td><td>22.95</td><td>-</td><td>-</td><td>-</td>
  </tr>
</table>

Table 21: Ablation study of G-Retriever on reasoning path lengths.
<table>
  <tr>
    <th rowspan="2">Entities Count</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
    <th colspan="3">≥ 5</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>Single-hop</td><td>70.38</td><td>71.96</td><td>75.66</td><td>68.14</td><td>61.95</td><td>63.10</td><td>66.36</td><td>57.94</td><td>72.53</td><td>74.18</td><td>78.46</td><td>67.69</td>
  </tr>
  <tr>
    <td>Multi-hop</td><td>26.15</td><td>29.52</td><td>37.81</td><td>23.32</td><td>23.56</td><td>26.31</td><td>31.53</td><td>21.62</td><td>22.72</td><td>23.66</td><td>25.71</td><td>20.00</td>
  </tr>
  <tr>
    <td>Set</td><td>44.91</td><td>48.84</td><td>70.95</td><td>46.96</td><td>41.93</td><td>45.99</td><td>70.24</td><td>44.05</td><td>25.22</td><td>27.06</td><td>55.00</td><td>20.00</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>49.26</td><td>50.80</td><td>55.23</td><td>44.77</td><td>52.45</td><td>54.19</td><td>59.09</td><td>49.24</td><td>35.59</td><td>36.59</td><td>40.54</td><td>27.03</td>
  </tr>
</table>

Table 22: Ablation study of dense retriever on query entities count.

<table>
  <tr>
    <th rowspan="2">Entities Count</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
    <th colspan="3">≥ 5</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>Single-hop</td><td>62.67</td><td>63.76</td><td>66.37</td><td>62.39</td><td>68.13</td><td>69.36</td><td>72.90</td><td>64.49</td><td>66.68</td><td>68.11</td><td>71.54</td><td>66.92</td>
  </tr>
  <tr>
    <td>Multi-hop</td><td>15.91</td><td>16.59</td><td>18.02</td><td>15.19</td><td>14.30</td><td>15.06</td><td>16.21</td><td>14.41</td><td>7.62</td><td>7.79</td><td>8.57</td><td>5.71</td>
  </tr>
  <tr>
    <td>Set</td><td>35.68</td><td>38.71</td><td>57.09</td><td>39.19</td><td>34.69</td><td>36.76</td><td>50.00</td><td>35.71</td><td>17.67</td><td>22.72</td><td>35.00</td><td>25.00</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>37.08</td><td>38.45</td><td>42.44</td><td>35.47</td><td>37.87</td><td>38.53</td><td>40.15</td><td>38.64</td><td>30.63</td><td>31.02</td><td>32.43</td><td>27.03</td>
  </tr>
</table>

Table 23: Ablation study of sparse retriever on query entities count.

<table>
  <tr>
    <th rowspan="2">Entities Count</th>
    <th colspan="3">3</th>
    <th colspan="3">4</th>
    <th colspan="3">≥ 5</th>
  </tr>
  <tr>
    <th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th><th>MacF1</th><th>MicF1</th><th>Hit</th><th>Hit@1</th>
  </tr>
  <tr>
    <td>Single-hop</td><td>54.17</td><td>55.16</td><td>57.52</td><td>54.42</td><td>56.04</td><td>57.04</td><td>59.81</td><td>53.27</td><td>59.93</td><td>59.88</td><td>62.31</td><td>57.69</td>
  </tr>
  <tr>
    <td>Multi-hop</td><td>22.19</td><td>23.51</td><td>26.50</td><td>21.20</td><td>21.01</td><td>22.50</td><td>25.23</td><td>22.52</td><td>20.48</td><td>21.06</td><td>22.86</td><td>17.14</td>
  </tr>
  <tr>
    <td>Set</td><td>38.66</td><td>43.26</td><td>72.30</td><td>47.39</td><td>33.82</td><td>37.44</td><td>64.29</td><td>51.19</td><td>25.09</td><td>30.87</td><td>55.00</td><td>35.00</td>
  </tr>
  <tr>
    <td>Aggregation</td><td>37.54</td><td>38.41</td><td>40.70</td><td>37.21</td><td>42.19</td><td>43.10</td><td>45.45</td><td>42.42</td><td>32.43</td><td>32.43</td><td>32.43</td><td>32.43</td>
  </tr>
</table>

Table 24: Ablation study of G-Retriever on query entities count.

D.2 Formalization of Tree Sampling under Different Settings

In order to facilitate the understanding of tree sampling algorithms under different settings, we provide formalization in this section.

Single-hop Setting  We first perform C-BFS starting from node \( v_a \in V \), setting \( D_{\max} = 1 \):

\[
G_{T_{v_a}} = \text{C-BFS}(v_a, G, D_{\max} = 1, W_{\max}).
\] (5)

Then we randomly select \( n \) non-root nodes as query nodes, that is \( V_q = \{v_{q_1}, v_{q_2}, \ldots, v_{q_n}\} \subseteq V_{T_{v_a}} \setminus \{v_a\} \). To ensure the uniqueness of the answer, we check all common neighbors of the query nodes to ensure no other node shares the same relation. Formally, let \( \mathcal{R}(u, v) \) represent the relation between nodes \( u \) and \( v \) in the graph \( G \). To ensure that no other node shares the same relation with the query nodes, we define the condition as follows:

\[
\forall v \in \mathcal{N}_q, \exists v_{q_i} \in V_q, \mathcal{R}(v, v_{q_i}) \neq \mathcal{R}(v_a, v_{q_i}), \tag{6}
\]

where \( \mathcal{N}_q = \cap_{i=1}^n \mathcal{N}_{v_{q_i}} \) denotes the common neighbors of query nodes.

Multi-hop Setting  We first perform C-BFS starting from node \( v_a \in V \), setting \( D_{\max} \geq 2 \):

\[
G_{T_{v_a}} = \text{C-BFS}(v_a, G, D_{\max} \geq 2, W_{\max}). \tag{7}
\]

Then we randomly select \( n \) non-root nodes as query nodes, that is \( V_q = \{v_{q_1}, v_{q_2}, \ldots, v_{q_n}\} \subseteq V_{T_{v_a}} \setminus \{v_a\} \). We check the children nodes (level 1 nodes) of the answer node to ensure its uniqueness. Specifically, let \( v_a \) be the answer node, and \( \mathcal{C}(v) \) denote the set of children nodes (level 1 nodes) of node \( v \). Specifically, the parent nodes of the answer node \( v_a \) are those nodes that are directly connected to \( v_a \) at depth 1 in the tree \( G_{T_{v_a}} \). We define the condition as follows:

\[
\forall v \in \mathcal{N}_{c_v}, \exists v_c \in \mathcal{C}(v), \mathcal{R}(v, v_c) \neq \mathcal{R}(v_a, v_c),
\] (8)

where \( \mathcal{N}_{c_v} = \cap_{v_c \in \mathcal{C}(v)} \mathcal{N}_{v_c} \) denotes the common neighbors of the children nodes.

Set Setting  We first adopt the same searching approach as the single-hop setting:

\[
G_{T_v} = \text{C-BFS}(v, G, D_{\max} = 1, W_{\max}). \tag{9}
\]

To obtain the answer node set, we set relation consistency constraints to ensure the answer nodes share some similar properties:

\[
V_a = \{v_{a_1}, v_{a_2}, \ldots, v_{a_n}\} \subseteq V_{T_v} \setminus \{v\},
\]
\[
\forall v_{a_i}, v_{a_j} \in V_a, \mathcal{R}(v_{a_i}, v) = \mathcal{R}(v_{a_j}, v).
\] (10)
Algorithm 1 The pipeline of data construction

Input: Knowledge graph \( G = (V, E) \), construction iterations \( T \)
Output: Constructed data \( D = \{(q, V_q, v_a, \mathcal{P})\} \), including query \( q \), query nodes \( V_q \), answer nodes \( V_a \), and reasoning paths \( \mathcal{P} \)

1: \( D \leftarrow \{\} \)
2: for \( i = 1 \) to \( T \) do
3:     \( v_a \leftarrow \text{Sample}(V, 1) \) # Sample one node as the answer
4:     \( G_{T_{v_a}} \leftarrow \text{C-BFS}(v_a, G, D_{\max}, W_{\max}), \quad \text{where} \)
5:     \( G_{T_{v_a}} = (V_{T_{v_a}}, E_{T_{v_a}}) \)
6:     \( V_q \leftarrow \text{Sample}(V_{T_{v_a}} \setminus \{v_a\}, k) \) # Sample query nodes
7:     \( \mathcal{P} \leftarrow \{\} \)
8:     for \( v_q \in V_q \) do
9:         \( p_{v_q, v_a} \leftarrow \text{BackTrack}(v_q, v_a, G_{T_{v_a}}) \)
10:        \( \mathcal{P} \leftarrow \mathcal{P} \cup \{p_{v_q, v_a}\} \)
11:    end for
12:    \( q \leftarrow \text{LLM}_c(V_q, v_a, \mathcal{P}) \) # Query creation
13:    \( q \leftarrow \text{LLM}_r(q, V_q) \) # Query refinement
14:    \( V_q \leftarrow \text{EntityMatching}(q, V_q) \)
15:    \( \mathcal{P} \leftarrow \{\} \)
16:    for \( v_q \in V_q \) do
17:        \( p_{v_q, v_a} \leftarrow \text{BackTrack}(v_q, v_a, G_{T_{v_a}}) \)
18:        \( \mathcal{P} \leftarrow \mathcal{P} \cup \{p_{v_q, v_a}\} \)
19:    end for
20:    \( D \leftarrow D \cup \{(q, V_q, v_a, \mathcal{P})\} \)
21: end for
22: \( D \leftarrow \text{Filtering}(D) \)
23: return \( D \)

Finally, we use the common neighbors of the answer nodes as the starting points for multi-source BFS to ensure the connectivity of the search tree:

\[
G_{T_A} = \text{Multi-source BFS}(\mathcal{S}, G, D_{\max}, W_{\max}),
\]

where \( \mathcal{S} = \bigcap_{v_a \in V_a} \mathcal{N}_{v_a} \) denotes the common neighbors.

Aggregation Setting We first obtain the node set \( V_a = \{v_{a_1}, v_{a_2}, \ldots, v_{a_n}\} \) and search tree \( G_{T_A} = (V_{T_A}, E_{T_A}) \) using the same strategy as the set setting. Then we construct the final answer node using two strategies:

For the first strategy, we identify common relations shared by all answer nodes and find neighboring nodes connected.

\[
\exists r, s.t. \forall v_a \in V_a, \exists v \in \mathcal{N}_{v_a}, \mathcal{R}(v_a, v) = r.
\]

Then we add the newly retrieved nodes and edges to original search tree \( G_{T_A} \) and obtain \( G'_{T_A} \):

\[
E'_{T_A} = E_{T_A} \cup \{(v, r, v_a)|v_a \in V_a, v \in \mathcal{N}_{v_a}\},
\]
\[
V'_{T_A} = V_{T_A} \cup \{v|v \in \mathcal{N}_{v_a}, v_a \in V_a, \mathcal{R}(v_a, v) = r\},
\]
\[
G'_{T_A} = (V'_{T_A}, E'_{T_A}).
\]

For the second strategy, we construct the edge set \( E \) by taking all the edges adjacent to \( v_a \):

\[
E = \{(v_a, r, v) \mid v \in \mathcal{N}_{v_a}\}.
\]

For each edge \((v_a, r, v)\) in \(E\), iterate over all other nodes \(v' \in V_a\) (i.e., \(v' \neq v_a\)) and check whether the edge \((v', r, v)\) exists. If such an edge exists, remove the edge \((v_a, r, v)\) from \(E\). This condition can be formally expressed as:

\[
E' = \{(v_a, r, v) \in E \mid \nexists v' \in V_a,\ v' \neq v_a\}.
\]

Finally, we randomly select one edge \((v_a, r, v)\) from the remaining edge set \(E'\) and add the edge to the search tree \(G_{T_A} = (V_{T_A}, E_{T_A})\).

Editing Setting First, we modify the graph \(G\) according to the specified editing method:

\[
G' = \text{edit}(G).
\]

Next, we construct the tree based on the tree sampling method used in the single-hop or multi-hop settings.

Answerability Setting We first construct the reasoning tree, query and nodes using the same strategy as the single-hop or multi-hop settings. Then we sample some of the data and modify the graph structure \(G\):

\[
V' = V \setminus \{v_a\},
\]
\[
E' = \{(u, r, v) \in E \mid u \neq v_a, v \neq v_a\},
\]
\[
G' = (V', E').
\]

E Examples

Examples in six settings of our constructed dataset M^3GQA can be found in Table 25 and 26.

F Prompts

Prompts used in our paper are listed below.
Table 25: Cases of data in four general settings.

<table>
  <tr>
    <th> </th>
    <th>Question</th>
    <th>Answer</th>
    <th>Reasoning Paths</th>
  </tr>
  <tr>
    <td rowspan="1">Single-hop</td>
    <td>Which film production company produced the movies {The Pink Panther}, {A League of Their Own}, and {Awakenings}?</td>
    <td>Columbia Pictures</td>
    <td>[A League of Their Own, film.film.production_companies, Columbia Pictures], [The Pink Panther, film.film.production_companies, Columbia Pictures], [Awakenings, film.film.production_companies, Columbia Pictures]</td>
  </tr>
  <tr>
    <td rowspan="1">Multi-hop</td>
    <td>Which river, flowing through {Myanmar} and near {Phou Khe}, was partially affected by {Typhoon Mirinae} in {Asia}?</td>
    <td>Mekong</td>
    <td>["Laos", "location.location.partially_contains", "Mekong"],[Vietnam, location.location.partially_contains, Mekong],[Phou Khe, location.location.partially_containedby, Laos],[Typhoon Mirinae, meteorology.tropical_cyclone.affected_areas, Vietnam],[Myanmar, location.location.partially_contains, Mekong],[Asia, base.locations.continents.countries_within, Vietnam]</td>
  </tr>
  <tr>
    <td rowspan="1">Set</td>
    <td>What languages are spoken in a country with a {Presidential system}, located in the same region as where {Juba Arabic} and {Algerian Arabic} are spoken?</td>
    <td>Swahili Language, English Language, Arabic Language</td>
    <td>[Tanzania, location.country.languages_spoken, English Language],[Africa, base.locations.continents.countries_within, Tanzania],[Tanzania, location.country.languages_spoken, Swahili Language],[Tanzania, location.country.languages_spoken, Arabic Language],[Algerian Arabic, language.human_language.region, Africa],[Presidential system, government.form_of_government.countries, Tanzania],[Juba Arabic, language.human_language.region, Africa]</td>
  </tr>
  <tr>
    <td rowspan="1">Aggregation</td>
    <td>Which female literary figure influenced both {Sherwood Anderson} and {Ralph Ellison}, and subsequently impacted the works of {Ray Bradbury}, {William Faulkner}, {John Steinbeck}, and {Thomas Wolfe}?</td>
    <td>Mark Twain, Gertrude Stein, Walt Whitman</td>
    <td>[Sherwood Anderson, influence.influence_node.influenced_by, Gertrude Stein],[Sherwood Anderson, influence.influence_node.influenced_by, Walt Whitman],[Ralph Ellison, influence.influence_node.influenced_by, Mark Twain],[Thomas Wolfe, influence.influence_node.influenced_by, Sherwood Anderson],[Walt Whitman, people.person.gender, Male],[Gertrude Stein, people.person.gender, Female],[Ralph Ellison, influence.influence_node.influenced_by, Gertrude Stein],[William Faulkner, influence.influence_node.influenced_by, Sherwood Anderson],[Ralph Ellison, influence.influence_node.influenced_by, Walt Whitman],[John Steinbeck, influence.influence_node.influenced_by, Sherwood Anderson],[Sherwood Anderson, influence.influence_node.influenced_by, Mark Twain],[Ray Bradbury, influence.influence_node.influenced_by, Sherwood Anderson],[Mark Twain, people.person.gender, Male]</td>
  </tr>
</table>
Table 26: Cases of data in two special settings.

<table>
  <tr>
    <th> </th>
    <th>Question</th>
    <th>Answer</th>
    <th>Reasoning Paths</th>
  </tr>
  <tr>
    <td rowspan="2">Editing</td>
    <td>Who is a notable person born in the region that includes {Alameda County}, served by the {San Francisco Chronicle}, and covered by {Cairn Marketing}?</td>
    <td>Spike Lee</td>
    <td>[San Francisco Chronicle, book.newspaper.circulation_areas, San Francisco Bay Area],[San Francisco Bay Area, location.location.containedby, Northern California],[Alameda County, location.location.containedby, San Francisco Bay Area],[Northern California, location.location.people_born_here, 'Spike Lee],[Cairn Marketing, organization.organization.geographic_scope, San Francisco Bay Area]</td>
  </tr>
  <tr>
    <td>Which {Taylor Swift} concert tour featured performances of {"The Lucky One"}, {"Shake It Off"}, and {"If This Was a Movie"}?</td>
    <td>No</td>
    <td>None</td>
  </tr>
</table>
The prompt for query generation.

Please help me design a multi-entity (multi-hop) reasoning question. The question must include at least 3 of the following entities: {entities}

And the answer of this question must be: {answer entity}

You can create the question according to the following knowledge graph. The question must be natural and close to reality

The question needs to include the relationships in the knowledge graph. Just need to output the question, no need to output additional explanatory statements

Knowledge graph: {reasoning paths}

Question:

The prompt for query paraphrase.

Please polish the following question to make it more natural and more close to reality. Just need to output the question, no need to output additional explanatory statements.

Question:
The prompt for the first strategy of aggregation setting.

I will give you a question and its answer. I am now providing you with an additional edge related to the answer on the knowledge graph. Please help me modify the original question into a question involving further reasoning according to the additional edge, that is, further reasoning on the answer set.

For example, the original question is, which scientists in the field of deep learning have won the Turing Award? The answer if Hinton, Bengio, Lecun. We can modify it into a question for further reasoning about the answer: Who among the Turing Award winning scientists in the field of deep learning graduated from the University of Cambridge?

You should only output the modified question, no need to output an additional statement. If rewriting is not possible, output "No"

Original question: {original question}

The answer of the original question: {answer entities}

The additional edge related to the answer: {constrained edges}

The answer to the modified question must be: {final answer}

Modified question:
The prompt for the first strategy of aggregation setting.

I will give you a question and its answer. I am now providing you with the answer set of all neighboring nodes on the knowledge graph. Please help me modify it into a question involving further reasoning, that is, further reasoning on the answer set. Additionally, you also need to provide the answer of modified question to me.

The output format is a binary (question, answer), where the first item represents the modified question and the second item represents the answer to the question. If the answer is a set, please write the answer part in the form of a list, such as (question, [a1, a2, a3, a4]).

If rewriting is not possible, output "No"

For example, the original question is, which famous rivers pass through China? The answer is Yangtze River, Yellow River, the Yarlung Zangbo River, etc. We can transform it into a question for further reasoning about the answer: ("Among all the famous rivers that pass through China, which one is the longest?", "Yangtze River")
For example, the original question is, which scientists in the field of deep learning have won the Turing Award? The answer if Hinton, Bengio, Lecun. We can modify it into a question for further reasoning about the answer: ("Who among the Turing Award winning scientists in the field of deep learning graduated from the University of Cambridge?", "Hinton")

The answer to the modified question can also be a collection. For example: ("which deep learning scientists who have won the Turing Award are Canadian?", "[Hinton, Bengio]")

Please strictly follow the form: (question, answer)! No need to output additional explanatory statements.

Original question: {original question}

The answer of the original question: {answer entities}

Edges and neighbors related to the answer: {constrained edges}

Modified question:

The prompt for replacing the answer node in editing setting.

Give you an entity: {original answer entity}, please provide another entity that belongs to the same category as it.

You should only output the entity, no need to output additional explanatory statements.

New entity:
The prompt for LLM-based filtering.

Given a question and an answer, please use the given knowledge graph to determine if the answer is correct.

If the answer is correct, please output 'yes', otherwise, output 'no'. No need to output additional statement.

Question: {question}

Answer: {answer}

Knowledge graph: {reasoning paths}

Judgement:

The prompt for answer generation of LLM-only methods.

Given a question, please use one or more entities to answer it.

Please directly output the entity (entities) that meet(s) the requirements, without the need to output additional statements.

If the answer is multiple entities, please separate them with commas.

Question: {question}

Answer:

The prompt for answer generation of other baselines.

Given a question and a relevant knowledge graph, please use one or more entities to answer the question.

Please directly output the entity (entities) that meet(s) the requirements, without the need to output additional statements.

If the answer is multiple entities, please separate them with commas.

Question: {question}

Knowledge graph: {knowledge graph}

Answer:
The prompt for answer generation of answerability setting.

Given a question and a relevant knowledge graph, please check whether the question is answerable according to the knowledge graph.

If the question is answerable, please output 'yes', otherwise output 'no', without the need to output additional statements.

Question: {question}

Knowledge graph: {knowledge graph}

Answer:

Reviewer paper summaries: In this paper, the authors present the new GraphRAG benchmark to offer an extensive set of complex real-world applications involving multi-entity queries, a highly common yet challenging aspect in GraphRAG systems. They thoroughly describe the data collection process and perform a lot of experiments implementing the existing approach on the proposed benchmark.

This paper aims to design more complex benchmarks to evaluate GraphRAG methods more thoroughly. To verify that GraphRAG methods truly comprehend the content of graphs, this benchmark considers more challenging settings, which contain more entities in queries. The whole procedure of forming benchmarks consists of four stages: different tree sampling strategies for different settings, reasoning path backtracking, query generation by LLMs, and refinement. Some advanced GraphRAG methods are validated in these benchmarks.
Summary of strengths: The paper is well written and easy to follow. The motivation and the task formulation are very well described. The authors perform a big range of experiments reproducing well-known tasks. I think that the benchmark is useful for further researchers, so I am positive regarding this paper.

1.	This benchmark adopts a new QA construction approach, which does not rely on fixed templates. This method utilizes the reasoning path to construct the QA pairs, and the additional reasoning path is important for retrievers to learn how to find the ground truth.
2.	This paper is easy to follow and has a strong motivation. 
3.	This work has conducted experiments on some advanced GraphRAG methods and has clear tables and figures to show their results. The results show that there is significant room for improvement in GraphRAG methods on the proposed benchmark.
Summary of weaknesses: The main disadvantages of the paper that motivate a lower score are as following:
1. The automatic provenance of the dataset: no one can disagree that a properly created manual dataset is better than the automatically created one (even with the best models ever). This is ok for me, however, it may prevent others from using the dataset. Or, at least, you coulf manually verify the test set so it has more credibility.
2. Unclear manual curation section: no details about the manual curation step are given (how many items were verified, what is the annotation agreement, were there anyguidelines, etc.). This contributes to the negative view on the dataset.
3. Mentionning the four-step data construction as contribution is not persuasive: each step is not novel and I cannot find any other applications of this pipeline to benchmarking another task.

1.	This work mainly focuses on simple answers that are easy to validate for correctness. However, long context answer tasks like summary tasks are also important for GraphRAG methods [1]. 
2.	There are some missing baselines. Traditional dense RAG methods have two advantages: stronger LLMs (GPT-4) and better recall (as shown in Table 5). It would be better to control variables (such as using the same LLMs as G-Retriever) to better understand dense RAG's performance. Does its superiority stem from better LLMs or retrieval methods? 
3.	The details of manual filtering in Section 4.4 are lacking. Are there numerical results showing the correctness of QA pairs under human supervision, considering that LLMs were used to construct these pairs?

[1] From Local to Global: A Graph RAG Approach to Query-Focused Summarization
Comments, suggestions and typos: 1. Missing related work: TextGraphs 2024 Shared Task on Text-Graph Representations for Knowledge Graph Question Answering was a shared task that also considered KGQA with subgraphs and presented a dataset on KGQA (with manually created test set)
https://aclanthology.org/2024.textgraphs-1.9/
Would be nice to add the dataset into benchmarking (probably some modifications are needed)

2. line 401: Eq 3 -> Equation (use full form)
3. line 439: add citation to LLM query to Appendix, or describe how it was done
4. None information about how and on how many and on annotation agreement, guidelines is given about the manual filtering.
5. Why the number of reasoning hops is 0 for multihop in Figure 4b?
6. line 562: https://en.wikipedia.org/wiki/Widows_and_orphans avoid widows and orphans
7. line 107: real-world applications - how the could be real world if they are automatically created? How do you check that the questions are similar to the real ones?

Please address my concerns in Summary Of Weaknesses.
Overall assessment scores: 4

3