When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation
Authors: Zhishang Xiang
Chuanjie Wu
Qinggang Zhang
Shengyuan Chen
Zijin Hong
Xiao Huang
Jinsong Su
Venue: ICLR 2026 Poster (2026)
Keywords: RAG
GraphRAG
GraphRAG Benchmark
Primary area: datasets and benchmarks

Abstract: Graph retrieval-augmented generation (GraphRAG) has emerged as a powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning. Despite its conceptual promise, recent studies report that GraphRAG frequently underperforms vanilla RAG on many real-world tasks. This raises a critical question: Is GraphRAG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG-Bench, a comprehensive benchmark designed to evaluate GraphRAG models on both hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG-Bench features a comprehensive dataset with tasks of increasing difficulty, covering fact retrieval, complex reasoning, contextual summarize, and creative generation, and a systematic evaluation across the entire pipeline, from graph construction and knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application. All related resources and analysis are collected for the community at https://github.com/GraphRAG-Bench/GraphRAG-Benchmark.

Published as a conference paper at ICLR 2026
WHEN TO USE GRAPHS IN RAG: A COMPREHENSIVE ANALYSIS FOR GRAPH EETRIEVAL-AUGMENT GEN- ERATION
Zhishang Chuanje Wu Qinggang Zhang" Shengyuan Chen', Zijin Hong' Xiao Huang". Jinsong Su2.1.3f
Institute of Artificial Intelligence, Xiamen University, China School of Informatics, Xiamen University, China
KKey Laboratory of Digital Protection and Intelligent Processing of Intangible Cultural Heritage of Fujian and Taiwan (Xiamen University), Ministry of Culture and Tourism, China
"The Hong Kong Polytechnic University, Hong Kong SAR, China saanghhishang stu xmu edu. en; wuchuanjie@@ste qinggangg- zhangê connect polyu.hke hk; zi zijin. hon.gconnect........ooooooo....e pol yu. (sheng- Seheng-yuan... yuan. chen, xiao. huang) @polyu. edu. hk; jssu@xmu.edus
xmu, edu. en;
ABSTRACT
Graph retrieval-augmente generation (GraphRAG) has emerged as powerful paradigm for enhancing large language models (LLMs) with external knowledge. It leverages graphs to model the hierarchical structure between specific concepts, enabling more coherent and effective knowledge retrieval for accurate reasoning. Despite its conceptual promise, recent studies report that GraphRAG frequently un- derperforms vanilla RAG on many real-world tasks. This raises critical question: Is GraphRAGi AG really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems? To address this, we propose GraphRAG- Bench, comprehensive benchmark designed to evaluate GraphRAG models on both hierarchical knowledge retrieval and deep contextual reasoning. GraphRAG- Bench features comprehensive dataset with tasks of increasing difficulty, covering fact retrieval, complex reasoning, contextual summarize, and creative generation, and systematic evaluation across the entire pipeline, from graph construction and knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG and the underlying reasons for its success, offering guidelines for its practical application. All related resources and analysis are collected for the community https: / qttptbt..bb../..o/ com http//raaahaGGGaaaee Bench wwwpeanwcw.aacaaaaaaaaaaa.aa Bww.aaaaaaaaaaaaaaaaaaaaaaaaaaaa Benchmark
INTRODUCTION
Large language models (LLMss), like Claude (Anthropic) 2024) and GPT (OpenAl 2023) series, have surprised the world with their remarkable capabilities in many real-world tasks, like linguistic comprehension (Brown et al. 2020), question answering (Khashabi et al., 2020), mathematical reasoning (Hong al.] 2025). and content generation (Chowdhery et al. [023 Hong al. 2024
Zhang et al. 2024a). Despite the success, LLMs are always criticized for their inability to handle knowledge-intensive tasks and the tendency generate hallucinations (Huang al.
2023). especially
when faced with questions requiring specialized expertise (Chen et al. 2024b He et al.
2024 Tan
al. 2024). Retrieval-augmenter generation (RAG) (Gao al.] 2023 Lewis et al. 2020)
recently
offered a promising approach to adapt LLMs for specific or private domains. Rather than retraining
LLMs incorporate new knowledge and updates (Feng al. 2025 Fang et al. 2025 Jiang et al.
Wang et al. 2024b Zhang et al. 20255), RAG enhances these models by leveraging external knowledge from text corpora. This approach enables L.LMs to generate responses by leveraging not only their parametric knowledge but also real-time retrieved domain-specific information, thereby
2025
providing more accurate and reliable answers (Chen al. 2024a Li et al. 2024).
"Equal contribution. 'Corresponding author.

---

Published as a conference paper at ICLR 2026
However, traditional RAG systems often face critical challenges when dealing with large-scale, unstructured domain corpora (Edge et al. 2024 Peng et al. 2024). The textual documents in this corpus, collected from different sources, like research papers, textbooks and technical reports, often vary widely in accuracy and completeness (Guo et al.
2025) The information retrieved by RAG
systems be extensive, complex, and lack clear organization, since domain knowledge is typically scattered across multiple documents without clear hierarchical relationships between different con-
cepts (Sun al., 2024; Zhang et al. 2024b; Ma al. 2024). Although RAG systems (Borgeaud et al. Izacard et al. [0033] Jiang et al.
2023) attempt to manage this complexity by dividing documents into smaller chunks for effective indexing. this approach inadvertently sacrifices crucial contextual information, significantly compromising retrieval accuracy and contextual comprehension
for complex reasoning (Han et al.
2024 Zhang et al. 2025a Shengyuan et al. 2024).
To address this, graph
generation (GraphRAG) (Zhang et al.
2025a Peng et al.
2024| |Procko & Ochoa 2024) has recently emerged as a powerful paradigm that leverages external structured graphs to improve LLMs' capability on contextual comprehension (Han et al. 2024
Zhang et al. 2025a). Early efforts, like Microsoft GraphRAG (Edge et al.] 2024) and its variant
LazyGraphRAG (Darren Edge 2024), employ hierarchical community-basee search and combine
local/global querying for comprehensive responses. Building on this, LightRAG (Guo et al. 2024)
improves scalability through dual-level retrieval and graph-enhanced indexing, while GRAG (Hu et al.] (024) introduces a soft pruning technique to mitigate the impact of irrelevant entities in retrieved subgraphs and employs graph-aware prompt tuning to help LLMs interpret topological structure. Further extending these capabilities, StructRAG (Li et al. 2024) tailors data structures to specific tasks by dynamically selecting optimal graph schemas, while KAG (Liang et al. 2024) constructs domain expert knowledge using conceptual semantic reasoning and human- annotattd schemas, which significantly reduces noise present in OpenlE systems. These strategies used in GraphRAG models significantly improve retrieval precision and contextual depth, enabling LLMs to address complex, multi-hop queries more effectively.
Despite its conceptual promise, recent studies (Han et al. 2025 Zhou et al.) 2025 report that GraphRAG models frequently underperform traditional RAG approaches on many real-world tasks. Specifically, the previous study (Han et al. 2025) demonstrates that GraphRAG achieves 13.4% lower accuracy on Natural Question compared to vanilla RAG, with particularly poor performance on time- sensitive queries (e.g., 16.6% accuracy drop for questions requiring real-time knowledge updates). While graph retrieval improves reasoning depth by 4.5% on HotpotQA's multi-hop questions, introduces 2.3 higher latency on average (Zhou al. 2025). These inconsistencies between conceptual potential and practical efficacy raise critical questions: Is Graph really effective, and in which scenarios do graph structures provide measurable benefits for RAG systems?
It is crucial to identify the factors that are currently limiting GraphRAG's s real-world performance. However, quantitatively and fairly assessing the role of graph structures in RAG systems is challenging. Current benchmarks, including HotpotQA (Yang et al. 2018), MultiHopRAAG (Tang & Yang 2024) and UltraDomain (Qian et al.) (024)) fail to adequately evaluate the effectiveness of graph structures in RAG systems due to fundamental limitations in both their problem design and corpus composition. 1 First, existing benchmarks lack granular differentiation in task complexity. Existing benchmarks overemphasize retrieval difficulty, locating scattered facts from corpora, while neglecting reasoning complexity, which involves synthesizing interconnected facts into contextually grounded solutions. As shown in Figure 2. they predominantly focus narrow task categories, such as simple fact retrieval or linear multi-hop reasoning. without systematically capturing the spectrum of challenges encountered in real- weal-wrrld scenarios (Tang & Yang 2024). For instance, typical multi-hop question in existing benchmarks might ask, "Who founded Company Kjaer Weis, and in which city was this person born;," This requires only the extraction of several discrete facts and cannot extend to complex scenarios requiring hicrarchical reasoning and contextual synthesis. @ Second, corpora in existing RAG benchmarks suffer from inconsistent quality and low information density. Many datasets are built on generic sources like Wikipedia or news articles, which lack domain -specifcc knowledge or explicit logical connections. While some work, like UltraDomain (Qian et al., 2024), has tried to extract domain-specift corpora from textbooks, they often fail to encode implicit hierarchies for real-world applications. This makes it impossible to assess GraphRAG's 's core strengths in leveraging domain hierarchies. For example, a corpus with poorly defined conceptual hierarchies or loosely connected entities cannot meaningfully test whether graph-aware retrieval mechanisms improve multi-hop reasoning or preserve contextual coherence during knowledge acquisition. Additionally,
2

---

Published as a conference paper at ICLR 2026
Naive RAG
Knowledge retrieval implicit relationships
Graph RAG
Semantic Question
Gooat
Question
retroved
perriene
Inplit relztiorshiss
Implich
Corgen
Cunteni
Implic relatorehips
Corpus
Ieder Graga A Crerrar
-mmlkii
Hierarchical reasoning with indirect evidence
omrgtngg
(Prempting)
Generr by
Ansrerr
heoely
MMdli-op cherrs
Genoration with
2ir
Ie4son ng pults
Retrieved cxnsent
dependerncien
Rewieved content Hicrarchicdl reaseing
deperdoncios
Efficiency Sculability
Luss preprocesiing coul
Less token coal
Tock comploiity
Sigger praprocssing coo
Higher tolan coul
Test complesity
Prompt Ryforie
No graph counta oclion
Only keep passages with high scmaatiic sinuaariy
prompt EDE Minimll preproomargy
Graph cunstrardinn
Lisk MaRC relivedd
sraph
LOW prompt Mininal preprocessinc
Figure 1: RAG vs. GraphRAG. The pipelines of RAG and GraphRAG and their characteristics.
the absence of corpora with varying information densities, ranging from tightly structured domain knowledge to loosely organized real-world documents, further restricts the evaluation of graph structures' scalability and adaptability.
To bridge this gap, we propose GraphRAG-Bench, comprehensive benchmark designed to evaluate GraphRAG models on deep reasoning. GraphRAG- --nBhhchhh features comprehensive corpora with different information density, including tightly structured domain knowledge and loosely organized texts, and @ tasks of increasing difficulty, covering fact retrieval, multi-hop reasoning. Contextual Summarize, and creative generation, and @ systematic evaluation across the entire pipeline, from graph construction and knowledge retrieval to final generation. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG systems and the underlying reasons for its success, offering guidelines for its practical application.
2
PRELIMINARY STUDY
Before going into the details of our benchmark, we first examine the pipelines of RAG and GraphRAG, and conduct a comprehensive study to identify the primary limitation of existing benchmarks.
2.1 RAG VS. GRAPHRAG
We carefully compare GraphRAG's pipeline with traditional RAG's and summarize their characteris- ties in F Generally, RAG retrieves contextually relevant data from a corpus during inference, enabling real-time, domain-specific responses without model retraining. While efficient, its reliance on direct semantic similarity may overlook the broader contextual web of relationships, hierarchies, or implicit logic that binds concepts together. GraphRAG addresses this limitation by expanding the retrieval framework beyond semantic relevance. It structures background knowledge as a graph, where nodes represent entities, events, themes, and edges define their logical, causal, associative connections. When processing a query. GraphRAG retrieves not only directly related nodes but also traverses the graph capture interconnected subgraphs, uncovering latent patterns such as thematic evolution, indirect dependencies. or multi-step reasoning chains. This approach enables the model to synthesize insights from dispersed data points, making it particularly good tasks demanding complex logical inference. For instance, while RAG might retrieve isolated facts about a topic, GraphRAG could identify related events, causal chains, or thematic clusters, thereby enabling more coherent and comprehensive responses. To sum up. the primary distinction between these two paradigms lies in their handling of contextual depth. RAG excels scenarios requiring rapid access to discrete information, while GraphR AG emphasizes deep contextual analysis for tasks requiring nuanced understanding of interconnected data. More detailed analysis is included in Appendix[[]
2.2 CURRENT RAG BENCHMARKS
Existing benchmarks, such as HotpotQA YYang et al. 2018). MultiHopRAG TTagg & Yang 2024) and UltraDomain QQian et al. 2024). were primarily designed to evaluate traditional text-centric
3

---

Published as a conference paper at ICLR 2026
RAG frameworks. While these benchmarks have advanced the field, they exhibit critical limitations when applied to assessing GraphRAG.
First, existing benchmarks narrowly focus testing retrieval difficulty, the ability to locate scattered information from the corpus, while neglecting the equally critical challenge of reasoning ng difficulte which involves integrating interconnected concepts/facts by capturing the latent logic. While these benchmarks include "multi-hop" questions to test a model's ability in complex reasoning, they do not reflect real-world scenarios demanding complex logical synthesis.
For instance, a typical multi-hop question in existing benchmarks might ask, "Who founded Company Kjaer We s, and in which city this person born?" This requires only the extraction of several discrete facts from the corpus. However, real-world problems, such as explaining why Company Kjaer Weis failed a specific market, demand synthesizing financial reports. competitor analyses, consumer trends, and regulatory changes into coherent narrative. GraphRAG's strength lies in mapping these interdependencies (e.g.. "Market entry timing + supply chain disruptions + regulatory fines brand erosion") through graph traversals. However, current benchmarks lack tasks that explicitly require such synthesis, reducing "multi-hop" queries to sequential fact retrieval within narrow contexts, failing to evaluate how models infer domain-specifi hierarchies.
Table I: Categorization of tasks by complexity, ranging from factual retrieval to creative generation.
Category
Task Name
Brief Description
Example
Level
Fact Rerrieval
Roquire retrieving isolated knowledge points with minimal reasoning: mainly test procise keywoed matching.
Which region of France is Mont St. Michel located?
Lovel
Complex Reasoning
Require chaining multiple knowledge points across documents via logical connections.
How did Hinze's agreement with Felicia relate to the perception of England's rulers?
Level
Contextual Summurize
Involve symthesizing fragmented information coherent, structured answer; emphasize logical coherence and context.
What role does John Curgenven play as
Comish boatman for the visitors exploring this region?
Lovel
Creative Generation
Require inforence beyond retrieved content,
Retell the scene of King Arthur's 's comparison
often involving hypothetical or novel sccnarios.
John Curgenven and the exploration of the
Comiss coustline as a newspoper article.
Second, the corpora used in existing benchmarks suffer from inconsistent quality and low information density, Most datasets are built on generic sources like Wikipedia or news arti- cles, which lack structured domain- specific knowledge or explicit logical connections. A corpus with poorly de- fined conceptual hierarchies or loosely connected entities cannot meaning- fully test whether graph-aware re- trieval mechanisms improve multi- hop reasoning or preserve contextual coherence during knowledge integra- tion. While some work, like UltraDo- main, has tried construct domain- specific corpora using textbooks. they often fail encode implicit hierar-
Table 2: Average number of entities and relations across benchmarks. (Details are in Table 13 in Appendix E)
Metric
Ultradomain
MultiHop-AG
HotpotQA
Avg Entities Avg. Relations
170.6 73.2
10.1
39.3
3.82
12.7
Fact Retrieval Contestual Summarize
Complex Reasoning Creative Generation
100
97.
Percentage( %%)
52.2
50
44.9
19
1.8 1.2
2.9
UltraDomain
MultHHopRAG
HotpotQA
Figure 2: Distribution of question difficulty levels.
chies for real-world applications. As shown in Table 2 and Table 13 in Appendix domain concepts and their hierarchical dependency appear sparsely the corpus. This sparsity falls far below the threshold of multi-hop reasoning, which makes it impossible assess GraphRAG's core strengths in leveraging domain hicrarchies. Additionally, the absence of corpora with varying infor- mation densities, ranging from tightly structured domain knowledge to loosely organized real-world documents, further restricts the evaluation of graph structures' scalability.
Third, current benchmarks fall short in evaluating GraphRAG since their evaluation metrics focus solely on the final outputs, answer accuracy or fluency. while treating GraphRAG's internal processes (graph construction, retrieval, and generation) black boxes. Such evaluations hardly measure
4

---

Published as a conference paper at ICLR 2026
The Pipeline of Benchmark Construction
Tasks with Varying Complexity
Corpus Collection
4. Question Generation
uacailas
mgiinn Midhel
Fact Retrieval
Contextual Sommarize
Curgcoven play explaring
exideace
Abilties
Complex Reasoning
Creative Generation
agreeneent Felicia
compariess cxpleration Cornish contlize urw araspiee ontite.
2. Logic Mining
Check&Correct
perception Eaglead's
Duraker
Chock evideace
Brintion Cheskeg?
Point-to-Point Evaluation Metrics
Noda:Rotation
3. Evidence Collection
6. Refinement
Coane
$-3-2-0
dissing
Rarran
Fedewred
Reflne
Refise stoweer
Derrees
Clasterigg
Evedence
Fusthfahr
Bridiins
Graph Quality
Retrieval Performance
Generation Accuracy
Figure The overall framework of GraphRAG- It consists of three key components: (i) pipeline of benchmark construction (left), (ii) task classification by difficulty (upper right), and (iii) multi-stage evaluation framework covering indexing, retrieval, and generation (lower right).
how graph structures contribute to the retrieval and reasoning processes. truly assess GraphRAG, more holistic evaluation is necessary, encompassing the entire pipeline. This includes examining the efficiency graph construction and the quality of the resulting graph, the structure and relevance of the knowledge retrieved via the graph, and finally, the faithfulness of the generated answer to this graph-derived context. Such a comprehensive view essential to understand the actual impact and benefits of graph structures within retrieval augmented generation systems. Detailed corpus statistics and analysis for these benchmarks can be found in Section of the Appendix.
3 GRAPHR BA-BHAAG-BENN
In this section, we present GraphRAG-Bente a novel benchmark specifically designed to assess GraphRAG systems through comprehensive task hierarchies and structured knowledge integration. Specifically, GraphRAG-Bena covering fact retrieval, multi-hop reasoning, Contextual Summarize, and creative generation, and (ii) real-world corpora with different information density, and (iii) a systematic evaluation across the entire pipeline, from graph construction and knowledge retrieval to final generation.
consists a comprchensive dataset with (i) tasks of increasing difficulty,
3.1 TASK FORMULATION
Traditional benchmarks focus on tasks with simple fact retrieval linear multi-hop reasoning, where answers depend on linking concepts or facts across limited set of documents. While these tasks test model's ability to locate scattered information (retrieval difficulty). they do not reflect real-worke scenarios demanding complex logical synthesis (reasoning complexity). Our benchmark addresses this gap by designing four different tasks that progressively scale both retrieval difficulty and reasoning complexity. As shown in Table I these four tasks ensure more comprehensive evaluation: lower-level tasks validate retrieval capability, while higher levels assess reasoning depth, ensuring models balance precise fact extraction with clear contextual comprehension.
3.2 DATASET CONSTRUCTION
Corpus collection. Existing datasets often derive from generic sources like Wikipedia or news articles, which, while broadly accessible, lack explicit logical connections and structured domain expertise, unable to evaluate systems that require reasoning over implicit relationships or contextual hierarchies. We address these issues by (i) integrating tightly structured domain data from NCCN medical guidelines to embed explicit hierarchies and standardized protocols, which provide dense conceptual relationships (e.g., treatment protocols linking symptoms, drugs, and outcomes) at scales exceeding typical domain corpora, and (ii) collecting loosely organized texts sare-20th-centuur novels) from Gutenberg library to simulate real-world documents with implicit, non-linear narratives, ensuring the corpus reflects the complexity of unstructured knowledge while minimizing pretraining contamination. This combination ensures the corpus balances unstructured, real-works ambiguity with
5

---

Published as a conference paper at ICLR 2026
domain-specific hierarchies, enabling rigorous evaluation of both retrieval robustness and reasoning depth. We include more details about these datasets (Novel and Medical Datasets) in Appendix C
Logic and evidence extraction. To overcome the superficial treatment of reasoning in existing bench- marks, where multi-hop queries often reduce to linear fact retrieval, our framework systematically transforms raw text into structured domain ontologies. These ontologies preserve not only entities but also their contextual relationships and hierarchical dependencies, enabling the extraction of fine-grained evidence that reflects both localized factual clusters and interconnected reasoning chains. Where prior work struggles to represent latent logical synthesis (e.g.. inferring causal pathways from dispersed market factors), our evidence extraction process isolates self self-contained subgraphs for basic retrieval while reconstructing multi-hop relational sequences that expose deeper inferential patterns.
Question generation. generate the questions according the complexity of the underlying evidence. Rather than treating difficulty function of factual scarcity or hop count, we calibrate questions by progressively integrating evidence types, from isolated subgraph fragments for retrieval tasks to global topology-aware reasoning for synthetic reasoning. This ensures that complex questions necessitate not merely aggregating discrete facts but synthesizing contextual hierarchies and domain- specific ontologies. By anchoring questions in structured evidence packages that mirror real-world knowledge interdependencies, our benchmark evaluates how models derive insights from both explicit logical frameworks and unstructured contextual ambiguity, thereby addressing the critical gap in assessing reasoning depth beyond simple retrieval.
Relevance check and refinement. To ensure the accuracy and practical relevance of the dataset, implemented rigorous validation and refinement processes after initial construction. Full methodolog. ical details are provided in Appendix C while the visualization of our datasets is in Appendix B
3.3 EVALUATION METRICS
Existing benchmarks primarily focus on the accuracy or fluency of final outputs, while how to measure the graph's contribution the retrieval and reasoning processes remains an open challenge. To address this, we design stage-specific metrics that evaluate the entire workflow from graph construction and retrieval to final generation. In this section, we introduce these metrics accordingly.
Graph Quality. GraphRAG constructs graphs to represent the domain concepts and their relations, which enables structured and effective knowledge organization. To evaluate its effectiveness, we design structure-based metrics to assess the quality of graphs built in different GraphRAG.
NODE COUNT quantifies the number of entities extracted during knowledge graph construction. Higher values imply broader domain coverage and finer- finer-grained knowledge representation.
EDGE COUNT measures the number of relations among entities. Higher values indicate denser semantic connectivity, facilitating multi-hop reasoning and complex query handling.
AVERAGE DEGREE captures global connectivity by averaging the number of edges per node. Higher values of AVERAGE DEGREE indicate more integrated knowledge representations, enabling efficient cross-node traversal. It is computed as:
AVERAGE DEGREE
Σ DEV
deg(u),
(1)
M
where is the set of nodes, and deg(v) is the degree of node v.
AVERAGE CLUSTERING COEFFICIENT evaluates local neighborhood connectivity via triad
completion. Higher values, common domain-specific clusters (e.g., disease treatment symptoo in medical graphs), indicate coherent subgraphs that support localized reasoning. It can be obtained by:
2- T(u)
AVERAGE CLUSTERING COEFFICIENT
Σ C(v),
C(v) deg(v) (deg(v) - 1)
(2)
M
VEV
Here, C(v) is the clustering coefficient of node v, with T(v) denoting its centered triangle number.
Retrieval Performance. To evaluate the retrieval performance of GraphRAG, we argue that an effective system should not only ensure the completeness of retrieved information (i.e., high recall) but

---

Published as a conference paper at ICLR 2026
Table 3: Results of Generate Evaluation using GPT-4o-mini, covering tasks of varying complexity.
Category
Model
Faet Retrieval ROUGE-L
Complex Reasoning ROUGE-L
ACC
Contextual Summarize Cuv
Creative Generation
Novel Donuset
Basic RAG
RAG twlo rerank)
58.76
37.35
41.35
15.12
50.08
82.53
41.52
47.46
37.84
RAG (w rerank)
60.92
36.08
42,93
15.39
51.30
83.64
38.76
49.21
40.04
MS GraphRAG [ideo
2026
49.29
26.11
50.93
24.09
64.40
75.58
39.10
55,44
35.65
HlpppRRG Guleere
Gruph RAG
HippoRAG2
m0
60.14
31.35
53.38
33:42
70.84
48.28
49.84
30.95
LightR ACT Lin 2f a
2012
58.62
35.72
49.07
24.16
48.85
63.05
23.80
57.28
25.01
Fast-GimhhRA RORMinn 11-11
2024
56,95
35.90
48.55
21.12
56.41
80.82
46. 18
57.19
36,99
RAPTOR Samti
49.25
23.74
11.66
47.10
82.33
38.01
70.85
35.88
Lazy-OrphiB.a
Edge
58.20
Mediicul Dusused
Basic RAG
RAG (wla reran)
63.72
29.21
57.61
13.9%
63.72
77.34
58.04
35.88
57.87
RAG (wl rerank)
64,73
30.75
58.64
15.57
65.25
28.54
60.61
36.74
58.72
MS-GraphhAAG
[ddc
M02
38.63
26.80
47.04
22.98
53.01
52.65
HippoRAG
GespRAG
HippoR M32
66.28
36.97
46.13
58.78
LightRAT AGT Libt 2f 2024
63.32
37.19
61.32
24.9%
63.14
51.16
67.91
78.76
51.58
Fast. GraphRM
[
LEVIMMI 31 2024
60.93
31,04
61.73
21.37
67.98
52.07
65.93
56.07
44.73
RAPTOR (Sati
54.07
17.93
53.20
11.73
58.23
78.28
62.38
59,98
63.63
Lazy-CiphiRAM
Fage
47.82
43.79
also reduce irrelevant content (i.e., high relevance). We introduce two corresponding retrieval-quality- based metrics: 1) CONTEXT RELEVANCE measures how well the retrieved content aligns with the question's intent by calculating the semantic similarity between the question and the retrieved context. 2) EVIDENCE RECALL measures retrieval completeness by assessing whether all critical components required to correctly answer the question are captured. Details are provided in Appendix F
Generation Accuracy. After retrieval, GraphRAG system is expected to generate accurate answers based on the retrieved contexts. To evaluate the quality of the generation, we introduce four key metries: 1) LEXICAL OVERLAP: Measures word-level similarity between the generated and reference answers using longest common subsequence matching. 2) ANSWER ACCURACY: Assesses both semantic similarity and factual consistency with the reference answer. 3) FAITHFULNESS Evaluates whether the relevant knowledge points in a long-form answer are faithful to the given context. 4) EVIDENCE COVERAGE: Measures whether the answer adequately covers all knowledge relevant to the question. We provide details of these widely used metrics in Appendix
4 EXPERIMENT
This section evaluates GraphRAG against RAG through comprehensive experiments on our new benchmarks. We aim to address the following research questions: Q1 (Generation Accuracy): How does GraphRAG perform compared to RAG GraphRAG retrieve higher-quality and less redundant information the retrieval process? Q3 (Graph complexity): Does the constructed graph correctly organize the underlying knowledge? Q4 (Efficiency): Does GraphRAG introduce significant token overhead during retrieval?
benchmark? Q2 (Retrieval Performance): Does
4.1 GENERATION ACCURACY (Q1)
To address Q1, we evaluate seven representa- tive GraphRAG frameworks on our benchmark, using tailored metrics for different question types. For Type I (retrieval) and Type 2 (reason- ing) questions, we assess answer quality with ROUGE scores and accuracy. For Type 3 (sum- marization) questions, we introduce evidence coverage to measure the comprehensiveness of the generated answers. For Type 4 (creative gen- eration) questions, factual consistency. Main results in Table and Appendix lead to following observations:
Accuracy
Accuvocy
Contest Relevance
Evidence
Context
Eridence
Effeclency
Effeciency
Fact Retrieval
Complex Reasoning
Vanilla-RA
Accuracy
Graph-RAG
Accuracy
faithfulness assess
Connect
Evidence Recall
Contast
Evidence Recall
Referunce
Relevance
Obs.1 Basic RAG Matches GraphRAG in simple fact retrieval task: basic RAG is com- parable to or outperforms GraphRAG in simple fact retrieval tasks that does not require
Effeccooyy
Effeciency
Contextual Summarize
Creative Generation
Figure 4: Retrieval and generation performance of RAG and GraphRAG across four different tasks.

---

Published as a conference paper at ICLR 2026
Table 4: Results of Retrieval performance using GPT-4o-mini, covering tasks of varying complexity.
Category
Model
Fact Retrieval Relevance
Complex Reasmiigg Relevance
Contextual Summarize Relevance
Creative Generation Relevance
Novel Dotavet
Basic RAG
RAG (w/o rerank)
RAG (wl remank)
83.21
61.37
77.77
74,60
64.47
59, 89.80
82.08
80.82
73.38
69.08
83.10
$0.05
32.48
39.59
K2.84
78.73
MSS-immhhRAAie Edde
30
43.13
53.55
35.07
HippoRAGi
87.91
Graph RAG
Hippok Conterrer or
70.29
79.25
09:77
85.75
82.50
$7.82
42.18
79.10
LightR AG Cuo el 2020
73.69
33.08
85.52
37.46
87.59
38.02
71.22
38.06
Fast: GraphRAG
2024
61.48
47.86
73.51
55.21
78.58
49.74
56.31
46.27
RAPTOR Supti
266
Lazy-GraphRR.4 Darren Fdge
30.76
57.73
42.98
77.38
43.62
55.22
31.94
Mediiaal
Datuset
Basic RAG
RAG renink)
RAG rurunk)
87.83
86.24
63.71
64.73
84.97
86.49
85.56
84.11
84.
85.87
89.90
91.35
44,88
45.23
60.50
58.73
M15-GraphR.Ae (Edec
39.66
HiggoRAG (Gulenre d
87.25
52.44
83.80
42.19
45.03
Graph RAG
HipooRAG2 Tuntorrs or
78.70
87.96
77.00
80.94
77.40
86.85
61.12
78.64
Light AG Lsoo Bi. 2024
80.32
41.27
82.91
42.79
85.71
43.11
81.34
45.17
Fast. GragbRRGG
Cirsehlis- 2024
45.86
74.93
38.80
77.27
47.58
62.99
25.15
RAPTOR
Lazy GrapbRrapRRA
Darren Edge
2024
74.29
19.90
78.65
17.50
78.72
21.35
83.11
15.09
plex reasoning across connected concepts. This suggests that in less complex scenarios, basic RAG's straightforward retrieval method is sufficient, while GraphRAG's extra graph-based processing may introduce redundant noisy information for simpler queries, degrading answer quality.
Obs.2. GraphRAG excels in complex tasks: GraphRAG models show a clear advantage in complex reasoning, Contextual Summarize, and creative generation. This is intuitive, as these tasks require bridging the complex relations among multiple concepts, which is naturally a graph structure.
Obs.3. GraphRAG ensures greater factual reliability in creative tasks: RAPTOR scores highest in faithfulness (70.9%) on the novel dataset, though RAG covers more evidence (40.0%), likely because GraphRAG's fragmented knowledge retrieval and complicates broad scope generation. This trade-off highlights GraphRAG's strength in precision but limitations in wide-ranging synthesis.
4.2 RETRIEVAL PERFORMANCE (Q2)
To quantitatively compare the retrieval effectiveness of the two paradigms, we adopt two comple- mentary metrics: Evidence Recall, which measures how completely the retrieved context covers the gold evidence, and Context Relevance, which measures the semantic alignment between the retrieved content and the imput query. As shown in Table and Appendix G we have the following observtions:
Obs.4. . RAG excels at retrieving discrete facts for simple questions that do not require complex logics, achieving 83.2% Evidence Recall on the novel dataset (vs. HippoRAG2's best Context Relevance). Medical dataset results confirm this pattern, suggesting relevant evidence for Level questions typically resides single passages. It is because the graph used GraphRAG introduces several logically relevant but redundant information in these scenarios.
Obs.5. GraphRAG's advantages emerge clearly as questions grow more complex, For Level 2-3 questions on the novel dataset, HippoRAG achieves remarkable Evidence Recall (87.9-90 (87.9-90.9% while HippoRAG2 leads in Context Relevance (85.8-87.8%) Medical dataset results reinforce this trend, demonstrating GraphRAG's unique ability to connect information across distant text segments, crucial for multi-hop reasoning and comprehensive summarization.
0bs.6. RAG and GraphRAG show trade-off creative tasks requiring broad knowledge synthesis. Global-Graphie achieves superior Evidence Recall (83.1%), though RAG maintains better Context Relevance (78.8%). While GraphR accesses relevant information overall, its retrieval approach naturally introduces some redundancy compared to RAG's more focused results.
4.3 GRAPH COMPLEXITY (Q3)
During the indexing phase, GraphRAG extracts entities and relations from the corpus to construct a knowledge graph. By indexing the graph structure, GraphR AG establishes logical and semantic
8

---

Published as a conference paper at ICLR 2026
connections between the knowledge graph and the original context, resulting in well-structured and knowledge-comple complete index graph. To reveal the structural characteristics of the index graph and highlight differences introduced by various GraphRAG, we introduce the following metrics: number of nodes, number of edges, average degree, and average clustering coefficient.
Obs.7 The index graphs generated by different GraphRAG implementations demonstrate substantial structural variation.
As illustrated in Figure Hip- poRAG2 produces significantly denser graphs, with node and edge counts that far surpass other frame- works. Specifically, on the novel dataset, HippoRAG2 has an average of 2,310 edges and 523 nodes, while on the medical dataset, it has an average of 3,979 edges and 598 nodes (per 10k corpus tokens). This enhanced graph density improves both information connectivity and coverage, ultimately contributing to superior retrieval and generation capa- bilities. This observation is consistent with the retrieval performance, which shows that HippoRAG2 achieves higher recall than other baselines.
bovel sodes movell-llees
meelical-ondes
medical-edge
311
MGNGBRAG
Heyingg
1igHRAC
Sen bencrroiigg
Micrplitot)
Hapwrs
Haperagh
Figure 5: The relational structure of different methods.
Table 5: The Graph statistics across RAG methods.
Metric
MS-GraphR.G
HippoR AG2
TighRAGG
Fast-GrrphhRAG
HippoRAG
Nevel
Average Degree
Aug. Coaff
Medical Dutiect
Degnee
1822 0,100
1331 0.497
5,00
2.0
Ang. Clas. Coeff
0,199
0.547
00K]
4.4 EFFICIENCY (Q4)
GraphRAG retrieves relevant knowl- edge by traversing the constructed graph. While this approach allows for more structured knowledge organization, can also lead to substantial increase in token cost. To better understand the associated efficiency and cost implications, we conduct a dedicated analysis on prompt statistics across different GraphRAG models.
Obs.8 Compared to vanilla RAG, GraphRAG significantly increases prompt length due to the additional steps involved in knowledge retrieval and graph-based aggregation. Specifi- which in-
Table 6: Ave token cost of different GraphRAG (Part 1).
Asg Tokens
V-RAG
MS-GraphRAGdecal)
MS-GraphhAGgghhhll
HippoRAG2
Novel Medical
879
38707
331375
1008
954
39821
532881
1020
cally, MS- MiS-GraphRAGGglobba corporates coammunit--summarizae mechanism, reaches a prompt size of up to 4 x 10s tokens. LightRAG also produces lengthy prompts (== 104 to- kens). In contrast, HippoRAG2 main- tains a more compact prompt size ( 103 tokens), showing better efficiency. These results highlight
Table 7: Ave token cost of different GraphRAG (Part 2).
Avg Tokens LightRAG
Fast-GraphRAG
RAPTOR
HippoRAG
Novel Medical
100832
4204
3441 3510
7208
100310
4298
7342
that GraphRAG's structured pipeline incurs non-trivial token overhead.
Obs.9. As task complexity and the number of required knowledge points increase, GraphRAG's prompt length exhibits clear upward trend. Notably, ASSGGaapRRAGllloballsss prompt size expands from 7,800 to 40,000 tokens across tasks of increasing difficulty. This excessive token accumulation often introduces redundant information, which in turn degrades context relevance during retrieval. These findings underscore critical trade-off: while GraphRAG improves retrieval breadth, may also introduce noisy context due to prompt inflation, especially in complex tasks.
5 CONCLUSION
Graph-base based Retrieval- Augmeente Generation (GraphRAG) emerges as a pioneering approach that introduces graph structures to explicitly model entity relationships and hierarchical dependencies,

---

Published as a conference paper at ICLR 2026
enabling coherent and effective knowledge retrieval. Despite its conceptual promise, empirical studies report that GraphRAG often fails to outperform vanilla RAG on many NLP tasks, raising questions about its real-world effectiveness. This paper systematically investigates when and why GraphRAG succeeds, offering practical guidelines for its application. Specifically, we first conduct an extensive analysis GraphRAG AG due to the lack of domain-specific corpora and oversimplified task granularity. Based on the findings, we propose comprehensive benchmark designed to evaluate GraphRAG models in terms of hierarchical knowledge retrieval and deep contextual reasoning.
existing benchmark datasets and identify that they inadequately assess
ETHICS STATEMENT
The benchmark introduced in this paper constructed from publicly available internet data. We have taken care to ensure our data collection process respects user privacy by filtering for personally identifiable information and complies with the terms of the source platforms. All models used in our evaluation are open-source, promoting transparency and reproducibility. While we have focused on the technical aspects of reasoning, advise users be mindful of potential societal biases that may exist in the source data and to consider the broader context of their applications.
REPRODUCIBILITY STATEMENT
To ensure the reproducibility of our research, we have made dataset, evaluation code, and detailed experimental settings publicly available. We have open-sourced the original data for our benchmark and all the code required to replicate the results presented in this paper. These resources are available in the repository at //tp::///ithubll com Gr GraphRAG- Beephhnn-BenchhGGe- GaapprraphhGGGGG Benchmark rk Furthermore, we provide detailed description of our dataset construction process in Appendix C The specific hyperparameter settings used for all baseline models evaluated in our experiments are also fully documented in Appendix H.2
ACKNOWLEDGMENTS
This work was supported by the Natural Science Foundation of Fujian Province of China (No. 2024J01 (024) and the Public Technology Service Platform Project of Xiamen (No. 3502 We thank the anonymous reviewers for their insightful comments.
REFERENCES
Anthropic. The claude model family: Opus, sonnet, haiku. Claude-3 Model Card, 2024.
Tu Ao, Yanhua Yu, Yuling Wang. Yang Deng, Zirui Guo, Liang Pang. Pinghui Wang, Tat-Seng Chua, Xiao Zhang, and Zhen Cai. Lightprof: A lightweight reasoning framework for large language model on knowledge graph. In Conference on Artificial Intelligence (AAAI), 2025.
Md Adnan Arefeen, Biplob Debnath, and Srimat Chakradhar, Leancontext: Cost-efficient domain- specific question answering using lims. Natural Language Processing Journal, 2024.
Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. Self-rag: Learning to retrieve, generate, and critique through self-reflection. In International Conference on Learning Representations (ICLR), 2023.
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean- -aptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. Improving language models by retrieving from trillions of tokens. In International Conference Machine Learning (ICML), 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. In Advances in Neural Information Processing Systems (NeurIPS). 2020.
10

---

Published as a conference paper at ICLR 2026
Boyu Chen, Zirui Guo, Zidan Yang. Yuluo Chen, Junze Chen, Zhenghao Liu, Chuan Shi, and Cheng Yang. Pathrag: Pruning graph-base based retrieval augmented generation with relational paths. arXiv preprint arXiv:2502.14902, 2025a.
Rubing Chen, Xulu Zhang, Jiaxin Wu, Wenqi Fan, Xiao-Yong Yong Wei, and Qing Li. Multi-level querying using knowledge pyramid. arXiv preprini arXiv:2407.21227, 2024a.
Shengyuan Chen, Qinggang Zhang, Junnan Dong, Wen Hua, Qing Li, and Xiao Huang. Entity alignment with noisy annotations from large language models. arXiv preprint arXiv:2405. M4kiv:240..16000 16806, 2024b.
Shengyuan Chen, Chuang Zhou, Zheng Yuan, Qinggang Zhang, Zeyang Cui, Hao Chen, Yilin Xiao, Jiannong Cao, and Xiao Huang. You don't 't need pre-built graphs for rag: Retrieval augmented generation with adaptive reasoning structures. arXiv preprint arXiv:2508.0610 2025b.
Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung. Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. The Journal of Machine Learning Research (JMLR), 2023.
CircleMind-Al.
Streamlined and promptable fast graphrag framework designed for interpretable, high-precision, agent-driven retrieval workflows, 2024. URL https://github/ com/
Jonathan Larson Darren Edge, Ha Trinh. Lazygraphrag: Setting a new standard for quality and cost. Microsoft Blog, 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local global: A graph rag approach query- guery-focused summarization. arXiv preprint arXiv:2404. 36130, 2024.
Junfeng Fang, Houcheng Jiang, Kun Wang, Yunshan Ma, Shi Jie, Xiang Wang, Xiangnan He, and Tat-Seng Chua. Alphaedit: Null-space constrained knowledge editing for language models. JCLR, 2025.
Yujie Feng, Liming Zhan, Zexin Lu, Yongxin Xu, Xu Chu, Yasha Wang, Jiannong Cao, Philip S Yu, and Xiao-Ming Wu. Geoedit: Geometric knowledge editing for large language models, arXiv preprint arXiv:2502. 19953, 2025.
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. aarrivaa-auummeneed generation for large language models: survey. arXiv preprint arXiv:2312. 10997, 2023.
Michael Glass, Gactano Rossiello, Md Faisal Mahbub Chowdhury, Ankita Rajaram Naik, Pengshan Cai, and Alfio Gliozzo. Re2g: Retrieve, rerank, generate. arXiv preprint arXiv:2207.06300 2022.
Kai Guo, Harry Shomer, Shenglai Zeng, Haoyu Han, Yu Wang, and Jiliang Tang. Empowering graphrag with knowledge filtering and integration. arXiv preprint arXiv:2503. 10003///000 /3804, 2025.
Zirui Guo, Lianghao Xia, Yanhua Yu, Tu Ao, and Chao Huang. Lightrag: Simple and fast retrieval- augmented generation. arXiv preprint 24X 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Yu Gu, Michihiro Yasunaga, and Yu Su. Hipporag: Neu- robiologically inspired long-term memory for large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024.
Bernal Jiménez Gutiérrez, Yiheng Shu, Weijian Qi, Sizhe Zhou, and Yu Su. From rag to memory: -on-marametric continual learning for large language models. arXiv preprint arXiv:2502.14802. 2025.
Haoyu Han, Yu Wang, Harry Shomer. Kai Guo, Jiayuan Ding, Yongjia Lei, Mahantesh Halappanavar, Ryan A Rossi, Subhabrata Mukherjee. Xianfeng Tang, et al. Retrieval- generation with graphs (graphrag). arXiv preprint arXiv:2501.003000 2024.

---

Published as a conference paper at ICLR 2026
Haoyu Han, Harry Shomer, Yu Wang, Yongjia Lei, Kai Guo, Zhigang Hua, Bo Long, Hui Liu, and Jiliang Tang. Rag vs. graphrag: A systematic evaluation and key insights. arXiv preprint 1rXii:250221137,,, 2025.
Yancheng He, Shilong Li, Jiaheng Liu, Yingshui Tan, Weixun Wang, Hui Huang, Xingyuan Bu, Hangyu Guo, Chengwei Hu, Boren Zheng, et al. Chinese simpleqa: A chinese factuality evaluation for large language models. arXiv preprint urXiv:2411.071444 2024.
Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next-generation database interfaces: A survey of Ilm-based text-to-sql. arXiv preprint wrXv:24406008266, 2024.
Zijin Hong, Hao Wu, Su Dong, Junnan Dong. Yilin Xiao, Yujing Zhang, Zhu Wang, Feiran Huang, Linyi Li, Hongxia Yang, et Benchmarking large language models via random variables. arXiv preprint wrXiv:2501.17790 2025.
Yuntong Hu, Zhihan Lei, Zheng Zhang. Bo Pan, Chen Ling, and Liang Zhao. Grag: Graph retrieval- augmented generation. arXiv preprint arXiv:2405. 16506,, 2024.
Yuntong Hu, Zhihan Lei, Zhongjie Dai, Allen Zhang, Abhinav Angirekula, Zheng Zhang, and Liang Zhao. Cg-rag: Research question answering by citation graph serieval--aumenner llms. arXiv preprint arXiv:2501 15067, 2025.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng, Bing Qin, et al. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. ACM Transactions on Information Systems (TOIS), 2023.
Yigian Huang, Shiqi Zhang, and Xiaokui Xiao. Ket-rag: A cost-efficient multi-granular indexing framework for graph-rag. arXiv preprint arXiv:2502.099300 2025.
Gautier Izacard, Patrick Lewis, Maria Lomeli, Lucas Hosseini, Fabio Petroni, Timo Schick, Jane Dwivedi- Yu, Armand Joulin, Sebastian Riedel, and Edouard Grave Atlas: Few-shot learning with retrieval augmented language models. The Journal of Machine Learning Research (JMLR), 2023.
Houcheng Jiang, Junfeng Fang, Ningyu Zhang, Guojun Ma, Mingyang Wan, Xiang Wang. Xiangnan He, and Tat-seng Chua. Anyedit: Edit any knowledge encoded language models. ICMI,, 2025.
Wenqi Jiang, Shuai Zhang, Boran Han, Jie Wang, Bernie Wang, and Tim Kraska. Piperag: Fast retrieval-augmennedd generation via algorithm-system co-design. In Conference on Knowledge Discovery and Data Mining (KDD), 2024.
Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhiqing Sun, Qian Liu, Jane Dwivedi- Yu, Yiming Yang, Jamie Callan, and Graham Neubig. Active retrieval augmented generation. Empirical Methods in Natural Language Processing (EMNLP), 2023.
Daniel Khashabi, Sewon Min, Tushar Khot, Ashish Sabharwal, Oyvind Tafjord, Peter Clark, and Hannaneh Hajishirzi. Unifiedqa: Crossing format boundaries with single qa system. In Findings of Empirical Methods in Natural Language Processing (EMNLP). 2020.
Angeliki Lazaridou, Elena Gribovskaya, Wojciech Stokowiee, and Nikolai Grigorev. Internet- augmented language models through few-shot prompting for open-domain question answering. arXiv preprint arXiv:2003.05115, 2022.
Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, Mike Lewis, Wen-tau Yih, Tim Rocktäschel, al. Retrieval-augmented genera- tion for knowledge-intene inttgg--ntenivee nlp tasks. In Advances in Neural Information Processing Systems (NeurIPS), 2020.
Zhuoqun Li, Xuanang Chen, Haiyang Yu, Hongyu Lin, Yaojie Lu, Qiaoyu Tang, Fei Huang, Xianpei Han, Le Sun, and Yongbin Li. Structrag: Boosting knowledge intensive reasoning of llms via inference- time hybrid information structurization. In International Conference on Learning Representations (ICIR), 2024.
12

---

Published as a conference paper at ICLR 2026
Lei Liang, Mengshu Sun, Zhengke Gui, Zhongshu Zhu, Zhouyu Jiang, Ling Zhong, Yuan Qu Peilong Zhao, Zhongpu Bo. Jin Yang, et al. Kag: Boosting Ilms in professional domains via knowledge augmented generation. arXiv preprint arXtv:2409.13731 2024.
Chin Yew Lin. Rouge: A package for automatic evaluation of summaries. Text summarization branches out, 2004.
Hongyin Luo, Yung-Sung Chuang, Yuan Gong, Tianhua Zhang, Yoon Kim, Xixin Wu, Danny Fox, Helen Meng, and James Glass. Sail: Search-augmented instruction learning. In Findings of Empirical Methods in Natural Language Processing (EMNLP), 2023.
Renqiang Luo, Huafei Huang. Shuo Yu, Xiuzhen Zhang, and Feng Xia. :airGT::a fairness-aware graph transformer. In Proceedings of the 32rd International Joint Conference on Artificial Intelligence, pp. 449-457 457, 2024.
Renqiang Luo, Huafei Huang, Ivan Lee, Chengpei Xu, Jianzhong Qi, and Feng Xia. FairGP: A scalable and fair graph transformer using graph partitioning. In Proceedings of the 39th Annual AAAI Conference on Artificial Intelligence, pp. 12319-12327, 2025.
Shengjie Ma, Chengjin Xu, Xuhui Jiang. Muzhi Li, Huaren Qu. and Jian Guo. Think-on-gran -onkkapppo-phh--- 2.0: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval. In International Conference on Learning Representations (ICLR), 2024.
Yuning Mao, Pengcheng He, Xiaodong Liu, Yelong Shen, Jianfeng Gao, Jiawei Han, and Weizhu Chen. inneeration-augmentte retrieval for open-domain question answering. In Association for Computational Linguistics (ACL), 2020.
OpenAl. Gpt-4 technical report. OpenAl Blog, 2023.
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph retrieval-augmmented generation: A survey. arXiv preprint arXiv:24088. 08921, 2024.
Tyler Thomas Procko and Omar Ochoa. Graph retrieval- -uggented generation for large language models: A survey. Conference on AI, Science, Engineering, and Technology (AISEET), 2024.
Hongjin Qian, Peitian Zhang, Zheng Liu, Kelong Mao, and Zhicheng Dou. Memorag: Moving towards next-gen rag via memory-inspee inspired knowledge discovery. arXiv preprint 2xXvv:240900559/ 2024.
Parth Sarthi, Salman Abdullah, Aditi Tuli, Shubh Khanna, Anna Goldie, and Christopher Manning. Raptor: Recursive abstractive processing for tree-organized retrieval. In International Conference on Learning Representations (ICLR), 2024.
Chen Shengyuan, Yunfeng Cai, Huang Fang, Xiao Huang, and Mingming Sun. Differentiable neuro- symbolic reasoning on large-scale knowledge graphs. Advances in Neural Information Processing Systems, 36, 2024.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-YYung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In International Conference on Learning Representations (ICLR), 2024.
Yingshui Tan, Boren Zheng, Baihui Zheng, Kerui Cao, Huiyun Jing, Jincheng Wei, Jiaheng Liu, Yancheng He, Wenbo Su, Xiangyong Zhu, et al. Chinese safetyqa: A safety short-form factuality benchmark for large language models. arXiv preprint arXiv:2412.1522 2024.
Xiaqiang Tang. Qiang Gao, Jian Li, Nan Du, Qi Li, and Sihong Xie. proach for adaptive aeuienel-augmeetted generation through question complexity. arXiv preprint arXiv:2412.01572 2024.
Mba-rag: bandit ap-
Yixuan Tang and Yi Yang. Multihop-rag: Benchmarking retrieval augmented generation for multi-hop queries. In Conference Language Modeling (COLMM), 2024.
13

---

Published as a conference paper at ICLR 2026
Jinyu Wang. Jingjing Fu, Rui Wang, Lei Song, and Jiang Bian. Pike-rag: specialized knowledge and rationale augmented generation. arXiv preprint arXiv:2501.11551,250 2025a.
Shu Wang, Yixiang Fang, Yingli Zhou, Xilin Liu, and Yuchi Ma. Archrag: Attributed community- based hierarchical retrieval- -uugmettd generation, 2025b.
abs/
2502. 09891
Tianshu Wang, Xiaoyang Chen, Hongyu Lin, Xianpei Han, Le Sun, Hao Wang, and Zhenyu Zeng. Dbcopilot: Natural language querying over massive databases via schema routing. 2025c.
Xiaohua Wang, Zhenghua Wang, Xuan Gao, Feiran Zhang, Yixin Wu, Zhibo Xu, Tianyuan Shi, Zhengyuan Wang, Shizheng Li, Qi Qian, al. Searching for best practices in retrieval- retrieval-augmennte augmented generation. Empirical Methods Natural Language Processing (EMNLP), , 2024a.
Yu Wang. Nedim Lipka, Ryan A Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. Knowledge graph prompting for multi-document question answering, In Conference on Artificial Intelligence (AAAI), 2024b.
Feng Xia, Ciyuan Peng, Jing Ren, Falih Gozi Febrinanto, Renqiang Luo, Vidya Saikrishna, Shuo Yu, and Xiangjie Kong. Graph learning. Foundations and Trends in Signal Processing, 19(4):311-551 2025. ISSN 1932-8346. doi: 10.1561/2000000113. URL httt:// dx www. wrw.oo...rg/1.111500 2000000137
Yilin Xiao, Chuang Zhou, Yujing Zhang, Qinggang Zhang, Su Dong, Shengyuan Chen, Chang Yang, and Xiao Huang. Lag: Logie-augmented generation from cartesian perspective. arXiv preprint arXiv:2508.05500 055009 2025.
Yilin Xiao, Chuang Zhou, Qinggang Zhang, Bo Li, Qing Li, and Xiao Huang. Reliable reasoning path: Distilling effective guidance for I1m reasoning with knowledge graphs IEEE Transactions Knowledge and Data Engineering. 2026.
Fangyuan Xu, Weijia Shi, and Eunsol Choi. Recomp: Improving retrieval-augmented Ims with
compression and selective augmentation. In International Conference Learning Representations (ICLR), 2023.
Chang Yang, Chuang Zhou, Yilin Xiao, Su Dong, Luyao Zhuang, Yujing Zhang, Zhu Wang, Zijin Hong, Zheng Yuan, Zhishang Xiang, et al. Graph-based agent memory: Taxonomy, techniques, and applications. arXiv preprint ar a6Xiv:260..05665, 2026.
Zhilin Yang, Peng Qi. Saizheng Zhang, Yoshua Bengio, William W Cohen, Ruslan Salakhutdinov, and Christopher D Manning. Hotpotqa: A dataset for diverse, explainable multi-hop question answering. In Empirical Methods in Natural Language Processing (EMNLP), 2018.
Yongjing Yin, Fandong Meng. Jinsong Su, Chulun Zhou, Zhengyuan Yang, Jie Zhou, and Jiebo Luo.
A novel graph- graph-based based multi-modal fusion encoder for neural machine translation. Proceedings of the 58th annual meeting of the association for computational linguisties, pp. 3025-3033 2020.
Yongjing Yin, Jiali Zeng, Jinsong Su, Chulun Zhou, Fandong Meng, Jie Zhou, Degen Huang, and Jiebo Luo. Multi-modal graph contrastive encoding for neural machine translation. Artificial Intelligence, 323:103986, 103986, 2023.
Hao Yue, Shaopeng Lai, Chengyi Yang, Liang Zhang, Junfeng Yao, and Jinsong Su. Towards better graph-base based cross-document relation extraction via mon-bridge entity enhancement and prediction debiasing. In Findings of the Association for Computational Linguistics: ACL 2024, pp. 680-691, 2024.
Qinggang Zhang, Jumnan Dong, Hao Chen, Wentao Li, Feiran Huang, and Xiao Huang. Structure guided large language model for sql generation. arXiv preprint arXiv:2402.1 13284, 2024a.
Qinggang Zhang, Jumnan Dong, Hao Chen, Daochen Zha, Zailiang Yu, and Xiao Huang. Knowgpt: Knowledge graph based prompting for large language models. In Advances in Neural Information Processing Systems (NeurIPS), 2024b.
14

---

Published as a conference paper at ICLR 2026
Qinggang Zhang, Shengyuan Chen, Yuanchen Bei, Zheng Yuan, Huachi Zhou, Zijin Hong, Junnan Dong, Hao Chen, Yi Chang, and Xiao Huang. A survey of graph retrieval- -aageeeed- generation for customized large language models. arXiv preprint arXIv:2501.13958, 2025a.
Qinggang Zhang. Zhishang Xiang, Yilin Xiao, Le Wang, Junhui Li, Xinrun Wang, and Jinsong Su. Faithfulrag: Fact-level conflict modeling for context- faithfu retrieval-aus aerioval-uugmeneed generation. arXiv preprint arXiv:2506.0893 2025b.
Xuejiao Zhao, Siyan Liu, Su- Yin Yang. and Chunyan Miao. Medrag: Enhancing retrieval- augmented generation with knowledge graph-elicited reasoning for healthcare copilot. In International World Wide Web Conference (WWW), 2025.
Yingli Zhou, Yaodong Su, Youran Sun, Shu Wang, Taotao Wang, Runyuan He, Yongwei Zhang, Sicong Liang, Xilin Liu, Yuchi Ma, al. In-depth analysis of graph- rag in unified framework, arXiv preprint arXiv:2503.04338, 2025.
Appendix
CONTENTS
Introduction
Preliminary Study
2.1 RAG vs. GraphRAG
Current RAG Benchmarks
3
Beach
Task Formulation
Dataset Construction
Evaluation Metrics
4
Experiment
Generation Accuracy (Q1)
4.2
Retrieval Performance (Q2)
8
Graph Complexity (Q3)
4.4
Efficiency (Q4)
5
Conclusion
Frequently Asked Questions (FAQs)
17
A.I
Code and Leaderboard
17
A.2
Why important to include tasks with varying complexity
17
A.3
How to control the complexity of each task?
18
A.4
Why construct two datasets from both novels and the medical corpus?
18
A.5
Why Not Use Other Formats for Question Construction?
18
A.6
What is the advantage of using stage-specific evaluation metrics?
18
A.7
Comparison with existing benchmarks and analysis papers.
19
15

---

Published as a conference paper at ICLR 2026
B Takeaway Findings
19
Benchmark Construction
20
Corpus Collection
20
Logic Mining
20
Evidence Collection
21
Question Generation
Check & Correct
21
C.6 Refinemen
23
D
Extending GraphRAGG-ench to New Domains
23
Dataset Statisties and Visualization
24
Evaluation Metrics
25
Additional Experiments
26
Experiments on more GraphRAG frameworks
26
G.2
Experiments on open-source model
26
G.3
Experiments on graph construction efficiency
26
G.4
Experiments the impact of LLM size
27
G.5
Experiments on Scalability with Corpus Size
27
Implementation Details
28
Implementation Details of Representative RAG Models
28
Configuration of GraphRAG Models
29
Related Work
31
Traditional RAG and their Limitations
1.2
GraphRAG and its Advantages
31
Existing Benchmarks and Analysis
33
Limitation
33
Broader Impact
34
The Use Large Language Models (LLMs)
34
16

---

Published as a conference paper at ICLR 2026
A FREQUENTLY ASKED QUESTIONS (FAQS)
A.1 CODE AND LEADERBOARD
To promote transparency and reproducibility, we have uploaded our code to GitHub at httpa Benchmarrk This repository includes the 7/gi thub. wwt..oooomraaeeee Cww.hhaa--ennchhaaeeaa SeC---eeechhcraplRR--Bnnnnneee
source code and scripts for evaluation, ensuring that researchers have full access to the resources required to reproduce and extend our work. In addition, will continue maintain and update the repository to reflect future improvements. Besides that, the leaderboard visualization is provided in Figure We have also updated the related resources the leaderboard.
Leaderboard
GruphRAG-Bench (Novel)
Comples Krastming
Sammanze
Model;
Bank?
Average:
Date]
ACC:
ROUGEA!
ACC:
ROUGE4L
ACC:
Corl
ACC|
ES:
Cavt
HippoRA02
56.45
2025-62
60.14
31.35
53388
33.42
641
70.34
48.25
1981
30.95
Past Past-Lirapitt.
32.02
2024-10
20.35
15.9
48.55
21.12
56.41
80.52
46.18
17.19
36.99
MS Mi.Ciii
(Bocal)
50.93
2034-04
49.29
36.11
5091
2600
644
75.58
39.1
15.4
35.65
Leny-iimphhtti
30.59
2024-09
51.35
16.97
29
25.48
$8.99
70.94
43.25
30.69
39.74
StusRAG
49.13
2024-10
53.34
16.73
46.2
23.49
5425
63.56
42.16
$2.61
36.75
RAG tenark)
48.35
00.32
16.08
42.91
15.39
$1.3
83.64
38.26
49.1
40.04
KGP
48.01
2013-08
14.15
34.73
46.3
16.91
$1.21
64.3
40.37
52.55
34.65
RAG rele reank)
47.93
$8.76
17.35
41.35
15.12
50.03
K2.53
41.52
47.46
37.84
KKERAG
47.67
2005-02
55.00 39
17.59
16.99
25.9K
57.47
69.2
46
16.73
31.68
LighRAGG
45.09
2024-10
58,02
48.55
63.05
25.01
HippottA
44.75
2024-05
20.00
85.35
35.55
38.97
MS-GGnpRAGG
44.52
2024-04
17.32
43.17
36.57
80.35
41.11
15.15
30.34
RAPTOR
13.74
K2.33
36.01
GraphR G-Bench (Medical)
Retrinal
Comples Reaseming
Contectual Summarias
Creative Goneration
Model;
Rank]
Average:
Date]
Link
ROUGE44;
ROUGE4L;
ACC1
Cor]
Carl
HippoRtAGi
64.35
2025-02
36.69
61.88
36.97
63.05
46.13
68.05
18.78
51.54
64.12
2024-10
11.04
67.55
52,07
65.33
16.00
44.73
LightLAG
62.59
37.19
26.95
63.11
51.16
67.91
51.58
remak)
78.54
remak)
61.00
29.21
5760
13.98
63722
77.34
58.94
35388
57.87
HiyooRAri
59.08
2034-05
20.95
99.5
62.73
64.43
63.56
StuceRAG
59.56
62.45
65.56
56.33
2023-08
54.51
63.77
45277
29.04
43.04
(ooal))
GrapBRGG
47.04
41.87
22.98
53.11
32.65
(global)
GrapBRAG
20.51
Figure 6: Overview of the leaderboard, ranked by average generation performance (ACC).
A.2 WHY IS IT IMPORTANT INCLUDE TASKS WITH VARYING COMPLEXITY?
Task complexity is critical for assessing GraphRAG models because their core value lies in navigating interconnected knowledge structures and synthesizing latent logical relationships. Real-world chal- lenges demand more than locating scattered facts since they require integrating hierarchical domain expertise with ambiguous, context-dependent narratives to form coherent insights. By evaluating models tasks of varying complexity, from factual retrieval to creative generation. expose whether they truly leverage graph structures to reason like humans: inferring causality, resolving conflicting contexts, and extrapolating insights beyond explicit data. Without measuring how models
17

---

Published as a conference paper at ICLR 2026
handle complexity, evaluations risk overestimating their utility for real applications, where success hinges on synthesizing interconnected concepts, not just retrieving them. Complexity assess- ment ensures GraphRAG systems validated their ability to map. traverse, and reason through domain-specific ontologies, the very features that distinguish them from traditional RAG frameworks.
A.3 HOW TO CONTROL. THE COMPLEXITY OF EACH TASK?
GraphRAG-Ben -aneecchhhhh leverages structural information to control complexity. We begin by extracting information from corpora to build ontologies or knowledge graphs. Taking the Novel dataset as example, we first construct a knowledge graph from the corpus, then form logic chains using relevant triples from this graph. Then, we control difficulty by adjusting the number of involved triples and the information span they cover. Specifically, we define question complexity through two dimensions: Knowledge Breadth (measured by the count of triples required answer question) and Reasoning Depth (measured by the number of inference hops between these triples). The relevant statistical results are shown in Table g
Table 8: Problem Complexity Statistics of GraphRAG- AeGBh....
Problem Complexity
Fact Retrieval
Complex Reasoning
Contextual Summarize
Creative Generation
Novel Dataset
Knowledge Breadth
1.40
2.60
3.51
7.11
Reasoning Depth
1.69
6.25
4.64
7.81
Medical Dotaset
Knowledge Breadth
1.25
3,45
5.1
10.14
Reasoning Depth
1.82
5,23
4.27
8.27
A.4
WHY CONSTRUCT TWO DATASETS FROM BOTH NOVELS AND THE MEDICAL CORPUS?
Including two distinct datasets, medical guidelines and unstructured novels, is essential to evaluate GraphRAG models under conditions that mirror wal-world knowledge ecosystems. Medical cor- pora provide explicit, hierarchical relationships, testing model's ability navigate rigid domain logic and standardized protocols. Conversely, Novel corpora introduce implicit, context-dependent dependencies, like socio-historical factors shaping character decisions, challenging models infer latent connections without predefined rules. This approach ensures the benchmark assesses both precision following formal hierarchies and adaptability interpreting ambiguous, open-ended contexts, critical for applications where models must integrate structured expertise with unstructured, real-world narratives.
A.5 WHYY NoT USE OTHER FORMATS FOR QUESTION CONSTRUCTION?
In this paper, we aim to figure out in which scenarios do graph structures provide measurable benefits for RAG systems. In other words, we care more about the retrieval and reasoning complexity of task, instead of the task type or question format. It is because the specific task type or format (like QA, multiple-choice, fact-checking) doesn't 't really change: 1) reasoning difficulty: the reasoning steps the model needs to take, 2) retrieval difficulty: how to locate scattered information from the corpus.
Given question of "What famous universities are in San Diego?", Whether this is asked as: An open QA question, or multiple-choice question (e.g., "Is UCSD in San Diego? A) Yes B) No"), or A fact-checking task (e.g., "Check this: UCSD has campus in San Diego")... doesn't 't change the core need: The RAG system must still find information about San Diego and reason to answer correctly.
A.6 WHAT IS THE ADVANTAGE OF USING
EVALUATION METRICS?
Stage-specific evaluation metrics are crucial because they provide granular insights into how well a GraphRAG model performs at each phase of its pipeline, rather than relying solely end-to-end output metrics like answer accuracy. Traditional benchmarks often treat the entire process as a "black box", obscuring whether failures stem from flawed knowledge graph construction, suboptimal
18

---

Published as a conference paper at ICLR 2026
WHEN TO USE GRAPH IN RAG
Single-step problems only require single retrieval
How to build an effective GraphRAG?
x
Scenarios with constraints reasoning speed and context length
Prioritize precise retrieval
Corpus knowledge is well-structured, with numerous entities and tightly connected relationships
Maximiac key information
while minimizing codundancy
Build quality graphs, not just large ones
Enable efficient multihop knowlodge traversal while minimizing codundant relationships
Multi-hop reasoning problems that require clear logical relationships
Actively manage context growth
Summarization problems that require knowledge integration capabilities
Establish search boundaries prevent contect explosion roduce reasoning costs
Figure 7: An overview of building an effective GraphRAG system. It consists of two key parts: (i) the crucial principles for system design (left), and (ii) the most suitable application scenarios (right).
retrieval, or weak reasoning. By designing metrics tailored to individual stages, such as graph completeness during logic mining, retrieval relevance in evidence collection, contextual coherence in question generation, we isolate and diagnose weaknesses in specific components.
A.7 COMPARISON WITH EXISTING BENCHMARKS AND ANALYSIS PAPERS.
Existing studies (Han 2025 Zhou et al., 2025) focus on architectural comparisons using homogeneous datasets, missing how models synthesize hierarchical expertise and unstructured narratives. To this end, we propose GraphRAG-Benchh a comprehensive benchmark designed to evaluate GraphRAG models on deep reasoning. It features hybrid corpora (medical guidelines + novels) with tasks increasing complexity and stage-specific metries to expose why models fail, whether in graph construction, knowledge retrieval, or contextual synthesis. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG systems and the underlying reasons for its success, offering guidelines for its practical application.
B TAKEAWAY FINDINGS
In this paper, we not only build GraphRAG-Bende to evaluate existing GraphRAG systems, but more importantly, we provide insightful recommendations for future GraphRAG research, as illustrated in Figure
PRIORITIZE PRECISE RETRIEVAL: Effective frameworks should focus on how maximize key information retrieval while minimizing redundancy. It's critical to pinpoint the key facts needed to answer question, while at the same time avoiding pulling in unnecessary details. This keeps the context clean and focused, which helps the model reason better and improves overall efficiency.
BUILD QUALITY GRAPHS, NOT JUST LARGE ONES: While GraphR AG forms knowledge graphs (entities/rclaaionnsipp) for efficient searching, more relationships * better performance. Optimal graphs require tightly connected communitics, which create denser structures rich in implicit multi- hop knowledge enabling faster graph traversal.
ACTIVELY MANAGE CONTEXT GROWTH:
Unlike traditional RAG (fixed context via vector search), GraphRAG retrieves entities, relationships, and raw text snippets, risking sudden context explosion and hig reasoning costs. Future solutions need search boundaries to curb context growth and significantly lower costs.
19

---

Published as a conference paper at ICLR 2026
Table 9: Results of Generation Evaluation using GPT-4o-mini, covering tasks of varying complexity.
Category
Model
Fact Retrieval ROUGE-L
Complex Reasoning ROUGE-L
ACC
Contextual Summarize
ACC
Creative Generation
Busic R.GG
RAG (wf remmk)
XAG (w perank)
58.
00.92
36.08
37.35
42.93
15.39
15.12
51:30
K2.53
83.64
38.26
41,52
49.21
46
20.04
37.84
MS- G5GGGagg (local)
iodee et 2025
49.29
26.11
50.93
24.09
54.40
75.
29.10
55.44
35.66
Graph RAG
MS-GrsphRRA AG (eloha))
HippuRAAG Quiddr
52.95
16.92
26.65
17:32
43.17
38.52
11.16
15.12
68.70
56.87
85.55
80.55
41.11
38.85
71.53
75.15
38,97
50.34
HlipoRAG2
70.84
LighsRMG
Fast GrapRRAG
Lavv Cav mi A321
56.95
35.90
48.55
21.12
56.41
80.82
16.18
57.19
36.99
RAPTOR Stgtire 3i 2i14
49.25
23.74
$8.59
166
47.10
K2.33
38.01
70.85
35.88
Lary
Gal--GnghRRa Xi (2iree Poog 0024
$1.65
36.97
49.22
23. 48
58.29
76.94
43.25
$0.69
39.74
KGP Wary 202-00
54.15
34.73
46.31
16.91
51.21
64. 34
40.37
52.55
34.65
53.84
26.73
66.27
23,
51.28
63.56
12.16
52.68
36.75
KET-RAG [Tang
Dananed
Busic RAG
RAG (wl rmmmk)
RAG (wl rerank)
63.72
64.73
30.75
29.21
58.64
15:57
13.98
63.72
65.75
78.54
60.61
38,
36.74
35 88
58.72
57.87
MS CraphRAG (local)
lodes ct 2025
38.63
26,80
$7.04
21.99
41,87
22.98
53.11
32,65
39.42
GraphRAG
MS-GasphRRMG
HlpppRAAG
Quildn
(olabal)
Panre mmo
56.14
16.42
20.95
46:00
55.87
15.61
$2.75
13.57
59.36
19.83
62.73
64.43
20.81
69.2]
65.56
13.64
HippoR
LighsRAG
37.19
24.98
78.76
Fast GrapRRAG
A221
60.93
31.05
61.73
21.37
57.88
$2.07
65.93
$6.07
44.73
RAPTOR Statir 3i 20143
54.07
17.93
53:20
11.73
58.73
78.25
62.38
59.88
63.43
Layg-inghhR.a R (Paren Fring 204
60.25
31.66
47.83
22.68
57.28
$5.02
62.22
30.95
43.79
KGP Warg
202-40
$2.34
21
51.53
11.09
$4.51
62.40
63.77
45.25
35.55
55.38
27.53
56.17
22.79
62,48
65.66
60.21
42.35
45.76
[Hang
C BENCHMARK CONSTRUCTION
To address the critical limitations of existing RAG evaluation frameworks, we present a novel bench- mark to assess GraphRAG systems through comprehensive task hierarchies and structured knowledge integration. Specifically, our benchmark is constructed through six stages that systematically inte- grate domain-specific logical hierarchies and contextual dependencies to enable precise control over question difficulty.
C.1 CORPUS COLLECTION
Existing benchmarks often rely on corpora with inconsistent quality and inadequate information density, partieularly in their inability to represent both loosely organized real-world knowledge and tightly structured domain-specific hierarchies. In contrast, our benchmark addresses this by constructing corpus that integrates two complementary datasets: (i) Medical Dataset. We integrate domain data from the National Comprehensive Cancer Network (NCCN) clinical guidelines, which provide standardized treatment protocols, drug interaction hierarchies, and diagnostic criteria. (ii) Novel Dataset. We curated a collection of pre-20th-century novels (narrative fictions) from the Project Gutenberg library, prioritizing lesser-known works to minimize overlap with pretraining data of LLMs. These texts were selected based on their length and narrative ambiguity, ensuring they simulate real-world documents with non-ling non-linear, inferential dependencies. These two complementary datasets ensure the corpus balances unstructured, real-world ambiguity with Nomain-specifis hierarchies, enabling rigorous evaluation of both retrieval robustness and reasoning depth.
C.2 LOGIC MINING
Raw text alone lacks explicit representations of the latent relationships that define real-world reason- ing, such as causality, hierarchy, or contradiction. Existing benchmarks often treat these relationships as implicit, leading to superficial evaluations of "multi-hop" queries as mere fact aggregation. To address this, we transform text into formalized ontologies using GPT-4.1, which codifies vertical hierarchies (e.g., sympiom + diagnosis) and horizontal dependencies (e.g., socio-economic factors influencing medical outcomes). This ontology acts as ground-truie map of domain logic, enabling precise identification of what constitutes reasoning step and how concepts interrelate. By making these relationships explicit, we establish a measurable foundation for distinguishing factual retrieval from genuine logical synthesis, prerequisite for controlling question difficulty.
20

---

Published as a conference paper at ICLR 2026
Table 10: Results of Retrieval performance using GPT-4o-mini, covering tasks of varying complexity.
Category
Model
Fact Retrieval Relevance
Complex Reasoming Relevance
Recall
Contextual Summarize Relesance
Creative Generation Relevance
Detour 74.66
Rasic RAG
RAG (wre rerank)
R-A (wl serank)
61.37
83.21
77:37
64.47
59.30
$2.05
80.82
09:08
73.38
80.05
83.10
32.4 48
39.59
78.73
$2.84
MS MRS-GGrahhRAGG (leaa)
Filee et X074
04
27. 30
73.03
39.09
$2.02
43.1
53.55
35.07
MS Graphi&AG relobal
62.27
09.37
86.68
14.36
59.68
15.35
$3.14
19.40
HippokAG2
19:25
42.18
79:10
LightRAG Lin 2i pile
73.69
33.08
83.52
37,46
$7.59
38.02
71.72
38.06
Fust-GrayeRAch CanpeRACCCiie (iRME 2021
61.48
47.86
73.51
55.21
78.58
49.74
56.31
46.27
RAPTOR (Saan 2072
02.14
54.08
67
61.26
75.79
633000
58.66
58.46
Graphi RAG
2026
59.25
30.76
57.73
42.98
77.38
43.62
55.24
31.94
KGP Waan 1034
23.71
64.30
67.57
KET-RAG Hibnggeei p025
63.55
39.11
56 93.93
32.59
67.35
39.05
53.40
36.74
Medical Daieret
Basic RAG
RAG (w/e rerank)
RAG (w rerank)
57.83
86.24
64.73
63.71
$4.97
$6.19
$4.11
85.55
$4.14
85.87
91.35
89.94
44.88
45.23
60.50
58.73
MS-CiraphR (Incal)
05.67
66.59
Graph
M55-GraphRRMG Colobal
HippoRAG Gullim vi
K7.25
52:44
33.80
42.19
81.66
02.73
45.03
HippaR
78.70
87.96
77.00
80.94
77.48
86.85
61.12
78.64
LightRAG
min Milk
$0.32
41.27
$2.91
42.79
85.71
43.11
81.34
45.17
Fust- Dol-GropRRG
2024
66.82
45.86
74
38,80
77.27
47.58
62.
25.15
RAPTOR
Lazy Graphh DISQUESiS
KGP Wang Cubli Pursid
57.51
27.34
$3.51
26.50
59.38
56.20
43.85
StruskRG Lioean 2002
63.25
37.26
61.75
35.88
62.55
32.01
62.76
46.15
KET RAG
[[bing crain 2022
96.44
57.00
90.62
30.86
39.02
44.59
44.06
32.38
C.3 EVIDENCE COLLECTION
Real-world reasoning complexity arises not only from the number of "hops" but from the structural and semantic properties of the knowledge being traversed. Isolating evidence into localized subgraphs (dense concept clusters) and multi-hop chains (logically linked sequences) allows to quantify difficulty through objective metrics like entity density, path length, and inferential distance. For instance, subgraph with high entity density tests model's ability to filter relevant facts within noisy context, while a long-chain dependency tests its capacity to maintain coherence across logical steps. This stage ensures that "difficulty" is not arbitrarily defined but rooted in the ontology's verifiable properties, aligning question design with the cogmitive demands of real analytical tasks. Crucially, this evidence extraction phase distinguishes retrieval questions are anchored contextually rich subgraphs, while complex reasoning tasks demand traversal of interconnected chains that reflect real-world problem-solving, such diagnosing a patient by integrating symptoms, lab results, and comorbidities.
approach by ensuring that even simple
C.4 QUESTION GENERATION
Questions are generated by aligning their cognitive demands with the structural properties of the underlying evidence. Retrieval-focused questions target isolated subgraphs, requiring models to recall clustered facts. Reasoning questions leverage short-range chains, demanding interpretation of relational predicates (e.g., causality or contraindication). Summarization tasks synthesize disjointed subgraphs into narratives, while creation questions extrapolate hypotheses from global graph topology (e.g., predicting policy impacts by traversing regulatory, economic, and clinical subgraphs). Difficulty is calibrated by the depth of contextual synthesis required-retrien
relies localized subgraphs, whereas creation necessitates integrating hierarchical, relational, and topological cues. The ontology's explicit logic ensures question complexity scales with measurable graph properties, like inferential distance between chain endpoints, avoiding the ambiguity of hop-count-bars metrics.
C.5 CIICK & CORRECT
To ensure the accuracy of both evidence and answers, perform verification and correction. We define the evidence validation process as follows: given the original corpus and the constructed evidence, assess whether the evidence criteria are strict: given a question, if no evidence triple can be inferred from the corpus, the corresponding question is discarded. Similarly, for answer correction, check whether the provided
be logically derived from the corpus. Our validation
21

---

Published as a conference paper at ICLR 2026
Question Generation: Fact Retrieval
Instructions:
You are professional dataset constructor. You generate questions based following requirement.
Please generate [num] factual English questions based on the following rules:
-Gnneration Guidelines:
Focus on explicit information:
Character relationships (e.g. family ties, romantic involvement)
Key events timelines
Physical descriptions
Professional roles'occpattins
Goographical locations
2. 2.basic rules:
each question has unique and accurate
Cover all characters and core relationships
- Avoid redundancy in questions
3 Prohibited content:
Subjective interpretations
Imp lied/non explicit information
Question Generation: Complex Reusoning
-Task Description:
You are an expert in constructing rensoning-busee datasets from narrative subgraphs. Your task is to generate [num) high-quality English reasoning questions from given context made up multiple logical chains. Each question must span multiple triplets and require reasening across distant elements within single chain across multiple chains.
-Generation Guidelines:
1. You generate type? reasoning questions:
Each question must require * "multi-hop hop reasoning** across **at least three triplets**.
Preforably, relevant triplets should *=oonnajjccemt* (i.c,, have some logical distance between them).
You may cither a *"ingle chain** ** combinc "rriplets from multiple chains"* * long the reasoning is clear and explicit.
2. The answer must be uniquely supported by the input triplers.
3. not up or infer anything
explicitly
triplcts.
Question Generation: Contextual Summarite
-Task Description:
You are professional datasct constructor. Your task is to generate summarization-style questions based on the following instructions. generate [num) high-level English questions that summarize or synthesize information
provided logical chains
-Generation Guidelines:
1. Questions must focus on summmriizng patterss, events, or roles** across the logical chain.
*the majority 3. You may combine information from **a single chain** ** compare across chains** ** when approprinte.
Each question should ideally
triplets** in
just
5. Avaid factual recall (like who did X to Y); instead, emphasize **storyline progression"* **role dynamies**. or **cause-effie effeect chains* unstated dctails. Use only **xxplici*** triplets.
7. Do NOT generate questions semantically similar to the existing list.
Question Generation: Creative Generation
-Task Description:
are professional dataset constructor working on understanding benchmark. tasked generating *ceettive expression questions", which require imaginstive re-narration grounded lactual evidence.
Please generate (num) creative-style questions in the following categories: Role-playing narration (first-person) character retells event their point view
Textual rewriting (in diary report format): the event re-expressed in another wriling style
Generation Guidelines:
Role-playing narration questions:
Ask the user to imagine being character and retelling or expressing an experience
explicit entities events triplets
-Ground the narratvve observed facts
2. Text rewriting questions:
for retelling
diary (subjective), news (objective)
Diary entries should reflect the character's perspective
News reports should be impersonal but factually accurate
Fach question supported hy
distinct triplets** evidence. The generated question synthesize
depend "broad scope of factual context". Do generate shallow narrowly grounded questions.
Figure 8: Example prompts used for constructing the Novel Dataset GraphRAG-Bent
22

---

Published as a conference paper at ICLR 2026
Table 1I: Results of Generation Evaluation using Qwen2.5-14B. covering tasks of varying complexity.
Category
Model
Fact Retrieval ROUGE-L
Complex Reasoning ROUGE-L
Contextual Summarize
Creative Generation ACC
Detoser
Busic RAGG
RAG (w remunk
46.74
19.11
42.36
11.90
51.55
83.00
38.23
52.27
38.76
M5-GmmphRAG AG (local) (ddee es
2024
39.89
25.93
46.12
27.30
65.28
69.10
39.57
54.64
52.42
MS Graghit (alobil)
Tooee pr
2014
39,89
11.34
42:22
19.10
50.41
73.45
36.59
84.00
34.30
Graph RAGG
HippRAc12 (illiem
54.79
90.16
50.45
30.30
61:14
68.09
40.52
52.24
52.05
LightRAG
11.00
12.22
60.27
9.91
52.07
86.20
39.74
78.73
39.67
Gas-GGuphRAG
74.45
RAFTOR
38.56
63.10
Medical
Dateret
Rusic R.G
RMG (W) rennk)
$7.56
21.14
56.01
12.71
61:95
79.32
60.91
53.84
47.91
MS GraphRA (local) lodae et al 2025
49.24
29.17
61.64
27.40
59.01
37.32
61.70
33.39
42.24
MS- MSS-GrsphiRGG (olaha)) Tome 2010
40.04
17.01
61.40
21.08
51.10
38.60
59.20
43.17
HippoRAGG2 Kuunén
64.05
60.77
39.33
LightRAC
Fist GraphRAG Cardesll
29.42
62,98
31,98
RAPTOR Striil o. di 2043
50.48
13.09
52.96
12.96
58.94
73.61
61.45
49.91
$1,46
*Due to time and resource constraints, we tested our benchmark on representative set GraphRAG models.
answer can be logically inferred from the evidence. This verification and correction process is supported by advanced models combined with human checking for final confirmation.
C.6 REFINEMENT
We observe that some generated questions may be overly direct, lacking sufficient contextual informa- tion, which may affect effective retrieval. To address this, we further enrich and refine the question by incorporating relevant background knowledge. Specifically, for each question, we locate the corpus segment from which the evidence was derived and employ GPT-4. to refine the original question by integrating this segment. This ensures that each question not only retains the necessary logical structure but also provides relevant background context, thereby enhancing the overall clarity and rationality of the question.
D EXTENDING GRAPHRAG-BENCH TO NEW DOMAINS
In this paper, we used medical dataset to represent domain knowledge in the initial version of GraphRAG-Bence The method we constructed the data can be used to extend the benchmark into other fields, such as law and finance. For researchers who wish to add other domains in future work, we provide some suggested methods.
For the legal domain, we offer the following suggestions: 1) CORPUS SELECTION: We suggest collecting data from the following sources: a) EU Case Law which contains 29.8K EU court decisions, mainly from the Court of Justice (CJEU), published in EUR-Lex. b) UK Case Law which contains 47K UK court decisions from the British and Irish Legal Information Institute (BAILII) database. c) US Case Law which contains 4.6M US decisions (opinions) from Court Listener, a web database hosted by the Free Law Project. 2) DATA MINING METHOD: We recommend a method that integrates both ontology and logic chains. Ontology provides structured representation of legal regulations and judicial interpretations by defining entities, actions, relationships, and the conditions under which specific laws apply. Logic chains, in parallel, capture the legal reasoning process and model potential multi-hop relationships between different laws.
For the finance domain, we suggest the following: 1) CORPUS SELECTION: We recommend focusing on publicly traded companies from the S&P 500 list for the period between 2015 and 2024. For these companies, we can collect their annual reports, quarterly reports, and reports on unscheduled events from the same period. These documents are primarily available in the EDGAR database.
2) DATA MINING METHOD: We suggest using an ontology to map company information, such as business structures and financial metrics,
unified schema. It is worth noting that the financial domain contains a large amount of numerical data, which may require further processing. The details statistics of Legal and Financial Corpora in Table T2
23

---

Published as a conference paper at ICLR 2026
Table 12: Statistics of Legal and Financial Corpora.
Source
Numbers
Tokens
Report Type
Numbers
Tokens
BU Casc Law
14.9K
89.25M
Annual Report
192
18.72M
UK Case Law
11.7K
92.10M
Quarterly Report
588
22.93M
U.S. Case Law
IK
114.23M
Current Report
1427
74.21M
Total
72.7K
295.58M
Total
2207
115.86M
(a) Statistics of Legal Corpora.
(b) Statistics of Financial Corpora.
Table 13: Corpus statistics for different benchmarks. We report the average number of entities and relations per Ik tokens (Avg. Entities, Avg. Relations), the proportion of non-isolated entities (Prop. of Non-isolated Entities), the average node degree (Avg. Degree), the proportion of entities with total degree greater than 1, 2, and (Prop. Degree k), and the geometric mean of the number of entities in connected components (Avg. Component Size).
Benchmark
Avg. Entities
Avg. Relations
Prop. of Non-isolated Entities
Avg. Degree
Prop- Degree 1
Prop. Degree
Prop. Degree 3
Arg. Component Size
UlraDommain
170.6
73.2
0.40
0.86
0.27
0.15
0.09
2.71
MalliHop-RAG
0.41
HotporQA
39.3
12.7
0.41
0.65
0.23
0.12
0.06
2.11
MuSiQQue
21.5
6.5
0.39
0.60
0.23
0.10
0.06
2.33
ZWik/MuhilopQAA
0,40
GraphiRAAi-Bae (novel)
0.66
GraphRAG- Bench (medical)
11.8
0.48
1.08
0.36
0.20
0.12
3.15
E DATASET STATISTICS AND VISUALIZATION
Corpus statistics We first conduct a statistical analysis of the composition of the corpus from existing bench- marks, shown in Table 13] We find that these benchmarks contain large number of redundant entities and relations. Although some bench- marks like UltraDomain have rela- tively high average numbers of enti- ties (170.6) and relations (73.2), their average degree remains low (with maximum of only 0.86), indicating lack of effective connectivity among entities. Further analysis reveals that the average component size remains
Table 14: Comparison between GraphRAG-Bent Bencch and other benchmarks. High Info Density indicates whether the corpus has high information density; Question Di- versity denotes whether questions are categorized by dif- ficulty levels; and Reference Answers indicates whether reference answers are provided.
High Info
Question
Reference
Dataset
#Questions
Density
Diversity
Answers
UltraDemain
2500
Multihop
HorpetQA
7405
GraphRAG AG-Bench
Fort Retrieval
Complex Reasoning Creative Generation
100
Costestual Summarioc
small (e.g.. 2.7 in UltraDomain and MultiHop-RAG), and the proportion of non-isolated entities also low (around 40%), suggesting sparse and fragmented graph structures. Addi- tionally, only a small fraction of enti- ties have degrees greater than (e.g.. 9% in UltraDomain), reflecting lim- ited semantic aggregation. Such struc- tural characteristies lead to low overall information density, making it difficult to effectively support retrieval tasks based graph structures. These findings highlight the limitations of current benchmarks in terms of information organization and provide strong motivation for our benchmark design.
Perceatage?
78.2
52.2
50
30.5
19.2
1.8
3.3
UltraDomain
MultHoo-RAG
HotpotQA
GraphRAG-Bench
Distribution of questions with varying diffi- culty levels across different benchmarks.
Figure 9:
Based on the above analysis, GraphRAG- Bench is designed to provide benchmarks with richer and more structured entity-relation graphs than existing datasets. In both the medical and novel subsets, the corpus contains higher average degrees (1.05 and 2.27, respectively). larger proportions of non-isolated entities (0.48 and 0.66), and increased average component sizes (3.15 and 3.99),
24

---

Published as a conference paper at ICLR 2026
indicating more coherent and connected graph structures. These improvements address the sparsity and fragmentation observed in prior benchmarks, and offer a more suitable foundation for studying the impact of graph-based information organization in retrieval-augie -aggmented generation. This setup enables controlled experiments for analyzing how different structural properties affect the behavior and effectiveness of GraphRAG systems.
Questions statistics We first categorize questions for existing benchmarks based on a predefined laxonomy of difficulty levels, with classification results shown in Figure 9 Our analysis reveals significant imbalance in the distribution of question levels across current benchmarks. Specifically, UltraDomain focuses mostly on Contextual Summarize questions (97%), while HotpoiQA mainly contains Fact Retrieval questions (78.2%), lacking coverage of deeper logical reasoning tasks. Although MultiHop-RAGi balances Complex Reasoning and Contextual Summarize questions better, it entirely lacks basic Fact Retrieval questions, making its evaluation coverage incomplete.
To address these limitations, GraphRAG-Bent Genheech introduces carefully designed taxonomy of question types to achieve more comprehensive evaluation coverage. We not only ensure more balanced distri- bution across the four core categories but also introduce a novel Creative Generation category, filling critical gap in assessing generative creativity which largely overlooked by existing benchmarks. This multi-level question design enables GraphRAG-Bent of both RAG and GraphRAG systems, offering unique advantages for analyzing model performance across tasks with varying levels of cognitive
to provide a more systematic evaluation
F EVALUATION METRICS
Retrieval Performance To evaluate the retrieval performance of GraphRAG, we argue that an effective system should both guarantee the completeness of retrieved information and reduce irrelevant content. We introduce two corresponding surrievalquualit--bee
metrics as detailed below
CONTEXT RELEVANCE measures how well the retrieved content aligns with the question's 's intent. quantifies the semantic similarity between the question and the retrieved evidence, with higher values indicating more focused and pertinent information. Specifically, it can be defined as:
1 CC]
Σ CEC
CONTEXT RELEVANCE
where C denotes the set of retrieved contexts, Q represents the question. denotes the set of evidence. and the operator R(-) determines whether context is relevant to the question Q and the evidence E
EVIDENCE RECALL measures retrieval completeness by assessing whether all critical components required to correctly answer a question are captured. Higher values indicate more comprehensive evidence collection. The formal definition as follows:
EVIDENCE RECALL 1 E((((c))
[
where R is the of reference claims, and the operator S(-) determines whether a claim i supported by the retrieved context C, providing the condition for the indicator function 1l ((.).
Generation Accuracy. After retrieval, GraphRAG system is expected to generate accurate answers based on the retrieved contexts. To evaluate the quality of the generation, we introduce four key metrics: 1) LEXICAL OVERLAP: Measures word-level similarity between the generated and reference answers using longest common subsequence matching. 2) ANSWER ACCURACY: Assesses both semantic similarity and factual consistency with the reference answer. 3) FAITHFULNESS: Evaluates whether the relevant knowledge points in long-form answer are faithful to the given context. 4) EVIDENCE COVERAGE: Mcasures whether the answer adequately covers all knowledge relevant to the question.
ROUGE-L quantifies text similarity through n-gram overlap between generated and reference answers, capturing both syntactic and semantic alignment (Lin 2004).
ANSWER ACCURACY provides dual assessment of answer quality: 1) Semantic alignment:
base similarity scores 2) Factual precision: Fine-grained statement-level verification
25

---

Published as a conference paper at ICLR 2026
This combined approach ensures answers are both contextually appropriate and factually accurate.
AC FC + (1 -a) S
(5)
where is the weight parameter, set it to 0.75 by default. FC is the Factual correctness and SS is Semantic similarity, they are defined as:
TP
FC TP+ +P FP + FN
SS = cos(fo
FAITHFULNESS specifically targets hallucination risks by measuring claim-to-evidence alignment. The metric calculates what percentage of answer assertions directly supported by the retrieved context, crucial for evaluating etrrieallgroundde generation. The computation follows:
IiE A ||S(c. S(c. C)}
FS
|A]
where FS is the faithfulness score, A is the set of claims in the response, C the retrieved context, and S(c, C) is boolean function indicating whether claim e supported by context C.
EVIDENCE COVERAGE evaluates answer completeness against reference standards. Rather than simple overlap, assesses whether all necessary knowledge components appear in the generated answer, particularly important for complex queries requiring comprehensive responses. The formal computation follows
He F ((. G))
Cov
[
where Cov the coverage score, is the set of reference evidences, the generated answer, and (e, G) is boolean function indicating whether evidence appears in G.
G ADDITIONAL EXPERIMENTS
G.1 EXPERIMENTS MORE GRAPHRAG FRAMEWORKS
We evaluated total of 11 GraphRAG frameworks in study. Due to space constraints in the main paper, only subset of the results is presented there. This appendix provides the complete and detailed results for all frameworks, which are shown in Table and Table
G.2 EXPERIMENTS ON OPEN-SOURCE MODEL.
In main experiments, employ GPT-4o-mini the default backbone model for generation. To evaluate how well different GraphRAG frameworks adapt across generation models, we use Qwen2.5-14B the open-source model. The experimental results are presented in Table 11 Due time and resource constraints, we tested representative subset of these models on Qwen2.5. - 14B. We summarize the observations as follows.
When integrated with the open-source Qwen2.5-14B model, several lightweight GraphRAG frame- works exhibit competitive performance. Notably, Fast- Bast-GraphRAG achieves the highest accuracy (60.08%) and ROUGE-L (41.31%) in fact retrieval, as well as strong performance in creative genera- tion (ACC 47.640..606 (0.6%) on the Novel dataset. On the Medical dataset, LightRAG leads in Contextual Summarize (ACC 69.37%) and creative generation (FS 70.84%), while HippoRAG2 obtains the best ROUGE-L scores for both fact retrieval (33.92%) and complex reasoning (33.02%). These results suggest that even under open-source settings, resource-efficien GraphRAG frameworks can effectively leverage graph-structured context to support various generation tasks.
G.3 EXPERIMENTS ON GRAPH CONSTRUCTION EFFICIENCY
To assess the practical deployment costs of different frameworks, we evaluated the graph construction efficiency the Novel dataset. We used one book (approximately 56k tokens) as the input corpus
26

---

Published as a conference paper at ICLR 2026
Table 15: Construction time and all token usage during the indexing phase across different GraphRAG.
Method
Time
Input Tokens
Completion Tokens
Tofal Tokens
GraphRAG [Edge 2024
292.45
535832
118841
654673
LightRAG (Guo el al. 2024
710.32
403401
70771
474172
GraphGGG
281.74
187765
64052
251817
HippoRAG (uttéérezz
2024
77.42
154990
22971
177961
HippoRAG2 (Gutiérezz
2025
96.71
283498
46495
329993
KGP /Wang et al. 2024b
32.01
68540
20675
89215
RAPTOR (Sarthi el al 2024
135.21
113641
1900
115541
KET-RAG [Huang al.
2025
350.43
433191
84360
517551
LAZY-GraphKAG
253.59
519394
71756
591150
and recorded the construction time, total input tokens, and complction tokens during the indexing phase. The quantitative results are presented in Table 15
We summarize the observations as follows. The efficiency varies significantly across different GraphRAG methods. Notably, KGP demonstrates the highest efficiency, completing the construction process in just 32.01 seconds with the lowest total token usage (89,215). HippoRAG and Hip- poRAG2 also perform efficiently, maintaining low time and token costs. In contrast, LightRAG and the MS-GraphR AG incur much higher resource consumption. LightRAG requires over 710 seconds index, while MSS-GrapRRAG consumes nearly 655,000 tokens. These results suggest that lightweight frameworks offer far more cost-effective solution for graph indexing, which is critical for applications with limited time or computational budgets.
Table 16: Generation Evaluation (ACC) on Medical dataset across Qwen series models (3B-14B)
Model
Fact Retrieval
Complex Reasoning
Contextual Summarize
Creative Generation
Avg
GruphRAG (lppooRAG2) (HipRRAGG()))))
Qwen2.5. 5.3h
19
58.11
58.69
$1.34
$7.08
Qwen2.5-7b
65.65
64.25
64.26
$1.85
61.50
Qwen2.5-148
63.98
62.62
64.95
62.09
63.91
Qwen2.5-38
58.66
52.25
58.71
54.89
56.13
Qwen2.56 5-7b
$7.45
53.53
60.52
56.79
Qwen2.5-14b
57.56
56.01
61.95
60.91
59.10
G.4 EXPERIMENTS ON THE IMPACT OF LLM SIZE
To investigate the dependence GraphRAG performance on the underlying LLM capability, we con- ducted experiments using the Qwen2.5 series (3B, 7B, and 14B) on the Medical dataset. We compared Standard RAG against GraphRAG, employing HippoRAG2 as the representative GraphRAG method. Since the graph index construction is offline process, utilized the same pre-constructed index (generated by GPT-4o-mini used in our main experiments across all model sizes. This experimental design allows to isolate the impact of model size specifically during the online generation phase. The quantitative results are presented in Table 16
GraphRAG exhibits higher sensitivity model capacity than Standard RAG. While RAG's per- formance remains relatively flat across scales (from Avg 56. 13 to 59.10), GraphRAG demonstrates substantial growth (from Avg 57.08 to 63.91), indicating greater reliance on the model's reasoning ability to synthesize structural information. Notably, we observe a distinct performance inflection point the 7B parameter scale (Avg increasing from 57.59 to 61.50), suggesting that 7B serves as a practical "minimum size" threshold where the model acquires sufficient reasoning power to effectively leverage graph-based context.
G.5 EXPERIMENTS ON SCALABILITY WITH CORPUS SIZE
To evaluate the scalability of GraphRAG compared to Standard RAG across varying data volumes, we conducted controlled experiments on the Novel dataset. We defined the corpus sizes based on token counts calculated via tiktoken: Small (~556k tokens, representing single book unit), Medium
27

---

Published as a conference paper at ICLR 2026
Table 17: Generation Evaluation (ACC) across Novel dataset under different corpus Sizes
Corpus Type
Corpus Size
Fact Retrieval
Complex Reasoning
Coniextual Summarize
Creative Generation
GraphRAG (HippoRAG2)
Small Mediom
56k
60.
53.38
64.10
48.28
59.99
56.67
65.77
50.06
Large
1132K
59.19
$4.29
62.63
51.18
Small
56k
64,73
58.64
65.75
60.61
Medium
6U3K
58.43
41.33
62.06
52.54
Large
1132k
58.04
43.20
62.43
47.19
(~603k tokens), and Large (~11 tokens, representing the full dataset). We utilized HippoRAG2 as the representative GraphRAG method and compared it against Standard RAG across these scales. The quantitative results are presented in Table 17
The comparison reveals critical insight regarding robustness. Standard RAG suffers significant performance degradation as the corpus size increases. Notably, its accuracy in Complex Reasoning drops heavily from 58.64% (Small) to 43.20% (Large). This supports the hypothesis that rector-based retrieval is prone to capturing high-similarity but irrelevant noise as the search space expands. In contrast, GraphRAG (HippoRAG2) exhibits remarkable stability, maintaining consistent performance across all scales (e.g., Fact Retrieval remains steady at 60%%.. This confirms that the performance stability is driven by the structural constraints of the graph (i.e., explicit entity and triple matching), which effectively filter out the retrieval noise that tends to accumulate with increasing corpus size.
H IMPLEMENTATION DETAILS
H.1 IMPLEMENTATION DETAILS OF REPRESENTATIVE RAG MODELS
Table 18: Implementation Details of different GraphRAG models.
Knewiedge Type
Index Content
Quers Inpet
Gratolarity
LLM cnntest
RX0
Fait
Query Estookitz
MS GraalRRGiilaa
Kree
Grapk.
MS Gr.gGGrgkB BRA vidoo
Textaa Kandolag Grapl.
Crumsenit
Query Entoskliu
Commaaaiy)
Liznd Rou.
HappeR Contnicetre E1
Kanekadge Graph
Euity
Exttics is Query
Muss: Clasis Cluck
Recansin pul
HippoRA
Kunsladge Grack
Plasae waa
Query Eaisokiiz
LiteRAd
Kanododge
Relainslpp
Clank
Litaal Liznd You
Elsnect
G
Textal Kinnlolaz Grapl.
Eatiyy
Eutlics Query
Enily Rewateaniiiiic..c. Clank
RAPTOS.
Tiee
Ticanade
Quey Enfookiig
Tree aode
Recsnsing pall
Pair Teet rer-dddddd
Tear C hunk
Quary limbadiing
Chankc Node (dyrumic)
Liarl Yine
sPaSICAL
[hennciical lonowiedge Craph
Ilier ecticaa Node
Query: limbadir
Noce, 1aah
Lisral Teel e Seaamnd Info
KET--AL
Krowlaagge-itteee Entunedd Inse
Keysent Enuity,
Query (mulr-nneas)
Keysoed, Liatyy Uhumi
Liant
As discussed in previous work (Gao et al. 2023 Lewis et al. 2020 Zhou et al., 2025 RAG
models comprise three core components: indexnig. retrieval, and generation, each with its specific implementation details as presented in Table 18 Some explanation should be given to the content in the "Knowledge Type" column of the table.A knowledge graph is constructed by extracting entities and relationships from each chunk, which contains only entities and relations, is commonly represented as triples. textual knowledge graph is a specialized KG (following the same construction step knowledge graph), which enriches entities with detailed descriptions and type information. A tree structure formed by document content and summary.
Implementation Details of RAG We follow the standard RAG paradigm: retriever model first retrieves relevant context from the corpus based on the given question, and then the question is concatenated with the retrieved context to form a query for the generation model to produce the final answer. Since existing RAG approaches often incorporate rerankers to improve retrieval quality, we consider two baselines: RAG-with-rerank and RAG-without-rerank
Implementation Details of GraphRAG We evaluate several representative GraphRAG frameworks on benchmark, including:
GraphRR based local retrieval granularity.
28

---

Published as a conference paper at ICLR 2026
MS-GGRAPHRAGGGLOBAAL) Microsoft-GraphRAA based on global retrieval granularity.
LIGITRAA: a framework that enhances graph efficiency by leveraging optimized graph structures and two-stage retrieval pipeline.
HIPPORAG: a framework inspired by the hippocampal memory indexing theory, integrating large language models, knowledge graphs, and personalized PageRank to enable efficient single-step multi-hop knowledge integration and retrieval.
HiPPORAG2: framework that achieves deeper contextual understanding by jointly incor- porating conceptual (phrase-level) and contextual (passage-level) nodes.
framework designed to improve retrieval speed and reduce computa- tional cost through efficient graph-basee querying.
RAPTOR: framework that constructs tree structure through recursive embedding, clustering, and summarization text segments, enabling efficient information retrieval across different levels of abstraction.
STRUCTRAG: a framework that boosts knowledge-intente reasoning of LLMs by dy- namically restructuring scattered information into hybrid, structured format inference time.
KGP: framework that improves multi-document question answering by constructing and traversing knowledge graph to formulate the right context for large language models.
LAZY-GRAPHRAG: a framework that achieves better cost-quality trade-off by using "lazy" approach to build a concept graph only at query time.
KET-RAG: framework that achieves cost-efficient and high-quality Graph-RAG by leveraging a multi-granular indexing approach combining a knowledge graph skeleton with text-keyword bipartite graph.
All of these GraphRAG methods construct graphs and refine retrieval strategies to boost RAG system performance across various specialized tasks. Due to time limits, we only assess several representative GraphRAG models. We will include more SOTA models, like ArchRAG (Wang et al. 20255) PIKE-RAG (Wang et 2025a) MedRAG (Zhao
2025) PathRAG (Chen et al. 2025a) CG-RAG (Hu et al. 2025).
DBCopilot (Wang et al.) 2025c) LightPROF (Ao et al. 2025
H.2 CONFIGURATION OF GRAPHRAG MODELS
In our experiments, we maintained consistent conditions for fair comparison. Specifically, all GraphRAG and RAG systems used the bge-large-en-v1.5 -n-115 embedding model during retrieval stage, and used generation temperature of 0.7 during generation stage. For GraphRAG systems, given the inherent differences in graph indexing, retrieval strategies, and generation mechanisms across frameworks, we preserved their default configurations (including graph indexing, retrieval strategies, and generation methods) without modification to assess their native performance. This approach ensures both comparability across systems and realistic evaluation of their practical capabilities. The detailed configuration parameters are following:
RAG Configuration
rembeddiing_odelll Soge-large-en-vi "reranker"; reeerrrraner--arrce
"retrieval_topk" "chunk "chunk_token wtwpetoooeeeeeee_ee_______ size": 256,
29

---

Published as a conference paper at ICLR 2026
Mi-CraapRAAiigbba&llcaa))) Configuration
"embedding_modell "bge-large-en-v..5",
"chunk_tookn_sizze" 1000,
henrk__oov_aa__tte__sissss 100,
cummaarzaa__eeecipiiionns..ssss maass_mx__lee_ _aass_mxx_eeehh:: 500, max_ccluster_siiz=: 10,
commmntts__rrgprrre_aaa_eeetttt 2000,
"community_reporrss aappu__lenghh"" 8000
LightRAG Configuration
"embedding_moder "bge-lazggeen--1.55" "query_tyee:: "hybrid", "chunk_token_siiz"" 1200, "retrieval_topk: 30, "cduun__overlaapttkkrssssss 100,
ttt__tttoeef____e______ for for__eet__uee "or____f__ttt__uit""" 4000,
httaa_tttee_foo_oo_e_ee for cerr__gobbalccoteexeeo 4000, "max token_tokee___e
Beoo_ttooee__fr_oooaaccoooetee 4000
FastGraphRAG Configuration
sittedddingmodeel"" bye-large-en-vi.55 "entitt__rankinggpollyy 0.005, "reelation_ramking_oolll. 64,
8,
HippoRAG2 Configuration
Vembeddingg_oceel ": bge-large-en-vi.5,
"retrieval_top_kee _" 5,
"Linking_top_k": 5,
"max_qa_steps"; 3,
"qa_top_k": 5,
"graph_type"" facts_and sim ind__oiieddddeeeioo.......
HippoRAG Configuration
"embeddinggmodel" bge-large-en-vl.5,
"chunk_token_size"" 1200,
fhaunk_overlap_token_sszz 100, "retrieve_topk": 20,
"entities_max_toeens"" 2000,
"xelationships_max max__ppp_______eem tokexs" 2000,
30

---

100, Published as a conference paper at ICLR 2026
RAPTOR Configuration
"embedding model": bge-Large-en-vl.5, "chuuk_tooke__size". 1200,
"num_layers": 5, "max__enggth_in__luuttt"""" 3500, "threshold"; 0.1,
=lluster_mttrie cosine, gereshoid_clustiee num' 5000
I RELATED WORK
1.1 TRADITIONAL RAG AND THEIR LIMITATIONS
The naive RAG systems (Lewis al. 2020) operate through three key steps: knowledge preparation, retrieval, and integration. During knowledge preparation, external sources such as documents, databases, or webpages are divided into manageable textual chunks and converted into vector representations for efficient indexing. In the retrieval stage, when user submits query, the system searches for relevant chunks using keyword matching or vector similarity measures. The integration stage then combines these retrieved chunks with the original query to create informed prompt for the LLM's ' response. Recent advancements in RAG systems have moved beyond basic text retrieval to structured knowledge integration Xiao et al. (2026 2025); Chen (2025b). Modern implementations employ hierarchical architectures maintaining document organization via layered retrieval processes (Chen al. 2024a; Li et al. 2024), while others enhance precision through multi-phase retrieval mechanisms that first broaden then refine context selection (Glass et al. 2022
Xu et al. 2023). Autonomous query parsing frameworks automatically break down intricate questions into executable subqueries (Asai et al.) 2023), complemented by context-aware systems that modify retrieval tactics in real-time according to query complexity and intent (Tang et al. 2024 Sarthi al. 2024). These strategies advance naive RAG systems by improving context awareness, retrieval accuracy, and handling complex queries.
Although researchers have extensively explored traditional RAG, there are still some unresolved limitations due to the constraints of the data structure itself. (i) Vector database architectures limit traditional RAG's ability to handle multi-hop reasoning, as they retrieve information only from text chunks containing anchor entities. While methods like query expansion (Mao et al.] 2020) and metadata enrichment (Wang al.) (2024a) attempt improve complex query handling, they remain constrained by the chunk-based knowledge structure that inherently disconnects related concepts. This structural limitation particularly hinders domain-specific reasoning requiring logical synthesis across distributed evidence. (ii) The chunking process sacrifices critical contextual relationships between specialized terms and abstract concepts, despite techniques like real-time retrieval alignment (Jiang et al., 2024) and external API integration (Lazaridou et al. 2022). Vector databases' flat organization fails preserve hierarchical conceptual dependencies essential for domain expertise utilization, leaving models unable to reconstruct professional knowledge networks from fragmented chunks. (iii) Vector similarity retrieval often overwhelms LLMs' fixed context windows (OpenAI) 2023 Anthropic 2024) with redundant content, exacerbating their limited capacity for long-range dependency modeling. While strategies like context pruning (Arefeen et al.) 2024) and LLM fine- tuning (Luo et al.] (202)) reduce input volume, they cannot compensate for the structural inability to establish explicit connections between retrieved chunks. This fundamental mismatch persists despite optimizations like sliding windows (Wang al., 2024a), vector-based approaches lack mechanisms for relational reasoning.
1.2 GRAPHRAG AND ITS ADVANTAGES
To address this, graph retrieval- -uggenedd generation (GraphRAG) (Peng et al. 2024 Procko & Ochoa 2024) has recently emerged powerful paradigm that leverages external structured graphs
31

---

Published as a conference paper at ICLR 2026
Question Generation: Fact Retrieval
IInstructions:
generate questions based following requirement. Pleuse generale [num] factual Einglish questions based the following rules:
You professional dataset constructor.
-Generation Guidelines:
1.Create information seeking questions that require direct factual retrieval from relations list.
Focus these biomedical categories:
Cancer Type ddeniiicaaiio Discase staging
associations
Treatment protocols
Diagnosric methods
Risk factors
Symptomatology
babai rules:
each question has unique and uccurate unswer
Caver all medical concept, Fissuee proportional coverage of all cancept categories
Avoid redundancy in qucstions**
Probibited
Subjectivee interpretations
Implied/non explicit eoppiipliiitttttt
Question Generation: Complex Reasoning
-Task Description:
You are a professional dataset constructor. You can generate questions based following requirement.
Generate [num) Reasoning English questions, each questions requiring muliiple inferential leups uccessing several of information from different locations or sources to arrive at an answer. Generate based on the following rules:
-Generation Guidelines:
Generation Guidelines:
1.Focus on explicit information:
the relations that have
entity find the connections of three evidences and combin
complex iden.
2.basic rules:
question a unique accurate answer question cannot answered by relying on just one evidence requires
understanding and linking the information from different sentences. Avoid redundancy in questions
3.Do not make up or infer anything that is not explicitly stated in the input.
Question Generation: Contextual Summarize
-Task Deseription:
You are a professional dataset constructor. You can generate questions hased on the following requirement. Generate 5-10 **ummary--yp English questions**, where each question prompts coneise synthesis biomedical knowledge related to cancer based on the input data.
-Generation Guidelines:
1.Avaid factual recall (like who did X to Y inisttcd, focus on synthesis of explicit information:
Summarize categorics, patterns, properties connected single concept (e.g., diagnostic methods used for NSCLC),
Consolidate related items (e-g., all symptouns of candition, treatments available specific stage)
2. Each question must have one clear, unique answer and should ideally cover **tbe majority of the ontology **
3.vvoid redundancy and overly narrow questions.
4.Do NOT infer invent unstated details any information not explicitly stated the input antology.
Question Generation: Creative Generation
-Task Deseription:
working novel understanding benchmark. You tasked with generating *creative expression questions*, which require imaginative ro-narration grounded in factual evidence. Please generate clinica syathesis tasks following categories:
You professional dataset
Rooee-bssed medical tasking: the model is asked take professional medical role (e.g., clinician, radiologist, medical patient)
and perform complex task like diagnosis formulation, treatment planning, or case summary writing.
-Format transformation: rewrites modical facts specific document format such as a clinical report, referral note, discharge summary, public health advisory.
-Generation Guidelines:
1.Role-based medical tasking:
the imagine they medical professional patient
Task exumples: write diagnosis, generate u treatment plan, construct a referaa letter, etc.
The response should reflect professional medical tone and reasoning
F.Format transformation:
Ask the user rewrite the case facts into formal medical document
Examples: Discharge summary, SOAP note, progress note, guideline draft, public notice Use explicit entities from the triplcts follow relevant medical document conventions
Figure 10: Example prompts used for constructing the Medical Dataset in GraphRAG-Benn
32

---

Published as a conference paper at ICLR 2026
to improve LLMs' capability on contextual comprehension (Han et al.
2024 Zhang al. 2025a).
Early efforts. like Microsoft GraphRAG (Edge et al.) 2024) and its variant LazyGraphhAa (Dar
ren Edge 2024), employ hierarchical community- based search and combine local/global querying for comprehensive responses. Building on this, LightRAG (Guo et al. 2024) improves scalability through dual-Ievel retrieval and graph-enhanced indexing, while GRAG (Hu et al. (024) introduces a soft pruning technique to mitigate the impact of irrelevant entities in retrieved subgraphs and employs graph-aware prompt tuning to help LLMs interpret topological structure. Further extending these capabilities, StructRAG (Li et al. 2024) tailors data structures to specific tasks by dynamically selecting optimal graph schemas, while KAG (Liang et al. (2024) constructs domain expert knowledge using conceptual semantic reasoning and human-annotated schemas, which significantly reduces noise present in OpenlE systems. These strategies used GraphRAG models significantly improve retrieval precision and contextual depth, enabling LLMS to address complex, multi-hop queries more effectively.
GraphRAG offers several significant advantages over traditional RAG systems (Peng al. 2024). enhancing the capabilities of Al-powered information retrieval and generation. First, its graph- knowledge representation captures hierarchical relationships and multi-hop dependencies between entities, enabling nuanced contextual reasoning and discovery of latent connections (Yang et al., 2026). This structured approach resolves ambiguity by evaluating multiple semantic paths during query processing. Besides, the graph structure allows unified integration of structured databases, semi-structured formats, and unstructured text within single graph, supporting cross-modar queries that combine textual, numerical, and multimedia data (Yin et al., 2023) 2020; Yue et al. (022)). This interoperability maximizes value from heterogeneous organizational knowledge assets (Procko & Ochoa 2024, Luo et al. [2024] 2025 Xia et al. 2025). Third, users can audit decision pathways by visualizing entity relationships traversed during retrivval. Combined with LLMs, this transparent architecture supports multi-hop logical synthesis, critical for specialized domains like healthcare and
finance (Procko & Ochoa 2024 Han et al. 2024).
1.3 EXISTING BENCHMARKS AND ANALYSIS
It is crucial to identify the factors that are currently limiting GraphRAG's real-world performance.
However, quantitatively and fairly assessing the role of graph structures in RAG systems is challenging. Current benchmarks, including HotpotQA (Yang al. 2018), MultiHopRAG (Tang & Yang 2024)
and UltraDomain (Qian et al. 2024). fail to adequately evaluate the effectiveness of graph structures in RAG systems due fundamental limitations in both their problem design and corpus composition. A few studies, like DIGIMON (Zhou et al. 2025) and another analysis paper (Han et al. (0225). have recently tried analyze the effect of different GraphRAG models. Despite their effort, they mainly focus on architectural comparisons using homogencous datasets, missing how models synthesize hierarchical expertise and unstructured narratives. To this end, we propose GraphRAG-Bennd comprehensive benchmark designed to evaluate GraphRAG models on deep reasoning. It features hybrid corpora with tasks of increasing complexity and stage-specific metrics to expose why models fail, whether in graph construction, knowledge retrieval, contextual synthesis. Leveraging this novel benchmark, we systematically investigate the conditions when GraphRAG surpasses traditional RAG systems and the underlying reasons for its success, offering guidelines for its practical application.
a
J LIMITATION
While our benchmark advances GraphRAG evaluation by systematically addressing reasoning com- plexity beyond traditional retrieval- -eriival--entric paradigms, it inherits constraints inherent to its design scope. Most notably, the framework operates exclusively within unimodal (text-based) contexts, omitting the challenges and opportunities posed by multimodal data integration. Real-world applica- tions of GraphRAG often necessitate synthesizing heterogeneous information types, such as visual diagrams, tabular datasets, or sensor- generated temporal sequences, to resolve complex queries. This limitation mirrors a broader gap in RAG benchmarking, as existing frameworks similarly neglect multimodal interplay despite its growing practical relevance. Future iterations will expand this work to incorporate multimodal evaluation, testing how graph-bases retrieval and reasoning mechanisms generalize hybrid knowledge representations while preserving contextual fidelity across data types.
33

---

Published as a conference paper at ICLR 2026
K BROADER IMPACT
Our work introduces a paradigm shift how to comprehensively evaluate GraphRAG systems, with broader implications for AI's role in knowledge-inlensive domains such healtheare, legal analysis, and scientific research. By rigorously assessing not only the outputs but also the structural and procedural integrity of knowledge representation and reasoning, our benchmark advances the development of AI systems capable of contextually grounded, logically coherent moblem--ollving This progress addresses critical gap in current Al evaluation methodologies, which often prioritize superficial fluency over semantic and causal fidelity, thereby risking the deployment of systems that generate plausible ungrounded or fragmented insights.
From technical perspective, our framework establishes precedent for holistic evaluation, encourag- ing the community to move beyond answer-centric metrics and instead prioritize the traceability of reasoning processes. This shift could catalyze innovations in graph-based knowledge representation, fostering models that explicitly encode domain hierarchies, causal relationships, and contextual dependencies. capabilities essential for real- world applications like clinical decision support policy analysis. For instance, by evaluating how faithfully a system traverses medical guideline graphs synthesize treatment recommendations, dowain-aware AI, reducing reliance on opaque black-box reasoning.
approach incentivizes the development of reliable,
L THE USE OF LARGE LANGUAGE MODELS (LLMs)
In the preparation of this manuscript, we used a large language model as a writing assistant. Its main role help improve our English writing, such as correcting grammar and refining sentences for clarity and style. Additionally, was used to help set up the initial format for several tables. The authors made all final decisions on the content, carefully checking and editing all suggestions from the model to ensure the scientific accuracy and integrity of this work.
34


Reviews:
[{"review_id": "0QMeZSHJi5", "reviewer": "Reviewer_zK5v", "summary": "This paper discusses and tests the advantages and disadvantages of GraphRAG compared to vanilla RAG. To achieve this goal, the authors discusses the limitations of existing benchmarking datasets for GraphRAG/RAG-related systems and propose GraphRAG-Bench, a benchmark on knowledge retrieval and contextual reasoning. GraphRAG-Bench not only focus on testing a model’s ability to correctly retrieve necessary information, but also tests the model’s ability to reason in complex scenarios. It contains 4 types of tasks: Fact Retrieval, Contextual Summarize, Complex Reasoning and Creative Generation. The process of constructing the dataset is also introduced, with 6-steps - Corpus collection, Logical Mining, Evidence Collection, Question Generation, Check, and Refinement. With this benchmark, authors systematically checked the condition when a series of GraphRAG works.", "questions": "1.\tThis reviewer is interested to know the reason why some GraphRAG systems, such as ToG, ToG2.0, GNN-RAG, SubgraphRAG are not taken into comparison.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1.\tThis paper studied a fundamental and important problem. GraphRAG is trending, demonstrating good motivation. The proposed benchmark provides rooms to systematically examine the advantage of different GraphRAG systems.  \n\n2.\tThis manuscript identifies key limitations of existing RAG benchmarking datasets, including neglecting the evaluation of logical reasoning, limited corpus coverage, and focusing only on end results.\n\n3.\tApart from general QA, the proposed benchmark includes a set of tasks, including Fact Retrieval, Contextual Summarize, Complex Reasoning and Creative Generation, which allows users to comprehensively evaluate retrieval and reasoning capabilities of RAG systems from different perspectives.", "weaknesses": "1.\tThe dependence of GraphRAG on LLM ability. The authors tested two models (GPT-4o-mini and Qwen2.5-14B) on the benchmark. However, the analysis of how GraphRAG depends on the ability (size) of LLMs is missing. It would be nice if the author could give some analysis on the minimum size for a successful GraphRAG.\n\n2.\tSome of the multi-hop QA datasets, which contains QA pairs with various question types and different difficulties, such as CWQ, MuSiQue, and 2WikiMultihopQA, are not discussed in this manuscript. \n\n3.\tThis paper lacks a concrete explanation of how the raw text corpus is organized into an ontology by GPT-4.1. This task can be challenging, hence one may need to recognize entities and relations, and then perform necessary disambiguation procedures. \n\n4.\tThe proposed benchmark does not evaluate the efficiency of graph construction. Ignoring graph construction efficiency is justifiable only if the constructed graph can be reused across a wide range of scenarios.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6BwT83wfIG", "reviewer": "Reviewer_dvvB", "summary": "This paper systematically investigates when graph-based retrieval augmentation (GraphRAG) surpasses traditional RAG, revealing that GraphRAG excels in complex reasoning and structured knowledge integration while RAG remains more efficient for simple retrieval tasks.", "questions": "As discussed above in weakness.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "This work tackles the key open question of when graph-based retrieval truly benefits RAG systems, bridging a major gap in empirical understanding. Through dense, well-controlled experiments across domains and tasks, it provides strong, data-driven evidence supporting its conclusions and design insights.", "weaknesses": "1. The proposed four-level task hierarchy is claimed that task difficulty increases along the retrieval difficulty and reasoning complexity. However, this paper do not provide formal or operational definitions for these levels—there are no explicit thresholds for evidence quantity, reasoning steps, or context length that determine the boundaries between levels.\n\n2. The benchmark does not disclose the absolute corpus size for the Novel and Medical datasets. Consequently, it remains unclear how GraphRAG’s performance scales with corpus size. Without controlled experiments across small, medium, and large datasets, the current results cannot confirm whether the observed improvements stem from the graph structure itself or from differences in data volume and density.\n\n3. The paper does not empirically analyze how different indexing contents—such as entities, relationships, phrases, or communities—affect GraphRAG performance. Although Table 15 summarizes the indexing types used by various implementations, these differences are only descriptive and intertwined with other design factors like retrieval strategies and graph density.\n\n4. The paper does not evaluate GraphRAG on time-sensitive or temporally evolving datasets. The current study’s exclusive focus on static corpora (medical guidelines and classical novels) means it cannot assess how graph-based retrieval performs under temporal drift or continuous knowledge refresh.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "y2bSVNmto2", "reviewer": "Reviewer_ebWh", "summary": "This paper proposes GraphRAG-Bench, a comprehensive benchmark for systematically evaluating GraphRAG models in terms of hierarchical knowledge retrieval and deep contextual reasoning. \n\nThis benchmark covers both tightly structured domain data (medical data from NCCN guidelines) and loosely organized texts (novels from Gutenberg library), and focuses on 4 tasks with different difficulty levels, from easy to difficult, including fact retrieval, complex reasoning, contextual summarize, creative generation. This paper provides an evaluation framework with clear quantitative metrics that assesses graph construction, retrieval, and generation performance. \n\nThis paper conducted extensive experiments to evaluate multiple GraphRAG methods against RAG, with GPT-4o-mini and Qwen-2.5-14B as base LLMs. The experiments show that GraphRAG outperforms RAG in complex reasoning, summarization, and creative tasks, while it does not hold an advantage in simple retrieval. Based on the experimental results, the paper offers practical guidelines, indicating that GraphRAG is suitable for multi-hop reasoning and knowledge integration tasks, while it is not applicable for single-step retrieval or latency-sensitive scenarios.", "questions": "Refer to Weaknesses.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This benchmark focuses on tasks with different difficulty levels, reflecting real-world scenarios demanding complex logical synthesis.\n- This benchmark provides a comprehensive evaluation for GraphRAG with clear quantitative metrics, including graph structure, retrieval performance, efficiency, and final output quality.\n- Based on thorough experiments across multiple GraphRAG methods, LLMs, and tasks, this paper offers clear and practical guidelines on when GraphRAG outperforms traditional RAG.", "weaknesses": "- The calculation process for some evaluation metrics is simplistic and unclear. For example, the paper does not explain how it determines whether a claim $c$ is supported by the context $C$ when calculating EVIDENCE RECALL and FAITHFULNESS.\n- The process of Logic and Evidence Extraction is not clearly described either in the main text or the appendix. While the use of GPT-4.1 is mentioned, the specific extraction procedure and output format are not illustrated clearly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ohgmeWN6VD", "reviewer": "Reviewer_bvTB", "summary": "This paper systematically investigates when and why graph structures benefit retrieval-augmented generation systems, introducing **GraphRAG-Bench** to evaluate GraphRAG models across hierarchical knowledge retrieval and deep contextual reasoning, and providing empirical insights and guidelines for their practical application.", "questions": "As discussed above in Weakness.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1. This paper addresses the key research question of when graph structures truly improve RAG performance.\n2. This paper provides extensive experimental evidence through the new GraphRAG-Bench benchmark.\n3. This paper offers concrete recommendations and design principles for advancing the GraphRAG community.", "weaknesses": "1. **The implementation details of the evaluation metrics are not clearly specified.** It seems that both *Faithfulness* and *Evidence Coverage* rely on external judgment models, such as *LLM-as-a-Judge*, but the paper does not disclose the exact implementation of this component.\n\n2. **The scale of retrieved content among methods appears inconsistent**. According to Section H.2 *Configuration*, the retrieval strategies of RAG and different GraphRAG variants are not aligned—for example, RAG restricts the Top-K to 5. This inconsistency may affect the fairness of performance comparisons.\n\n3. **The potential mechanisms through which *Graph Indexing content* influences various tasks deserve further study**. Statistics in Table 15 show that different GraphRAG strategies employ distinct types of indexed content, such as text chunks, communities, entities, and relationships. For instance, could the inclusion of *community* information better support *contextual summarization* tasks?\n\n4. **How corpus characteristics affect the choice of GraphRAG remains an open question**. Although the paper thoroughly investigates when to use GraphRAG from the perspective of task properties and performance, Sections C.1 and D further discuss the traits of the selected *Medical* and *Novel* corpora, as well as the yet-unexplored *Legal* and *Financial* domains. Extending the study of \"When to use GraphRAG\" to domain-specific corpora would significantly strengthen the work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)