Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation
Authors: Mufei Li
Siqi Miao
Pan Li
Venue: ICLR 2025 Poster (2025)
Keywords: Knowledge Graphs
Large Language Models
Retrieval-Augmented Generation
Retrieval
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Large Language Models (LLMs) demonstrate strong reasoning abilities but face limitations such as hallucinations and outdated knowledge. Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) addresses these issues by grounding LLM outputs in structured external knowledge from KGs. However, current KG-based RAG frameworks still struggle to optimize the trade-off between retrieval effectiveness and efficiency in identifying a suitable amount of relevant graph information for the LLM to digest. We introduce SubgraphRAG, extending the KG-based RAG framework that retrieves subgraphs and leverages LLMs for reasoning and answer prediction. Our approach innovatively integrates a lightweight multilayer perceptron (MLP) with a parallel triple-scoring mechanism for efficient and flexible subgraph retrieval while encoding directional structural distances to enhance retrieval effectiveness. The size of retrieved subgraphs can be flexibly adjusted to match the query's needs and the downstream LLM's capabilities. This design strikes a balance between model complexity and reasoning power, enabling scalable and generalizable retrieval processes. Notably, based on our retrieved subgraphs, smaller LLMs like Llama3.1-8B-Instruct deliver competitive results with explainable reasoning, while larger models like GPT-4o achieve state-of-the-art accuracy compared with previous baselines—all without fine-tuning. Extensive evaluations on the WebQSP and CWQ benchmarks highlight SubgraphRAG's strengths in efficiency, accuracy, and reliability by reducing hallucinations and improving response grounding.

Published as a conference paper at ICLR 2025
SIMPLE IS EFFECTIVE: THE ROLES OF GRAPHS AND LARGE LANGUAGE MODELS IN KNOWLEDGE-GRAPH:
BASED RETRIEVAL-AUGMENTE GENERATION
Mufei Li", Siqi Miao", Pan
Georgia Institute of Technology {mufei. 1i, siqi miao, panli)@gatech.edu
"Equal contribution.
ABSTRACT
Large Language Models (LLMs)) demonstrate strong reasoning abilities face limitations such hallucinations and outdated knowledge. Knowledge Graph (KG)-based Setrieval-Augmented Generation (RAG) addresses these issues by grounding LLM outputs in structured external knowledge from KGs. However, current KG-based RAG frameworks still struggle to optimize the trade-off be- tween retrieval effectiveness and efficiency identifying a suitable amount of relevant graph information for the LLM to digest. We introduce SubgraphR extending the KG-based RAG framework that retrieves subgraphs and leverages LLMs for reasoning and answer prediction. Our approach innovatively integrates lightweight multilayer perceptron (MLP) with parallel siple-scoring mecha- nism for efficient and flexible subgraph retrieval while encoding directional struc- tural distances to enhance retrieval effectiveness. The size of retrieved subgraphs can be flexibly adjusted match the query's 's needs and the downstream LLM's capabilities. This design strikes a balance between model complexity and rea- soning power, enabling scalable and generalizable retrieval processes. Notably, based on our retrieved subgraphs, smaller LLMs like Llama3. ISSttrutt deliver competitive results with explainable reasoning, while larger models like GPT-4o achieve state-of-the-ar accuracy compared with previous baselines-all without fine-tuning. Extensive evaluations the WebQSP and CWQ benchmarks high- light SubgraphRAGr strengths in efficiency, accuracy, and reliability by reducing hallucinations and improving response grounding. Our implementation avail- able at
INTRODUCTION
Large language models (LLMs) have increasingly demonstrated remarkable capabilities (Brown ct al. 2020; Kojima et al. 2022: Wei et al. 2022
Bubeck et al. [023 Yao et al. 2023 Huang & Chang 2023). However, issues like hallucinations (Ji ct al. 2023 Huang et al. 2023 Zhang al.
2023). outdated knowledge (Dhingra et al.) 2022 Kasai et al.) 2023), and a lack of domain exper- tise (Li et al., 2023b) undermine their trustworthiness. Retrieval-augmentte generation (RAG) has emerged as promising strategy to mitigate these problems by grounding LLM outputs in external knowledge sources (Shuster et al., 2021 Borgeaud et al. 2022 Vu et al.) 2024 Gao al. 2024b).
Despite the effectiveness of text-based RAG, graph- structures offer more efficient alternative for organizing knowledge (Chein Mugnier (208). Graphs facilitate explicit representation of relationships, reduce information redundancy, and allow for more flexible updates (Robinson al.) 2015). Recent studies have explored using graph- graph-structured knowledge, particularly knowledge graphs (KGs), as RAG resources (Pan et al. 2024] Peng et al. 2024 Edge al. 2024). However, developing effective and efficient frameworks for KG-based R.G remains limited due to the unique challenges posed by information retrieval from KGs.
Firstly, traditional text retrieval methods, such as BM25 (Robertson et al., 1994 Robertson & Zaragoza 2009) and dense retrieval (Karpukhin al. 2020), are insufficient for supporting L.LMS

---

Published as a conference paper at ICLR 2025
"Which organize limss have business teeesspppp wilh
SubgraphRAG
The answars: (Nvidla, Nasa)
founded respactiely Elan Musk Jifl Gatee founded any
Topic entitt T extractions
Stop 2 Subgraph Gig extraction:
The reasontngg as tollows.
amazon e Oie Baat paei
Step 2.1 Structuraa Festure Construction
Option Directional Distance Encocingg (used)
Nvidla has business
TERLN
portnerships with
it FG e). bidirectional propagationss) bidfeecioonlloogggniorrr
founded Blen
PayPal
& one-hot encodingg of
Musi
Amazon founded by leff Benos
Option (not used)
SPACEK
Fetract Relewant Triples Paraile)
3. Microsoft founded by Bit cases
Microsoft
IILE Iusa
Top k triples in Peftile:)) 4)
Relotiorships
Nvidie was nut founded by Eton Musk, left Beros,
comcany founder
Stop 3- LLMs for reasoning siver Syt
CAERAPT
business partnendip
"Ue
as C
ovidence to answer the question"
Figure I: The SubgraphRAG framework. Retrieved subgraphs are flexible in form and size, consist- ing of triples extracted in parallel. This example subgraph has a complex form beyond a path.
in complex reasoning tasks (Sun al. 2018). For instance, a query like "What is the most famous painting by contemporary of Michelangelo and Raphael?" requires not only retrieving works by their contemporaries but also reasoning about relationships beyond Michelangelo and Raphael themselves. Thus, KG retrieval goes beyond basic entity linking, requiring the extraction of non- local entities connected through multi-hop and semantically relevant relationships to support rea- soning (Jiang et al. 2023a Luo et al.) 2024b] Sun et al. 2024a). Such information is often best represented as KG subgraphs, whose retrieval enables more effective downstream reasoning.
Second, KG-based RAG faces significant computational challenges. In order to handle online queries and adapt to dynamic KG updates (Trivedi et al. 2017; Liang et al. 2024), efficiently iden- tifying relevant structural information while meeting latency requirements is crucial for designing a practical KG-based RAG framework. Efficient approximate methods for nearest neighbor search like locality-sensitive hashing (Indyk & Motwani 1998) are not suitable for structure extraction.
Third, retrieved structural information must cover the critical evidence needed to answer the query while respecting the reasoning capacity of LLMs. Simply expanding the context window increases computational cost and can degrade RAG performance due to the introduction of irrelevant infor- mation (Xu et al. 2024), causing the "lost in the middle" phenomenon (Liu et al.! (024b). To pre- vent these issues, redundant structural information should be pruned to keep only relevant evidence within the LLMs' processing limits, improving accuracy and avoiding hallucinations of LLMs.
Existing KG-based RAG frameworks often struggle to effectively balance retrieval with reasoning over the retrieved data, limiting their ability to address these challenges. For example, many ap- proaches rely on multiple LLM calls for iterative KG search, introducing significant complexity for
each query (Kim et al.] Gao al. 2024a Wang et al.) 2024, Guo et al.) [024] Ma et al.] 2024] [uun et al. 2024a| Jiang et al., [024 Jin et al. [2024] Xiong et al. 2024). Furthermore, the mismatch between large KG search spaces and constrained LLM context windows poses challenges to the coverage of LLM-based retrieval. Conversely, some methods employ lighter models, such Long Short-Term Memory or Graph Neural Networks, for both retrieval and reasoning (Zhang et al.) 2022
Liu al. 2024a Sun al. 2019). Despite being more efficient, these methods constrained by the inferior reasoning capabilities of the lighter models. Additionally, certain approaches retrieve fixed types of subgraphs (e.g., paths) for convenience (Zhang et al. 2022 Luo et al., 2024b), which restricts the coverage of critical evidence for LLM reasoning-2 point we explore further in Sec. 3.]
Design Principles We argue that there is inherent tradeoff between model complexity and rea- soning capability. To effectively search over KGs, which are expected to grow rapidly, knowledge retrievers should remain lightweight, flexible, generalizable, and equipped with basic reasoning abil- ities to efficiently filter (roughly) relevant information from vast amounts of irrelevant data while delegating complex reasoning tasks to LLMss As LLMS continue to demonstrate increasingly so- phisticated reasoning capabilities and are likely to improve further, this division of labor will be- come more sensible. As long as the retrieved information fits within the LLM's reasoning capacity, LLMMs-levrrggigg their superior reasoning power-can then perform more fine-grained analysis and provide accurate answers with appropriate prompting. This approach extends the two-stage Re- call & Ranking (rough to fine) framework commonly used in traditional information retrieval and recommendation pipelines, with the key advance being that each stage is paired with appropriately tiered reasoning capabilities from AI models to meet the demands of answering complex queries.
2

---

Published as a conference paper at ICLR 2025
Besides, for questions proven too challenging, the concept of iterating the above process can be adopted. However, this consideration is beyond the scope of the current work.
Present Work Our KG-based RAG framework, SubgraphRAG (Fig. D. follows pipeline that first retrieves a relevant subgraph and then employs LLMs to reason over it. While this approach mirrors some existing methods (He al. 2021 Jiang et al. 2023b), SubgraphRAG introduces novel design elements that significantly improve both efficiency and effectiveness by adhering to the aforemen- tioned principles. For efficiency, we employ lightweight multilayer perceptron (MLP) combined with parallel triple-scoring for subgraph retrieval. To ensure effectiveness, we encode tailored struc- tural distances from the query topic entities as structural features. This enables our MLP retriever to outperform more complex models, such as GNNs, LLMs. and heuristic searches, in terms of covering the triples and entities critical for answering the query while maintaining high efficiency. Additionally, the retrieved subgraphs have flexible forms, with adjustable sizes to accommodate the varying capacities of L.LMs. SubgraphRAAG employs unfine-tuned L.LMs, maintaining generaliza- tion, adaptability to updated KGs, and compatibility with black-box LLMs.
We evaluate SubgraphRAG on two multi-hop knowledge graph question answering bench- marks-WebQSP and CWQ, Remarkably, without fine-tuning, smaller models like Llama3.1-81 Instruct can achieve competitive performance. Larger models, such as GPT-4o, deliver state-of-the- art results. Furthermore, SubgraphRAG shows robust multi-hop reasoning capabilities, excelling on more complex, multi-hop questions and demonstrating effective generalization across datasets de- spite domain shifts. Ablation studies on different retrievers highlight the advantage of our retriever, which consistently outperforms baseline retrievers and is key to our superior KGQA performance. Additionally, our method also exhibits substantial capability of reducing hallucination by generat- ing knowledge-grounded answers and explanations for its reasoning.
2 PRELIMINARIES
A KG can be represented set of triples, denoted by G [(h, h, E,r R}. where & represents the set of entities and R. represents the set relations. Each triple denoted by r = (h, r, () characterizes fact that the head entity h and the tail entity / follow directed relation r. In practice entities and relations are often associated with a raw text surface form friendly for LLM reasoning.
KG-based RAG enhances LLMs with KG knowledge. Given a query q. LLMs can access relevant knowledge represented by triples in the KG to address The challenge lies in efficiently searching for relevant knowledge within the often large-scale KG and reasoning for accurate prediction.
Entity Linking is often the first step in KG-based RAG, whose goal is to identify the set of entities Ty C directly involved in the query q. The entities in Tg. named topic entities, provide valuable inductive bias for retrieval as the triples relevant to g are often close to Ty-
Knowledge Graph Question Answering (KGQA) is an important application often used evalu- ate KG-based RAG, where the query q is a question that requires finding answers that satisfy certain constraints. The answer(s) A, often correspond to a set of entities in the KG. Questions that re- quire multiple triples in the KG as evidence to identify an answer entity are classified as complex questions, as they demand multi-hop reasoning, in contrast to single-hop questions.
3 THE SUBGRAPHRAG FRAMEWORK
Our proposed framework, SubgraphRAG, adopts
pipeline for KG-based
RAG. Specifically, given a query q, SubgraphRAG first extracts subgraph Gg C 9 that represents predicted relevant knowledge, and then LLMs generate the response by reasoning over Gg While some existing works adopt a similar pipeline (Zhang et al. 2022 Luo et al. 2024b; Wu et al. 2023 Wen et al. 2023), our approach is novel in addressing three key challenges: First, the extracted sub- graph 9g is designed to comprehensively cover the relevant evidence for answering while adhering to size constraint K, which can be adjusted according the capacity of the downstream LLM. Second, the extraction of G, is highly efficient and scalable. Third, we employ tailored prompting to guide the L.M in reasoning over 9g and generating grou-ggo answer with explanations.
3

---

Published as a conference paper at ICLR 2025
3.1 EFFICIENT, FLEXIBLE AND EXPRESSIVE SUBGRAPH RETRIEVAL
Problem Reduction We formulate the subgraph retrieval problem and reduce it to an efficiently solvable problem. An LLM models P( q) that takes queries and evidence subgraphs. Given a query a and its answer Ay. the best subgraph evidence for this LLM is denoted as 9 arg ca P(A, g). However, solving this problem is impossible as it requires the knowl- edge of A.. Instead, we aim to learn a subgraph retriever that generalizes to unseen future queries.
Specifically, let the subgraph retriever be a distribution Qa(-ly: 9) over the subgraph space of the KG, where / denotes the parameters. Given a training set of question-answer pairs D, the subgraph retriever learning problem can be formulated as the following problem:
max E(g.A,))DD. Qu IP(A, Gq. 2).
This problem is hard to solve due to the complexity of the L.M (IP). Evaluating P(A, | 9g. g) requires the LLM to generate the particular answer Aq. which can be costly and is only applicable to grey/white-box LLMs with accessible output logits, let alone the non-computable gradient AF z5,"
To solve the problem in Eq. [ adopt the following strategy. If we know the optimal subgraph Gg. the maximum likelihood estimation (MLE) principle can be leveraged to train the retriever maxe Big_A,)>~ Q9 | G,u). However, getting 9 for an even known question-answer pair (q, Ag) is computationally hard and I.L.-dependent. Instead, we use (g, Ag) to construct surrogate subgraph evidence with heuristics ((q,AA) and train the retriever based on MLE:
max (OAA)) (G, 9, where 9g = -((g. Ag). 9
An example Gy the shortest paths between the topic entities Ty and answer entities Ay. Eq. conceptually similar to the weak supervision in Zhang et al.] (2022). However, the formulation in Eq. D iiniicaees that the sampled subgraph does not necessarily follow fixed type like path. Instead, the retriever distribution Qo can by construction factorize into a product of distributions over triples, allowing efficient training and inference, flexible subgraph forms, and adjustable subgraph sizes.
Triple Factorization We propose to adopt retriever that allows subgraph distribution factoriza- tion over triples given some latent variables 2, 2= q) (to be elaborated later):
Qo 99 9,9)
II (T 2r, 9) II (1 po (r (2, 9)) T
(3)
rEgiG,
This strategy is inspired by the studies on graph generative models Kipp & Welling 2016) and enjoys four benefits: Efficiency in Training- The problem Eq. 2 can be factorized as maxç (A)WD Erea, log Po(p 27,q) + Eregre log(1 pa(T 2r, 4)); Efficiency in Sam- pling After computing 2r, we can select triples from G in parallel: Flexibility Triple combi- nations can form arbitrary subgraphs; Adjustable Size Subgraphs formed by top- triples with different K values can accommodate various LLMSs with diverse reasoning capabilities. In practice, Qo can be further simplified given topic entities Ty (He et al.) 2021 Jiang et al., 2023b), by only considering subgraphs close to the topic entities, i.c., Pe(T 2r. for 7 that is far from Ty.
Relevant Designs Previous approaches often adopt heuristics and focus on particular subgraph types, such as constrained subgraph search (e.g., connected subgraphs (He et al. 2024)), ), constrained path search from topic entities, often imposing constraints in path counts and lengths (uoo et al., 2024b Mavromatis & Karypis 2024) or employing potentially expensive iterative processes (Zhang et al. 2022, Wu et a [2023; Sun et al., 2024a Liu et al.) 2024a] (unn et al., 2024b), and entity se- lection followed by extracting entity-induced subgraphs, where all triples involving an entity are included together (Yasunaga et al. 2021; Taunk et al.] 2023). The loss in flexibility narrows the space of possible retrieved subgraphs, which eventually harms the RAG effectiveness.
Directional Distance Encoding (DDE) zr (G, q) models the relationship between triple and the query 9 given G. Graph neural network (GNN) is one option for this modeling purpose through message passing with entity, relation, and question embeddings (Yasunaga al. 2021 Kang al.)
2023 Mavromatis & Karypis 2024 Liu et al. 2024a). However, it faces fundamental limitations in
its representation power (Xu al. 2019 Murphy al. 2019
Morris et al. 2019 Chen al. 2020
4

---

Published as a conference paper at ICLR 2025
System: Bosed on the tripies retrieved from knowledge graph, please onswer the question. Please return formatted onswers list, eoch prefived with "ons""
User: Triplets: (5y. <g) \/ (fa- (g) 'n....
Question....
// //L example
Assistant: the question, have find From triples that Therefore, the answers // onz....
// /LLL example
User: Triplets: (fa. Tap- (s) (ee Ted eg) In 'n
Question:
// the cvoluction question
Assistant: onswer the question, hove to find.... From the triples that Therefore, the onswers are: // ons:
// onswer
Figure 2: The prompt used in SubgraphRAG. Concrete examples can be found in Appendix
One critical limitation is its inability to compute distances between entities and topic entities [Li
et al. 2020), which can be particularly problematic for multi-hop question answering.
The structural relationship between and q provides valuable information complementing their semantic relationship. Inspired by the success of distance encoding and labeling trick in enhancing the structural representation power of GNNs (Li et al.. 2020 Zhang et al., (02T), we propose a
DDE as z- z-(9, q) to model the structural relationship. Given topic entities Ty. let se be a one-hot encoding representing whether E Ty or not. For the 1+ 1-th round, we perform feature propagation
and compute se ((+1) = ( (1) | (e',,,e) E g}. and through the reverse direction to account e for the directed nature of 9. s (., = MEAN(s (5:)) | (e,,,e) E 9). where s (rr.)) = =s!)) We |s. ( leading to triple encodings 2 (9. q) = [s/]|s that concatenate the encodings of h and f. In section 4.1 we
concatenate all results to obtain the final entity encodings se = [s. ||s
compare different approaches to compute 2- 9) using GNNs, DDE, or only one-hot encodings of Ty. and DDEs perform the best. See Appendix A for additional illustrations and discussions.
A Lightweight Implementation For (o(-l=-(G,9)) We present lightweight implementation of Po that integrates structural and semantic information. Following previous approaches (Karpukhin jet al. 20200) Gao et al. 2024b), we employ off-the-shelf pre-trained text encoders to embed all enti- tiss/rlaatins in a KG based on their text attributes. These semantic text embeddings are computed and stored in a vector database during the pre-processing stage for efficient retrieval. For a newly arrived question q, we embed g to obtain 2q and retrieve embeddings Zr. 2t from the vector database for the involved entities and relations. After computing DDEs 2,, an MLP employed for binary classification using the concatenated input [=g]|>a|| ||-/||==||||]]
Relevant Designs We considered several alternatives but found them less preferable due to concerns about efficiency and adaptability to KG updates. Cross-encoders, which concatenate question and a retrieval candidate for joint embedding (Wolf et al. 2019), potentially offer better retrieval performance. However, due to the inability to pre-compute embeddings, this approach significantly reduces retrieval efficiency when dealing with a large number of retrieval candidates, as is the case in triple retrieval. Li et al. (2023a) embeds each triple whole rather than individual entities and relations. However, this approach incurs higher computational and storage costs and exhibits reduced generalizability to the triples that are combinations of old entities and relations. Our lightweight implementation allows for fast triple scoring while maintaining good generalizability.
3.2 PROMPTING- BASSED LLM REASONING
We employ an LLM reason over G. by incorporating the linearized list of triples forming G, into the prompt. This approach allows the LLM to ground its reasoning in the retrieved structured knowl- edge and select the answers A, from the entities within G.,. mitigating issues like hallucinations and outdated knowledge (Lin et al. 2019 Shuster et al., 2021 Vu et al. 2024). Specifically, we prompt explanations besides the answers, which further effec- tively reduces hallucinations. We adopt in-context learning (ICL) (Brown et al., 2020) and design dedicated prompt templates with explanation demonstrations to guide LLM reasoning (see Fig. 2)
the LLM to generate
Without fine-tuning L.MMs, we reduce computational costs, enable the usage of state-of-the-ar (SOTA) black-box LLMs, and maintain the generalizability for unseen KGs. Fine-tuning also risks enhancing prediction accuracy at the cost of general reasoning and explaining capabilities. High- quality text- explanations are typically unavailable for question-answerti a result, previous KG-based RAG approaches that rely on fine-tuning often depend on larger, unfine- tuned LLMs like GPT-4 to generate explanations additional training data (Luo al. 2024b).
tasks in practice. As
Regarding the size K of the retrieved subgraph, while increasing K in principle improves the cov- erage of relevant information, it also risks introducing more irrelevant information that may harm
5

---

Published as a conference paper at ICLR 2025
Table 1: Evaluation results for retrieval recall and wall-clock time. Best results are in bold. Being training- faeei, cosine similarity and G-Retriever stay unchanged in generalization evaluations.
Wedrgs
CWg
CWQ-WWapp
Wetrges-ccoo
Toglcs
Extile
Tigles
Exiles
Triple
Eatiics
Tryfes
Stcrtor Iaads
CIPT-4
Assincr
Time (s)
Staetcal
GPT-46
Assiver
Tire (V)
Shartion Paah
CIPT-44
Shoricel
CPE-44
Acawer
ariiantty
0,234
0.219
11:118
3
01567
11:592
113
0:711
0.711
13.70h
0.48%
0.367
11.522
SateNAS in 123:
0:457
0.334
11/117
133
Retiive
Rowile-Aawe
0.855
0.507
945
0.688
0.500
0.841
5327
0..00
0.223
0.635
0.301
0.330
0.412
G.Resiceer
0.234
0.855
B.54
073
0.158
0.917
0:803
1339
0.201
0.225
0:345
0.158
0.217
0.570
RAN
0.328
8406
0.51
68
0.500
0.586
0841
100
0.440
0.361
0.891
0.484
0.301
0.000
0.905
0.811
0.840
0.014
12
0.982
0.023
Table 2: Breakdown of recall evaluation over # hops. Best results are in bold.
WVKgSP
(W)
Wingsp
CWU
CWQ
(35,05))
(55.855
(34.25)
(25.075)
(67.575)
(34.275)
(38 075)
(74)
(6.37%)
COnTR similatty
0.874
0--135
0.50M
0.412
0.353
0.587
0.155
0..33
0.311
0.854
0:315
01:33
11.913
0.:72
0.289
SR+NSM
0.563
0.322
0.500
0.376
0.916
00:311
RoG
0.991
0.415
9.706
0.257
0.558
0.466
0.271
0.344
01233
0.120
0.874
0.077
0.827
0.025
G Retricess
0,333
0.216
0.184
0.250
6.368
0.345
0.264
0.150
0.210
0.226
0.506
0.486
0.257
0.254
0,553
GNN RAG
0.622
0.502
0.315
0.458
0.446
0.384
0.445
0,225
0.015
0.418
0,530
0.311
01553
0.841
0.787
MIP rnpicrsiit
0.911
0.755
0.551
0.750
0.300
0.775
0.788
0.511
0.876
0.815
0.056
[1883
11913
01953
0.748
0.531
0.820
0.626
11908
0. 0,609
0.823
0.400
0.753
0,877
0.881
01310
0.918
07711
LLM reasoning (Xu et al. different sized context windows and also exhibit distinet capabilities in reasoning over long-context retrieval results (Dubey et al. 2024). As such, although the training of SubgraphR retriever is LLM-agnostic. the size K needs to be properly selected per LLM and constraints. In Section 4.2 we
2024
Liu et al., 2024b). Different L.MSs are inherently equipped with
empirically verify that more powerful LLMs can benefit from incorporating a larger-sized retrieved subgraph, demonstrating the benefit of size-adjustable subgraph retrieval in SubgraphRAG.
EXPERIMENTS
We design empirical studies examine the effectiveness and efficiency of SubgraphRAG in addressing the various challenges inherent to KG-hased RAG, covering both retrieval and reasoning aspects. Q1) Overall, to meet the accuracy and low-latency requirements of KG-based RAG, does SubgraphRAG effectively and efficiently retrieve relevant information? Q2) For complex questions involving multi-hop reasoning and multiple topic entities, does SubgraphRAG properly integrate structural information for effective retrieval? Q3) How effectively does SubgraphRAG perform on KGQA tasks, and how is its accuracy influenced by different factors? Q4) To what extent can our pipeline provide effective noowledge-groundee explanations for question answering?
Datasets. We adopt two prominent and challenging KGQA benchmarks that necessitate multi-hop reasoning WebQSP (Yih et al., 2016) and CWQ (Talmor & Berant 2018). Both benchmarks utilize Freebase (Bollacker et al. 2008) as the underlying KG. To evaluate the capability of LLM reasoners in knowledge- showledge-grounded hallucination-free question answering. we introduce WebQSP-sub and CWQ-sub, where we remove samples whose answer entities are absent from the KG.
4.1 EVALUATION FOR RETRIEVAL (Q1 & Q2)
Baseline Retrievers. Cosine similarity performs structure-free triple retrieval based on triple embeddings (Li et al.] 2023a). SR+NSM w E2E (Zhang et al.] 2022) and Retrieve-Rewrite- Answer (Wu al. (202)) propose constrained path search based on pre-trained text encoders.
RoG (Luo et al. (024b) adopts a similar strategy but fine-tunes an LLM for relation path prediction. G-Retriever (He al. 2024) combines cosine similarity search with combinatorial optimization to
construct a connected subgraph. GNN-RAG employs lightweight GNNs to predict answer entities and extracts shortest paths between them and topic entities (Mavromatis & Karypis 2022 2024).
Implementation Details. We adopt gte-large-en-vi.5 (Li et al. 2023c) as the pre-trained text en- coder for both the cosine similarity baseline and SubgraphRAG, a 434M model that achieves good balance between efficiency and English retrieval performance, evidenced by the Massive Text Em- bedding Benchmark (MTEB) leaderboard (Muennighoff et al. (023). ). For supervision signals, there no ground-trute relevant subgraphs for query q. Previous path-based subgraph retrievers adopt the shortest paths between the topic and answer entities for the weak supervision signals (Zhang et al. 2022
Luo al. 2024b). We also utilize these shortest paths as the heuristic relevant sub-
6

---

Published as a conference paper at ICLR 2025
Sharteet Trol faca!
GFT-4o
frower Cretty
2.0
1.00
1.00
1.90
3.10
2.70
1.70
2.40
3.60
3.50
1.00
2.50
0.50
3.40
230
0.00
1.00
220
0.30
76
TCB
2013
4000
74
TE0
332
1040
110 735
2007
M6.7
NLP arillt
G-fatrlew
GHIMAAG
GrepisAdE sapi wriitt
Fel
Figure 3: Retrieval effectiveness on CWQ across a spectrum of K values for top-K K triple retrieval.
graphs Gg to train Qe as in Eq. 2 To reduce the size of candidate triples G. we construct subgraphs centered at topic entities Tq. following previous works. See Appendix C.1 for more details.
Evaluation Metrics. Our evaluation covers both effectiveness and efficiency. For effectiveness, we employ three recall metrics, respectively for short- est path triples (i.e., 9g), GPT-44-ideenified rele- vant triples, and answer entities within retrieved sub- graphs. The first metric assesses the retrieval of weak supervision signals used for training. For a more accurate assessment, we employ GPT-4o to identify s 20 relevant triples for all test samples, po- tentially capturing relevant triples beyond the short- est paths (see Appendix B for labeling details). We calculate individual recall values per question and then take the average. For efficiency, we measure wall clock time on a 48GB NVIDIA RTX 6000 Ada GPU in seconds. To focus on model computational efficiency, we exclude KG query time. For text em- bedding, employed by SubgraphRAG variants and most baselines (cosine similarity, G-Retriever, and GNN-RAG), we only count the time for question embedding the entity and relation embeddings for the entire KG can be pre-computed. Such pre- computation may yield time-memory trade-off.
Table 3: Question-answering performance on WebQSP and CWQ. Best results are in bold. By default, reasoners use the top 100 retrieved triples. Results with 200 and 500 triples (indicated in parentheses) shown. Results with (++) evaluate retriever generalizability, where the retriever is trained on one dataset and applied to the other.
also
WengSP
CWQ
Hii
Macn-Fi
UBIKGOA
72.2
43.0
SRINSM EZE
45.8
(GPT-)
74.00
Renieve Ansker
79.36
G Remrever
53.41
73 13:40
RoG-Kaa
20.26
66.67
54.6
61.91
RoCS-Sep
06.45
$2.19
59887
04.55
$2.6
SubgraphiRGG Lianá 1-88
70.57
86.61
47.16
$6.98
SuppraghRAG: AG Lame3 1-700
76.20
50
51.76
57.00
Sungraghtt AG ChatiPT
04.21
K3
89.13
56.27
SubgraptR.Gi
77:45
00.11
54.13
Sulgreghhi
SubgnaphiaG GPTHomini (200)
77.82
00.54
$4.89
63.40
SutgraghitA: AG GPTTo (200)
26.26
9811
51.42
67.09
SubgsaghB.AG GPTio-miti (500)
77.67
91.22
55.41
64.97
GPTo
58.04
SubgraphikaG GPTio-mini (+6, 500g
76.20
01
50.30
60.00
Sutgraghitk:: AG GiFTio (10)
74.27
87 17.10
$7.55
54339
Overall Evaluation (Q1). Table [ shows that Sub-
SubgnapABAGG GPT (+i, 5000
77.12
89.65
55.14
62.98
graphRAG consistently outperforms all other ap- proaches in retrieval effectiveness. RoG achieves the most competitive baseline performance for retrieving shortest path triples. However, when evaluated formance drops significantly (45.6% for WebQSP, 52.2% for CWQ), while SubgraphRAG remains robust (2.0% decrease for WcbQSP 3.6% increase for CWQ). This stark difference, despite using the same training signals, empirically validates that SubgraphRAG's individual triple selection mecha- nism allows for more flexible and effective subgraph extraction compared RoG's s constrained path search approach. Regarding efficiency, SubgraphRAG is only slightly slower than the cosine simi- larity baseline on WebQSP while being one to two orders of magnitude faster than other baselines, including GNN-RAG. For the cosine similarity baseline and SubgraphRAG, we report recall met- rics based on the top-100 retrieved triples (2.3% of total candidate triples on average). This budget consistently yields robust reasoning performance across LLMs in subsequent experiments.
GPT-4o-labeled triples, RoG's per-
Generalizability. We further examine the generalizability of the retrievers by training them on dataset A and evaluating dataset B, denoted as A - B in Table I Despite anticipated performance degradation, SubgraphRAG consistently outperforms the alternative approaches.
Ablation Study for Design Options and Retrieval Size. To evaluate individual component tributions, we conduct an ablation study with several variants. MLP is a structure-free variant employing only text embeddings. We consider GraphSAGE (Hamilton et al., 2017). popular
7

---

Published as a conference paper at ICLR 2025
Mecro-
Mocro
Upper Bound
Upper Bound GPT40-mini
Ulppr Bound GPT44-min
Upper Bound
GPT4c-miri
GPT40 mini
Llama3.1 1888
Llame8.1 1.88
Liame5. 1:88
Liamas. 1:88
62-
65-
100
200
100
202
130
400
505
530
Number Triples
Number Triples
Number Triples
Number Triples
(a) WebQSP-sub
(b) CWQ-sub
Figure 4: Ablation studies on the number of retrieved triples used in LLM reasoners.
GNN, to update entity representations prior to the triple scoring. We further augment both MLP and GraphSAGE with a one-hot-encoding topic entity indicator (MLP- + topic entity and GraphSAGE topic entity). Following Sun et al. (2018), we incorporate Personalized PageRank (PPR) (Haveli-
wala 2002). seeded from the topic entities, to integrate structural information (MLP + topic entity PPR). To account for both the varying capabilities downstream LLMs and inference cost/latency
constraints, we evaluate these variants across broad spectrum of retrieval sizes (K).
Fig. presents the results on CWQ, with baselines included for reference. Larger retrieval sizes uniformly improve recall across all variants. Equipped with DDE, SubgraphRAG outperforms other variants, even at the relatively small average retrieval sizes of the baselines. This demonstrates that SubgsaphRAG's superiority is not solely attributable to larger retrieval sizes. Regarding design options, the topic entity indicator invariably leads to an improvement. In contrast, GNN variants often result in performance degradation compared to their MLP counterparts. We suspect that the diffusion of semantic information introduces noise in triple selection. Finally, PPR fails to reliably yield improvements. For the results WebQSP, see Appendix
C.3
which are also consistent.
Multi- Hop and Multi- Topic Questions (Q2). To evaluate the effectiveness of various approaches in capturing structural information for complex multi-hop and multi-topic questions, we group ques- tions based on the number of hops and topic entities. Table 2 presents the performance breakdown by hop count. SubgraphRAG consistently outperforms other methods on WebQSP and achieves the best overall performance on CWQ. Notably, while the cosine similarity baseline and RoG demon- strate competitive performance for single-hop questions, their performance degrades significantly for multi-hop questions. Appendix C.4 provides performance breakdown for single-topic and multi-topic questions, focusing exclusively on CWQ due to the predominance of single-topic ques- tions in the WebQSP test set (98.3%) SubgraphRAG consistently exhibits superior performance across all metrics for both single-topic and multi-topic questions. Our comprchensive analysis high- lights the remarkable effectiveness of DDE in capturing complex topic-centered structural informa- tion essential for challenging questions involving multi-hop reasoning and multiple topic entities.
4.2 KGQA RESULTS (Q3 & Q4)
KGQA Baselines. Besides the baselines in retrieval evaluation, we include more KGQA methods, including UniKGQA Tiang et al. 20235), KD-CoT (Wang et al. 2023a), StructGPT (Jiang et al. 2024a). For RoG, we present two entries: RoG- 2023a), ToG (Sun et al. 2024a). and EtD (Liu et al.
Joint and RoG-Sep. Onginally, RoG fine-tuned its LLMs on the training sets of both WebQSP and CWQ. Yet, this joint training approach leads to significant label leakage, with over 50% of WebQSP test questions (or their variants) appearing in CWQ's s training set, and vice versa. Therefore, we re-trained RoG on each dataset separately, denoted by RoG-Sep.
Evaluation Metrics. Along with the commonly reported Macro-Fl and Hill we also include Micro- Fl to account for the imbalance in the number of ground-truth answers across samples and Hit@) for a more inclusive evaluation. To further assess model performance, we introduce score,, inspired by (Yang et al., 20044) which evaluates how truth- south-groudedd the predicted answers are and the degree of hallucination. This metric penalizes hallucinated answers while favoring missing answers over incorrect ones. Scores are normalized truth-grounding in the model's answers. The scoring strategy is described in detail in Appendix D Experiment Settings. We use the vllm (Kwon al., 2023) framework for efficient LLM inference. For UniKGQA, KD-CoT,
range of O to 100, with higher scores indicating better
Rewiite--wwtttennwwweee ToG, and EtD, reference their published
'The baselines claim to report Hit@ 1, but they actually compute Hit, which measures whether at least one correct answer appears in the L.M response.
8

---

Published as a conference paper at ICLR 2025
performance on WebQSP-sub and CWQ-sub. Best results are in bold. By default, our reasoners use the top 100 retrieved triples. Results with 200 and 500 triples (indi- cated in parentheses) are also shown. Results with (++) evaluate retriever generalizability, where the retriever is trained on one dataset and applied to the other.
Table
WetQSP sab
CWQ.sib sib
Macro-011
Micre-I11
Scotty
Macm-HI
Micm-
Scoms
SK-NSM EZE
37.15
68.63
64.44
G G-Risrioeer
54.13
23:84
74.52
67.56
67.97
Rots Jotee
72.81
47:70
03.50
$2.02
76.13
58.61
52.11
66:33
61.17
15.15
RoCi-cp
67.94
43.10
$4.00
77.61
72.79
$7.00
52.83
64.64
00.64
$4.51
SubgmajoRAG
59.69
51.64
85.57
ChaGPT
70.81
44.73
55.15
$0.82
31.55
56.50
64.40
00.99
61.31
SahgmjoRAt: GPIio-mi
76.36
18.33
131
12.21
70.01
65 4K
64.20
SabgnylRRAG a GPT4
77.61
56.78
91.44
$5.40
$1.55
65.99
63.18
73.91
68.39
66.57
SibgrijoRAi GPTS (200)
78.66
50.65
51.73
$7.14
81.58
01.50
57.47
71.45
65.87
63.65
SabgrajdRAGG
(200)
58.91
82:46
66.48
61.30
75.14
09:42
SyhgmjeRAA Gor-o-mi (500)
37.05
$1.95
50.56
06:57
SahgmpoRAG Llumad.
42.79
25
80.09
43.00
55.09
47.5%
56.75
SahgmaplBAAG GPT4-mmm (+o)
74.42
40.41
$9.10
$4.87
$1.55
49.47
45.16
60.18
54.18
$9.86
SibgrajeRAli - GPT mi (e%, 500g
76.85
$2.01
92.35
87.43
81.05
55.58
09.13
67.24
99.99
19.87
SalgmykRRGG GPT4 (44)
74.98
50.67
$8.25
94.73
83.24
52 40
52.28
00.36
56.39
63.70
SakgmpoRAGGG GPT (+e, 5000
77.56
56.38 40
06
$6.90
83.
61.90 96
54.48
70:47
64.86
86
Table 5: Breakdown of QA performance by reasoning hops.
WebrgSP-suk sub
CWQ-uu>
(91.29)
(28.0%)
Marco F]
Marco F1
Hir
Murco F1
Hit
Marco F1
Hi
Marco Ft
201
G. Retiiver
$6.41
75.30
45.73
65:35
RoC--fnint
72.05
81
91
20
68.56
41-21
Sep
74.50
55.02
73.45
66.20
59.45
67.17
31.33
33.33
Syngrayb Jama 1-318)
75.50
K362
51.56
6500
57.52
41.85
47.37
SubgaaphRRAG (CPP440-mii)
$0.55
92.56
74.11
88.51
57.36
67.34
63.85
72.74
$1.14
$4.39
results due to various reproducibility difficulties. For StructGPT, we directly use their provided processed files obtain results for WebQSP. For SR, we include their reported results for WebQSP and CWQ, and further evaluate it on WebQSP-sub as we could not run through SR's 's code on the two CWQ datasets. For the remaining baselines, we successfully reproduced their results and evaluated them on additional metrics and datasets, including WebQSP-sut and CWQ-sub. However, we do not include results for G-Retriever on CWQ as required over 200 hours of computation on NVIDIA RTX 6000 Ada GPUs If not specified, the I.M.M reasoners in SubgraphRAG use the top 100 retrieved triples; results using more triples are explicitly noted, All I.lama models used are their instruction fine-tuned versions, i.e., L.ama3. 1-801101) are gpt-3.5-turb-- 1106, gp1-4o-mini-2024-07-11 and gpt-40-2024-08-- used in SubgraphRAG, both the temperature and the seed are set to 0 to ensure reproducibility.
The used GPT models' versions For all LLM reasoners
Overall Performance. Tables 3 and present the results, where SubgraphRAG achieves SOTA on WebQSP and WebQSP-sub Even with 8B LLMs, method surpasses previous SOTA ap- proaches by < 4% in Macro-Fl and Hit metrics (excluding RoG-Joint due to test label lcakage). With larger models like 1--ma3.--70B-Inseee
and GPT-40,, SubgraphR achieves greater perfor- mance, showing Σ 12% improvement in Macro-F1 and S 9% in Hit. On the more challenging CWQ and CWQ-sub datasets, which require more reasoning hops. SubgraphRAG performs competitively with smaller 8B models. When paired with advanced LLMs like GPT-4o, SubgraphRAA achieves results second only to ToG on CWQ. Notably, SubgraphRAG requires only a single call GPT-4o, whereas ToG requires 6-8 calls, which increases computational cost, and we were unable to repro- duce ToG's performance using their published code. On CWQ-sub, SubgraphRAG demonstrates gains of < 9% in Macro-F1 and < 11% in Hit, indicating that tasks with greater reasoning com- plexity benefit substantially from more powerful I.LMs. Our method also excels in truth-grounded QA, with Score, consistently outperforming baselines by < 12%, driven by our prompt design that encourages retrieval-grounded reasoning. See Appendix D for further discussions.
Additionally, our framework generalizes well, with retrievers trained on one dataset performing effectively on others. Although moderate performance decay occurs due to domain shift, this can largely be mitigated by including more triples extracted by the retriever. Notably, the decay is gener- ally minor on WebQSP and WebQSP but more pronounced on CWQ and CWQ-sub, potentially due to greater label leakage from the WebQSP test set to the CWQ training set.

---

Published as a conference paper at ICLR 2025
Table 6: Ablation studies with different retrievers, using the same prompt and Llama3,1-8B-1 as the reasoner. Rand refers to random triple sampling, RandNoAns removes triples with ground- truth answers after random sampling, and NoRetriever directly asks questions without KG info.
WrKgsP
Weigg5h-ua ui
CWQ-sn
Rand + atkkkt Rewster
37.00
61.16
21:34
15:25
37.33 (M
17:79
6AL14
54.23
66.16
23:57
20:15
21.15
55.31
$2.55
Kanthwkas atokA Remener
21-18
22:11
23.61
33.00
21:33
18:47
19:42
43.7V
25:64
55.57
23.66
$3.95
usus stallaiy Sukgi pMRAG Romene:
$8.41
24.14
34,55
43.61
59.36
57.31
75.43
11.20
73.16
3933
35.45
-0.02
43.66
55.72
Smmestif sibgraphiRA: Rewennrr
25.00
62.05
44:33
75.00
13.57
19192
t-knnnner suhgraphi Ati Reaceerr
48.1
60:00
26-4
14.00
-2.00
28.08
06.98
$3.00
72.54
11.45
30.33
25.17
26.33
59:22
Sey Sulp g4R Remene:
76.65
GNBURAG
62.54
$4.35
17.00
$0.56
5500
53.36
59.76
67.55
GAN-RAL (++) + stigg quii Reascner
62317
71.36
34.55
43.13
53.01
34.35
72.05
57.00
75.37
3946
35.00
00.54
64.91
57.42
(--)
65.42
83.42
$1.96
48.37
67.91
42.79
85.25
91.21
30.00
43,03
40.33
$3.09
47.38
55.78
For efficiency, studies show that overall LLM latency is dominated by # LLM calls, followed by # output tokens and then # input tokens SubgraphRAG, despite potentially varying # input tokens, utilizes only one LLM call per question, whereas various baselines require multiple calls.
the performance breakdown by reasoning hops. Our method, even with an 8B LLM, shows significantly better results multi-hop questions. This can be attributed to our design philosophy, which does not restrict the retrieved subgraph form and leverages unfine-tuned LLMS with ICL examples, fully unlocking their reasoning capabilities. Interestingly, SubgraphR AG shows relatively lower performance on some 1-hop questions in CWQ- sub. Crose-cefe with Table which provides detailed breakdown of whether the answers are truth- gouhhgroouned, we find that previous methods may generate many correct answers that are not supported by the retrieved results. Further investigation suggests this discrepancy may partly result from dataset leakage, giving ouning-based methods an advantage: In CWQ. over 60% of test samples have answers that have appeared in the training set, making it easier for these methods to succeed without robust reasoning. Thus, future studies are needed to fully investigate the potential leakage.
Performance Breakdown & Analysis.
Performance with Different Retrievers. To assess the impact of our retriever on the overall per- formance, we conducted extensive studies by replacing the retrieval results with those from baseline retrievers, while keeping all other settings constant (same prompts, same LLM reasoners). Tables 6 and / show the results using L 1 8B and GPT4o-mini the LLM reasoners, respectively. For retrievers from Retrieve-Rewee Rewrite-Answer StructGPT, G-Retriever, and RoG, we used all their retrieved triples, while for other cases, kept 100 triples. Clearly, pairing with our retriever yields significantly better performance on all metrics and datasets compared to using baseline retrievers, indicating that our superior results are largely due the effectiveness of improved retriever.
Impact of Retrieval Size. Irrelevant context can mislead LLMS (Wu et al., 2024), and retrieving more triples may not help. We conducted experiments (Fig by feeding Llama 3.1 T8B and GPT-4o- mini different numbers of retrieved triples. The upper bound is defined such that if the answer entity present in the retrieved results, it is considered correct answer. The results show a significant performance decay in Llama 3.1 8B with more triples, while GPT-4o-mini performs even better with additional retrieved triples. This suggests that different LLMs have varying capacities for robust reasoning, and users should input proper numbers of retrieved results to match the capacity of the used I.LM and the available computational budget. See Appendix Appendix[]foo more results.
Explainability. By retrieving flexible, high-quality evidence subgraphs as context and exploiting the strong reasoning capabilities of pre-trained LLMs, SubgraphRAG can natively provide effective explanations along with answer predictions. In contrast, explainable predictions with fine-tuned LLMs necessitate extra labeling efforts for preserving explainability (Luo et al.] (024bb or post hoc explainability approaches (Krishna et al., 2023). To illustrate the explainability of our framework, we provide multiple examples of our reasoner's responses (see Appendix F))
5 CONCLUSION
We propose SubgraphRAG, a KG-based RAG framework that performs efficient and flexible sub- graph retrieval followed by prompting unfine-tuned LLMss. SubgraphRAG demonstrates better or comparable accuracy, efficiency, and explainability compared to existing approaches.
For LLMss like GPT-4 and Claude-3.5, an additional input token typically adds about 10-2x x the latency of an extra output token and 10 10="x that of additional L.L call [Vivkk 2024).
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGMENTS
M. Li, S. Miao, and Li are partially supported by NSF awards PHY-2117997, IIS- 223956,, IIS- 2428777, and CCF-2402816; DOE award DE-FOA-0002785; JPMC faculty awards; OpenAl Re- searcher Access Program Credits; and Microsoft Azure Research Credits for Generative Al.
REFERENCES
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collab- oratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD International Conference Management of Data, pp. 1247-1250, 2008
Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Milli- can, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, Diego De Las Casas, Aurelia Guy, Jacob Menick, Roman Ring, Tom Hennigan, Saffron Huang, Loren Maggiore, Chris Jones, Albin Cassirer, Andy Brock, Michela Paganini, Geoffrey Irving, Oriol Vinyals, Simon Osindero, Karen Simonyan, Jack Rae, Erich Elsen, and Laurent Sifre. Improving language models by retrieving from trillions of tokens. Proceedings of the 39th International Conference Machine Learning, pp. 2200-224,, 2022.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D) Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss. Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. Language models are few-shot learners. Advances in Neural Information Processing Systems, pp. 1877-1901, 2020.
Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes Gehrke, Eric Horvitz, Ece Kamar, Peter Lee, Yin Tat Lee, Yuanzhi Li, Scott Lundberg, Harsha Nori, Hamid Palangi, Marco Tulio Ribeiro, and Yi Zhang. Sparks of artificial general intelligence: Early experiments with gpt-4. arXiv preprint arXiv:2303.1 12712, 2023.
Michel Chein and Marie-Laure Mugnier. Graph-based knowledge representation: computational foundations of conceptual graphs. Springer Science & Business Media, 2008
Zhengdao Chen, Lei Chen, Soledad Villar, and Joan Bruna. Can graph neural networks count substructures? In Advances in Neural Information Processing Systems, pp. 10383-10395, 2020.
Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 Conference of the North American Chapter the Association for Computational Linguistics: Human Language Technologies, Volume (Long and Short Papers), pp. 4171 11166 2019.
Bhuwan Dhingra, Jeremy R. Cole, Julian Martin Eisenschlos, Daniel Gillick, Jacob Eisenstein, and William W. Cohen. Time-aware language models as temporal knowledge bases. Transactions of the Association for Computational Linguistics, 10:257-273, 2022.
Abhimanyu Dubey et al. The llama 3 herd of models. arXiv preprint arXiv:2407.21783, 2024.
Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. From local to global: graph rag approach to query-focused summariza- tion. arXiv preprint arXiv:2404. 161130 2024.
Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop on Representation Learning on Graphs and Manifolds, 2019.
Yifu Gao, Linbo Qiao, Zhigang Kan, Zhihua Wen, Yongquan He, and Dongsheng Li. Two-stage generative question answering on temporal knowledge graph using large language models. arXiv preprint 4rXiv:2402.1633 2024a.
11

---

Published as a conference paper at ICLR 2025
Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, Meng Wang, and Haofen Wang. Retrieval augmenttd generation for large language models: A survey. arXiv preprint arXiv:2312. 2024b.
Tiezheng Guo, Qingwen Yang. Chen Wang. Yanyi Liu, Pan Li, Jiawei Tang, Dapeng Li, and Yingyou Wen. Knowledgenavigator: Leveraging large language models for enhanced reasoning over knowledge graph. Complex & Intelligent Systems, pp. 1-14, 2024.
Aric A. Hagberg, Daniel A. Schult, and Pieter J. Swart. Exploring network structure, dynamics, and function using networkx. In Proceedings of the 7th Python in Science Conference, pp. 11-15, 2008.
Will Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems, 2017.
Taher H. Haveliwala. senicvsen pagerank. In Proceedings of the 11th International Conference World Wide Web, pp. 517-526, 2002.
Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. Improving multi-hop knowl- edge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM international conference on web search and data mining, pp. 553-561, 2021.
Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bres- son, and Bryan Hooi. G-retriever: Retreeval-augmented generation for textual graph understand- ing and question answering. arXiv preprint arXhv:2402.0763 0766000000.0.... 2024.
Yuntong Hu, Zhihan Lei, Zheng Zhang, Bo Pan, Chen Ling, and Liang Zhao. Grag: Graph retrieval- augmented generation. arXiv preprint 240.::::005.660066 2024.
Jie Huang and Kevin Chen-Chuan Chang. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 1049-1065, 2023.
Lei Huang, Weijiang Yu, Weitao Ma, Weihong Zhong, Zhangyin Feng, Haotian Wang, Qianglong Chen, Weihua Peng, Xiaocheng Feng. Bing Qin, and Ting Liu. A survey on hallucination in large language models: Principles, taxonomy, challenges, and open questions. arXiv preprint arXiv:2311.05252. 2023.
Piotr Indyk and Rajeev Motwani. Approximate nearest neighbors: towards removing the curse of dimensionality. In Proceedings of the Thirtieth Annual ACM Symposium on Theory of Comput- ing, STOC '98, pp. 604-613, New York, NY, USA, 1998. Association for Computing Machin- ery. ISBN 0897919629. doi: 10.1 10.1145/27669882766666 URL
276698 276876
Ziwei Ji, Nayeon Lee, Rita Frieske, Tiezheng Yu, Dan Su, Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12), 2023.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye, Wayne Xin Zhao, and Ji- Ji-Rong Wen. Structgpt: A general framework for large language model to reason structured data. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Processing, pp. 9237-9251, 2023a.
Jinhao Jiang, Kun Zhou, Xin Zhao, and Ji-Rong Wen. UniKGQA: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. In International Conference on Learning Representations, 2023b.
Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong Wen. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowl- edge graph. arXiv preprint arXiv:2402.1 163, 2024.
Bowen Jin, Chulin Xie, Jiawei Zhang, Kashob Kumar Roy, Yu Zhang, Suhang Wang, Yu Meng, and Augmenting large language models by reasoning on graphs. arXiv preprint arXiv:2404.07 103, 2024.
Jiawei Han. Graph
12

---

Published as a conference paper at ICLR 2025
Minki Kang, Jin Myung Kwak, Jinheon Back, and Sung Ju Hwang. Knowledge suppb-uugmented language models for 2023.
dialogue generation. arXiv preprint 1ath::335.18845.
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. Dense passage retrieval for open-domain question answering. In Proceedings of the 2020 Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 6781, 2020.
Jungo Kasai, Keisuke Sakaguchi, yoichi takahashi, Ronan Le Bras, Akari Asai, Xinyan Velocity Yu, Dragomir Radev, Noah A. Smith, Yejin Choi, and Kentaro Inui. Realtime QA: What's the answer right now? In Shirth-eeventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track, 2023.
Jiho Kim, Yeonsu Kwon, Yohan Jo, and Edward Choi. Kg-gpt: A general framework for reason- ing on knowledge graphs using large language models. The 2023 Conference on Empirical Methods in Natural Language Processing.
Thomas N Kipf and Max Welling. arXiv:1611.07300 2016.
Variational graph auto-encoders. arXiv preprint
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. Large language models are zero-shot reasoners. In Advances in Neural Information Processing Systems, 2022.
Satyapriya Krishna, Jiaqi Ma, Dylan Z Slack, Asma Ghandeharioun, Sameer Singh, and Himabindu Lakkaraju. Post hoc explanations of language models can improve language models. In Advances in Neural Information Processing Systems, 2023.
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, and Ion Stoica. Efficient memory management for large language model serving with pagedattention. In Proceedings of the ACM SIGOPS 29th Symposium on Operating Systems Principles, 2023.
Benjamin Lefaudeux, Francisco Massa, Diana Liskovich, Wenhan Xiong, Vittorio Caggiano, Sean Naren, Min Xu, Jieru Hu, Marta Tintore, Susan Zhang, Patrick Labatut, Daniel Haziza, Luca Wehrstedt, Jeremy Reizenstein, and Grigory Sizov. xformers: A modular and hackable trans- former modelling library. https::/githtii comp/ com/ http///abbcekercc//eeeeoeoee 2022.
Pan Li, Yanbang Wang, Hongwei Wang, and Jure Leskovec. Distance encoding: Design provably more powerful neural networks for graph representation learning. Advances in Neural Information Processing Systems, 33, 2020.
Shiyang Li, Yifan Gao, Haoming Jiang, Qingyu Yin, Zheng Li, Xifeng Yan, Chao Zhang, and Bing Yin. Graph reasoning for question answering with triplet retrieval. In Findings of the Association for Computational Linguistics: ACL 2023, pp. 3366-3377 2023a.
Xianzhi Li, Samuel Chan, Xiaodan Zhu, Yulong Pei, Zhiqiang Ma, Xiaomo Liu, and Sameena Shah. Are ChatGPT and GPT-4 general-purpor solvers for financial text analytics? study on several typical tasks. In Proceedings of the 2023 Conference on Empirical Methods in Natural Language Industry Track, pp. 408 422, 2023b.
Processing:
Zehan Li, Xin Zhang, Yanzhao Zhang, Dingkun Long, Pengjun Xie, and Meishan Zhang. Towards general text embeddings with multi-stage contrastive learning. arXiv preprint arXiv:2308.0325 2023c.
Zijian Li, Qingyan Guo. Jiawei Shao, Lei Song, Jiang Bian. Jun Zhang, and Rui Wang. Graph neural network enhanced retrieval for question answering of llms. arXiv preprint arXiv:2406.0657. 2024.
13

---

Published as a conference paper at ICLR 2025
Ke Liang. Lingyuan Meng, Meng Liu, Yue Liu, Wenxuan Tu, Siwei Wang, Sihang Zhou, Xinwang Liu, Fuchum Sun, and Kunlun He. A survey knowledge graph reasoning on graph types: Static, dynamic, and multi-modal. IEEE Transactions Pattern Analysis and Machine Intelligence, 2024.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. Kagnet: Knowledge-aware graph networks for commonsense reasoning, In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural 2019.
Language Processing (EMNLP-IJCNLP)) pp.
Guangyi Liu, Yongqi Zhang, Yong Li, and Quanming Yao. Explore then determine: A gnn-llm syn- ergy framework for reasoning over knowledge graph. arXiv preprint arXiv:2406.01145, 2024a.
Nelson Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. Lost in the middle: How language models long contexts. Transactions of the Association for Computational Linguistics, 12:157-173, 2024b.
Haoran Luo, Haihong E, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, Yifan Zhu, and Anh Tuan Luu. ChatKBQA: A generate- then-retrieve framework for knowledge base question answering with fine-tuned large language models. In Findings of the Association for Computational Linguistics: ACI. 2024, pp. 2039-2056, 2024a.
Linhao Luo. Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. Reasoning on graphs: Faithful and interpretable large language model reasoning. International Conference on Learning Repre- sentations, 2024b.
Shengjie Ma, Chengjin Xu, Xuhui Jiang, Muzhi Li, Huaren Qu, and Jian Guo. Think-on-graph 2.0: Deep and interpretable large language model reasoning with knowledge graph-guided retrieval. arXiv preprint arXiv:2407.1080) 2024.
Costas Mavromatis and George Karypis. ReaRev: Adaptive reasoning for question answering over knowledge graphs. In Findings of the Association for Computational Linguisties: EMNLP 2022, pp. 2447-2458, 2022.
Costas Mavromatis and George Karypis. Gan-rag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.201333 2024.
Christopher Morris, Martin Ritzert, Matthias Fey, William L. Hamilton, Jan Eric Lenssen, Gaurav Rattan, and Martin Grohe Weisfeiler and leman go neural: higher-order graph neural networks. In Proceedings of the Thirty-Thire AAAI Conference on Artificial Intelligence and Thirty-First First Innovative Applications of Artificial Intelligence Conference and Ninth AAAI Symposium on Ed- ucational Advances in Artificial Intelligence, 2019.
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. MTEB: Massive text em- bedding benchmark. In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pp. 2014-2037, 2023.
Ryan L.. Murphy, Balasubramaniam Srinivasan, Vinayak Rao, and Bruno Ribeiro. Janossy pooling: Learning deep commuttaionninvarian functions for variable-size inputs. In International Confer- ence on Learning Representations, 2019.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. IEEE Transactions on Knowledge and Data Engineering, 36(7):3580-355 -3999, 2024.
Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Kopf, Edward Yang, Zachary Vito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high bigh-peeformance deep learning library. In Advances in Neural Information Processing Systems, pp. 8024-8035. 2019.
14

---

Published as a conference paper at ICLR 2025
Boci Peng, Yun Zhu, Yongchao Liu, Xiaohe Bo, Haizhou Shi, Chuntao Hong, Yan Zhang, and Siliang Tang. Graph aetrieval-uggmene generation: A survey, arXiv preprint TrXiv:2408.0892. 2024.
Stephen Robertson and Hugo Zaragoza. The probabilistic relevance framework: Bm25 and beyond. Found. Trends Inf. Retr. 3(4):333-389 2009.
Stephen E. Robertson, Steve Walker, Susan Jones, Micheline Hancock-Beaulieu, and Mike Gatford. Okapi trec-3. In 7REC, volume 500-225 of NIST Special Publication, pp- 109-126, 1994.
Ian Robinson, Jim Webber, and Emil Eifrem. Graph databases; new opportunities for connected data. " O'Reilly Media, Inc.", 2015.
Kurt Shuster, Spencer Poff, Moya Chen, Douwe Kiela, and Jason Weston. Retrieval augmentation reduces hallucination in conversation. In Findings of the Association for Computational Linguis-
tics: EMNLP 2021, pp. 3784-388 3803,,80,2200 2021.
Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Pro- cessing, pp. 4231-4243 2018.
Haitian Sun, Tania Bedrax-Weiis and William Cohen. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pp. 2380-2390, 2019.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung- Yeung Shum, and Jian Guo. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In International Conference on Learning Representations, 2024a.
Bleung-Yeung
Lei Sun, Zhengwei Tao, Youdi Li, and Hiroshi Arakawa. Oda: Observation agent for inte- grating Ilms and knowledge graphs. arXiv preprint arXiv:2404006677 2024b.
Alon Talmor and Jonathan Berant. The web as a knowledge-bare for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume I (Long Papers), pp. 641- 651, 2018
Dhaval Taunk, Lakshya Khanna, Siri Venkata Pavan Kumar Kandru, Vasudeva Varma, Charu Sharma, and Makarand Tapaswi. Grapeqa: Graph augmentation and pruning to enhance question- answering. In Companion Proceedings of the ACM Web Conference 2023, pp. 1138-1 144, 2023.
Rakshit Trivedi, Hanjun Dai, Yichen Wang. and Le Song. Know-evolve: Deep temporal reasoning for dynamic knowledge graphs. In international conference on machine learning, pp. 3462-3471. PMLR, 2017.
Vivek. Understanding input/output tokens vs latency tradeoff.
https://minusx.att
blog/inpute vs oustuttttuuuuuuuuuuuuouu tokens/ 2024. URL
input- ---------uus output tokens/|
Accessed: 2024-10-27.
Tu Vu, Mohit lyyer, Xuezhi Wang, Noah Constant, Jerry Wei, Jason Wei, Chris Tar, Yun-Hsuan Sung, Denny Zhou, Quoc Le, and Thang Luong. FreshLLMs: Refreshing large language models with search engine augmentation. In Findings of the Association for Computational Linguistics ACL 2024, pp. 13697-13720, 2024.
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. Knowledge-drive cot: Exploring faithful reasoning in Ilms for knowledge- intensive question answering. arXiv preprint 2308 /3259, 2023a.
Xintao Wang, Qianwen Yang, Yongting Qiu, Jiaqing Liang, Qianyu He, Zhouhong Gu, Yanghua Xiao, and Wei Wang. Knowledgpt: Enhancing large language models with retrieval and storage access knowledge bases. arXiv preprint arXiv:2308. 1/761, 2023b.
15

---

Published as a conference paper at ICLR 2025
Yu Wang, Nedim Lipka, Ryan A Rossi, Alexa Siu, Ruiyi Zhang, and Tyler Derr. Knowledge graph prompting for multi-document question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pp. 19206-19214, 2024.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, brian ichter, Fei Xia, Ed H. Chi, Quoe V Le. and Denny Zhou. Chain of thought prompting elicits reasoning in large language models. In Advances in Neural Information Processing Systems, 2022.
Yilin Wen, Zifeng Wang, and Jimeng Sun. Mindmap: Knowledge graph prompting sparks graph of thoughts in large language models. arXiv preprint orXiv:2308.09729, 2023.
Thomas Wolf, Victor Sanh, Julien Chaumond, and Clement Delangue. Transfertransfo: A transfer learning approach for neural network based conversational agents. arXiv preprint 1rXiv:1901.08149 2019.
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Remi Louf, Morgan Funtowiez, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander Rush. Transformers: State-of-the-art natural language processing. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing: System Demonstrations, pp. 38 45, 2020.
Siye Wu, Jian Xie, Jiangjie Chen, Tinghui Zhu, Kai Zhang, and Yanghua Xiao. How easily do irrelevant inputs skew the responses of large language models? arXiv preprint urXiv2440.03302, 2024.
Yike Wu, Nan Hu, Guilin Qi, Sheng Bi, Jie Ren, Anhuan Xie, and Wei Song. Retrieve-rewrite- answer: A kg-to-text enhanced Iims framework for knowledge graph question answering. arXiv preprint arXiv:2309. 1/206, 2023.
Guanming Xiong, Junwei Bao, and Wen Zhao Interactive- Interactive-KBQA: Multi-turn interactions for knowledge base question answering with large language models. In Proceedings the 62nd Annual Meeting of the Association for Computational Linguistics (Volume I: Long Papers), pp- 10561-10582, 2024.
Keyulu Xu, Weihua Hu, Jure Leskovee, and Stefanic Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations, 2019.
Peng Xu, Wei Ping, Xianchao Wu, Lawrence McAfee, Chen Zhu, Zihan Liu, Sandeep Subramanian, Evelina Bakhturina, Mohammad Shoeybi, and Bryan Catanzaro. Retrieval meets long context large language models. In International Conference on Learning Representations, 2024.
Xiao Yang, Kai Sun, Hao Xin, Yushi Sun, Nikita Bhalla, Xiangsen Chen, Sajal Choudhary, Rongze Daniel Gui, Ziran Will Jiang, Ziyu Jiang, et al. Crag-comprehensive rag benchmark. arXiv preprint arXiv:2406.044: 04744, 2024.
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Thomas L. Griffiths, Yuan Cao, and Karthik R Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. In Ad- vances Neural Information Processing Systems, 2023.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. In North Amer- ican Chapter of the Association for Computational Linguistics (NAACL). 2021.
Wen-tau Yih, Matthew Richardson, Chris Meek, Ming-Wes Chang, and Jina Suh. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pp- 201-206, 2016.
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang. Jie Tang, Cuiping Li, and Hong Chen. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Anmual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp- 5773-5784, 2022.
16

---

Published as a conference paper at ICLR 2025
Muhan Zhang, Pan Li, Yinglong Xia, Kai Wang, and Long Jin. Labeling trick: A theory of using graph neural networks for multi-node representation learning. Advances in Neural Information Processing Systems, 34:9061-9073, 2021.
Yue Zhang, Yafu Li, Leyang Cui, Deng Cai, Lemao Liu, Tingchen Fu, Xinting Huang, Enbo Zhao, Yu Zhang, Yulong Chen, Longyue Wang, Anh Tuan Luu, Wei Bi, Freda Shi, and Shuming Shi. Siren's s song the ocean: survey on hallucination large language models. arXiv preprint arXiv:2309.01219, 2023.
17

---

Published as a conference paper at ICLR 2025
Revvrro Fommn
Figure 5: A visual illustration of DDE, where the leftmost column highlights the topic entities in color. From left right, iteratively perform graph diffusion the directed edges. The color intensity indicates the magnitude of the diffusion weights for individual entities.
A DIRECTIONAL DISTANCE ENCODING (DDE)
Fig. prooviees a visual illustration of DDE. By passing entity labeling information in both directions multiple rounds, DDE efficiently approximates measures such as the average distance between an entity and all topic entities. This process implicitly reveals information like whether entity lies at the intersection paths of particular lengths connecting multiple topic entities.
B ADDITIONAL RELATED WORK
The field of KGQA has evolved significantly over time. Early approaches to KGQA do not rely
on LLMs for answer generation (Yasunaga et al.) 2021 Taunk et al. 2023 Zhang et al. 2022 Lin
al. 2019 Sun et al. 2019). though they often employ pre-trained language models (PLMs) like
BERT text encoders (Dellin et al. 2019). These methods typically search for answer entities with
GNNs Yasunaga
(202)); Taunk et al. (2023): Lin al. (2019) or LSTMM-aased models Sun al. Zhang et al. (2022).
2019
With the rapid advancement LLMs in recent years, researchers began to leverage them in KGQA. For instance, recent work has explored using LLMs as translators, converting natural language ques- tions into executable SPARQL queries for KG databases to retrieve the answers (Jiang et al. 2023a
Wang et al. 2023b
Luo al. 2024a] Xiong et al. 2024).
Contemporary approaches have further expanded the role of LLMs, utilizing them for both knowl-
edge retrieval from KGs and reasoning (Kim et al., Gao al. 2024a Wang et al. 2024 Guo al., 2024 Ma et al. 2024 Sun et al. 2024a] Jiang et al. 2024 Jin et al. 2024). The strength of this strat- egy lies LLMs' ability to handle multi-hop tasks by breaking them down into manageable steps. However, as discussed in Section [ this often necessitates multiple LLM calls, resulting in high latency, To mitigate this issue, some frameworks have attempted to fine-tune LLMs memorize knowledge, but this reduces their ability to generalize to dynamically updated or novel KGs (Luo) al. 2024b) Mavromatis & Karypis 2024). Other models have explored fine-tume tine-tuning adapters em- bedded in fixed LLMss to better preserve their general reasoning capabilities while adapting to spe-
cific KGs (He et al. 2024 Gao al. 2024a Hu al. 2024).
In parallel with these developments, several approaches have emerged that, like our approach, allow LLMs to reason over subgraphs (Kim et al. Liu et al., 2024a Li et al. 2023a Guo et al. 2024]
Wu et al., 2023 Li et al. 2024| Wen et al. (003), though they employ different retrieval strategies.
Kim et al. breaks queries into sub-queries, retrieving evidence for each sub-query before reasoning
the collected evidence. Guo et al. (2024) uses PLM-based entity extraction followed by multi- hop expansion for retrieval. Li et al. (2023a) linearizes KG triples into natural language for global
triple retrieval using BM25 dense passage retrievers, Wen al. (003) extracts topic entities and
18

---

Published as a conference paper at ICLR 2025
merges triples into the retrieved subgraph using two heuristic methods: connecting topic entities via paths and retrieving their 1-hop neighbors.
While these subgraph retrieval strategies share similarities with SubgraphRAG, they lack several key advantages of it, such as retrieval efficiency, adjustable subgraph sizes, and flexible subgraph types. Consequently, they frequently result in suboptimal coverage of relevant information in the retrieved subgraphs. SubgraphRAG addresses these limitations, offering a more comprehensive and adaptable approach KGQA that builds upon and extends the capabilities of existing methods, while fully leveraging the power of advanced LLMs.
C ADDITIONAL EXPERIMENT DETAILS FOR SUBGRAPH RETRIEVAL
C.I ADDITIONAL IMPLEMENTATION AND EVALUATION DETAILS
RoG, G- Gettriieerr GNN-RAG, and SR+NSM w E2E, we utilize their official open-source implementations for training and evaluation. While a pre-trained RoG model publicly available, it was jointly trained on the training subset of both WebQSP and CWQ, causing label leakage issue due to sample duplication across the two datasets. To address this, we retrain the RoG model separately on the training subset of WebQSP and CWQ. Both Retrieve-Rewrite-AA Neerite-Answer and G-Retriever were originally only evaluated on the WebQSP dataset. We managed to adapt the G- G-Retriever codebase for an evaluation the CWQ dataset. While SR+NSM w E2E originally also reports the results for CWQ, we failed in running the official codebase for this dataset.
For Retrieve-Rewrite-
RoG, G- G-Retriever, the cosine similarity baseline, GNN-RAG, and all SubgraphRAG variants per- form relevant subgraph retrieval from the identical rough subgraphs. These subgraphs are centered around the topic entities and are included in the released RoG implementation. In contrast, Retrieve- Rewrite-Answer directly loads and queries the raw KG.
GraphSAGE originally only deals with node attributes and propose straightforward extension of it for handling both the node attributes and edge attributes, Let z, be the text embedding of an entity & and z, be the text embedding of relation r E R. A GraphSAGE layer updates entity representations with
1+1 ((e))
MEAN l
(4)
2+11
(12
(5)
2 Ze:
(6)
where o-)) is an MLP. Empirically, we find that I-layer GraphSAGE yields the best performance.
C.2 IMPLEMENTATION DEPENDENCIES
Our implementation is based on the following packages: PyToorch (Paszke et al. (2019). Transform- ers (Wolf et al. 2020), xFormers (Lefaudeux et al., (002)), NetworkX (Hagberg et al. 2008). and
PyTorch Geometric (Fey & Lenssen] 2019). We employ the built-in implementation of PPR from NetworkX.
C.3 ADDITIONAL EXPERIMENT RESULTS FOR VARYING K IN TOP-K TRIPLE RETRIEVAL
Fig. 6 presents the performance of SubgraphRAG variants WehQSP across spectrum of K values for top-K triple retrieval, which is consistent with the observations made for CWQ.
C.4 EVALUATION BREAKDOWN OVER TOPIC ENTITY COUNT
See table
D SUPPLEMENTARY KGQA RESULTS
Truth-grounded QA Analysis. Table & provides a detailed analysis truth- grounding in generated answers, showing that previous methods often produce correct answers that unsupported by the
19

---

Published as a conference paper at ICLR 2025
1.00
Shorter Hith
ANBART Enity
3.00
3.00
0.70
0.40
1.60
0.20
Scapteace
Craphsaule 1at cnlly
U/B AMG
Figure 6: Retrieval effectiveness on WebQSP across various K values for top- triple retrieval.
Table 7: Breakdown of recall evaluation for CWQ over topie entity count. Best results are in bold.
Model
Sbortest Puth Triples
GPT-to-labeled Triples
Answer Entities
Single (46.3%)
Multiple (53.7%)
Single (46.356)
Mulriplc (53.7%)
Single (46.3%)
Maltiple (50.755
cosine similarity
0.596
0.587
0.409
RoG
0.711
0.547
0.324
0.275
0.700
0.5KG
G Retrieever
0.245
0.129
0.288
0.158
0.350
0,300
GNN-RAG
0.496
0.503
0.369
0.003
0.829
0
0.788
0.777
MLP topic entity
SubgraphRAG
0.877
0.764
0.871
0.820
0.911
0.916
retrieved results, increasing the risk of hallucination. In particular, our method is significantly less likely to generate answers that present in the retriever results (the NR answers in the table). For instance, on CWQ-sub, more than 10% of RoG's correct answers are NR, while SubgraphRAG keeps this to just 1%. SubgraphRAG can also decline to answer when there is insufficient evidence, with refusal rates increasing from 2% to 19% on WebQSP and from 7% to 29% on CWQ for ques- tions lacking answers in the KG, compared to questions with KG-supported answers. In contrast, baseline models consistently provide answers even when supporting evidence is absent. Together, these qualities make SubgraphRAG a more trustworthy and truth-grounded KGQA framework.
To obtain single metric to assess the degree of truth-grounded QA performance, we follow (Yang) et al., 2024) to develop scoring strategy to aggregate the results of these tables. Specifically, for samples with answer entities present in the KG, correct answers receive score of +1, incorrect swers -1, and missing answers O. For samples without answer entities in the KG, missing answers, answers with entities found in the retrieved results, and answers with entities not found in the trieved results are scored +1, -1, and -1.5, respectively. Denote 8; as the score yielded for sample i and a; the number of answers outputted by the LLM reasoner for this sample. For dataset with
samples: Score, = Normalize (+ E St ). where we use min-max normalization with min O and max = 100. This metric penalizes hallucinated answers while favoring missing answers over incorrect ones.
Impact Retrieval Size. Fig. 4shows how our SubgraphRAG would perform with different num- bers of retrieved triples. To further demonstrate the superior performance of our proposed retriever module, we also scale RoG's retriever with more retrieved triples by increasing its beam search sizes (noting that this substantially increases computational costs). As shown in Fig. 7 clearly, even when using RoG's retriever with more triples in SubgraphR our proposed retriever consistently achieves significantly better results, further demonstrating the effectiveness of retriever module.
E PROMPT TEMPLATES
Fig. is the detailed prompt template used in our experiments for all samples, and Fig, provides the prompt employed to label relevant triples via GPT-4o.
20

---

Published as conference paper at ICLR 2025
Macro-
66
60
100
Number Triples
Triples
Number Triples
Number Triples
(a) WebQSP-sub
(b) CWQ-sub
Figure 7: Ablation studies on the number of retrieved triples and the used retrievers in Sub- graphRAG. The triples from RoG's retriever are generated using beam search with sizes {3. 6, 9, 12]. and all the experiments use SubgraphRAG's un-fine-tuned LLM reasoners.
Table 8: Detailed performance of truth guuh-grounded QA. No Ans Samples refers to cases where LLM reasoners refuse to answer; NR (Not Retrieved) indicates answers not present in the retrieved triples, while (Retrieved) indicates answers found within the retrieved triples.
Sample
Sumples Estitics
No AB Sacales
Arong Ae1
Copas dor INS
S0.46
Stoge 2031
(V1
11764405
19%
fers
Sep
(11559
6867/12401
555
32115534
349470667
30R
71/162
43/162 27%
SahgmpoRACC (1.lem 1-80))
2001559 =
1997/990
3.%%
5066655 =: 166
RA1 90F 856
1389 19%
16/00 15%
71/007 669
SuhymphleAG (GPPT4ominn)
12/1559 =
1802/R011
1.%
1266319 16
17671002
7699 105%
17683 2056
51883 11%
365106709
00%
1833/229 67%
Sop
318366129
341/2966
1295/3183
41%
00
1840/2620
Su MG Illaana&1 SubgmyhlRAG AG (GPT44-mini)
2102548
2417/51288
21/2611
199663
114/1052-
51910052
200/2646 7%
219405206 42%
30/001 154
19W22194
1371683 3066
70/053 - 18%
564/953 - 59%
Input Prompts for KGQA
System:
Based the triples retrieved from knowledge graph, please the question. Please return formarted list, cach prefixed with "ans:".
User: ICT, example
Triplets:
(Lou Scal, sports mascoLieam.SSon Francisco Giants)
(San Francisco seantsssports.spppe sports leam cas..chanpioosspps.200 World Series)
(San Francisco Giants.sports, sporsssprsssssossss
pprpts.urttt.chmmionsssig wwwpp//oommiip.eeeeeeeaa. geen.champion,220 2014 World Series)
(San Francisco Giants, time. participani.evente 201 Major League Baseball season)
Francisco Giants.time,
aerticipant.eeen..2.
World Series)
(San Francisco Giants,time, participant.event eee.tttveeet22.11. Major League Basehall season)
(San Francisco Giants.sports sports. wtt.sspoossssssee.s Jeam. championships.oom
World Scries)
(San Francisco Siants.spotss.spp sports seam maseot, Crazy Crab)
(San Francisco Giunts.sporis. siants.sporss.sprri.. Jeam championships
World Series)
(San Francisco Giants.sports, professional wtt.ssioonnssptsssssaaooomm pepporiseeam.oaaaa Seam.wwnrr_..BB Bill Neukom)
(San Franciscu Giants, Lime, purticipant.event eveni,222e World Series)
cennciscn.ports.se.. sports team lecation.temms.Su Francisco Giants)
(San Francisco Giants, sports. sports Jeam arena stadium, AT&T Park)
(AT&T nutklocatioo.ocatiion ooeetton.events.... 2012 2 World Scries)
Question:
What year did the team with named Lou Scul the World Seriex?
Assistant:
example
To find the year the team with mascot named Lou Seal won the World Series, we need find the team with mascot named Lou then find the year they won the World Scries.
From the tripless, we see that Lou Seal the mascot of the San Francisco Giants. Now, we need to find the year the San Francisco Giants wun the World Series. From the triplets, So, the team with mascot named Lou Seal (San Francisco Giants) won the World Scries in 2010, 2012, and 2014.
that San Francisco Giants
2010 World Series and 2012 World Series and 2014 World Series.
Therefore, formatted answers
ans: 2014 (2014 World Series)
ans: 2012 (2012 World Series)
2010 (2010 World Series)
User: Sample infer
Triplets:
(ea: Fab, eh) Retrieved triples
(Co, Fed-Sa) 92)
Question:
what's ...? Q datasets
Figure 8: Detailed prompt for LLM cnestioon-answering used in our experiments for all samples.
21

---

Published as a conference paper at ICLR 2025
Table 9: Ablation studies with different retrievers, using the same prompt and GPT4o-mini as the reasoner. Rand refers to random triple sampling, RandNoAns removes triples with ground-truth answers after random sampling, and NoRetriever directly asks questions without KG info.
WrRgSP
WedggSh-op -ih
CWi-sn
Rand e Suhy akikAG Sukg--ttiKA Reasee:
47.20
66.31
35.23
29.52
47. 13
23:34
08.53
54.39
06.20
13.55
38.26
<2.36
41.43
5431
Kasthwkas stOAA Remenzr
26.11
15.65
usus stallait Sukgiagd 4MRAG Remene:
6454
78.19
41.23
48.22
05.36
43.34
78.00
14.15
73.03
4507
42.27
53.53
40.33
$7.00
SmeriiP subgrighiAA: Resennar
82.08
32387
$0.61
$3.45
L-lanner Phgr-phskAA: Reaceeer
78.25
26.78
62.23
15.10
78.00
12.07
76.31
78.55
25:47
$3.30
57.67
GNNURAG
13:27
GAN-AAU (we) sibgr qUiEAA Rearener
01.19
73.99
13:42
$3:37
61.3
42.02
79.19
16.05
76.11
432
50.00
52.12
90.15
SebgmptäL50
23.81
55.08
44.59
54.21
04.42
00.10
$4.07
$1.35
49.47
43.16
00.18
54.18
53.8
Input Prompts for Labeling Relevant Triples via GPT-4o
System:
Based the triplets retrieved from knowledge graph, please select relevant triplets for answering the question. Please retur formatted triplets list, each prefixed with "evidence:""
User:
// Sample to infor
Triplets:
(ea, tn) / Retrieved triples
(eey Feas eg)
Question:
what's Q datasets
Figure 9: Detailed prompt for labeling triples via GPT-4o.
22

---

Published as a conference paper at ICLR 2025
F EXPLAINABLITY EXAMPLES
To demonstrate the superior explainability of our approach, provide multiple example responses from our LLM reasoners below, which cover questions requiring different reasoning hops and logic chains. We have also included examples of our LLM reasoners refusing to answer due to insufficient information Appendix F22 which helps to provide more outh-grounded answers. Due to limited space, do not include retrieved triples for all the examples, but one will be able to find full prediction files in our released repository.
F.1 LLM RESPONSES WITH EXPLAINABLE REASONING
Question:
The people from the country that contains Nord-Ouest Department speak what languages today?
Retrieved Triples:
(Ilati.locationcc wowtiioonooon......o.. wow.ammrnnnnyyyyumuueeeee innguuges.spoken.. spp.eepoooaaiii Haitian Creole)
(Hairi, locaticonnn.. cataaiionoonntty...oom country.first_lee sel__lee__diiiiionss. _diiiiiinss,..i. Nowdnns,ooodooo Nord-Quest Department)
(Hairi [Hliti.location, www.l//ccctccommete. offfccy. language. French)
(French, language human langunge.comi countries spoken HaiHaiti
WNrd-OOus www.eoooooooooouuaaeeeeeee.eeeee Hanguagesssspot wwp.spo.rorache
peruttment.oocattio administrative division first level
divisioon of Jive.vioee.iiiiioon
Haiti)
locitioooation.c country
(Haiti location..ooom www.:/m.oouurry.unn.... administrativve www..trr....iiiiitttttiieeeeeeee
Noriiinn..orOOOooaa Department)
(Haiti.lccation wttiillaattinnoooc www.toy.nnoofccc.ooom official Janguage.Haiienn Haitian Creole)
(Nord-Oues Department. location location containedby.Hait)
(Haitian Creole, Creole,language human langunge main country:Haitt)
LLM Response:
To find the languages spoken in the country that contains Nord Ouest Department, find the country that contains Noed-Quest Quest Department. From the triplets,
Nord-Ouest Department contained by Haiti. So, the country that contains Nord-Ouest Department is Haiti.
From triplets, we can see Haiti speaks Haitian Creole French.
Therefore, the people from the country that contains Nord-Ouest Department (Haitl) speak Haitian Creole and French today. The formatted answer is: Haitian Creale
ans: French
Ground-truth:
Haitian Crcole
French
23

---

Published as a conference paper at ICLR 2025
Question:
team mascot Hank play
Retrieved Triples:
(Milwaukee Brewers. sprrssooosssststttt jeam.ieem mascot, lank)
Slank,,sorts.msscc leam, leaaeeeelammniieeeee Milwankee Brewers)
(Milwaukee Brewers.sports sports sports.leam Jeam previously Snoon..s.Seallll Pilots)
(Seattle Pilots, sports defunct sports team. later known as, Milwankee Brewers)
sports
(Milwaukee Brewers, seovers.ports.. sports team arena stadium, Miller Park)
(Milwaukee Brewers,sports wwrrerrs.porsssggg. sports.tcame weatssstam..oo.. location.ottooo Milvakke)
(Miller Park. sports sports facility iearlyiemmmmiiiiii Milwaukee Brewers)
(Milwauker, www.eee.ssssss....ee. spertsppeamm htt.e/aammaaaiiiia Jocation..nnoom teammn.tmmmsMMM..
teams,,ilwauker Milwaukee Brewers)
(Miller locatoocation.os location.contaieddly.. containeely.Millwaeee... Milwaukee)
(Milwaukee Brewers, baseball buseball.
cwwches,m005277688)
LLM Response:
find the stadium where the tearn with muscal named Hank plays, we need to lind the tearn with muscol named Hank and then find the stadium they play
From the triplets, we can see that Hank is the mascor of the Milwankee Brewers. we
stadium From the triplets, we can see that the Milwaukee Brewers play at Miller Park.
Brewers play.
Therefore, the formatted answer is: Miller Park
Ground-truth:
Miller Park
CWQ-WebQTm 922154777777333306666600044446
Question:
Which of
Tom Ticks sports tearn in the American League Wext?
Retrieved Triples:
(Texas Rangers, wwwssspooppooee professioni sprrpp.ooott.ooeiiooooa sports team.owner s,Tom Hicks)
(Tom Bicksssportt sports team ownerteamss www.d, Rangers)
seorts.team
(Dallas Stars, pors.sports.ppp professional. sports ream owner_s,Tom Hicks)
(Seuttle Floossspprts..u
defusus defunci, sports leam laler known as, Milwaukee Brewers)
(Tom bocks.spors..porte settsssssssssmm.s....mm Jeam_/weerrereeeeee
Dallas Stars)
(United States of America, base. locations.countrics statesttttttteeeessss stwatesprooiinnee...ssssssssssse provinces (ibbi.,ssihhn,,.aamaa))))
(Houston
wpwrtt.ppofeesseeseseeepp professional sports team
s,Drytoon Mclane,
(m 1n..0erd222ppon sports league participation team,Los Angeles Angels of Anaheim)
(United States of America. base.aarcas, schema..hcc..c administrative. htt.s/aaraaaaaa. addministraive.chileee
(m mm.orrd31..ppp ssposstssssppo sportsssports.eegee.... Jeague. www.pcippaiioon.ooom Anahhii Angels)
(United States of America, base aureas, bhemma.dmiiistaatii aww....dmiiistiiiii childdenn
wwwilirenn.asshhngiine Washington, D.C.)
(United States of America. www..e.ocaiion.ccommm countries petttesspoiiccss..e within Missisip
(m mmrOetdd Om.Oerrdvv..prrtss.. sppttrtssssstttss....... Jearts wwtppppatciipiioo.ttee. htwppptttttttmmmmommmmmm Califomiaa Angels) (United
America. nareas ahhema.ddministraiii. reeeniministetive children, Mississiiisssssippi))
(American League West baseball bascball division bisisall.divsiion teams, Texas Rangers)
L.M Response:
find the owned by Tom Hicks that the American League West, need find the owned Tom Hicks then find the teams in the American League West.
Rangers Now, we need find the teams in the American League West.
From triplets, we can see
Tom Hicks
Stars.
From the triplets, we can see that the teams in the American League West are Texas Rangers, Los Angeles Angels of Anaheim. Seattle Mariners, Oakland Athletics, and Seatlle Pilots.
So, among the teams owned by Tom Hicks (Texas Rangers and Dallas Stars), only Texas Rangers in American Teague West.
Therefore, the formatted ans: Texas Rangers
Ground-truth:
Texas Rangers
24

---

Published as a conference paper at ICLR 2025
Question:
film
Winged Monkey who played Dorothy?
LLM Respouse:
To find the actor who played Durohh the film with the character named Winged Monkey #7, need to find the film that Winged Monkey #7 in and then find the actor who played Dorothy in that fll.
the lilm Wizurd Now, we need to find the actor who played Dorothy in "The Wizard of Oz". From triplets, we can see Judy Garland (m 026y72) played Dorothy "The Wizard Oz". Therefore, the formatted answer is: ans: Judy Garland
From triplets, we see Winged Monkey
(m.0yjóur7).
Ground-truth:
Judy Garland
Question:
What city home fur Lauren Courad and had a mayor named Willie Brown?
LLM Respons:
To find the city that home for Lauren Conrad and had mayor named Willie Brown, we need to find the city where Lauren
Conrad lives and city where Willie Brown the
From the triplcts, we can see that Lauren Conrad lived in Los Angeles (m3344)j44) and San Francisco (m.04hy_14).
Fruncisco. Therefore, the city that home for Lauren Conrud and had mayor named Willie Brown is San Francisco.
triplets, we see
Brown the mayor
So,
answer
San Francisco
Ground-truth:
San Fruncisco
Question:
Which movie did Angelina Jolie direct, that was produced by Erwin Stoff?
LLM Response: To find the movie that Angelina Jolie directed, which was produced by Erwin StofT, we need to the movies that Angelina Jolie directed and the movies that Frwin Stoff produced.
From the triplcts, we can see that Angelina Jolie directed the following movies:
and
Honey
Unhroken
By the Sea
From the triples,
that Erwin Stoff produced the following movies:
Unbroken
Day Earth Stood Still
Edge of Tomorrow
for Elephants
Austio Powers: The Spy Who Shagged Me
The Blind Side
The Replacements
1 Am eggnnd
Hardball
- The Lake House
Kings
-47 Ronin
Now: we need to find common movie between Angelina Jolie's is directed movies and Erwin Stoll's produced movies.
is Unbroken. Therefore, the formatted answer is:
The
Unbroken
Ground-truth:
Unbroken
25

---

Published as a conference paper at ICLR 2025
Question:
is
Retrieved Triples:
(Sweden location. location.ocaaion..o location lime Centrll European Time Zone)
groken,Finnish Finnish Lngunge)
(Finnish Language.Junguage lunguage humun langunge countries.puoee..n..k
(Sweden,locule country languages spoken, Turoyo Langunge)
(Sweden. location country official language. Swedish Language)
Swweden.location
(S location. location time zones, Centrl Europcan Time Zoac)
Soaan_.ccation
"Stockholm..oc:e acation.location time zones, Central European Time Zone)
(Unitary state, govegoomenn.....mmmm form gww.ggvweeen..om coumtries, Sweden)
(Swedich
human_language wwtpaagc..onrrssau.. seontrre_spokcn.in..ii
(Sweden, Jocatoonaiinoo.o. country. lunguages, spuken, Swedish Language)
LLM Respouse:
To find the timesone Sweden, we can look for the triplets that contain Sweden and have a property related to timezones. From the triplets,
that Sweden located the Central European Time Zone.
Therefore, the answer is:
European
Ground-truth:
Central European Time Zone
WebQSP-WebQTeste
Question:
where did bristal pulin school
Retrieved Triples:
B...kylyyl.eduucain education institution, Wasilla High School)
(Wasilla High School.education educational institution students graduates, (ittuutessm.Ohy442))
Liniiiiionnnnnsttutttooooooo (Juncau-oowe Douglass High School,cducation educational Linstitution students wttpsssgrdduuuu..oom 0oppkbbwp)
(m..0ppbve bespkbwp.educcte eeeducatiooncco educctioocccooooooocoo institution Jouell-Deaalll High School)
(Bristol Palin.peuple. pww.pplppprso.....m educationnn.... cducation,,m.0e114zl) (m.0b1422l)
(Bristol palipepeoppl.pp person, education,m. m..ppp/b.mm.
(Bristol Palin. peuile person, education, (x.pkbbwp)
(m me..cpp44ll.co cdacation.education instintion, West Anchorage High School)
(West Anchorage High School. www.ccttonocccoocc www.ccciioonaa....c instiiution students, students.graduutss.o gww.gaates..mmepe. Oeppk-bbb)
(m edde-itliedccationn educaticnnnnnnno student, Bristol Palin)
(mn mm..kpp44bl,p we..cbllddcctton.. cet.uattoo.eductioonn student, Bristol Palin)
student.Bristal
(m Oopkbwppppkeecc
wookbwp.dducation edw.ationnnnccm education student. Bristol Palin)
(m 04tmm l...ducation..ddeeeee.o
Wasilla High School)
(m.00lg77g..ol 77..ecccduationn education. instituiionn, Wasilla High School)
edlucation.nsstiution
(Wasila High School, education educational institution students graduates,m. 0.40mmm11111 1z)
(Wasilla High educatinn educational inn.iuuttoonnttttiiiooooooooooo studentss wwwdeetts.grduuusseees graduates.mmmon (nl1g777.m0
(Bristol Palin.pcople person.place birth, Wasilla)
(Wasilla location. location, people born here hereeBristol Palin)
ducatioo.people
(Bristol Palin, Talin.peoplee.ee wew.llepesoo..... nationaliry, United United States of America)
(Levi oohnston.poopl.. person, place of Wasill))
LLM Respouse:
school, need find educational institutions that Bristol Palin atlended. From the triplets, we can see that Bristal Palin attended the following educational institutions:
To lind where Bristol Palin
Wasilla High School (m Obyl4zl)
Juncan-Douglas High School (m. (cpkbpp))
Anchorage High
Depk4hh)
Therefore, the formatted answers arc:
ans: Wasilla High School
Juneau Douglas High School
ans: West Anchorage High School
Ground-truth:
Anchorage High Schwol
Wasilla High School
Juneuu Douglas High School
26

---

Published as a conference paper at ICLR 2025
WebOSP-WebQTese
Question:
parry
helong to
Retrieved Triples:
(Barack Obana.inlluenen wtt.uiiillennceinlleeeee influence node influenced.. Su.Suul Alinsky)
(Barack Ohamn,orammiztion orgganiattioorggnnzzationnn
gegairattonnffunderrr founder. aaaanddrrrgnniztiions,
National Commission on Fiscal Responsibility and
Reform)
(Barack Ohama presidential campuign, 2012, 2012 2..eggermmeet.llee
campuign party, Democratic Party)
(Barack Dhana.government politician election campaigns, Barack Obama presidential primary campaign, 2008)
(Barack Drama.govermmen politician party.m.0 paryy....gllwww)
(Burack Gournu.overmment politician.eleeeeeoc election campaigns. Barack Obama 1996 Illinois Senate Campaign)
(Barack Ohama 2004 US Semale pumpiigg.guvemmeer ieveemeet,election... eleceeeennnnneeeee wwmpaign..atttt...... Democratic Party)
c (Barack Ohana,influenc inflnence pwwee.onffiuun iode.influenee._. by. Reinhold Nichuhr)
(m.003g/ww.ggre) w.goveemmeett political party Jenure party, Demiocaticc Party)
(Ranck Ohama,orgaa Cramu.organization. organization founder founder.organzzaiin organizations founded, White House Task Force to Protect Students from
Assault)
(Barack Obama presidential primary campaign, 2008, wov..ooveeeeetccoom www.:eaeeeeeeeeeeeooooooooooeeee campaign. party, Democratio Party)
(Burack Ohara 1996 Illinois Semale gevpppiigggeeemmmm..... election wttp:iinwwmmpiggggaattt.e.aaea Det.paayaaeereaaaiiii. Party)
(Barack Ohama,hase politicalconventionn
paanvennioo--pssieentllll presidential pow.nnee..mminnn
200, Democratic National Convention)
(Barack Obama govermmettetett politician govermeent powtppoostittseeee held.m hell..m.33h96h66)
(Barack Ohama Soamuugovermment politician elecctio campuigns,Bare Obama 2004
Campaign)
LLM Response:
From the triplets, we can see that Barack Ohama associated with the Democratic Purty in several instances:
(Barack Ohama.govermeen politician murty...03gihww)
2. (Barack Obama 2004 US Senate Campaign.govemmaee covermenn.clecioo.ccom hamppiinn.caaainnnnn party, Democratic Party)
Obama Illinois
Campaign. camppaign.goeemmmelll.. www.gggooeeeeleeeee.mmo ellection campaign Democcatiii Party)
4. (Barack Obama presidential primary campaign, 2008 goverment election campaign, party, Democratic Party)
5. (Democratic Party presidential primarics, 3X08_oovermmett.pinig pereetion.party... www.teaa.prt..emmmce.... Democratic Part))
w.ggeeenmmeti, political.part Jenure Jww.reeaatty.eeeeaaeee.. Dent.reia Purty) Therefore, the formatted answer is: Democratic Party
Ground-Iruth: Democratic Party
WehQSP-WehQTess-ff
Question:
what was jesse james killed with
LLM Response:
death. From the triplets, we can see that Jesse James was killed by Assassination.
Jesse
Now, we need to find cause of Assussination.
Assassination includes Assassination by 'Accident', Assussination by firearm, and Assussination by 'Suicide'. However, we kooking for Jesse James
From triplets,
death such Assassination ways which appear natural,
From the triplets, we can see that Firearm is cause of death that includes causes of death such as Assassination by firearm. Fircarm.
Therefore, Jesse So, the answer is:
ans: Firearm
Ground-truth:
Firearm
27

---

Published as a conference paper at ICLR 2025
F.2
LLM REFUSAL TO ANSWER DUE TO INSUFFICIENT INFORMATION
SehhSs--ebQTeet-ee
Question:
how deep lake merritt oakland
LLM Response:
To find the depth of Lake Merritt Oakland, we need to look for any information related the depth of Lake Merritt. From triplets,
seo that Lake Merritt lake, but there direct information about its depth.
However, we can see that Lake Merril is National Historic Landmark (m.06 4414, protected
cultural site listinn designation
or_cultural site, 44(4) National
I Landmark 04f44 protected_site sites natural perrrrrulll prc_utumil_stee letttsttssiinn Jisting category oe, priteeii,, National Historic Landmark).
We can also see that Lake Merritt is lake (Lake Merritt, common. wtwp://mm.ppiic.moooo.oo.o notable Lake) and located in Oakland (Lake Merritt Incation. location.com contiioncontaineehyy. Oaklaad)).
But there is no information about the depth of Lake Merritt. However,
see there mention 7L50), but this does not provide any information about the depth of the lake. Therefore, hased ans: not available
geolocation 0c.70000 Monitt, location location geoeloongooooaiinnn
given triplets,
any information depth Merrit.
Ground-truth:
3.048
WebOSP-WehQTest-0e
Question:
who does jeremy shockey play for in 2012
LLM Response:
find out who Jeremy Shockey played 2012, we need to the team he played and the year he played that team. From the triples, we
that Jeremy Shockey played for the New Orleans Saints, Carolina Panthers, Miami Hurricanes fontball, and New York Giants. However, need find
he played each of these From the triplcts, we can see that Jeremy Shockey played for the New Orleans Saints 2007,
the Carolina Panthers for the Miami lluriicaees football in 2001.
and the New York Giants in 2002.
However, there information ahout Jeremy Shockey playing for any of these 2012.
Therefore, we cunnot determine which team Jeremy Shockey played for in 2012. based on the given triplets. available
Ground-Iruth:
Carolina Panthers
CWQ-WehQTest-337 00338800044455555000000000222220
Question:
Harvard Board of Overseers is the school with what colors?
LLM Response:
To find the school with the colors that Harvard Roard of Overseers is part of, we need to find the parent organization of Harvard Board Overseers.
From the triplts,, we can see that Harvard Board of Overseers is child of Harvard University.
Now, we need find colors Harvard University.
From the triplets, we see that there no direct information about the colors of Harvard University, However, we see that Harvard University is a purent of Harvard Bourd Overseers, and Harvard Board of Overseers child Harvard University. itself (Harvard
But also that Harvard University Harvard Harvard University), which means it's an educational institution. wtt.c/cuattonndcccatto ctaiiiiooaiiissttttiiio cuucitttion.cmppusee However, there information about the colors Harvard University.
Ground-truth: Crimson
28

---

Published as a conference paper at ICLR 2025
movie starring Logan htg.://ss....222000da21222222aad
Question:
was edited Nordberg?
LLM Respouse:
To find the movie starting Logan Lerman that edited Thomas Nordherg, to find the movies edited by Thomas J
Nordherg and then find the ones starring Logan Lerman.
From uiplets, we see following muvies edited by Nordberg:
Scary Movie
Host
Love You Phillip Mortis
Turn
Any Given Sunday
The Guardian
Alexander
Now; we need to find the movies among these that star Logan I Lerman.
From triplets, we can see that Logan Lerman starred the following movies:
-hat Women Want
'The Perks Being Wallflower
Fury
Percy Jackson: Sca of Monsters
Noah
Riding in Cars with Boys
3:10 Yuma
However, But, we can see that Fury not edited by Thomas Northeer but Fury is related another film edited by Thomas Nordberg. The film Fary related Fury (2014) which is film edited by Thomas Nordherg.
these movies among edited Thomas Nordberg.
Ground-truth:
What Women Want
Question:
Who his govemmenntal position from December 12, 1808 Governor of Ohio
Respome:
To find the answer, need to find person who held governmental position from December 12, 1808 and was Govemor of
201
From the triplets,
that there direct information about person holding govermeenal position from December 12,
1808. However, we can find a person who was Governor of Ohio 2011.
From triplets,
Mike DeWine
Governor
However, there no information about Mike DeWine bolding governmental position from December 12, 1808. We
information about any other person holding govermeental position December
Therefore, we cannot find person who held governmental position from December 12, 1808 and was Govemor of Ohio in 2011.
ans: not available
Ground-truth:
Meigs.
29


Reviews:
[{"review_id": "jsxoxrEFiI", "reviewer": "Reviewer_A8Bg", "summary": "This paper introduces the SubgraphRAG framework for the first time, a knowledge-graph (KG)-based generation method specifically designed to optimize the efficiency and accuracy of large language models (LLMs) in retrieval-augmented generation (RAG) tasks. By incorporating a lightweight multilayer perceptron (MLP) and a parallel triple-scoring mechanism, SubgraphRAG efficiently retrieves subgraphs relevant to the query and improves retrieval precision through Directional Distance Encoding (DDE). This method effectively balances retrieval accuracy and computational complexity, achieving adaptive subgraph retrieval tailored to different LLM reasoning capabilities for the first time.", "questions": "1. Could LLM-driven retrieval better identify information required for multi-hop reasoning?\n\n2. Does topic entity bias lead to misleading results in ambiguous queries, and is there a more generalized retrieval strategy available? \n\n3. Should triples be directly converted to natural language for LLMs to better understand and utilize them?\n\n4. Could a complete example using a typical query illustrate the entire retrieval-reasoning workflow of SubgraphRAG?\n\n5. How does SubgraphRAG perform on other mainstream KGQA datasets, and is it equally effective?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "**Quality**: The experimental design is comprehensive, covering two major multi-hop datasets in KG-based question answering tasks (WebQSP and CWQ) and providing detailed comparisons with multiple baseline methods. Results show that SubgraphRAG outperforms existing baselines across several metrics, demonstrating higher retrieval efficiency and answer accuracy, particularly in multi-hop reasoning and complex structure retrieval tasks. Additionally, ablation studies validate the independent contributions and effectiveness of each component, such as DDE and MLP.\n\n**Clarity**: The paper is well-structured, with clear explanations progressing from the background of KG-augmented generation tasks to the design of each module within SubgraphRAG. Not only does the paper provide a flowchart illustrating the SubgraphRAG framework, but it also explains each step’s design principles and implementation details in depth. Moreover, the experimental section offers a detailed analysis of different design variations, which helps readers understand the role of each component.\n\n**Significance**: SubgraphRAG provides an innovative and efficient retrieval-reasoning method for KG-augmented generation tasks. Compared to existing methods, SubgraphRAG achieves strong generalizability and extensibility through flexible subgraph retrieval strategies and LLM reasoning without fine-tuning. It offers more robust support for knowledge-driven generation tasks, contributing to the broader and deeper application of KGs in practical generation tasks.", "weaknesses": "1. **Insufficient Exploration of LLMs’ Potential for Retrieval Support**: SubgraphRAG primarily relies on an MLP and Directional Distance Encoding (DDE) for subgraph retrieval, yet LLMs inherently excel in handling complex semantics and relational structures. Directly assigning the retrieval process to LLMs could improve flexibility in multi-hop reasoning scenarios and adaptivity to complex KG structures, potentially enhancing the framework’s ability to handle diverse types of queries.\n\n2. **The Limitation of Topic Entity Bias on Generalizability**: The paper assumes that all queries can be guided by topic entity-induced inductive biases for retrieval. However, this topic-driven approach may restrict information coverage or introduce noise in cases involving ambiguous or polysemous queries. Relying on topic entities might lead to suboptimal performance for non-topic-focused queries.\n\n3. **Limitations of Structured Triples**: SubgraphRAG presents triples to LLMs in structured form, rather than transforming them into natural language descriptions, which misses an opportunity to leverage LLMs’ strengths in natural language understanding and reasoning. Converting triples to natural language may allow LLMs to more fully exploit the triple information and achieve more accurate reasoning.\n\n4. **Lack of a Complete Retrieval Process Example**: Although the paper details the algorithmic flow of SubgraphRAG, it lacks a full example from query to retrieval and reasoning to answer generation. Providing a concrete example, such as with a typical query from WebQSP or CWQ, would enhance readers’ understanding of the procedural details.\n\n5. **Limited Experimental Dataset Scope**: Experiments are conducted solely on WebQSP and CWQ datasets, leaving out other mainstream multi-hop KGQA benchmarks, such as HotpotQA. A broader range of dataset tests could further verify the method’s generalizability and robustness across various KGQA tasks.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "pEJAZXzPPR", "reviewer": "Reviewer_Jjux", "summary": "This manuscript introduces a framework called SubgraphRAG, which is a KG-based RAG system designed to enhance the reasoning capabilities of LLMs by integrating structured information from knowledge graphs.\n\n-  The framework employs a lightweight multilayer perceptron (MLP) combined with a parallel triple scoring mechanism, proposing to adopt a retriever that allows for subgraph distribution factorization. This means that each triple can be optimized independently, which improves the efficiency of subgraph retrieval.\n-  By encoding directional structural distances as structural features using DDE, the accuracy of retrieval is enhanced.", "questions": "- Some Ambiguous Expressions Lack Further Mathematical Definition or Explanation\n  - What is ‘scalable’ and how to prove whether the method is ‘scalable’ or not?\n  - What constitutes suitable prompting?\n\n- The authors noted that baselines report Hit@1 but compute Hit, assessing if any correct answer is in the LLM's response. There's also confusion over metrics like extract match and Hit@1 in the research community. Clarification of these metrics in the appendix is needed, and defining another \"Hit metric\" is not advised.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- Efficiency and Scalability: As the authors present in Table 1, SubgraphRAG demonstrates high efficiency and scalability in the retrieval process. Additionally, the flexible form of retrieved subgraphs, with adjustable sizes—subgraphs formed by top-triples can accommodate various LLMs with diverse reasoning capabilities.\n- By combining DDE and MLP, SubgraphRAG surpasses more complex models, such as RoG and G-Retriever, in terms of covering key triples and entities.", "weaknesses": "- I affirm: The optimization objective defined in the article states that the SubgraphRAG retriever distribution Qθ can be factorized into distributions over triples, which means that the retrieved subgraphs do not necessarily have to follow a fixed type (such as trees or paths). This design allows for efficient training, efficient subgraph retrieval, flexible subgraph types, and adjustable subgraph sizes, rather than relying solely on path retrieval. This differs from the methods widely discussed in the current research community for finding retrieval triples and performing graph RAG, such as RoG and G-Retriever, which the article compares.\n\n  - However，The main experiments require more graph retrieval methods, especially path retrieval, for comparison. For example:\n\n  > Haoran Luo,et al., ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models. ACL (Findings) 2024\n\n  > Guanming Xiong, et al., Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models. ACL (1) 2024\n\n\n  - I am also curious as to why the ToG method is not reproducible, as it seems to be a strong contender in the Hit results. The calculation of ToG's hit@1 is actually based on ToG only finding the first match in the answer text, which is considered as the answer by the large model to calculate Hits@1. However, I do not believe this is the reason for the irreproducibility. Interactive-KBQA has reproduced ToG's results.\n\n\n[Subtle weaknesses]\n\n- It was not clearly stated whether the experimental results of comparison methods such as RoG and G-Retriever were reproduced a second time or referenced from the original text. This is because the referenced StructGPT missed many tests, while the results of baseline methods like RoG should have been reproduced by the authors themselves.\n\n[Subtle weaknesses]\n\n- Typos and Formatting Issues\n  - Naming  Eq.3 and q)). \n  - line 052 ``\n  - line 161 .\n  - line 986 . \n  - line 966 ，968，970 .\n- There are some grammar errors, not listed one by one", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OwBiSW6rvy", "reviewer": "Reviewer_Yzun", "summary": "This paper proposes SubgraphRAG, extending the KG-based RAG framework. To be specific, this method integrates a lightweight multilayer perceptron (MLP) with a parallel triple-scoring mechanism for efficient subgraph retrieval while enhancing retrieval accuracy by encoding directional structural distances. In addition, SubgraphRAG strikes a balance between model complexity and reasoning power as the size of retrieved subgraphs can be flexibly adjusted to match the query’s need and downstream LLM’s reasoning power. Extensive experiments demonstrate the effectiveness of the proposed methods.", "questions": "Please see in **Weaknesses** above.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The idea of employing lightweight models for retrieval and using large language models for reasoning is interesting, sensible, and intuitive.\n2. The proposed method is efficient in terms of training time, requiring only the training of an MLP.", "weaknesses": "1. **May Lack novelty.** The novelty behind the proposed method may be insufficient as embedding-based retrieval methods for triplets [1] resemble existing approaches in the literature. Embedding-based retrieval methods for triplets utilize the embeddings of questions and triplets to retrieve the relevant triples. However, the proposed approach merely concatenates $z_\\tau$ with the previous embeddings, which may lack sufficient innovation.\n2. **Baseline methods.** As the proposed methods follow the retrieved-based paradigms, they would be better compared to more retrieved-based methods, such as UniKGQA [2], Subgraph Retrieval (SR) [3], and GNN-RAG [4], which are relevant to the proposed method and derive new SOTA for this task.\n3. **Experimental Results.** As shown in Table 1 for the evaluation for retrieval recall, it would be better to present the number of extracted triplets in a new column for each baseline to ensure fair comparison. The same for Table 3 to demonstrate the QA performance.\n4. In line 226, the paper claims that \"GNNs are known to have limited representation power,\" which is why GNNs were not chosen for the retriever. However, as shown in Table 5, the paper presents different variants of SubgraphRAG by incorporating various retrieval methods. Including a GNN-based retrieval method for comparison would strengthen the evaluation, such as RevRea [5] as a GNN-based retriever.\n\n\n[1] Li, Shiyang, et al. \"Graph Reasoning for Question Answering with Triplet Retrieval.\" Findings of the Association for Computational Linguistics: ACL 2023. 2023.\n\n[2] Jiang, Jinhao, et al. \"UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph.\" The Eleventh International Conference on Learning Representations.\n\n[3] Zhang, Jing, et al. \"Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering.\" Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.\n\n[4] Mavromatis, Costas, and George Karypis. \"GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning.\" arXiv preprint arXiv:2405.20139 (2024).\n\n[5] Mavromatis, Costas, and George Karypis. \"ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs.\" 2022 Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "hX2ht64z0n", "reviewer": "Reviewer_y2dZ", "summary": "The paper presents SubgraphRAG, a graph retrieval method for Knowledge Graph Question Answering (KGQA). SubgraphRAG trains a retrieval module to score relevant triplets of the KG based on training questions. During inference, SubgraphRAG retrieves top-scored relevant triplets based on the question and uses them as additional context to the LLM for KGQA. Experiments are performs on WebQSP and CWQ benchmarks, evaluating retrieval and downstream KGQA performance.", "questions": "Please refer to the previous comments. Additionally, I have few further questions/comments:\n- Q1) Could you provide some citations on what \"locality-sensitive hashing, designed for similarity search\" in Line 076 means?\n- Q2) Line 101 (necessity for lightweight retrieval) and the final choice of MLP contradict Line 095 (LSTMs and GNNs are limited because their lightweight). \n- Q3) In Line 323, SubgraphRAG  continues to use shortest paths, despite their limitations noted in Line 097. \n- Q4) The prompt for GPT-4o scoring (Fig.7) does not include the ground-truth answer. Could the evaluation results change if we include the ground-truth answers?\n- Q5) Line 429 mentions that RoG suffers from label leakage. However, SubgraphRAG utilized GPT-4o, whose training data is unknown and could also be affected by training leakage.", "limitations": "", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "The paper strengths are summarized below:\n\n- S1) The retrieval module is lightweight pre-trained text encoder, followed by MLPs. This allows fast retrieval and does not incur significant latency during KGQA with LLMs.\n- S2) The experimentation includes ablation studies in both retrieval and downstream KGQA performance, which are helpful to understand the benefits of each of the competing methods.\n- S3) The paper reads nicely and is easy to follow.", "weaknesses": "The main weaknesses of the paper are summarized below:\n\n- W1) SubgraphRAG emphasizes the necessity for  lightweight retrieval for KGQA. However, GNNs are already established as lightweight retrievers (as also shown in Table 1) and specifically designed for handling KGs. Although the paper references some GNN-based approaches [1,2], it does not compare SubgraphRAG with them. Without these comparisons, it is unclear why SubgraphRAG favors MLPs over GNNs and what the rationale behind this choice is.\n\n- W2) The retrieval evaluation, as presented in Table 1, relies solely on the recall metric. However, SubgraphRAG retrieves a larger number of triplets (e.g., top-100), which should naturally achieve higher recall than the baselines. To ensure a fair comparison, additional metrics such as precision, F1 score, or recall @ k should also be included. Furthermore, in the KGQA results, SubgraphRAG utilizes more advanced LLMs, such as GPT-4o, compared to the baselines. This raises questions about the advantages of SubgraphRAG when using the same LLMs. For instance, SubgraphRAG combined with LLaMa 3.1-8B does not outperform RoG in the CWQ dataset.\n\n[1] EtD: Explore then determine: A gnn-llm synergy framework for reasoning over knowledge graph. arXiv preprint arXiv:2406.01145, 2024a. \n[2] GNN-RAG: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139, 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "UNxjsRPrbq", "reviewer": "Reviewer_q2Wy", "summary": "This paper proposes a novel KBQA method named SubgraphRAG. Instead of incorporating more powerful LLMs and/or advanced SFT techniques, SubgraphRAG focuses on the retrieval part. It extends the existing KG-based RAG framework with an MLP and a newly proposed triple-scoring mechanism, which shows to be effective and efficient. The retrieved subgraphs allow the proposed method to achieve SOTA performance on smaller LLMs without SFT.", "questions": "1) As mentioned in Section 3.2, the authors mentioned that the designed prompt template is shown in Figure 2. In addition, this reviewer is interested to know some more *details* about the design of “in-context demonstrations”.\na) Does SubgraphRAG utilize the same “in-context demonstration” for all the questions?\nb) How were these in-context demonstrations designed? Specifically, what are the criteria for the selection of these demonstrations? How do you generate the explanations? \n\n2) The authors raise concerns about the reproducibility issue of the ToG baseline. Since ToG relies on closed-source LLMs such as GPT4, this reviewer believes that the performance can be greatly affected by the specific versions of LLMs. SubgraphRAG also uses closed-source LLMs for evaluation. Which specific version of GPT4o is being used in this work? (Please also include this in the *implementation* part in the paper.)", "limitations": "", "rating": 8, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "1) The problem is clearly defined. (Section 3.1, equation 1 and 2)\n2) The proposed method is easy to follow. It also includes several details for the reader to\nreproduce the results, e.g. the initialization of embeddings, the introduction of DDE.\n3) The proposed retrieval method is efficient and effective. Better retrievers enhance the\nperformance ceiling of RAG.\n4) This paper includes detailed ablation study (Q1, Q2, and Q5), which proves the effective-\nness of the retriever part.\n5) The proposed method achieves SOTA performance on low-scale pretrained LLMs.", "weaknesses": "1) This reviewer suggests the authors move the main results and related analysis to an earlier\nsubsection of the experiments part, and group the ablation studies together.\n2) The WebQSP dataset does not include official explanations for its answers. However, the proposed method requires explanations to perform reliable in-context learning, which relies on external knowledge other than the dataset or the KG.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)