Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Authors: Mufei Li, Siqi Miao, Pan Li
Primary area: applications to computer vision, audio, language, and other modalities
Keywords: Knowledge Graphs, Large Language Models, Retrieval-Augmented Generation, Retrieval

Abstract:
Large Language Models (LLMs) demonstrate strong reasoning abilities but face limitations such as hallucinations and outdated knowledge. Knowledge Graph (KG)-based Retrieval-Augmented Generation (RAG) addresses these issues by grounding LLM outputs in structured external knowledge from KGs. However, current KG-based RAG frameworks still struggle to optimize the trade-off between retrieval effectiveness and efficiency in identifying a suitable amount of relevant graph information for the LLM to digest. We introduce SubgraphRAG, extending the KG-based RAG framework that retrieves subgraphs and leverages LLMs for reasoning and answer prediction. Our approach innovatively integrates a lightweight multilayer perceptron (MLP) with a parallel triple-scoring mechanism for efficient and flexible subgraph retrieval while encoding directional structural distances to enhance retrieval effectiveness. The size of retrieved subgraphs can be flexibly adjusted to match the query's needs and the downstream LLM's capabilities. This design strikes a balance between model complexity and reasoning power, enabling scalable and generalizable retrieval processes. Notably, based on our retrieved subgraphs, smaller LLMs like Llama3.1-8B-Instruct deliver competitive results with explainable reasoning, while larger models like GPT-4o achieve state-of-the-art accuracy compared with previous baselines—all without fine-tuning. Extensive evaluations on the WebQSP and CWQ benchmarks highlight SubgraphRAG's strengths in efficiency, accuracy, and reliability by reducing hallucinations and improving response grounding.

Decision: accept (poster)
Reviewer scores: 6, 5, 6, 5, 8

Reviews and discussion:
<thread>
creation date:
1729505466151

thread_type:
official_review

<thread_object>
creator: reviewer_q2wy

<content>
summary:
This paper proposes a novel KBQA method named SubgraphRAG. Instead of incorporating more powerful LLMs and/or advanced SFT techniques, SubgraphRAG focuses on the retrieval part. It extends the existing KG-based RAG framework with an MLP and a newly proposed triple-scoring mechanism, which shows to be effective and efficient. The retrieved subgraphs allow the proposed method to achieve SOTA performance on smaller LLMs without SFT.

strengths:
1) The problem is clearly defined. (Section 3.1, equation 1 and 2)
2) The proposed method is easy to follow. It also includes several details for the reader to
reproduce the results, e.g. the initialization of embeddings, the introduction of DDE.
3) The proposed retrieval method is efficient and effective. Better retrievers enhance the
performance ceiling of RAG.
4) This paper includes detailed ablation study (Q1, Q2, and Q5), which proves the effective-
ness of the retriever part.
5) The proposed method achieves SOTA performance on low-scale pretrained LLMs.

weaknesses:
1) This reviewer suggests the authors move the main results and related analysis to an earlier
subsection of the experiments part, and group the ablation studies together.
2) The WebQSP dataset does not include official explanations for its answers. However, the proposed method requires explanations to perform reliable in-context learning, which relies on external knowledge other than the dataset or the KG.

questions:
1) As mentioned in Section 3.2, the authors mentioned that the designed prompt template is shown in Figure 2. In addition, this reviewer is interested to know some more *details* about the design of “in-context demonstrations”.
a) Does SubgraphRAG utilize the same “in-context demonstration” for all the questions?
b) How were these in-context demonstrations designed? Specifically, what are the criteria for the selection of these demonstrations? How do you generate the explanations? 

2) The authors raise concerns about the reproducibility issue of the ToG baseline. Since ToG relies on closed-source LLMs such as GPT4, this reviewer believes that the performance can be greatly affected by the specific versions of LLMs. SubgraphRAG also uses closed-source LLMs for evaluation. Which specific version of GPT4o is being used in this work? (Please also include this in the *implementation* part in the paper.)
</content>
</thread_object>

</thread>
<thread>
creation date:
1730051332160

thread_type:
official_review

<thread_object>
creator: reviewer_y2dz

<content>
summary:
The paper presents SubgraphRAG, a graph retrieval method for Knowledge Graph Question Answering (KGQA). SubgraphRAG trains a retrieval module to score relevant triplets of the KG based on training questions. During inference, SubgraphRAG retrieves top-scored relevant triplets based on the question and uses them as additional context to the LLM for KGQA. Experiments are performs on WebQSP and CWQ benchmarks, evaluating retrieval and downstream KGQA performance.

strengths:
The paper strengths are summarized below:

- S1) The retrieval module is lightweight pre-trained text encoder, followed by MLPs. This allows fast retrieval and does not incur significant latency during KGQA with LLMs.
- S2) The experimentation includes ablation studies in both retrieval and downstream KGQA performance, which are helpful to understand the benefits of each of the competing methods.
- S3) The paper reads nicely and is easy to follow.

weaknesses:
The main weaknesses of the paper are summarized below:

- W1) SubgraphRAG emphasizes the necessity for  lightweight retrieval for KGQA. However, GNNs are already established as lightweight retrievers (as also shown in Table 1) and specifically designed for handling KGs. Although the paper references some GNN-based approaches [1,2], it does not compare SubgraphRAG with them. Without these comparisons, it is unclear why SubgraphRAG favors MLPs over GNNs and what the rationale behind this choice is.

- W2) The retrieval evaluation, as presented in Table 1, relies solely on the recall metric. However, SubgraphRAG retrieves a larger number of triplets (e.g., top-100), which should naturally achieve higher recall than the baselines. To ensure a fair comparison, additional metrics such as precision, F1 score, or recall @ k should also be included. Furthermore, in the KGQA results, SubgraphRAG utilizes more advanced LLMs, such as GPT-4o, compared to the baselines. This raises questions about the advantages of SubgraphRAG when using the same LLMs. For instance, SubgraphRAG combined with LLaMa 3.1-8B does not outperform RoG in the CWQ dataset.

[1] EtD: Explore then determine: A gnn-llm synergy framework for reasoning over knowledge graph. arXiv preprint arXiv:2406.01145, 2024a. 
[2] GNN-RAG: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139, 2024.

questions:
Please refer to the previous comments. Additionally, I have few further questions/comments:
- Q1) Could you provide some citations on what "locality-sensitive hashing, designed for similarity search" in Line 076 means?
- Q2) Line 101 (necessity for lightweight retrieval) and the final choice of MLP contradict Line 095 (LSTMs and GNNs are limited because their lightweight). 
- Q3) In Line 323, SubgraphRAG  continues to use shortest paths, despite their limitations noted in Line 097. 
- Q4) The prompt for GPT-4o scoring (Fig.7) does not include the ground-truth answer. Could the evaluation results change if we include the ground-truth answers?
- Q5) Line 429 mentions that RoG suffers from label leakage. However, SubgraphRAG utilized GPT-4o, whose training data is unknown and could also be affected by training leakage.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730387713024

thread_type:
official_review

<thread_object>
creator: reviewer_yzun

<content>
summary:
This paper proposes SubgraphRAG, extending the KG-based RAG framework. To be specific, this method integrates a lightweight multilayer perceptron (MLP) with a parallel triple-scoring mechanism for efficient subgraph retrieval while enhancing retrieval accuracy by encoding directional structural distances. In addition, SubgraphRAG strikes a balance between model complexity and reasoning power as the size of retrieved subgraphs can be flexibly adjusted to match the query’s need and downstream LLM’s reasoning power. Extensive experiments demonstrate the effectiveness of the proposed methods.

strengths:
1. The idea of employing lightweight models for retrieval and using large language models for reasoning is interesting, sensible, and intuitive.
2. The proposed method is efficient in terms of training time, requiring only the training of an MLP.

weaknesses:
1. **May Lack novelty.** The novelty behind the proposed method may be insufficient as embedding-based retrieval methods for triplets [1] resemble existing approaches in the literature. Embedding-based retrieval methods for triplets utilize the embeddings of questions and triplets to retrieve the relevant triples. However, the proposed approach merely concatenates $z_\tau$ with the previous embeddings, which may lack sufficient innovation.
2. **Baseline methods.** As the proposed methods follow the retrieved-based paradigms, they would be better compared to more retrieved-based methods, such as UniKGQA [2], Subgraph Retrieval (SR) [3], and GNN-RAG [4], which are relevant to the proposed method and derive new SOTA for this task.
3. **Experimental Results.** As shown in Table 1 for the evaluation for retrieval recall, it would be better to present the number of extracted triplets in a new column for each baseline to ensure fair comparison. The same for Table 3 to demonstrate the QA performance.
4. In line 226, the paper claims that "GNNs are known to have limited representation power," which is why GNNs were not chosen for the retriever. However, as shown in Table 5, the paper presents different variants of SubgraphRAG by incorporating various retrieval methods. Including a GNN-based retrieval method for comparison would strengthen the evaluation, such as RevRea [5] as a GNN-based retriever.


[1] Li, Shiyang, et al. "Graph Reasoning for Question Answering with Triplet Retrieval." Findings of the Association for Computational Linguistics: ACL 2023. 2023.

[2] Jiang, Jinhao, et al. "UniKGQA: Unified Retrieval and Reasoning for Solving Multi-hop Question Answering Over Knowledge Graph." The Eleventh International Conference on Learning Representations.

[3] Zhang, Jing, et al. "Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering." Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 2022.

[4] Mavromatis, Costas, and George Karypis. "GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning." arXiv preprint arXiv:2405.20139 (2024).

[5] Mavromatis, Costas, and George Karypis. "ReaRev: Adaptive Reasoning for Question Answering over Knowledge Graphs." 2022 Findings of the Association for Computational Linguistics: EMNLP 2022. 2022.

questions:
Please see in **Weaknesses** above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730648397735

thread_type:
official_review

<thread_object>
creator: reviewer_jjux

<content>
summary:
This manuscript introduces a framework called SubgraphRAG, which is a KG-based RAG system designed to enhance the reasoning capabilities of LLMs by integrating structured information from knowledge graphs.

-  The framework employs a lightweight multilayer perceptron (MLP) combined with a parallel triple scoring mechanism, proposing to adopt a retriever that allows for subgraph distribution factorization. This means that each triple can be optimized independently, which improves the efficiency of subgraph retrieval.
-  By encoding directional structural distances as structural features using DDE, the accuracy of retrieval is enhanced.

strengths:
- Efficiency and Scalability: As the authors present in Table 1, SubgraphRAG demonstrates high efficiency and scalability in the retrieval process. Additionally, the flexible form of retrieved subgraphs, with adjustable sizes—subgraphs formed by top-triples can accommodate various LLMs with diverse reasoning capabilities.
- By combining DDE and MLP, SubgraphRAG surpasses more complex models, such as RoG and G-Retriever, in terms of covering key triples and entities.

weaknesses:
- I affirm: The optimization objective defined in the article states that the SubgraphRAG retriever distribution Qθ can be factorized into distributions over triples, which means that the retrieved subgraphs do not necessarily have to follow a fixed type (such as trees or paths). This design allows for efficient training, efficient subgraph retrieval, flexible subgraph types, and adjustable subgraph sizes, rather than relying solely on path retrieval. This differs from the methods widely discussed in the current research community for finding retrieval triples and performing graph RAG, such as RoG and G-Retriever, which the article compares.

  - However，The main experiments require more graph retrieval methods, especially path retrieval, for comparison. For example:

  > Haoran Luo,et al., ChatKBQA: A Generate-then-Retrieve Framework for Knowledge Base Question Answering with Fine-tuned Large Language Models. ACL (Findings) 2024

  > Guanming Xiong, et al., Interactive-KBQA: Multi-Turn Interactions for Knowledge Base Question Answering with Large Language Models. ACL (1) 2024


  - I am also curious as to why the ToG method is not reproducible, as it seems to be a strong contender in the Hit results. The calculation of ToG's hit@1 is actually based on ToG only finding the first match in the answer text, which is considered as the answer by the large model to calculate Hits@1. However, I do not believe this is the reason for the irreproducibility. Interactive-KBQA has reproduced ToG's results.


[Subtle weaknesses]

- It was not clearly stated whether the experimental results of comparison methods such as RoG and G-Retriever were reproduced a second time or referenced from the original text. This is because the referenced StructGPT missed many tests, while the results of baseline methods like RoG should have been reproduced by the authors themselves.

[Subtle weaknesses]

- Typos and Formatting Issues
  - Naming  Eq.3 and q)). 
  - line 052 ``
  - line 161 .
  - line 986 . 
  - line 966 ，968，970 .
- There are some grammar errors, not listed one by one

questions:
- Some Ambiguous Expressions Lack Further Mathematical Definition or Explanation
  - What is ‘scalable’ and how to prove whether the method is ‘scalable’ or not?
  - What constitutes suitable prompting?

- The authors noted that baselines report Hit@1 but compute Hit, assessing if any correct answer is in the LLM's response. There's also confusion over metrics like extract match and Hit@1 in the research community. Clarification of these metrics in the appendix is needed, and defining another "Hit metric" is not advised.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730733718442

thread_type:
official_review

<thread_object>
creator: reviewer_a8bg

<content>
summary:
This paper introduces the SubgraphRAG framework for the first time, a knowledge-graph (KG)-based generation method specifically designed to optimize the efficiency and accuracy of large language models (LLMs) in retrieval-augmented generation (RAG) tasks. By incorporating a lightweight multilayer perceptron (MLP) and a parallel triple-scoring mechanism, SubgraphRAG efficiently retrieves subgraphs relevant to the query and improves retrieval precision through Directional Distance Encoding (DDE). This method effectively balances retrieval accuracy and computational complexity, achieving adaptive subgraph retrieval tailored to different LLM reasoning capabilities for the first time.

strengths:
**Quality**: The experimental design is comprehensive, covering two major multi-hop datasets in KG-based question answering tasks (WebQSP and CWQ) and providing detailed comparisons with multiple baseline methods. Results show that SubgraphRAG outperforms existing baselines across several metrics, demonstrating higher retrieval efficiency and answer accuracy, particularly in multi-hop reasoning and complex structure retrieval tasks. Additionally, ablation studies validate the independent contributions and effectiveness of each component, such as DDE and MLP.

**Clarity**: The paper is well-structured, with clear explanations progressing from the background of KG-augmented generation tasks to the design of each module within SubgraphRAG. Not only does the paper provide a flowchart illustrating the SubgraphRAG framework, but it also explains each step’s design principles and implementation details in depth. Moreover, the experimental section offers a detailed analysis of different design variations, which helps readers understand the role of each component.

**Significance**: SubgraphRAG provides an innovative and efficient retrieval-reasoning method for KG-augmented generation tasks. Compared to existing methods, SubgraphRAG achieves strong generalizability and extensibility through flexible subgraph retrieval strategies and LLM reasoning without fine-tuning. It offers more robust support for knowledge-driven generation tasks, contributing to the broader and deeper application of KGs in practical generation tasks.

weaknesses:
1. **Insufficient Exploration of LLMs’ Potential for Retrieval Support**: SubgraphRAG primarily relies on an MLP and Directional Distance Encoding (DDE) for subgraph retrieval, yet LLMs inherently excel in handling complex semantics and relational structures. Directly assigning the retrieval process to LLMs could improve flexibility in multi-hop reasoning scenarios and adaptivity to complex KG structures, potentially enhancing the framework’s ability to handle diverse types of queries.

2. **The Limitation of Topic Entity Bias on Generalizability**: The paper assumes that all queries can be guided by topic entity-induced inductive biases for retrieval. However, this topic-driven approach may restrict information coverage or introduce noise in cases involving ambiguous or polysemous queries. Relying on topic entities might lead to suboptimal performance for non-topic-focused queries.

3. **Limitations of Structured Triples**: SubgraphRAG presents triples to LLMs in structured form, rather than transforming them into natural language descriptions, which misses an opportunity to leverage LLMs’ strengths in natural language understanding and reasoning. Converting triples to natural language may allow LLMs to more fully exploit the triple information and achieve more accurate reasoning.

4. **Lack of a Complete Retrieval Process Example**: Although the paper details the algorithmic flow of SubgraphRAG, it lacks a full example from query to retrieval and reasoning to answer generation. Providing a concrete example, such as with a typical query from WebQSP or CWQ, would enhance readers’ understanding of the procedural details.

5. **Limited Experimental Dataset Scope**: Experiments are conducted solely on WebQSP and CWQ datasets, leaving out other mainstream multi-hop KGQA benchmarks, such as HotpotQA. A broader range of dataset tests could further verify the method’s generalizability and robustness across various KGQA tasks.

questions:
1. Could LLM-driven retrieval better identify information required for multi-hop reasoning?

2. Does topic entity bias lead to misleading results in ambiguous queries, and is there a more generalized retrieval strategy available? 

3. Should triples be directly converted to natural language for LLMs to better understand and utilize them?

4. Could a complete example using a typical query illustrate the entire retrieval-reasoning workflow of SubgraphRAG?

5. How does SubgraphRAG perform on other mainstream KGQA datasets, and is it equally effective?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732500797556

thread_type:
official_comment

<thread_object>
creator: area_chair_e8yf

<content>
title:
Please reply to the authors' response.

comment:
Dear reviewers,

The ICLR author discussion phase is ending soon. Could you please review the authors' responses and take the necessary actions? Feel free to ask additional questions during the discussion. If the authors address your concerns, kindly acknowledge their response and update your assessment as appropriate.


Best,
AC
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper introduces SubgraphRAG, a framework designed to enhance retrieval-augmented generation by integrating a lightweight multilayer perceptron (MLP) with a parallel triple-scoring mechanism for efficient subgraph retrieval. The framework aims to balance retrieval accuracy and efficiency, improving response grounding and reducing hallucinations. The paper demonstrates its strengths, including efficiency and scalability in retrieval processes, flexibility in adjusting subgraph sizes to match different LLM reasoning capabilities, and strong performance in retrieval efficiency and answer accuracy, particularly in multi-hop reasoning tasks. 

However, there are concerns about the method's generalizability across different knowledge graphs and datasets, and the evaluation metrics used. Reviewers suggested including more metrics like precision and F1 score for a fairer comparison and addressing issues with reproducing results of some baseline methods like ToG. Detailed evaluations across different input lengths and practical latency costs were also recommended. The authors have provided detailed responses to reviewers' concerns, including additional experiments and comparisons with new baselines. Still, they are encouraged to include more discussion or experiments on the retrieval size effects (e.g., reducing the number of retrieval triplets) in the final manuscript.

Overall, the reviewers' comments indicate that the pros outweigh the cons, leading to a recommendation for acceptance.