Title: Can Knowledge-Graph-based Retrieval Augmented Generation Really Retrieve What You Need?
Venue: NeurIPS 2025
Accepted: yes
Keywords: graph machine learning

knowledge graph

LLM
Abstract: Retrieval-Augmented Generation (RAG) based on knowledge graphs (KGs) enhances large language models (LLMs) by providing structured and interpretable external knowledge.
However, existing KG-based RAG methods struggle to retrieve accurate and diverse information from text-rich KGs for complex real-world queries.
Process Reward Models (PRMs) offer a way to align the retrieval process of KG-based RAG with query-specific knowledge requirements, 
but they heavily rely on process-level supervision signals that are expensive and hard to obtain on KGs.
To address this challenge, we propose GraphFlow, a framework that efficiently retrieves accurate and diverse knowledge required for real-world queries from text-rich KGs.
GraphFlow employs a transition-based flow matching objective to jointly optimize a retrieval policy and a flow estimator.
The flow estimator factorizes the reward of the retrieval outcome into the intermediate retrieval states.
Such reward factorization guides the retrieval policy to retrieve candidates from KGs in proportion to their reward.
This allows GraphFlow to explore high-quality regions of KGs that yield diverse and relevant results. 
We evaluate GraphFlow on the STaRK benchmark, which includes real-world queries from multiple domains over text-rich KGs. 
GraphFlow outperforms strong KG-RAG baselines, including GPT-4o, by 10\% on average in hit rate and recall. 
It also shows strong generalization to unseen KGs, demonstrating its effectiveness and robustness.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposed to use flow estimation as progress reward models in the training of KG-RAG LLMs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n1. The idea is interesting: this paper propose to learn a flow function as progress reward models for KG-RAG. I like the idea of introducing intermediate rewards and the idea of using Gflow network/ flow matching.\n\n----\nWeaknesses:\n1. **The learning method is problematic.** In the paper, the authors defined states to be a collection of documents. The flow estimation $F(s_{t+1})$ is calculated by putting all documents of $s_{t+1}$ into a LLM. More specifically, it is calculated by the last  token embedding transformed by a MLP. However, the conditional probability $P(s_{t+1}|s_{t})$ is calculated in a similar way. The only difference between these two is the MLP and the prompt. Within the same documents as input, I doubt how much difference we can have. The learning objective is the MSE between $\\log F(s_{t})-\\log F(s_{t+1})$ and $-\\log P(s_{t+1}|s_{t})$. I don't think this makes sense.\n\n2. **Weak Evaluation** The authors evaluated their method on the STaRK benchmark, which is a less common choice. Using STaRK has no problem, but I would like to see more results since I doubt the effectiveness of methodology. A more common choice of KG-RAG is either Freebase KG (datasets like WebQSP and CWQ), Wikidata KG (2WikiMultihopQA), or even Wikipedia (HotpotQA). The Wikipedia corpus has hyperlinks between entities and has a document for each entity, which exactly matches the description of text-attributed KG in the paper. Moreover, the baselines are weak. The ToG method is the strongest one among the baselines. However, according to my own experience, I cannot reproduce the result reported in the ToG paper. (Its official GitHub has many reproduction issues.) So, I doubt the effectiveness of this method.\n\nQUESTIONS:\nQuestions:\n\nPlease refer to Weaknesses.\n\nAlso, since the training process relies on sampled trajectories who cover the final answer, does this means it eventually relies on terminal reward?\nConsidering recent progress of RL4LLM (e.g., the GRPO method) who only depends on terminal rewards , how different between intermediate supervised and terminal supervised in KG-RAG?\n\nSuggestion:\nMaybe finding a new way to model the flow estimation?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes GraphFlow, a new framework for improving Retrieval-Augmented Generation (RAG) using knowledge graphs (KGs). Existing KG-based RAG methods often fail to retrieve both accurate and diverse information for complex queries. Unlike prior approaches that require costly process-level supervision, GraphFlow uses a flow-based retrieval policy trained with only final outcome rewards. Inspired by GFlowNets, GraphFlow models retrieval as a multi-step decision process, jointly learning a policy and a flow estimator using a detailed balance objective. This enables effective exploration of high-reward regions in KGs. Experiments on the STaRK benchmark show that GraphFlow outperforms strong baselines (including GPT-4o) by around 10% in both accuracy and diversity, and generalizes well to unseen domains.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n- The writing and presentation is clear.\n- Their methodology is novel and well-motivated. Leveraging GFlowNets for this problem is an inspiring prospect.\n- Their empirical results show strong performance over other supervised methods, making it a potentially practical system.\n\nWeaknesses:\n- Although STARK is a reasonably good benchmark for evaluating KG-based RAG systems, it is specifically design to test performance on a very high quality KG coupled with documents. The more general setting of KG-based RAG, in which a KG is automatically extracted from text [GraphRAG, HippoRAG, LightRAG, RAPTOR] and then leveraged for more effective retrieval, would be a more interesting and impactful testing ground for this framework. This is not grounds for rejection but a comment on the potential significance on this method.\n\nQUESTIONS:\n- Why is GraphFlow deployed using LLaMA3-8B-Instruct while ToG uses LLaMA3-8B?\n- I can't find the training/dev/test dataset statistics used for SFT, PRM and GraphFlow. Please make sure that this information is clearly provided in the final paper?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper explores the use of Knowledge Graphs to enhance Retrieval-Augmented Generation for large language models. The authors propose a novel framework called GraphFlow that aims to efficiently retrieve accurate and diverse knowledge required for complex queries from text-rich KGs. GraphFlow employs a detailed balance objective with local exploration to jointly optimize a retrieval policy and a flow estimator. This approach allows GraphFlow to explore relevant regions of KGs, yielding diverse and accurate results. The framework is evaluated on the STaRK benchmark, demonstrating significant performance gains over existing KG-based RAG methods.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n+ The introduction of GraphFlow, which uses a detailed balance objective and local exploration, is a novel contribution that addresses the limitations of existing KG-based RAG methods. GraphFlow shows a 10% performance gain in both retrieval accuracy and diversity metrics compared to strong baselines, including GPT-4o.\n+ The framework demonstrates strong generalization capabilities by effectively retrieving information from unseen KGs to support new-domain queries. The authors provide extensive experimental results on the STaRK benchmark, covering multiple domains such as e-commerce, academic, and biomedical KGs.\n\nWeaknesses:\n- The commonly used KGVQA datasets CWQ and WebQSP are not included in the experiments. Including results on these standard benchmarks would help better illustrate the generalizability and robustness of the proposed framework.\n- The effectiveness of GraphFlow relies on the availability of high-quality, text-rich KGs, which may not always be accessible or well-maintained.\n\nQUESTIONS:\n1.\tThe supplementary material notes that only a handful of KG-based RAG models have been evaluated on the text-rich STaRK benchmark, whereas most prior work relies on simpler KGQA datasets such as CWQ and WebQSP. Could the authors explain why CWQ and WebQSP were omitted from the experiments? Reporting results on these widely used benchmarks would provide a clearer picture of the framework’s generalizability.\n2.\tCan the authors provide more examples of real-world applications where GraphFlow has been successfully implemented? This would help illustrate the practical benefits of the framework.\n\nLIMITATIONS:\nPaper discuss the limitations of GraphFlow on complexity and dependence on high-quality data.  Regarding potential negative societal impacts, the paper does not explicitly address this aspect.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper tackles the problem of retrieval-augmented generation over text-enriched knowledge graphs for complex, multi-facet queries. The authors observe that prior KG-based RAG methods underperform when the query demands (i) fusing structural and textual evidence, and (ii) returning a diverse set of supporting items, and while procedural reward models could in principle give step-wise guidance, they need costly process-level supervision unavailable for large KGs Thus, the paper introduces GraphFlow, a GFlowNet-inspired framework that models KG retrieval as a stochastic decision process. GraphFlow jointly learns a retrieval policy that samples multi-hop sub-graphs, and a flow estimator that factorizes the terminal reward backward onto each intermediate step, promoting both accuracy and diversity without explicit step-wise labels. Experiments on the STaRK benchmark show that GraphFlow exceeds strong baselines, becoming an effective and robust solution for complex KG-based RAG scenarios.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- This paper is very well written and joyful to read. It clearly and reasonably frames related works to retrieval-based and agent-based and uses proper examples to illustrate their idea, showing the authors' good understanding of this area.\n\n- The paper's core technical idea for providing step-wise guidance using a trained LLM is both intuitive and elegant. The experiments also demonstrate the effectiveness and generalizability of the proposed method.\n\nWeaknesses:\n\n- Missing baselines and analysis. The experimental section omits some widely‐used STaRK baselines such as AVATAR [1], MFAR [2], KAR [3], and MOR [4]. My own inspection shows that on the Amazon and MAG domains the proposed method still lags these baselines by a noticeable margin. The impressive result of this paper lies in the PRIME domain, where by far it's the best results I've seen and leads by a large margin. However, there is a lack of analysis of this domain-variant result in this paper.\n\n- Does the paper sufficiently answer the question in the title? More analysis is needed to illustrate whether a retrieval method (previous and proposed) retrieves useful documents. For example, you may use SePer [5] to quantitatively estimate the utility of information at the intermediate steps.\n\n- Efficiency comparison unclear. One of the key claimed advantages is the efficiency of using the flow estimator for graph retrieval. However, there is a lack of experiments on the cost in time and money compared to retrieval-based and other agent-based retrieval methods.\n\nWould consider improving the score if more sufficient experiments are shown.\n\n[1]  Avatar: Optimizing llm agents for tool usage via contrastive reasoning\n\n[2]  Multi-field adaptive retrieval\n\n[3] Knowledge-Aware Query Expansion with Large Language Models for Textual and Relational Retrieval\n\n[4] Mixture of Structural-and-Textual Retrieval over Text-rich Graph Knowledge Bases\n\n[5] SePer: Measure Retrieval Utility Through The Lens Of Semantic Perplexity Reduction\n\nQUESTIONS:\nWhy can this method outperform ToG by such a large margin? Is it because of the gap between the limited context window size and the large degree of nodes in STaRK, or other reasons?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-03"}