Title: Query-Aware Graph Attention for Precise Subgraph Retrieval in Knowledge-Augmented Reasoning
Venue: NeurIPS 2025
Accepted: no
Keywords: Knowledge Graph Question Answering

Large Language Models

Retrieval-Augmented Generation

Graph Neural Networks
Abstract: In recent years, Retrieval-Augmented Generation (RAG) has demonstrated great potential in enhancing the factual accuracy of large language models (LLMs) in open-domain question answering. Incorporating knowledge graphs (KGs) as external knowledge sources into the RAG paradigm is a promising direction. However, KG-RAG systems for complex multi-hop reasoning tasks still face significant challenges in precisely retrieving structured evidence highly relevant to the query. Existing approaches struggle to dynamically and accurately retrieve graph-based evidence by effectively leveraging query semantics and relational information. To address these challenges, we propose a novel framework called Query-aware Subgraph Retrieval Augmented Generation (QSRAG), centered around a new attention-based architecture termed Query-Relational Graph Attention Network (QR-GAT). QR-GAT is a graph attention mechanism that learns expressive representations of triples by capturing intricate interactions between the query context and relation types. Based on these representations, a scoring module assigns fine-grained relevance scores to triples in the KG, enabling precise subgraph retrieval for downstream reasoning. These structured evidence subgraphs, enriched with confidence scores, are then provided to an LLM to enhance its reasoning capability. Extensive experiments on two widely-used multi-hop Knowledge Graph Question Answering (KGQA) datasets, WebQSP and CWQ, demonstrate that our approach achieves state-of-the-art retrieval performance, particularly excelling in identifying complex multi-hop evidence. KGQA results further show that QSRAG delivers state-of-the-art or competitive performance on both datasets. Our work highlights the effectiveness of query-aware graph attention for accurate structured evidence retrieval, and its potential to enhance knowledge-augmented reasoning with large language models.


Peer reviews:
comments: SUMMARY:
This paper proposes a new approach for using knowledge graphs for KGQA. It retrieves structured evidence in form of fact triples which is then used as context by LLM to answer a query. Drawbacks of existing work include - inefficient retrieval and not incorporating the query for retrieval. Their proposed approach uses a GNN with an attention mechanism for dynamic query-aware retrieval. In the reasoning stage, the retrieved subgraph and its confidence scores are given to the LLM to guide final answer generation. Results are reported on ComplexWebQuestions and WebQuestionSP. The authors demonstrate improvement over SubGraphRAG, the main baseline.

STRENGTHS AND WEAKNESSES:
Strengths:
1. The motivation for introducing query dependency in GNN attention is intuitive.
2. The authors report significant improvement over baselines like SubgraphRAG and GNN-RAG.


Weaknesses:
1. Missing baselines: The paper misses some recent baselines on CWQ - HGNet [1] and FiDeLiS [2].  HGNet reports Hits@1=68.9 on CWQ.

[1] Chen, et al. 2022. Outlining and flling: hierarchical query graph generation for answering complex questions over knowledge graphs. IEEE Transactions on Knowledge and Data Engineering

[2] Sui, et al. 2024. FiDeLiS: Faithful Reasoning in Large Language Model for Knowledge Graph Question Answering.

QUESTIONS:
What is the meaning of the last sentence in "Implementation Details" (sec 5.1)? Do the authors skip comparing to some baselines? Please explain the reason clearly.

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 3
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-22

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Query-aware Subgraph Retrieval Augmented Generation (QSRAG), a framework for knowledge graph question answering that addresses the challenge of accurately retrieving structured evidence from knowledge graphs. The core contribution is the Query-Relational Graph Attention Network (QR-GAT), which extends graph attention networks by incorporating query semantics and relation types into the attention mechanism. The authors argue that existing KG-RAG methods fail to dynamically incorporate query semantics into their retrieval process, leading to less focused evidence retrieval. The proposed QR-GAT computes attention scores by combining structural features with query-relation interactions, enabling triple-level relevance scoring. The framework is evaluated on WebQSP and CWQ datasets, showing improvements over several baseline methods in both retrieval and question answering metrics.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths:\n\n- Significance: The paper addresses an important gap in KG-RAG systems - the lack of sufficiently good query-aware retrieval mechanisms that can dynamically adapt to query semantics\n- Quality: Comprehensive empirical evaluation. Of particular noteworthy is the inclusion of a variety of baseline methods and the author's meticulous attempt at reproducing all of them. This significantly enhances the trustworthiness of the empirical results.\n- Clarity: The paper is written quite well and is easy to follow. In particular, the introduction is written in a straightforward way that it is easy to understand which existing literature gap the authors are attempting to address.\n\n# Weaknesses:\n\n## Originality:\n\nIt seems to me that the core idea of incorporating query embeddings into GAT's attention computation is quite straightforward, and I do not see much discussion of what technical challenges that require special handling, novel theoretical principles, or particular engineering compromises and tradeoffs, that the authors encountered when incorporating the query embeddings into the attention. Overall, I think the work lacks a bit in technical depth and originality.\n\n## Significance\n\nInsufficient explanation of why existing methods (RoG, SubgraphRAG) fail to incorporate query semantics - the criticism in Related Work is too vague and abstract. The paper would benefit if there were a concrete, motivating example, or better yet using a figure, with which the author can explain how the design of RoG or SubgraphRAG would fail to capture the nuances in the query semantics.\n\n## Quality:\n\n1. There seems to be one significant training assumption made for the method to work - the ground-truth triplets must form paths between topic and answer entities. If my understanding of this is correct, then the method may have limited generalizability to more complex queries with more sophisticated topologies. \n2. In the model equations in section 4.1 - Structured Evidence Retrieval, it is unclear which part comes from the original GATv2 design and which part is the original one proposed by the authors to address the problem at hand.\n3. Although it is noteworthy that the experimental section compares with a lot of baselines, the number and variety of datasets seem to be limited - only WebQSP and CWQ, which are relatively simple. There are more recent and challenging KGQA benchmarks designed for LLMs that are available. \n\n## Clarity:\n\n1. Critical technical details are missing in the model equations in section 4.1,  including definitions of key terms (W_e, W_q, W_r, vector a), the attention head indices (appears later in $h_{i, k}^{(l)}$ but not in the original equation for $a_{i, j}^{(l)}$\n2. Other critical technical details are also missing for sufficient reproducibility, such as how the confidence score is computed in Section 4.2.\n\nQUESTIONS:\n1. Can the authors describe in much more details, perhaps using a motivating example, why can't methods like RoG and SubgraphRAG, which do process queries, achieve similar query-aware retrieval?\n\n2. Can the authors explain, in the proposed modified GAT attention computation, which parts come from the original GATv2 architecture, and which parts are the novel design proposed by the authors? Have the authors faced any challenges when experimenting with this designs, and if so, what kind of tradeoff or special design or principles the authors have applied to make the method finally work as expected? Similarly, including more detailed ablation studies on these components would equally strengthen the paper.\n\n3. How would the method perform on more complex queries whose evidence may not form simple, directed paths but require more complicated topologies? Would it be possible to benchmark on more recent, complex KGQA benchmarks, such as STARK, among others? (I understand that the rebuttal period is short so it may not be possible to experiment the method and all baselines on a new benchmark. Nevertheless, simply understanding whether this method is applicable to those newer benchmark is also some useful information since it concern's the method's generalizability)\n\nLIMITATIONS:\nThe authors discuss some limitations in their analysis section, including sensitivity to top-k selection and potential noise in large subgraphs. However, they should more explicitly address the limitation of their path-based training assumption and its impact on generalizability to complex multi-hop scenarios. The potential negative societal impacts appear minimal given the academic nature of the work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors propose Query-aware Subgraph Retrieval Augmented Generation (QSRAG) framework to improve how large language models (LLMs) answer complex questions using knowledge graphs (KGs). They identify the key challenge in existing systems, where essential subgraphs to support LLMs for multi-hop reasoning are difficult to retrieve precisely. Therefore, the QR-GAT is designed to dynamically incorporate the questions' semantics and specific types of relations into the attention. The relevance scores to each fact in KGs would be captured during training, and the retrieved high-confident triplets would be injected into LLMs as context to generate final answers without task-specific funetuning. Experiments illustrate the state-of-the-art retrieval performance and competitive results on QA tasks.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n- S1. The paper is exceptionally well-organized and easy to follow. The introduction clearly highlights the challenges and motivation. The methodology and experiments sections are also well-explained, including the mathematical formulation and detailed hyperparameters and setup.\n\n- S2. The contribution of QSRAG is novel, particularly the QR-GAT component, which explicitly injects query and relation semantics into the attention mechanism rather than learning graph and query representations separately. The design allows the retrieval evaluation to be easily investigated by selecting different numbers of top-k triplets, offering flexibility and control over the precision/recall trade-off.\n\n- S3. The paper provides a robust empirical evaluation. QSRAG achieves state-of-the-art performance on two standard KGQA benchmarks. The improvement is substantial, especially on the more complex CWQ dataset. This improvement is consistent across different inference LLMs and settings. Moreover, QSRAG can achieve better performance even when providing fewer, higher-quality triplets.\n\n**Weaknesses**:\n\n- W1. While the experimental results are impressive, the authors do not include error bars or confidence intervals from multiple runs. I could not find a similar performance trend in the original SubgraphRAG paper [1] due to different evaluation metrics. Furthermore, the Macro-F1 and Hit metric numbers reported for SubgraphRAG differ from those in the original paper, while the results for RoG, GNN-RAG, and other baselines are identical to the numbers reported in [1].\n\n- W2. As mentioned in the NeurIPS checklist, the paper should discuss the computational efficiency of the proposed algorithm and its scalability with dataset size. While the analysis shows that the number of retrieved triplets can be reduced, a comparison of the end-to-end time cost against baselines is missing.\n\n- W3. As a RAG solution, the authors are encouraged to provide more analysis of the retrieved triplets. The paper analyzes recall via the downstream LLM performance, but other quantitative methods would be insightful, such as analyzing the triplet overlap with other solutions, direct ground-truth recall, or providing graph structure visualizations of the retrieved subgraphs.\n\n- W4. The authors emphasize retrieving \"precise subgraph structures,\" but the learning and retrieval mechanism focuses primarily on scoring individual triplets/edges rather than on substructure topology. The connectivity of the retrieved triples should be discussed more explicitly if a key contribution is to learn and match entire subgraphs. And there is no ablation to compare QR-GAT to other SOTA retriever part (except SubgraphRAG because I raised one question in W1).\n\n[1] Mufei Li, Siqi Miao, Pan Li. Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation. In ICLR, 2025.\n\nQUESTIONS:\nQ1. The paper states that a temperature of 0 was used for LLM inference. Did the authors experiment with non-zero temperatures to test for variance in the LLM's output, and if so, were any notable differences observed?\n\nQ2. The paper does not appear to discuss knowledge transfer. Was the retriever's ability to transfer to other knowledge graphs tested? For example, could a model trained on a KG like Freebase be effectively applied to another KG with a different schema or domain?\n\nQ3. The main contribution is the QR-GAT, but the ablation study on the retriever is limited. Have the authors considered comparing QR-GAT against other graph representation learning methods for the retrieval task? To better establish the model's effectiveness and demonstrate the novelty, it would be also valuable to include results from simpler retrieval strategies, such as randomly selected triplets and the ground-truth triplets directly to LLMs.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-05"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces Query-aware Subgraph Retrieval Augmented Generation (QSRAG), a novel framework designed to enhance the factual accuracy and reasoning capabilities of Large Language Models (LLMs) in Knowledge Graph Question Answering (KGQA) tasks. The core of QSRAG is the Query-Relational Graph Attention Network (QR-GAT), an attention-based architecture that learns expressive representations of knowledge graph triples by capturing intricate interactions between query context and relation types. This enables precise subgraph retrieval from knowledge graphs (KGs) by assigning fine-grained relevance scores to triples. The retrieved, confidence-scored subgraphs are then provided to an LLM for enhanced multi-hop reasoning. Experimental results on WebQSP and CWQ datasets demonstrate that QSRAG achieves state-of-the-art performance in retrieval metrics and competitive or state-of-the-art results in KGQA, particularly for complex multi-hop evidence identification.\n\nSTRENGTHS AND WEAKNESSES:\nThe introduction of Query-Relational Graph Attention Network (QR-GAT) with its query-aware and relation-guided attention mechanism is a significant novel contribution. It explicitly addresses the limitation of existing GNNs in dynamically incorporating query semantics for fine-grained triple-level relevance scoring. \n\nClarity of \"Confidence Scores\" Impact: The paper states that \"Confidence scores play an important role in guiding the LLM,\" but the specific mechanism of how the LLM uses these numerical confidence scores to improve reasoning is not fully detailed or analyzed beyond \"focusing on more relevant triples.\" More explicit evidence or a dedicated analysis section on this interaction would strengthen the claim. \nEfficiency Analysis: While the paper mentions that some existing methods increase inference latency due to iterative LLM calls, it does not provide a direct comparison of the inference efficiency (e.g., latency, computational cost) of QSRAG against these methods. This would reinforce the practical benefits of the plug-and-play approach\n\nQUESTIONS:\nCould the authors elaborate on the specific mechanism by which the LLM utilizes the numerical confidence scores of the retrieved triples? A more detailed explanation or an ablation study focusing on the impact of confidence scores on LLM reasoning would be valuable.\n\nAre there specific types of multi-hop reasoning questions where QSRAG demonstrates the most significant improvements, and conversely, any where it still struggles? A qualitative analysis with diverse examples beyond the appendix would offer deeper insights.\n\nLIMITATIONS:\nYes、", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-10"}