Title: Deliberation on Priors: Trustworthy Reasoning of Large Language Models on Knowledge Graphs
Venue: NeurIPS 2025
Accepted: yes
Keywords: Knowledge Graph-based Retrieval Augmentation

Knowledge Graph Question Answering

Large Language Models
Abstract: Knowledge graph-based retrieval-augmented generation seeks to mitigate hallucinations in Large Language Models (LLMs) caused by insufficient or outdated knowledge. However, existing methods often fail to fully exploit the prior knowledge embedded in knowledge graphs (KGs), particularly their structural information and explicit or implicit constraints. The former can enhance the faithfulness of LLMs' reasoning, while the latter can improve the reliability of response generations. Motivated by these, we propose a trustworthy reasoning framework, termed Deliberation over Priors (\texttt{DP}), which sufficiently utilizes the priors contained in KGs. Specifically, \texttt{DP} adopts a progressive knowledge distillation strategy that integrates structural priors into LLMs through a combination of supervised fine-tuning and Kahneman-Tversky Optimization, thereby improving the faithfulness of relation path generation. Furthermore, our framework employs a reasoning-introspection strategy, which guides LLMs to perform refined reasoning verification based on extracted constraint priors, ensuring the reliability of response generation. Extensive experiments on three benchmark datasets demonstrate that \texttt{DP} achieves new state-of-the-art performance, especially a H@1 improvement of 13% on the ComplexWebQuestions dataset, and generates highly trustworthy responses. We also conduct various analyses to verify its flexibility and practicality. Code is available at [https://github.com/mira-ai-lab/Deliberation-on-Priors](https://github.com/mira-ai-lab/Deliberation-on-Priors).


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe manuscript introduces Deliberation on Priors (DP), a reasoning framework for knowledge graph-based retrieval-augmentation generation (KG-RAG) that utilizes priors embedded in knowledge graphs (KGs). DP first fine-tunes an LLM on a given KG and then distills its knowledge using Kahneman-Tversky optimization. The distilled LLM generates relation paths for each entity mentioned in a question. These paths are instantiated and validated using manually-defined constraints extracted from the question. If validated, the path is used by another LLM to generate an answer. Experiments on three KG question-answering (QA) datasets show that DP outperforms existing approaches.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The manuscript is clearly written and easy to follow. Figures and tables are used appropriately and effectively to enhance understanding. Additionally, terms and notations are consistently defined before use, contributing to overall clarity.\n2. The idea of validating relation paths using constraints extracted from the questions is compelling. These constraints help ensure that the validated paths align with the question's intent, thereby enhancing the precision of the reasoning process.\n\n**Weaknesses**\n1. The constraints are manually defined, requiring manual effort for each domain and question type. This heuristic limits the generalizability of the proposed method. Moreover, manually crafted constraints may not be optimal, which could cap the performance of the framework.\n2. In the distillation stage, an LLM is fine-tuned on a specific KG and before knowledge distillation. This approach limits the framework’s scalability, as it requires separate fine-tuning for each KG, reducing its applicability across diverse domains.\n3. The supervised learning baselines employ different language models, making it difficult to attribute performance differences solely to the framework. To ensure a fair comparison, experiments using the same underlying language model for both DP and the baselines are necessary.\n\n**Minor Issue**\n1. Line 97: ‘a instantiation’ should be corrected to ‘an instantiation’.\n\nQUESTIONS:\n1. In the knowledge distillation stage, any path from the source entity to the ground truth entity that involves the source entity’s neighbor within k hops is used as ground truth. However, such paths may be irrelevant to the question. Why are these potentially irrelevant paths not filtered before being used for training?\n2. When both DP and the baselines are evaluated using the same language model, does DP still achieve state-of-the-art performance?\n3. The test sets for each dataset were uniformly sampled by the authors. Are these samples identical to those used to evaluate the baseline models?\n4. How well do naïve LLMs perform on these tasks? In line 261, while the authors state that ‘DP can enhance the reasoning reliability of various LLMs,’ the performance of naïve LLMs is not reported, which weakens this claim.\n\nLIMITATIONS:\nThe proposed framework, DP, requires fine-tuning on a specific KG. Consequently, applying it to a different KG requires additional fine-tuning, which limits its scalability and applicability across diverse KGs.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes \"Deliberation on Priors\" (DP), a trustworthy reasoning framework for Large Language Models (LLMs) on Knowledge Graphs (KGs). The framework addresses hallucinations in LLMs by better exploiting prior knowledge embedded in KGs, specifically structural information and constraints. DP consists of four modules: (1) Distillation - using progressive knowledge distillation with supervised fine-tuning and Kahneman-Tversky optimization to integrate structural priors; (2) Planning - generating candidate relation paths; (3) Instantiation - grounding paths in KG triples; and (4) Introspection - verifying reasoning paths against extracted constraints with backtracking capability. The authors evaluate DP on three KGQA benchmarks (WebQSP, CWQ, MetaQA) and demonstrate state-of-the-art performance with improved faithfulness and reliability.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**\n1. The challenge addressed in this paper, enhancing the faithfulness and reliability of response generation, is highly relevant to KG-based Retrieval-Augmented Generation (RAG). To this end, the authors introduce a trustworthy reasoning framework, DP, which enables large language models to generate faithful and reliable responses through deliberate reasoning over priors embedded in knowledge graphs. \n\n2. Extensive experiments across three datasets with multiple LLMs (LLaMA, GPT variants) demonstrate the framework's flexibility and consistent improvements. The ablation studies provide good insights into component contributions.\n\n3. The method requires significantly fewer LLM calls (2.5-2.9 vs 10-22 for baselines) while maintaining superior performance, making it practically valuable.\n\n**Weaknesses**\n1. One limitation of the proposed method is the manual definition of constraint types. As shown in Table 1, these types are predefined rather than automatically extracted. However, in real-world applications, it is often necessary to extract such constraints automatically. Although the authors include experiments comparing the performance of DP with and without manually defined constraints, the performance of response generation degrades without manual input. \n\n2. Testing on only 500 samples from WebQSP/CWQ due to \"computational constraints\" raises questions about the robustness of conclusions, especially given the relatively small performance margins in some cases. The shortest-path heuristic for collecting supervision signals may not capture all valid reasoning paths, potentially limiting the diversity of learned patterns. The paper acknowledges this limitation but doesn't thoroughly investigate its impact.\n\nQUESTIONS:\n1. Is it feasible to design an automatic constraint extraction strategy to replace the manual predefinition of constraint types? This would further improve the scalability and practicality of your approach in real-world applications, where manual annotation may not be viable.\n2. In Section 3.1, you employ Dijkstra's algorithm to identify all shortest relation paths from the subject entity $e_s$ to the ground-truth answer entity $e_t$. In this setting, is there always a unique shortest path for a specific entity pair, or can multiple shortest paths of equal length exist?\n3. Table 3 provides a comprehensive ablation study across WebQSP and CWQ datasets, showing the contribution of each component in the DP framework. I notice that removing any of the three modules leads to a similar drop in performance. Could you elaborate on why these three perturbations result in comparable degradation? Does this imply that each component contributes independently and equally to the overall performance?\n4. Appendix B.5 (Figure 6) shows the token consumption and LLM invocation statistics for different model integrations. I am curious why integrating DP with **Qwen3-8B** leads to the highest token consumption compared to other LLMs. Is this due to specific characteristics of the model, such as verbosity in generation, or differences in decoding strategies?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper tackles hallucinations and factual inaccuracies in knowledge graph-based retrieval-augmented generation by introducing Deliberation on Priors (DP). This two-stage framework leverages structural and logical priors from KGs. In the Progressive Knowledge Distillation stage, LLMs are fine-tuned using KG-derived reasoning paths with a combination of supervised learning and Kahneman-Tversky optimization to incorporate structural priors. The subsequent Reasoning-Introspection stage employs a deliberative verification process that utilises symbolic logical constraints to iteratively refine and filter reasoning paths, thereby reducing hallucinations and enhancing output reliability. Experiments on three KGQA benchmarks demonstrate improved performance, supported by ablation studies that validate the contribution of each component. The authors also experiment and report LLM calls and token counts for practical deployment consideration.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper presents a technically solid and methodologically rigorous approach addressing hallucination and faithfulness issues in knowledge graph-augmented large language model reasoning. The authors’ two-stage framework—combining progressive knowledge distillation, which incorporates structural priors through supervised fine-tuning augmented with Kahneman-Tversky optimization, and a reasoning-introspection module that leverages symbolic logical constraints for output verification. This is a well-motivated and novel integration. The experimental evaluation is good, covering multiple KGQA benchmarks with comprehensive ablations that isolate the contributions of each component. The inclusion of practical efficiency metrics, such as LLM call and token usage, demonstrates awareness of deployment constraints, thereby increasing the work’s applicability. The paper effectively reduces hallucinations while improving interpretability, offering a promising direction for integrating structured priors into LLM reasoning.\n\nI see a few weaknesses in the paper related to reliance on constraints in the introspection stage, effectiveness of the KTO optimization step, and experiment with different LLMs, which I discuss in the questions section below.\n\nQUESTIONS:\n* In L125, shouldn’t the function $M$ also have $G$ as the input? Additionally, it is somewhat confusing to write the function defined over the full question set; perhaps writing it over $q$ itself is better. \n* L128, what is the backup option when the reasoning depth is not available? \n* A minor suggestion: Eq (1), *P_w (q)* is a set, so maybe as a convention, write it in bold so that it is clear straightaway. \n* As with any KG or RAG-based framework, performance is highly dependent on the quality of the retrievals/ KG. Can the DP framework also be used to send feedback to the KG retrieval and improve it?\n* SFT and KTO are interesting approaches. In practice, how is this performed? First, SFT is done, and then KTO, or do we keep alternating between them? Was there any ablation performed on this?\n* L178: What does the author precisely mean by semantic alignment? How is this calculated?\n* L200: Backtracking is an interesting part of the framework, and it would be interesting to see in practice how often, for each dataset, backtracking was triggered. \n* Minor: I found Figures 7-9 very interesting and believe there should be discussion or references to them in the main paper as well. \n* Table 2: \n    - I believe the std-dev should be reported here itself rather than a different table in the appendix. \n    - Grouping the methods on the models would be better to see the performance increase. \n    - Why is that with ICL, GPT-4o, GPT-4.1, Llama3.1-8B are not experimented with?\n    - Minor: WebQSP: Llama2-Chat-7B also has 90.7. \n    - In terms of stochasticity in DP, what is the main source of randomness here? Is a higher temperature used in LLMs in the experiments?\n* Table 5, isn’t it a bit unfair comparison as DP has a substantial fine-tuning cost associated, which is not reflected here?\n* Table 4: Why is the performance close to 99 (H) for CE? If it is uniformly sampled, then it should roughly be close to what is presented in Table 2. What am I missing here? \n* In future, what do authors see as the possible methods to incorporate more dynamic constraints and big KGs with not pre-defined reasoning hops?\n\nLIMITATIONS:\nAuthors briefly discuss the limitations with the conclusions in the paper. Authors can discuss the limitations in detail in the appendix. I do not see any direct negative societal impacts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper presents a framework called DP to enhance the reasoning capabilities of LLMs when performing tasks such as KGQA. The framework incorporates a progressive knowledge distillation strategy that integrates the structural priors of KGs into LLMs, improving the faithfulness of relation path generation. Furthermore, DP employs a reasoning-introspection strategy, which verifies whether generated relation paths satisfy predefined constraints, thereby improving the reliability of responses. Extensive experiments on several benchmark datasets demonstrate that DP outperforms current methods, with a 13% improvement in Hit@1 on ComplexWebQuestions.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n\nThe paper provides a thorough methodology for improving LLM reasoning by incorporating priors from knowledge graphs, which is a relevant issue in current LLM research. The proposed framework (DP) is novel and is shown to outperform existing methods on multiple benchmark datasets.\n\nThe paper is well-structured and provides clear explanations of the methodology, experimental setup, and results. The figures and examples further aid in understanding the approach and its effectiveness.\n\nThe use of a combination of knowledge distillation and reasoning introspection for enhancing reasoning over knowledge graphs is a novel contribution to the field of LLMs and KG-based retrieval-augmented generation.\n\n**Weaknesses:**\n\n1. While the experiments are extensive, the paper could benefit from deeper analysis of the failure cases, especially those that involve complex, multi-hop reasoning where the proposed method might still struggle. While error analysis is presented, more detailed insights into these cases could help improve the robustness of the framework.\n\n2. The framework relies heavily on predefined constraints, which could limit its flexibility. The authors mention plans for automating the extraction of these constraints, but further discussion on how this would work in diverse domains would be valuable.\n\n3. The method relies on high-quality KGs, which limits its usefulness in real-world scenarios.\n\nQUESTIONS:\n1. The framework relies on predefined constraints for introspection. Could the authors elaborate on potential methods for automating the extraction of these constraints? How would this work in more diverse domains or with complex, domain-specific knowledge graphs?\n\n2. Can the authors provide further insights into the failure cases where DP struggled to generate reliable paths? This would help in understanding its limitations better, especially in more complex reasoning tasks.\n\n3. While the proposed method performs well on benchmark datasets, how does it handle real-world applications where knowledge graphs may be incomplete or noisy?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-04"}