Deliberation on Priors: Trustworthy Reasoning of Large Language Models
  on Knowledge Graphs
Authors: name: Jie Ma; name: Ning Qu; name: Zhitao Gao; name: Rui Xing; name: Jun Liu; name: Hongbin Pei; name: Jiang Xie; name: Linyun Song; name: Pinghui Wang; name: Jing Tao; name: Zhou Su
Published: 2025-05-21 07:38:45

Abstract: Knowledge graph-based retrieval-augmented generation seeks to mitigate
hallucinations in Large Language Models (LLMs) caused by insufficient or
outdated knowledge. However, existing methods often fail to fully exploit the
prior knowledge embedded in knowledge graphs (KGs), particularly their
structural information and explicit or implicit constraints. The former can
enhance the faithfulness of LLMs' reasoning, while the latter can improve the
reliability of response generation. Motivated by these, we propose a
trustworthy reasoning framework, termed Deliberation over Priors (DP), which
sufficiently utilizes the priors contained in KGs. Specifically, DP adopts a
progressive knowledge distillation strategy that integrates structural priors
into LLMs through a combination of supervised fine-tuning and Kahneman-Tversky
optimization, thereby improving the faithfulness of relation path generation.
Furthermore, our framework employs a reasoning-introspection strategy, which
guides LLMs to perform refined reasoning verification based on extracted
constraint priors, ensuring the reliability of response generation. Extensive
experiments on three benchmark datasets demonstrate that DP achieves new
state-of-the-art performance, especially a Hit@1 improvement of 13% on the
ComplexWebQuestions dataset, and generates highly trustworthy responses. We
also conduct various analyses to verify its flexibility and practicality. The
code is available at https://github.com/reml-group/Deliberation-on-Priors.