Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning
Authors: Mufan Xu, Gewen Liang, Kehai Chen, wei wang, Xun Zhou, Muyun Yang, Tiejun Zhao, Min Zhang
Accepted at: acl25

Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning

Mufan Xu1*, Gewen Liang1*, Kehai Chen1†, Wei Wang1†, Xun Zhou1,2
Muyun Yang1, Tiejun Zhao1, Min Zhang1
1School of Computer Science and Technology, Harbin Institute of Technology, China
2Pengcheng Laboratory, China
xmuffins0610@gmail.com, 24s051029@stu.hit.edu.cn,
{chenkehai,wangwei2019,zhouxun2023,yangmuyun,tjzhao,zhangmin2021}@hit.edu.cn

Abstract

Large language models (LLMs) have achieved remarkable performance on knowledge graph question answering (KGQA) tasks by planning and interacting with knowledge graphs. However, existing methods often confuse tool utilization with knowledge reasoning, harming readability of model outputs and giving rise to hallucinatory tool invocations, which hinders the advancement of KGQA. To address this issue, we propose Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning (MemQ) to decouple LLM from tool invocation tasks using LLM-built query memory. By establishing a memory module with explicit descriptions of query statements, the proposed MemQ facilitates the KGQA process with natural language reasoning and memory-augmented query reconstruction. Meanwhile, we design an effective and readable reasoning to enhance the LLM’s reasoning capability in KGQA. Experimental results demonstrate that MemQ achieves state-of-the-art performance on widely used benchmarks WebQSP and CWQ. Our code and data is available at https://github.com/HITIgw/MemQ.

1 Introduction

Large language models (LLMs) have demonstrated impressive reasoning capabilities in knowledge graph question answering (KGQA) task (Yu et al., 2022; Huang and Chang, 2023; Jiang et al., 2022). Using planning and interactive strategies, current LLM-based KGQA methods conduct the reasoning process on the knowledge graph based on SPARQL tools and achieve remarkable performance across benchmarks (LUO et al., 2024; Sun et al., 2024; Xu et al., 2024b). Typically, part of these studies directly strengthen the reasoning ability of LLMs to plan tool-based paths and retrieve information from the knowledge graph (Wang et al., 2023b; LUO et al., 2024). The others employ LLMs to construct knowledge reasoning agents that execute the reasoning process on the knowledge graph through continuous tool-based decision-making based on environmental observations (Gu et al., 2023; Jiang et al., 2024). These methods have achieved impressive results in the KGQA task.

However, existing methods often conflate tool invocations with the reasoning process, compromising readability and leading to tool invocation hallucinations. On one hand, current KGQA methods typically represent reasoning steps using SPARQL relations, which tend to be abstract and contain redundant information, significantly reducing readability. On the other hand, this complex representation places an unnecessary burden on the reasoning process and may introduce additional hallucination issues. For example, when answering the question “Who is Justin Bieber’s brother?”, existing methods fail to generate

![Comparing reasoning methods designed with knowledge graph query tools with proposed memory-augmented method MemQ.](page_1012_370_388_312.png)

Figure 1: Comparing reasoning methods designed with knowledge graph query tools with proposed memory-augmented method MemQ.
readable reasoning steps (Figure 1, top right). Instead, they rely on abstract SPARQL paths to represent the reasoning process (Figure 1, top left). In such cases, both poor reasoning and hallucinatory tool usage by LLMs can contribute to unreliable results, making it difficult for researchers to analyze the results (Figure 1, bottom left).

To address the issue, we propose Memory-augmented Query Reconstruction for LLM-based Knowledge Graph Reasoning (MemQ) to decouple LLM from the tool invocation task using an LLM-built query memory. To establish the query memory, we employ a rule-based strategy to decompose queries into statements, which are then described using the ICL capability of general-purpose LLM, facilitating an independent reasoning process. We design an effective reasoning strategy based on natural language, enhancing readability and generating explicit reasoning steps. Based on the developed steps, MemQ retrieves memory based on semantic similarity and reconstructs the final query to interact with the knowledge graph. By establishing this query memory, the MemQ approach enables the model to disengage from tool invocation and focus on generating readable knowledge reasoning steps. Our main contributions are:

• We proposed a novel memory-augmented KGQA method, MemQ, which enhances the LLM-based KGQA reasoning process by integrating fine-grained query information as an external memory.

• By designing a straightforward query decomposition strategy, MemQ decouples the LLM from tool invocation tasks, effectively improving the readability of the reasoning process and reducing the hallucination issues that arise from mixed tool invocations.

• The proposed MemQ achieved state-of-the-art performance on two widely used benchmarks WebQSP and CWQ.

2 Related Works

Memory-augmented LLM Generation. Though large language models have demonstrated remarkable performance across tasks, they still struggle to achieve consistent performance on complex reasoning tasks (Wang et al., 2024). In this context, the approach of constructing an external knowledge base to record key information has been proposed and shown to be beneficial (Hu et al., 2023; Anokhin et al., 2024). Researchers have proposed strategies to enhance LLM memory using external modules to support long-term dialogue history referencing (Lee et al., 2024; Rezazadeh et al., 2024). For tasks requiring extensive domain knowledge, methods for constructing memory banks either manually or using large models have also been proven effective (Cheng et al., 2024; Panda et al., 2024; Edge et al., 2024).

Knowledge Graph Question Answering. Early KGQA approaches focused on using networks like key-value memory and graph neural networks to represent inference paths (Miller et al., 2016; Yasunaga et al., 2021; Jiang et al., 2022), while other approaches teach models to build database queries such as SPARQL for direct answer retrieval (Gu and Su, 2022; Ye et al., 2022). With the rise of large language models (LLMs), methods utilize LLM’s graph reasoning capability to enhance the reliability of reasoning process (Zhong et al., 2024; Zhang et al., 2024; Zhu et al., 2024). Certain approaches are developed to leverage scaled models to directly interact with Knowledge Graphs or for generating labels that assist smaller models in distilling reasoning abilities (Sun et al., 2024; LUO et al., 2024; Xu et al., 2024b). Other efforts focus on constructing decision datasets based on annotated data to perform a supervised fine-tuning process, which enhance LLM’s understanding of the knowledge reasoning process and their ability to interact with knowledge graphs (Jiang et al., 2024). Since LLM-generated outputs are generally susceptible to hallucinatory behavior, some research has shifted to employing discriminative strategies instead of generative ones to reduce unfounded reasoning processes (Gu et al., 2023; Xu et al., 2024a).

However, the issue of confusing the tool invocation process with the knowledge reasoning process remains unresolved. The existing method often conducts reasoning based on SPARQL-formed edges like 'type.domain.property' or self-designed toolboxes, which diminishes the model’s focus on the reasoning process and suffers from hallucinatory tool invocation behaviors. In this paper, we propose a memory-augmented KGQA reasoning method that effectively decouples the reasoning process from tool invocation.
Figure 2: The overall framework of MemQ. During the memory construction stage, we describe the question with its query history using the LLMs to get the reasoning steps. In the inference stage, we reconstruct the query using the recalled query sentences based on the reasoning results.

3 Approach

In this section, we introduce the overall approach of MemQ; the overall framework is illustrated in Figure 2. We propose to facilitate the KGQA process using three tasks including memory construction, knowledge reasoning and query reconstruction. We first formulate the task and then showcase our method.

3.1 Memory Construction

Given the query history \( H \) that contains question \( q_i \) with its corresponding query \( query_i \), the memory construction task asks the model to build a memory \( M \) to represent the mapping function from natural language descriptions \( n_i \) to query statements \( s_i \):

\[
s_i = M(n_i),\ s_i \in query_i.
\]

Rule-based Decomposing. To establish a memory to augment the query reconstruction process, MemQ utilizes a rule-based strategy to decompose queries in training set and gather the description of each statement using a general-purpose LLM. We notice that Freebase contains the Compound Value Type (CVT) nodes that lack inherent semantic meanings. In order to acquire describable query statements, MemQ always uses non-CVT nodes as the starting or ending nodes when splitting the queries. As illustrated in Figure 3, this results in three types of query statements with different graph structures. To be specific, given a query from the training set, MemQ iterates through statements in the query. If no CVT node is encountered, the single statement will form a type 1 record. If a CVT node is encountered and it has not been recorded in any record in the memory, MemQ combines the next statement which connects to this CVT node to form a type 2 record. If the CVT node exists in a type 2 record in memory, MemQ combines the current statement with that record to form a type 3 record.

Description Collection. For each statement, we use the general-purpose LLM to provide a natural language description and store them in the query memory in pairs. We provide task instructions and examples in the context of conducting few-shot generation to ensure the quality of the description and prevent excessive differences between descriptions. We adopt GLM-4 as the description model to generate the descriptions. The prompt templates are shown in Appendix F. The memory construction process is actually a summarization and compression of historical search queries, providing readable hints for the future query reconstruction process.

3.2 Knowledge Reasoning

Given the question \( Q \), the mentioned entities \( E \), the knowledge reasoning task asks the model to develop an n-step reasoning plan \( P \) to answer the question. Here we regulate \( P \) with the rule that each reasoning step \( p_i \) is limited to searching or
<table>
  <tr>
    <th>Structure Type</th>
    <th>Query Statements</th>
    <th>Natural Language Description</th>
  </tr>
  <tr>
    <td>Type1</td>
    <td>?y location.country.currency_used ?x</td>
    <td>?x is the currency used in country ?y</td>
  </tr>
  <tr>
    <td>Type2</td>
    <td>?y people.person.sibling s ?evt; ?evt people.sibling_relation.sibling ?x</td>
    <td>?x is the sibling of person ?y</td>
  </tr>
  <tr>
    <td>Type3</td>
    <td>?y sports.pro_athlete.teams ?evt; ?evt sports.sports_team_roster.team ?z; ?evt sports.sports_team_roster.from ?x</td>
    <td>?x is the start date of the period during which ?y was part of the team ?z</td>
  </tr>
</table>

Figure 3: Structure example of the three types of statements acquired by decomposing training set queries.

examining only one entity. The n-step plan \( P \) can be represented as a set of reasoning steps:

\[
P = \{p_i|i = 1, 2, ..., n\}.
\] (2)

As shown in Figure 2, after obtaining the corresponding description of each query statement, MemQ uses those explanation-statement pairs to finetune the LLM to enhance its reasoning capabilities (bottom left). By adopting a memory-enhanced approach instead of using a model to directly generate query invocation content, MemQ only requires the LLM to focus on the reasoning process by generating reasoning steps based on the questions using natural language. The generated reasoning steps will be used for the memory reconstruction process.

3.3 Query Reconstruction

Given the generated reasoning plan \( P \) and the query memory \( M \), the query reconstruction task asks the model to first recall proper query statements \( s_i \) using \( M \) and then reconstruct the final query \( Q_f \) corresponding to the question \( Q \) using the set of collected statements:

\[
s_i = M(p_i),
\]
\[
Q_f = \text{Re-con}(S),
\]
\[
p_i \in P,\ s_i \in S.
\] (3)

During the query reconstruction process, MemQ iteratively perform memory recall and statements assembling based on the reasoning steps planned in the previous task, until the end of the reasoning steps is reached. As the query is reconstructed, it is executed to retrieve the final answer from the knowledge graph.

Adaptive Memory Recall Strategy. Given the developed reasoning steps, MemQ recalls relevant memory based on semantic similarity and employs rule-based methods to concatenate these statements to reconstruct a complete query. To measure the semantic similarity, we use Sentence-BERT to encode the reasoning steps and the explanations in the memory. Since the similarity scores of the top-N memory fragments can be nearly identical, MemQ adopts an adaptive recall strategy to retrieve the statements from the memory:

\[
N = \begin{cases}
1 & \text{if top-1 similarity} \geq \gamma_1, \\
k & \text{if top-1 similarity} < \gamma_1,
\end{cases}
\]
\[
k = \text{count}_{case}(\text{similarity} \geq \gamma_2).
\] (4)

Rule-based Reconstruct Strategy. MemQ adopts a rule-based reconstruction strategy that the most recently recalled sentence is appended to the end of the reconstructing query. In order for the recalled statements to be correctly appended to the reconstructed query in sequence, we ask the LLM to use the entity names (e.g., “person_n”) generated during the reasoning step to fill in the recalled statements.

4 Experiment

In this section, we first introduce the datasets and evaluation methods used by MemQ. After presenting the main experimental results, we will follow up with reports on several analytical experiments to examine the characteristics of the MemQ method compared to previous methods from various perspectives.

4.1 Benchmarks and Baselines

Benchmarks. To evaluate the KGQA capability of the proposed method, we choose two widely used
<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">WebQSP</th>
    <th colspan="2">CWQ</th>
  </tr>
  <tr>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Llama2-7b zero-shot (Touvron et al., 2023)*</td>
    <td>0.403</td>
    <td>0.293</td>
    <td>0.297</td>
    <td>0.272</td>
  </tr>
  <tr>
    <td>Llama3-8b zero-shot (Dubey et al., 2024)*</td>
    <td>0.303</td>
    <td>0.257</td>
    <td>0.305</td>
    <td>0.278</td>
  </tr>
  <tr>
    <td>Qwen2.5-7b zero-shot (Yang et al., 2024)*</td>
    <td>0.284</td>
    <td>0.237</td>
    <td>0.259</td>
    <td>0.241</td>
  </tr>
  <tr>
    <td>KV-Mem (Miller et al., 2016)</td>
    <td>0.467</td>
    <td>0.345</td>
    <td>0.184</td>
    <td>0.157</td>
  </tr>
  <tr>
    <td>GraftNet (Sun et al., 2018)</td>
    <td>0.664</td>
    <td>0.604</td>
    <td>0.368</td>
    <td>0.327</td>
  </tr>
  <tr>
    <td>QGG (Lan and Jiang, 2020)</td>
    <td>0.730</td>
    <td>0.738</td>
    <td>0.369</td>
    <td>0.374</td>
  </tr>
  <tr>
    <td>NSM (He et al., 2021)</td>
    <td>0.687</td>
    <td>0.628</td>
    <td>0.476</td>
    <td>0.424</td>
  </tr>
  <tr>
    <td>SR+NSM (Zhang et al., 2022)</td>
    <td>0.689</td>
    <td>0.641</td>
    <td>0.502</td>
    <td>0.471</td>
  </tr>
  <tr>
    <td>SR+NSM+E2E (Zhang et al., 2022)</td>
    <td>0.695</td>
    <td>0.641</td>
    <td>0.493</td>
    <td>0.463</td>
  </tr>
  <tr>
    <td>DECAF (DPR+FID-3B) (Yu et al., 2022)</td>
    <td>0.821</td>
    <td>0.788</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>UniKGQA (Jiang et al., 2022)</td>
    <td>0.751</td>
    <td>0.702</td>
    <td>0.507</td>
    <td>0.480</td>
  </tr>
  <tr>
    <td>KD-CoT (Wang et al., 2023a)</td>
    <td>0.686</td>
    <td>0.525</td>
    <td>0.557</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ToG w/ChatGPT (Sun et al., 2024)</td>
    <td>0.758</td>
    <td>-</td>
    <td>0.589</td>
    <td>-</td>
  </tr>
  <tr>
    <td>ToG w/GPT-4 (Sun et al., 2024)</td>
    <td>0.826</td>
    <td>-</td>
    <td>0.676</td>
    <td>-</td>
  </tr>
  <tr>
    <td>KG-Agent (Jiang et al., 2024)</td>
    <td>0.833</td>
    <td>0.810</td>
    <td>0.722</td>
    <td>0.692</td>
  </tr>
  <tr>
    <td>RoG (Top-3 relation path) (LUO et al., 2024)*</td>
    <td>0.795</td>
    <td>0.701</td>
    <td>0.567</td>
    <td>0.547</td>
  </tr>
  <tr>
    <th>MemQ (Ours)</th>
    <th><b>0.841</b></th>
    <th><b>0.858</b></th>
    <th><b>0.803</b></th>
    <th><b>0.830</b></th>
  </tr>
</table>

Table 1: The results of our method compared with previous approaches on WebQSP and CWQ. The asterisk * denotes the results we reproduced. Note that the Hits@1 result reported in the original RoG paper (WebQSP 0.857, CWQ 0.626) is not calculated in the right way, see the author’s response here.

benchmarks, WebQSP (Yih et al., 2016) and CWQ (Talmor and Berant, 2018).

Metrics. We choose commonly used metrics Hits@1 and F1 for the evaluation process following previous works. For the definitions of metrics, please refer to Appendix A.

Baselines. We select previous competitive approaches with planning strategy like RoG (LUO et al., 2024) or interactive reasoning strategy like ToG (Sun et al., 2024) as baselines. We also list representative methods and zero-shot performances of widely used LLMs for comparison. Since the experimental setup differs from other methods, we present the results for MemQ and the ChatKBQA (Luo et al., 2024) approach under the same settings in Table 6. We also finetune the LLMs with SPARQL queries as a strong baseline for ablation, see Section 4.4.

Base Model. To ensure fairness in comparison, we choose Llama2-7b (Touvron et al., 2023) as the base model following KGAgent (Jiang et al., 2024) and RoG (LUO et al., 2024). In analytical experiments, we adopt a stronger model Llama3-8b to better evaluate the effectiveness of MemQ.

4.2 Main Result

The performance of our MemQ framework on the WebQSP and CWQ datasets is presented in Table 1. Our method achieves state-of-the-art results on both benchmarks, as demonstrated by significant improvements in Hits@1 and F1 metrics. We also adopted the same experimental setup as ChatKBQA (Luo et al., 2024) for performance comparison. MemQ achieved significant improvements in both Hit and F1 metrics (see results in Appendix B). The results show the efficiency of proposed framework to decouple reasoning from tool invocation. By adopting a memory-augmented strategy, MemQ provides a new way to enhance the LLM-based reasoning process.

4.3 Reasoning Capability Enhancement

To investigate the improvements brought by our proposed reasoning framework, we conduct experiments to examine the discrepancies between the search graph of the reconstructed queries and that of the golden queries. We evaluate the quality of the developed subgraph from two aspects: 1) the structural accuracy and 2) the edge accuracy. Our analysis specifically targets these dimensions to identify the principal factors driving the observed performance improvements.

The structural accuracy GoldGED is defined as the Graph Edit Distance between the reconstructed graph \( G_{re} \) and the golden graph \( G_{gd} \):

\[
\text{GoldGED}(G_{re}) = \min_{\pi \in \Pi(G_{re}, G_{gd})} \text{num}(\pi).
\]
<table>
  <tr>
    <th>Total Hops</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
    <th>4</th>
    <th>5</th>
    <th>6</th>
    <th>7</th>
    <th>avg</th>
  </tr>
  <tr>
    <th colspan="9">Edge Hitting Rate <i>EHR</i></th>
  </tr>
  <tr>
    <td>RoG</td>
    <td><b>0.853</b></td>
    <td>0.644</td>
    <td>0.390</td>
    <td>0.276</td>
    <td>0.249</td>
    <td>0.230</td>
    <td>0.283</td>
    <td>0.377</td>
  </tr>
  <tr>
    <td>MemQ</td>
    <td>0.816</td>
    <td><b>0.844</b></td>
    <td><b>0.854</b></td>
    <td><b>0.851</b></td>
    <td><b>0.854</b></td>
    <td><b>0.861</b></td>
    <td><b>0.939</b></td>
    <td><b>0.860</b></td>
  </tr>
  <tr>
    <th colspan="9">Graph Edit Distance with Golden Graph <i>GoldGED</i></th>
  </tr>
  <tr>
    <td>RoG</td>
    <td>0.479</td>
    <td>2.494</td>
    <td>3.764</td>
    <td>4.505</td>
    <td>5.499</td>
    <td>7.193</td>
    <td>10.438</td>
    <td>4.910</td>
  </tr>
  <tr>
    <td>MemQ</td>
    <td><b>0.158</b></td>
    <td><b>0.465</b></td>
    <td><b>0.909</b></td>
    <td><b>1.364</b></td>
    <td><b>1.611</b></td>
    <td><b>2.531</b></td>
    <td><b>2.250</b></td>
    <td><b>1.327</b></td>
  </tr>
</table>

Table 2: We evaluate the Edge Hitting Rate and Graph Edit Distance with the golden graph for both our method and RoG. The results indicate that the reconstructed graphs achieve significantly higher accuracy and structural alignment compared to those generated by RoG.

The edge accuracy is quantified by the Edge Hitting Rate, which is computed using the hitting rate between edges in the golden graph \( G_{gd} \) and the edges in the reconstructed graph \( G_{re} \):

\[
\text{EHR}(G_{re}) = \frac{\text{num}(\{e| e \in G_{gd} \land e \in G_{re}\})}{\text{num}(\{e| e \in G_{gd}\})}
\]

(6)

The results is featured in Table 2. Specifically, MemQ achieves a significantly lower GoldGED, indicating more accurate structural alignment with reference graphs, especially in complex multi-hop scenarios. Additionally, MemQ sustains a higher EHR, demonstrating robust edge accuracy even as the number of reasoning steps increases. Overall, these results emphasize MemQ’s superior performance in producing accurate and structurally coherent graph-based reasoning across subgraphs.

4.4 Ablation Study

To further analyze the effectiveness of the proposed framework, we conduct experiments to ablate the strategies in MemQ and observe the change in performance. We design two finetune-based baselines to ablate our strategies. 1) For the query reconstruction process, we directly finetune the model utilizing the statements and the descriptions recorded in the memory (denoted as -w/o QRM) to evaluate the effectiveness of our proposed query memory; 2) For the whole MemQ framework, we finetune the model using queries to simulate a straightforward tool-based reasoning process (denoted as -w/o PE, QRM) to evaluate the effectiveness of the MemQ framework. The results are shown in Table 3.

According to the results, we can observe that: 1) Comparing MemQ with “-w/o QRM”, the proposed memory-augmented strategy significantly improves the stability of tool utilization process compared with LLM-based finetuning strategy; 2) Comparing “-w/o QRM” with “-w/o PE, QRM”, in the case of using a direct fine-tuning strategy, the method of direct fine-tuning that blends reasoning with tool invocation has lowered the overall F1 and EHR score. Furthermore, given that our method has also improved the overall Hits@1 and F1 scores compared to previous tool-based SOTA work, these results demonstrate the enhancement of the proposed decoupling strategy on the reasoning process of LLMs.

<table>
  <tr>
    <th rowspan="2">Strategy</th>
    <th colspan="3">WebQSP</th>
    <th colspan="3">CWQ</th>
  </tr>
  <tr>
    <th>Hits@1</th>
    <th>F1</th>
    <th>EHR</th>
    <th>Hits@1</th>
    <th>F1</th>
    <th>EHR</th>
  </tr>
  <tr>
    <td>MemQ</td>
    <td>0.857</td>
    <td>0.872</td>
    <td>0.858</td>
    <td>0.817</td>
    <td>0.845</td>
    <td>0.886</td>
  </tr>
  <tr>
    <td>-w/o QRM</td>
    <td>0.729</td>
    <td>0.743</td>
    <td>0.849</td>
    <td>0.588</td>
    <td>0.620</td>
    <td>0.864</td>
  </tr>
  <tr>
    <td>-w/o PE,QRM</td>
    <td>0.733</td>
    <td>0.731</td>
    <td>0.739</td>
    <td>0.556</td>
    <td>0.570</td>
    <td>0.806</td>
  </tr>
</table>

Table 3: Ablation study of MemQ with two settings.

4.5 Alleviation of Hallucination Issue

To figure out the impact of our decoupled reasoning strategy on the hallucination issue, we manually check and evaluate the error cases of MemQ and the “-w/o PE, QRM” baseline proposed in the ablation study. To guarantee an objective evaluation, we established criteria to check with the cases: 1) Correctness: whether the main reasoning steps contain errors, 2) Completeness: whether the reasoning logic lacks necessary filtering conditions, and 3) Redundancy: whether the reasoning logic includes irrelevant or unnecessary filtering conditions. We randomly sample 100 cases from the test set to record the frequency of each of the errors. Note that one sample may contain multiple errors at a time.

As shown in Table 4, our method significantly reduces the number of Correctness and Completeness errors, while errors in Redundancy slightly increase. The increment in Redundancy errors stems from our retrieval strategy, justified by the
<table>
  <tr>
    <th>Strategy</th>
    <th>Correctness</th>
    <th>Completeness</th>
    <th>Redundancy</th>
  </tr>
  <tr>
    <td>MemQ</td>
    <td>8</td>
    <td>16</td>
    <td>16</td>
  </tr>
  <tr>
    <td>-w/o PE,QRM</td>
    <td>39</td>
    <td>41</td>
    <td>9</td>
  </tr>
</table>

Table 4: We manually assess the reasoning plans based on Consistency, Completeness, and Redundancy, documenting the number of plans that exhibit errors in each of these categories.

presence of edges with similar semantic meanings in the Knowledge Graph (see Appendix D for details). The result indicates that our proposed decoupled reasoning strategy significantly reduces the errors brought by the confusing tasks, indicating an alleviation of the hallucinatory tool invocation issue.

4.6 Robustness towards erroneous data

![Two line plots showing Hits@1 results for WebQSP and CWQ across varying proportions of disturbed data.](page_256_670_512_384.png)

Figure 4: The Hits@1 results with Llama3-8b across varying proportions of disturbed data.

To investigate the robustness of MemQ towards erroneous data, we designed two strategies to add noise to the training data for simulating a small number of incorrectly labeled SPARQL: 1) Strategy 1: Randomly delete one statement from the SPARQL in a certain percentage of the training set; 2) Strategy 2: Instead of deleting, replace it with a random Freebase relation. We tested the impact of different levels of data perturbation for the proposed MemQ. The results are shown in Figure 4. The experimental results indicate that MemQ shows a certain degree of robustness to incorrectly annotated SPARQL queries, with significant performance degradation only appears when the perturbation proportion exceeds 20% of the total training data.

4.7 Error Analysis

To conduct a detailed error analysis, we categorize errors into two distinct types: 1) Main Path Error, where the primary reasoning path is incorrect, and 2) Filtering Error, which includes cases of excessive or insufficient filtering. This classification allows for a systematic evaluation of the inaccuracies in the reasoning process.

![Bar chart comparing number of errors for WebQSP and CWQ under three conditions: -w/o PE, QRM, RoG, and our method.](page_768_670_480_384.png)

Figure 5: Case frequency of two types of errors.

As shown in Figure 5, the Main Path Error of our method is significantly lower than the other two baselines in all datasets. In the CWQ dataset, our method achieves the lowest filtering error among all compared approaches. In the WebQSP dataset, our method achieves substantially lower filtering error compared to the setting without PE and QRM, though it is marginally higher than the RoG method. These results demonstrate the effectiveness of our method in reducing reasoning and filtering errors.

4.8 Data Efficiency Analysis

![Two line plots showing Hits@1 and F1 scores for WebQSP and CWQ across varying proportions of training data.](page_256_1170_512_384.png)

(a) WebQSP

(b) CWQ

Figure 6: Hits@1 and F1 scores of MemQ with Llama3-8b across varying proportions of training data.

To assess the data efficiency of our MemQ method, we evaluate the performance of planning expert LLM trained with varying levels of training data availability. In this experiment, we randomly selected 10%, 25%, 50%, 75%, and 100% of the step description data to fine-tune the LLaMA-3-8B-Instruct model. As illustrated in Figure 6, our method achieves an F1 score and Hits@1 of
<table>
  <tr>
    <th colspan="2">Case Study</th>
  </tr>
  <tr>
    <td colspan="2"><b>Question:</b> who was richard nixon married to?</td>
  </tr>
  <tr>
    <td colspan="2"><b>Generated Plan and Reconstruct Query:</b><br>
Step1: Find <b>the spouse</b> of richard nixon, assign it to ?x.<br>
Retrieved Query1: ns:m.06c97 ns:people.person.spouse_s ?cvt. ?cvt ns:people.marriage-spouse ?x<br>
Step2: Find <b>the type of union</b> between richard nixon and ?x, assign it to ?type_of_union.<br>
Retrieved Query2: ?cvt ns:people.marriage-type_of_union ?type_of_union<br>
Step3: Make sure <b>?type_of_union should be Marriage.</b><br>
Retrieved Query3: FILTER(?type_of_union = ns:m.04ztj)<br>
Finally the answer is?x.<br>
<b>Reconstruct Sparql:</b><br>
PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE{
ns:m.06c97 ns:people.person.spouse_s ?cvt .
?cvt ns:people.marriage-spouse ?x .
?cvt ns:people.marriage-type_of_union ?type_of_union .
FILTER(?type_of_union = ns:m.04ztj).
FILTER (!isLiteral(?x) OR lang(?x) = "" OR langMatches(lang(?x), 'en')).
FILTER(?x != ns:m.06c97) }
<b>Output:</b><br>
Pat Nixon (m.023v03)
    </td>
  </tr>
</table>

Figure 7: Case of MemQ, we retrieve memories based on the reasoning steps and reconstruct the final query.

approximately 0.7 with only 10% of the training data, steadily outperforming the zero-shot baseline across both datasets, indicating the method’s ability to scale effectively with additional data. These results show that our method can effectively utilize limited data, highlighting its strong data efficiency.

4.9 Further analysis

<table>
  <tr>
    <th rowspan="2">Base Model</th>
    <th colspan="2">WebQSP</th>
    <th colspan="2">CWQ</th>
  </tr>
  <tr>
    <th>Hits@1</th>
    <th>F1</th>
    <th>Hits@1</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>Vicuna-7b</td>
    <td>0.828</td>
    <td>0.846</td>
    <td>0.796</td>
    <td>0.826</td>
  </tr>
  <tr>
    <td>Llama2-7b</td>
    <td>0.841</td>
    <td>0.858</td>
    <td>0.803</td>
    <td>0.830</td>
  </tr>
  <tr>
    <td>Llama3-8b</td>
    <td>0.858</td>
    <td>0.872</td>
    <td>0.818</td>
    <td>0.845</td>
  </tr>
  <tr>
    <td>Qwen2.5-7b</td>
    <td>0.828</td>
    <td>0.850</td>
    <td>0.793</td>
    <td>0.818</td>
  </tr>
</table>

Table 5: Implementation with four widely-used LLMs.

Model Universality Analysis. To demonstrate the robustness and versatility of our MemQ, we conduct fine-tuning experiments on four distinct, widely-used large language models (LLMs) serving as the Planning Expert to generate the reasoning steps. The results in Table 5 demonstrate that all models achieved strong performance, indicating its adaptability to different LLM architectures and confirming its robustness as a model-agnostic solution for reasoning tasks.

Case study. To demonstrate the readability of MemQ, we present a case that highlight its capability to produce clear, logically consistent reasoning plans and accurate reconstruction queries in Figure 7. Refer to Appendix D for more cases.

Out-of-distribution Performance. When the question involves queries not present in the memory, MemQ can still use the next-best retrieval results from the memory repository to attempt finding the correct answer. We conduct experiments to verify this, the results is shown in Appendix D. It can be observed that the MemQ method demonstrates better generalization ability on these issues compared to previous work, which is likely stems from MemQ’s fine-grained memory and adaptive recall strategy.

5 Conclusion

In this paper, we propose decoupling LLM from tool invocation tasks using an LLM-built query memory to alleviate hallucinatory tool invocation issues. By facilitating the KGQA process using three tasks, we established a memory module to augment the query reconstruction process in the KGQA task. Based on the framework, we design an effective and readable reasoning strategy to enhance the LLM’s reasoning capability, which also alleviates hallucinatory behaviors in existing methods. Experimental results show that our proposed memory-enhanced framework has achieved the state-of-the-art (SOTA) performance on two commonly used benchmarks.
Limitation

Though our proposed MemQ framework has shown competitive KGQA performance and is proven to enhance the LLM’s reasoning capability, we identify several limitations that require further improvement. In the future, we will focus on the following directions to extend the current work:
1) Usage of Labeled data: Although our method effectively enhances LLM-based KGQA reasoning process and alleviates the hallucinatory tool invocations, we assume that we have the gold queries to construct the memory. However, it is noteworthy that the decomposing process of the query can be replaced by gathering all the relations and examples of the usage of relations from the Freebase itself. In the future, we will analyze the possibility of model the whole Freebase into a memory to get rid of the demand of gold queries.
2) Plug-and-play Capability: The proposed framework possesses good plug-and-play capability since the constructed memory is a portable module that can be adopted with other reasoning strategies and other tools. In the future, we will conduct experiments to showcase this kind of capability and testify our proposed memory-based framework under multi-tool or task transfer conditions.

Acknowledgment

We want to thank all the anonymous reviewers for their valuable comments. The work was supported by the National Natural Science Foundation of China under Grant (62376075, 62276077, 62472125 and U23B2055), National Natural Science Foundation of China (Research Fund for International Senior Scientists) 62350710797, Natural Science Foundation of Guangdong Province, China (2024A1515011205 and 2025A1515011258), Shenzhen Sustained Support for Colleges & Universities Program (GXWD20231128102922001) and Shenzhen Science and Technology Program (ZDSYS20230626091203008 and KQTD20240729102154066).

References

Petr Anokhin, Nikita Semenov, Artyom Sorokin, Dmitry Evseev, Mikhail Burtsev, and Evgeny Burnaev. 2024. Arigraph: Learning knowledge graph world models with episodic memory for llm agents. arXiv preprint arXiv:2407.04363.

Xin Cheng, Di Luo, Xiuying Chen, Lemao Liu, Dongyan Zhao, and Rui Yan. 2024. Lift yourself up: Retrieval-augmented text generation with self-memory. Advances in Neural Information Processing Systems, 36.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Darren Edge, Ha Trinh, Newman Cheng, Joshua Bradley, Alex Chao, Apurva Mody, Steven Truitt, and Jonathan Larson. 2024. From local to global: A graph rag approach to query-focused summarization. arXiv preprint arXiv:2404.16130.

Yu Gu, Xiang Deng, and Yu Su. 2023. Don’t generate, discriminate: A proposal for grounding language models to real-world environments. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4928–4949.

Yu Gu and Yu Su. 2022. ArcaneQA: Dynamic program induction and contextualized encoding for knowledge base question answering. In Proceedings of the 29th International Conference on Computational Linguistics, pages 1718–1731, Gyeongju, Republic of Korea. International Committee on Computational Linguistics.

Gaole He, Yunshi Lan, Jing Jiang, Wayne Xin Zhao, and Ji-Rong Wen. 2021. Improving multi-hop knowledge base question answering by learning intermediate supervision signals. In Proceedings of the 14th ACM International Conference on Web Search and Data Mining, WSDM ’21. ACM.

Chenxu Hu, Jie Fu, Chenzhuang Du, Simian Luo, Junbo Zhao, and Hang Zhao. 2023. Chatdb: Augmenting llms with databases as their symbolic memory. arXiv preprint arXiv:2306.03901.

Jie Huang and Kevin Chen-Chuan Chang. 2023. Towards reasoning in large language models: A survey. In Findings of the Association for Computational Linguistics: ACL 2023, pages 1049–1065.

Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, Yang Song, Chen Zhu, Hengshu Zhu, and Ji-Rong Wen. 2024. Kg-agent: An efficient autonomous agent framework for complex reasoning over knowledge graph. arXiv preprint arXiv:2402.11163.

Jinhao Jiang, Kun Zhou, Wayne Xin Zhao, and Ji-Rong Wen. 2022. Unikgqa: Unified retrieval and reasoning for solving multi-hop question answering over knowledge graph. arXiv preprint arXiv:2212.00959.

Yunshi Lan and Jing Jiang. 2020. Query graph generation for answering multi-hop complex questions from knowledge bases. In Proceedings of the 58th Annual Meeting of the Association for Computational
Linguistics, pages 969–974, Online. Association for Computational Linguistics.

Kuang-Huei Lee, Xinyun Chen, Hiroki Furuta, John Canny, and Ian Fischer. 2024. A human-inspired reading agent with gist memory of very long contexts. In ICLR 2024 Workshop on Large Language Model (LLM) Agents.

Haoran Luo, E Haihong, Zichen Tang, Shiyao Peng, Yikai Guo, Wentai Zhang, Chenghao Ma, Guanting Dong, Meina Song, Wei Lin, et al. 2024. Chatkbqa: A generate-then-retrieve framework for knowledge base question answering with fine-tuned large language models. In Findings of the Association for Computational Linguistics ACL 2024, pages 2039–2056.

LINHAO LUO, Yuan-Fang Li, Reza Haf, and Shirui Pan. 2024. Reasoning on graphs: Faithful and interpretable large language model reasoning. In The Twelfth International Conference on Learning Representations.

Alexander Miller, Adam Fisch, Jesse Dodge, Amir-Hossein Karimi, Antoine Bordes, and Jason Weston. 2016. Key-value memory networks for directly reading documents. In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 1400–1409, Austin, Texas. Association for Computational Linguistics.

Pranoy Panda, Ankush Agarwal, Chaitanya Devaguptapu, Manohar Kaul, et al. 2024. Holmes: Hyper-relational knowledge graphs for multi-hop question answering using llms. arXiv preprint arXiv:2406.06027.

Alireza Rezazadeh, Zichao Li, Wei Wei, and Yujia Bao. 2024. From isolated conversations to hierarchical schemas: Dynamic tree memory representation for llms. In The First Workshop on System-2 Reasoning at Scale, NeurIPS’24.

Haitian Sun, Bhuvan Dhingra, Manzil Zaheer, Kathryn Mazaitis, Ruslan Salakhutdinov, and William Cohen. 2018. Open domain question answering using early fusion of knowledge bases and text. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 4231–4242, Brussels, Belgium. Association for Computational Linguistics.

Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Lionel Ni, Heung-Yeung Shum, and Jian Guo. 2024. Think-on-graph: Deep and responsible reasoning of large language model on knowledge graph. In The Twelfth International Conference on Learning Representations.

Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651.

Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.

Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023a. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering. Preprint, arXiv:2308.13259.

Lei Wang, Wanyu Xu, Yihuai Lan, Zhiqiang Hu, Yunshi Lan, Roy Ka-Wei Lee, and Ee-Peng Lim. 2023b. Plan-and-solve prompting: Improving zero-shot chain-of-thought reasoning by large language models. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2609–2634.

Siyuan Wang, Zhongyu Wei, Yejin Choi, and Xiang Ren. 2024. Symbolic working memory enhances language models for complex rule application. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 17583–17604.

Mufan Xu, Kelai Chen, Xuefeng Bai, Muuyan Yang, Tiejun Zhao, and Min Zhang. 2024a. Llm-based discriminative reasoning for knowledge graph question answering. arXiv preprint arXiv:2412.12643.

Yao Xu, Shizhu He, Jiabei Chen, Zihao Wang, Yangqiu Song, Hanghang Tong, Guang Liu, Kang Liu, and Jun Zhao. 2024b. Generate-on-graph: Treat llm as both agent and kg in incomplete knowledge graph question answering. arXiv preprint arXiv:2404.14741.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. QA-GNN: Reasoning with language models and knowledge graphs for question answering. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 535–546, Online. Association for Computational Linguistics.

Xi Ye, Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, and Caiming Xiong. 2022. RNG-KBQA: Generation augmented iterative ranking for knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 6032–6043, Dublin, Ireland. Association for Computational Linguistics.
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206.

Donghan Yu, Sheng Zhang, Patrick Ng, Henghui Zhu, Alexander Hanbo Li, Jun Wang, Yiqun Hu, William Yang Wang, Zhigou Wang, and Bing Xiang. 2022. Decaf: Joint decoding of answers and logical forms for question answering over knowledge bases. In The Eleventh International Conference on Learning Representations.

Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5773–5784, Dublin, Ireland. Association for Computational Linguistics.

Yu Zhang, Kehai Chen, Xuefeng Bai, Zhao Kang, Quanjiang Guo, and Min Zhang. 2024. Question-guided knowledge graph re-scoring and injection for knowledge graph question answering. In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 8972–8985.

Wanjun Zhong, Lianghong Guo, Qiqi Gao, He Ye, and Yanlin Wang. 2024. Memorybank: Enhancing large language models with long-term memory. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19724–19731.

Yingjie Zhu, Xuefeng Bai, Kehai Chen, Yang Xiang, Jun Yu, and Min Zhang. 2024. Benchmarking and improving large vision-language models for fundamental visual graph understanding and reasoning. arXiv preprint arXiv:2412.13540.

A Metrics

In this section, we present the mathematical formulations and explanations for the metrics that were not fully elaborated in the main text.

Hits@1. Hits@1 quantifies the proportion of questions for which the top-ranked answer in the model’s output is correct. Let Answer represent the list of predicted answers, Golden denote the list of ground truth answers, and total num represent the total number of questions in the dataset. The formula of Hits@1 is defined as follows:

\[
\text{Hits@1} = \frac{\text{count}(Answer[0] \in Golden)}{\text{total\_num}}
\]

F1. Following previous methods, we use the Macro-F1 scoring method, which calculates the F1 for each test sample and then averages those F1 scores among the samples.

B Comparison with ChatKBQA

<table>
  <tr>
    <th rowspan="2">Method</th>
    <th colspan="2">WebQSP (llama2-7b)</th>
    <th colspan="2">CWQ (llama2-13b)</th>
  </tr>
  <tr>
    <th>Hit</th>
    <th>F1</th>
    <th>Hit</th>
    <th>F1</th>
  </tr>
  <tr>
    <td>ChatKBQA</td>
    <td>0.864</td>
    <td>0.835</td>
    <td>0.860</td>
    <td>0.813</td>
  </tr>
  <tr>
    <td>MemQ (Ours)</td>
    <td><b>0.890</b></td>
    <td><b>0.858</b></td>
    <td><b>0.883</b></td>
    <td><b>0.846</b></td>
  </tr>
</table>

Table 6: Performance of MemQ under same settings with ChatKBQA.

Since ChatKBQA (Luo et al., 2024) employs a different experimental setup (using llama2-13b on CWQ and using Hit as metric) compared to other approaches, we independently replicate and evaluate it under its original setting for fair comparison. Detailed comparative results are presented in Table 6. According to the results, MemQ achieved significant improvements in both Hit and F1 metrics.

C Prompt Template

The used prompt templates are listed in the following tables. We designs 3 templates for the three types of queries shown in Table 9, Table 10 and Table 11. Besides, for the finetuning process to enhance the LLM’s reasoning ability, we use the template in Table 12.

D Case study

Here, we present two additional cases generated by our method. As shown in Table 13, our method accurately constructs queries with "Order By" and "Limit" clauses in Step 5, demonstrating its ability to interpret the temporal meaning of "last time" in the question, which a nuance often overlooked by previous methods.

In Table 14, our method retrieves multiple queries with similar semantic meanings. While this approach may introduce redundancy, we argue that it is justified given the nature of the Freebase Knowledge Graph, where edges with similar semantic meanings do exist and can be challenging even for humans to distinguish. Consequently, retrieving all such edges ensures comprehensive coverage of potentially relevant answers.

E Out-of-distribution Performance

When the question involves query statements not present in the memory, MemQ can still use the next-best retrieval results from the memory repository to attempt finding the correct answer. To verify this, we filtered all questions from the WebQSP
and CWQ test sets that contain statements not present in the train sets (a total of 46), and tested and compared MemQ’s performance with previous methods on these questions. The results are shown in Table 7.

<table>
  <tr>
    <th></th>
    <th>RoG</th>
    <th>ChatKBQA(llama2-13b)</th>
    <th>MemQ</th>
  </tr>
  <tr>
    <th>Hit</th>
    <td>0.326</td>
    <td>0.369</td>
    <td>0.587</td>
  </tr>
</table>

Table 7: Performance on out-of-distribution cases.

"Hit" refers to the proportion of questions where the predicted answer list includes any of the correct answers. We use llama2-7b as the base model for MemQ and RoG. It can be observed that the MemQ method demonstrates better generalization ability on these issues compared to previous work. This improvement likely stems from MemQ’s fine-grained memory and adaptive recall strategy, improving its approximate retrieval capability in out-of-distribution scenarios.

F  Time efficiency of memory module

We calculate the average time cost for reasoning and query reconstruction in MemQ under different question complexities (indicate by hop counts) in Table 8.

<table>
  <tr>
    <th>Hop count</th>
    <th>1</th>
    <th>2</th>
    <th>3</th>
    <th>4</th>
    <th>5</th>
    <th>>5</th>
  </tr>
  <tr>
    <th>Reasoning</th>
    <td>0.62</td>
    <td>1.13</td>
    <td>1.36</td>
    <td>1.61</td>
    <td>1.92</td>
    <td>3.28</td>
  </tr>
  <tr>
    <th>Query reconstruction</th>
    <td>0.01</td>
    <td>0.02</td>
    <td>0.02</td>
    <td>0.03</td>
    <td>0.03</td>
    <td>0.05</td>
  </tr>
</table>

Table 8: Time cost of reasoning and query reconstruction.

The experiment is done using single NVIDIA H20 96GB GPU with Intel(R) Xeon(R) Platinum 8468V CPU. The results indicate that the additional time cost introduced by MemQ’s memory retrieval and query reconstruction steps is relatively low. Although complex questions may increase MemQ’s response time, the overall overhead remains manageable.
Prompt for Structure 1

Act as a SPARQL expert.
I need you to explain the meaning and function of a specific part of a SPARQL query.
You job is answer the Question for me. ONLY OUTPUT THE ANSWER, NOTING ELSE!!
### EXAMPLE1
Sparql:
?entity1 ns:location.country.currency_used ?entity2 .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the currency used in the country ?entity1.
### EXAMPLE2
Sparql:
?entity2 ns:location.country.currency_used ?entity1 .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the country that use ?entity1 as currency.
### EXAMPLE3
Sparql:
?entity2 ns:government.election_campaign.candidate ?entity1 .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the election campaign which ?entity1 is the candidate.
### EXAMPLE4
Sparql:
?entity1 ns:government.election_campaign.candidate ?entity2 .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the candidate in the election campaign ?entity1.
### EXAMPLE5
Sparql:
{ ?entity2 ns:sports.sports_championship_event.runner_up ?entity1 } UNION { ?entity2 ns:sports.sports_championship_event.champion ?entity1 }
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is either the runner-up or the champion of a sports championship event ?entity1.
### EXAMPLE6
Sparql:
{ ?entity1 ns:location.statistical_region.places_exported_to ?tmp0 .
?tmp0 ns:location.imports_and_exports.exported_to ?entity2 } UNION { ?entity1 ns:location.statistical_region.places_exported_from ?tmp1 .
?tmp1 ns:location.imports_and_exports.exported_from ?entity2 }
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the place that is either exported to or exported from the statistical region ?entity1.
### YOUR TURN
Sparql: {sparql}
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Prompt for Structure 2

Act as a SPARQL expert.
I need you to explain the meaning and function of a specific part of a SPARQL query.
You job is answer the Question for me. ONLY OUTPUT THE ANSWER, NOTING ELSE!!
### EXAMPLE1
Sparql:
?cvt ns:government.government_position_held.office_holder ?entity1 .
?entity2 ns:government.governmental_body.members ?cvt .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the governmental body that is held by ?entity1.
Answer: ?entity2 is the governmental body that has an office holder ?entity1.
### EXAMPLE2 Sparql:
?entity1 ns:film.actor.film ?cvt .
?cvt ns:film.performance.character ?entity2 .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the character played by the actor ?entity1.
### EXAMPLE3
Sparql:
?cvt ns:music.group_membership.member ?entity1 .
?entity2 ns:music.musical_group.member ?cvt .
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer: ?entity2 is the musical group that has the member ?entity1.
Answer: ?entity2 is the group that includes the member ?entity1.
### YOUR TURN
Sparql:
{sparql}
Question: How does ?entity2 related to ?entity1 ?
Please answer the question with "?entity2 is [noun phrase]" .
Answer:

Table 10: The prompt to get the explanation of Structure 2 graph
Prompt for Structure 3

Act as a SPARQL expert.
I need you to explain the meaning and function of a specific part of a SPARQL query.
You job is complete the answer for me. ONLY OUTPUT THE ANSWER, NOTING ELSE!!
### EXAMPLE1
Sparql:
?cvt ns:sports.sports_team_coach_tenure.position ?entity1 .
?cvt ns:sports.sports_team_coach_tenure.coach ?entity2 .
?entity3 ns:sports.sports_team.coaches ?cvt .
Question: How does ?entity3 related to ?entity1 and ?entity2 ?
Please answer the question with "?entity3 is [noun phrase]" .
Answer: ?entity3 is the sports team that has a coach ?entity2 who holds the position ?entity1 .
### EXAMLPE2
Sparql:
?entity1 ns:film.actor.film ?cvt .
?cvt ns:film.performance.character ?entity2 .
?cvt ns:film.performance.film ?entity3 .
Question: How does ?entity3 related to ?entity1 and ?entity2 ?
Please answer the question with "?entity3 is [noun phrase]" .
Answer: ?entity3 is the film in which the actor ?entity1 performs the character ?entity2.
Answer: ?entity3 is the film in which ?entity1 acted as a character ?entity2.
### EXAMLPE3
Sparql:
?entity1 ns:sports.pro_athlete.teams ?cvt .
?cvt ns:sports.sports_team_roster.team ?entity2 .
?cvt ns:sports.sports_team_roster.from ?entity3
Question: How does ?entity3 related to ?entity1 and ?entity2 ?
Please answer the question with "?entity3 is [noun phrase]" .
Answer: ?entity3 is the starting date when ?entity1 was part of the team ?entity2.
Answer: ?entity3 is the start date of the period during which ?entity1 was part of the team ?entity2.
### YOUR TURN
Sparql:
{sparql}
Question: How does ?entity3 related to ?entity1 and ?entity2 ?
Please answer the question with "?entity3 is [noun phrase]" .
Answer:

Table 11: The prompt to get the explanation of Structure 3 graph
Prompt for Plan Expert
You are given a problem to solve step by step. Each step should begin with either "Find", "Make sure" or "Rank". Finally, you need to output which one is the final answer.
The steps should logically follow from one another, where each step builds on the outcome of the previous steps.
Each step should be simple, clear, and directly related to achieving the overall goal.
Some topic entities you can use to start the plan are provided below.
Question:
{question}
Topic Entities:
{topic_entities}

Table 12: The prompt utilized for generating knowledge reasoning plans in the Planning Expert.

WebQTest-994

Question:
When was the last time the Dallas Cowboys won the superbowl?
Generated Plan and Reconstruct Query:
Step1: Find the championship won by the sports team dallas cowboys, assign it to ?x.
Retrieved Query1: ?entity1 ns:sports.sports_team.championships ?entity2
Step2: Find the end date of the event ?x, assign it to ?sk0.
Retrieved Query2: ?entity1 ns:time.event.end_date ?entity2
Step3: Find the championship associated with the sports championship event ?x, assign it to ?championship.
Retrieved Query3: ?entity1 ns:sports.sports_championship_event.championship ?entity2
Step4: Make sure ?championship should be super bowl.
Retrieved Query4: FILTER(?championship = ns:m.06x5s)
Step5: Sort the result based on ?sk0 in descending order and keep the first result.
Retrieved Query5: ORDER BY DESC(?sk0) LIMIT 1
Finally the answer is ?x.
Reconstruct Sparql:
PREFIX ns: <http://rdf.freebase.com/ns/> SELECT DISTINCT ?x WHERE{
ns:m.02896 ns:sports.sports_team.championships ?x .
?x ns:time.event.end_date ?sk0 .
?x ns:sports.sports_championship_event.championship ?championship .
FILTER(?championship = ns:m.06x5s).
FILTER (!isLiteral(?x) OR lang(?x) = "" OR langMatches(lang(?x), 'en')).
FILTER(?x != ns:m.02896) }
ORDER BY DESC(?sk0) LIMIT 1
Output:
Super Bowl XXX (m.076wz)

Table 13: Case Study 1
WebQTest-212_3c65a82425ff4488504b53ba502d0eb7

Question:
What is the name of the state that has the elected official Jan Brewer and also has the Colorado River flow through it?

Generated Plan and Reconstruct Query:
Step1: Find the location that partially contains *Colorado River*, assign it to?x.
Retrieved Query1: { ?entity1 ns:location.location.partially_containedby ?entity2 } UNION { ?entity2 ns:location.location.partially_contains ?entity1 }UNION { ?cvt ns:location.partial_containment_relationship.partially_contains ?entity1 . ?entity2 ns:location.location.partiallycontains ?cvt }UNION { ?entity1 location.location.partially_contained_by ?cvt ?cvt ns:location.partial_containment_relationship.partially_contained_by ?entity2}
Step2: Find the official who appointed the governing official ?x, assign it to ?appointed_by.
Retrieved Query2: ?entity1 ns:government.governmental_jurisdiction.governing_officials ?cvt . ?cvt ns:government.government_position_held.appointed_by ?entity2 .
Step3: Make sure ?appointed_by should be Jan Brewer.
Retrieved Query3: FILTER(?appointed_by = ns:m.02pkblc)
Finally the answer is ?x.

Reconstruct Sparql:
{ns:m.018qjq ns:location.location.partially_containedby ?x } UNION {?x ns:location.location.partially_contains ns:m.018qjq } UNION { ?cvt ns:location.partial_containment_relationship.partially_contains ns:m.018qjq . ?x ns:location.location.partiallycontains ?cvt } UNION {ns:m.018qjq ns:location.location.partially_contained_by ?cvt1 . ?cvt1 ns:location.partial_containment_relationship.partially_contained_by ?x }. ?x ns:government.governmental_jurisdiction.governing_officials ?cvt2 . ?cvt2 ns:government.government_position_held.appointed_by ?appointed_by .
FILTER(?appointed_by = ns:m.02pkblc).
FILTER (!isLiteral(?x) OR lang(?x) = " OR langMatches(lang(?x), 'en')).
FILTER(?x != ns:m.018qjq )

Output:
Arizona (m.0vmt)

Table 14: Case Study 2

Reviewer paper summaries: MemQ introduces a query memory module that decouples LLM from tool-calling tasks, allowing LLM to focus on the knowledge reasoning process, thereby improving the readability and accuracy of reasoning. By leveraging natural language reasoning strategies and semantic similarity retrieval, MemQ significantly enhances LLM’s reasoning ability in KGQA while mitigating hallucination-induced tool-calling issues.

This paper proposes a method to address the KBQA problem by leveraging memory, in-context learning, and LLMs. The approach involves storing relevant information related to questions and their corresponding correct SPARQL snippets in memory. When a new query is received, the stored information is used to help better transform the NLP query into SPARQL. Various techniques are employed in this approach, including question decomposition and in-context learning. Experimental results demonstrate the effectiveness of the proposed method.
Summary of strengths: 1. By separating reasoning from tool invocation, MemQ significantly reduces hallucinatory tool invocations.
2. MemQ achieves state-of-the-art performance on WebQSP and CWQ benchmarks, with notable improvements in Hits@1 and F1 metrics.
3. MemQ performs well even with limited training data, showing strong data efficiency.

The paper has the following strengths:

First, the performance is excellent, outperforming all baseline methods.
Second, the idea of utilizing memory is interesting.
Summary of weaknesses: 1. The introduction of memory retrieval and query reconstruction steps might affect reasoning speed, especially for complex queries.
2. How to handle queries that do not appear in memory?

The paper has the following weaknesses:

First, the writing needs improvement. For example, the second paragraph of the introduction section could be clearer in explaining the example. Currently, the explanation is brief and difficult to understand.

Second, it is unclear how the "Reasoning steps" are generated. The explanation needs to be clarified.

Third, for Rule-based Decomposing, it is unclear how this is done. Is in-context learning used for this process?

Fourth, what happens if there is no relevant information in memory, or if the information returned is not related to the reasoning steps?
Comments, suggestions and typos: n/a

line 161 type "the" to “that"
Overall assessment scores: 4

2