GNN-RAG: Graph Neural Retrieval for Large Language Model Reasoning

## Abstract

Abstract Knowledge Graphs (KGs) represent human-crafted factual knowledge in the form of triplets (head, relation, tail) , which collectively form a graph. Question Answering over KGs (KGQA) is the task of answering natural questions grounding the reasoning to the information provided by the KG. Large Language Models (LLMs) are the state-of-the-art models for QA tasks due to their remarkable ability to understand natural language. On the other hand, Graph Neural Networks (GNNs) have been widely used for KGQA as they can handle the complex graph information stored in the KG. In this work, we introduce Gnn-Rag , a novel method for combining language understanding abilities of LLMs with the reasoning abilities of GNNs in a retrieval-augmented generation (RAG) style. First, a GNN reasons over a dense KG subgraph to retrieve answer candidates for a given question. Second, the shortest paths in the KG that connect question entities and answer candidates are extracted to represent KG reasoning paths. The extracted paths are verbalized and given as input for LLM reasoning with RAG. In our Gnn-Rag framework, the GNN acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for ultimate KGQA. Furthermore, we develop a retrieval augmentation (RA) technique to further boost KGQA performance with Gnn-Rag . Experimental results show that Gnn-Rag achieves state-of-the-art performance in two widely used KGQA benchmarks (WebQSP and CWQ), outperforming or matching GPT-4 performance with a 7B tuned LLM. In addition, Gnn-Rag excels on multi-hop and multi-entity questions outperforming competing approaches by 8.9–15.5% points at answer F1.
We provide the code and KGQA results at https://github.com/cmavro/GNN-RAG .

## 1 Introduction

Large Language Models (LLMs) (Brown et al., ; Bommasani et al., ; Chowdhery et al., ) are the state-of-the-art models in many NLP tasks due to their remarkable ability to understand natural language. LLM’s power stems from pretraining on large corpora of textual data to obtain general human knowledge (Kaplan et al., ; Hoffmann et al., ). However, because pretraining is costly and time-consuming (Gururangan et al., ), LLMs cannot easily adapt to new or in-domain knowledge and are prone to hallucinations (Zhang et al., ).

Knowledge Graphs (KGs) (Vrandečić and Krötzsch, ) are databases that store information in structured form that can be easily updated. KGs represent human-crafted factual knowledge in the form of triplets (head, relation, tail), e.g., <Jamaica $\rightarrow$ language_spoken $\rightarrow$ English>, which collectively form a graph. In the case of KGs, the stored knowledge is updated by fact addition or removal. As KGs capture complex interactions between the stored entities, e.g., multi-hop relations, they are widely used for knowledge-intensive task, such as Question Answering (QA) (Pan et al., ).

Figure: Figure 1: Retrieval effect on multi-hop/entity KGQA. Our Gnn-Rag outperforms existing KG-RAG methods by 8.9–15.5% points.

Retrieval-augmented generation (RAG) is a framework that alleviates LLM hallucinations by enriching the input context with up-to-date and accurate information (Lewis et al., ), e.g., obtained from the KG. In the KGQA task, the goal is to answer natural questions grounding the reasoning to the information provided by the KG. For instance, the input for RAG becomes “Knowledge: Jamaica $\rightarrow$ language_spoken $\rightarrow$ English \n Question: Which language do Jamaican people speak?”, where the LLM has access to KG information for answering the question.

RAG’s performance highly depends on the KG facts that are retrieved (Wu et al., ). The challenge is that KGs store complex graph information (they usually consist of millions of facts) and retrieving the right information requires effective graph processing, while retrieving irrelevant information may confuse the LLM during its KGQA reasoning (He et al., ). Existing retrieval methods that rely on LLMs to retrieve relevant KG information (LLM-based retrieval) underperform on multi-hop KGQA as they cannot handle complex graph information (Baek et al., ; Luo et al., ) or they need the internal knowledge of very large LMs, e.g., GPT-4, to compensate for missing information during KG retrieval (Sun et al., ).

In this work, we introduce Gnn-Rag, a novel method for improving RAG for KGQA. Gnn-Rag relies on Graph Neural Networks (GNNs) (Mavromatis and Karypis, ), which are powerful graph representation learners, to handle the complex graph information stored in the KG. Although GNNs cannot understand natural language the same way LLMs do, Gnn-Rag repurposes their graph processing power for retrieval. First, a GNN reasons over a dense KG subgraph to retrieve answer candidates for a given question. Second, the shortest paths in the KG that connect question entities and GNN-based answers are extracted to represent useful KG reasoning paths. The extracted paths are verbalized and given as input for LLM reasoning with RAG.
Furthermore, we show that Gnn-Rag can be augmented with LLM-based retrievers to further boost KGQA performance.
Experimental results show Gnn-Rag’s superiority over competing RAG-based systems for KGQA by outperforming them by up to 15.5% points at complex KGQA performance (Figure 1).
Our contributions are summarized below:

- •
Framework: Gnn-Rag repurposes GNNs for KGQA retrieval to enhance the reasoning abilities of LLMs. In our Gnn-Rag framework, the GNN acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for ultimate KGQA. Moreover, our retrieval analysis (Section 4.3) guides the design of a retrieval augmentation (RA) technique to boost Gnn-Rag’s performance (Section 4.4).
- •
Effectiveness & Faithfulness: Gnn-Rag achieves state-of-the-art performance in two widely
used KGQA benchmarks (WebQSP and CWQ). Gnn-Rag retrieves multi-hop information that is necessary for faithful LLM reasoning on complex questions (8.9–15.5% improvement; see Figure 1).
- •
Efficiency: Gnn-Rag improves vanilla LLMs on KGQA performance without incurring additional LLM calls as existing RAG systems for KGQA require. In addition, Gnn-Rag outperforms or matches GPT-4 performance with a 7B tuned LLM.

## 2 Related Work

KGQA Methods. KGQA methods fall into two categories (Lan et al., ): (A) Semantic Parsing (SP) methods and (B) Information Retrieval (IR) methods. SP methods (Sun et al., ; Lan and Jiang, ; Ye et al., ) learn to transform the given question into a query of logical form, e.g., SPARQL query. The transformed query is then executed over the KG to obtain the answers. However, SP methods require ground-truth logical queries for training, which are time-consuming to annotate in practice, and may lead non-executable queries due to syntactical or semantic errors (Das et al., ; Yu et al., ). IR methods (Sun et al., , ) focus on the weakly-supervised KGQA setting, where only question-answer pairs are given for training. IR methods retrieve KG information, e.g., a KG subgraph (Zhang et al., ), which is used as input during KGQA reasoning. In Appendix A, we analyze the reasoning abilities of the prevailing models (GNNs & LLMs) for KGQA, and in Section 4, we propose Gnn-Rag which leverages the strengths of both of these models.

Graph-augmented LMs. Combining LMs with graphs that store information in natural language is an emerging research area (Jin et al., ). There are two main directions, (i) methods that enhance LMs with *latent* graph information (Zhang et al., ; Tian et al., ; Huang et al., ), e.g., obtained by GNNs, and (ii) methods that insert *verbalized* graph information at the input (Xie et al., ; Jiang et al., ; Jin et al., ), similar to RAG. The methods of the first direction are limited because of the modality mismatch between language and graph, which can lead to inferior performance for knowledge-intensive tasks (Mavromatis et al., ). On the other hand, methods of the second direction may fetch noisy information when the underlying graph is large and such information can decrease the LM’s reasoning ability (Wu et al., ; He et al., ). Gnn-Rag employs GNNs for information retrieval and RAG for KGQA reasoning, achieving superior performance over existing approaches.

Figure: Figure 2: The landscape of existing KGQA methods. GNN-based methods reason on dense subgraphs as they can handle complex and multi-hop graph information. LLM-based methods employ the same LLM for both retrieval and reasoning due to its ability to understand natural language.
Refer to caption: x1.png

## 3 Problem Statement & Background

KGQA. We are given a KG ${\mathcal{G}}$ that contains facts represented as $(v,r,v^{\prime})$, where $v$ denotes the head entity, $v^{\prime}$ denotes the tail entity, and $r$ is the corresponding relation between the two entities.
Given ${\mathcal{G}}$ and a natural language question $q$, the task of KGQA is to extract a set of entities $\{a\}\in{\mathcal{G}}$ that correctly answer $q$. Following previous works (Lan et al., ), question-answer pairs are given for training, but not the ground-truth paths that lead to the answers.

Retrieval & Reasoning. As KGs usually contain millions of facts and nodes, a smaller question-specific subgraph ${\mathcal{G}}_{q}$ is retrieved for a question $q$, e.g., via entity linking and neighbor extraction (Yih et al., ). Ideally, all correct answers for the question are contained in the retrieved subgraph, $\{a\}\in{\mathcal{G}}_{q}$.
The retrieved subgraph ${\mathcal{G}}_{q}$ along with the question $q$ are used as input to a reasoning model, which outputs the correct answer(s). The prevailing reasoning models for the KGQA setting studied are GNNs and LLMs.

GNNs. KGQA can be regarded as a node classification problem, where KG entities are classified as answers vs. non-answers for a given question.
GNNs Kipf and Welling (); Veličković et al. (); Schlichtkrull et al. () are powerful graph representation learners suited for tasks such as node classification.
GNNs update the representation ${\bm{h}}_{v}^{(l)}$ of node $v$ at layer $l$ by aggregating messages ${\bm{m}}^{(l)}_{vv^{\prime}}$ from each neighbor $v^{\prime}$. During KGQA, the message passing is also conditioned to the given question $q$ (He et al., ). For readability purposes, we present the following GNN update for KGQA,

$$ ${\bm{h}}_{v}^{(l)}=\psi\Big({\bm{h}}_{v}^{(l-1)},\sum_{v^{\prime}\in\mathcal{N}_{v}}\omega(q,r)\cdot{\bm{m}}_{vv^{\prime}}^{(l)}\Big),$ (1) $$

where function $\omega(\cdot)$ measures how relevant relation $r$ of fact $(v,r,v^{\prime})$ is to question $q$. Neighbor messages ${\bm{m}}_{vv^{\prime}}^{(l)}$ are aggregated by a sum-operator $\sum$, which is typically employed in GNNs. Function $\psi(\cdot)$ combines representations from consecutive GNN layers.

LLMs. LLMs for KGQA use KG information to perform retrieval-augmented generation (RAG) as follows. The retrieved subgraph is first converted into natural language so that it can be processed by the LLM. The input given to the LLM contains the KG factual information along with the question and a prompt. For instance, the input becomes “Knowledge: Jamaica $\rightarrow$ language_spoken $\rightarrow$ English \n Question: Which language do Jamaican people speak?”, where the LLM has access to KG information for answering the question.

Landscape of KGQA methods. Figure 2 presents the landscape of existing KGQA methods with respect to KG retrieval and reasoning. GNN-based methods, such as GraftNet (Sun et al., ), NSM (He et al., ), and ReaRev (Mavromatis and Karypis, ), reason over a dense KG subgraph leveraging the GNN’s ability to handle complex graph information.
Recent LLM-based methods leverage the LLM’s power for both retrieval and reasoning. ToG (Sun et al., ) uses the LLM to retrieve relevant facts hop-by-hop. RoG (Luo et al., ) uses the LLM to generate plausible relation paths which are then mapped on the KG to retrieve the relevant information.

LLM-based Retriever. We present an example of an LLM-based retriever (RoG;  (Luo et al., )). Given training question-answer pairs, RoG extracts the shortest paths to the answers starting from question entities for fine-tuning the retriever. Based on the extracted paths, an LLM (LLaMA2-Chat-7B (Touvron et al., )) is fine-tuned to generate reasoning paths given a question $q$ as

$$ $\text{LLM}(\text{prompt},q)\Longrightarrow\{r_{1}\rightarrow\cdots\rightarrow r_{t}\}_{k},$ (2) $$

where the prompt is “Please generate a valid relation path that can be helpful for answering the following question: {Question}”.
Beam-search decoding is used to generate $k$ diverse sets of reasoning paths for better answer coverage, e.g., relations {<official_language>, <language_spoken>} for the question “Which language do Jamaican people speak?”. The generated paths are mapped on the KG, starting from the question entities, in order to retrieve the intermediate entities for RAG, e.g., <Jamaica $\rightarrow$ language_spoken $\rightarrow$ English>.

## 4 Gnn-Rag

We introduce Gnn-Rag, a novel method for combining language understanding abilities of LLMs with the reasoning abilities of GNNs in a retrieval-augmented generation (RAG) style. We provide the overall framework in Figure 3. First, a GNN reasons over a dense KG subgraph to retrieve answer candidates for a given question. Second, the shortest paths in the KG that connect question entities and GNN-based answers are extracted to represent useful KG reasoning paths. The extracted paths are verbalized and given as input for LLM reasoning with RAG. In our Gnn-Rag framework, the GNN acts as a dense subgraph reasoner to extract useful graph information, while the LLM leverages its natural language processing ability for ultimate KGQA.

Figure: Figure 3: Gnn-Rag: The GNN reasons over a dense subgraph to retrieve candidate answers, along with the corresponding reasoning paths (shortest paths from question entities to answers). The retrieved reasoning paths –optionally combined with retrieval augmentation (RA)– are verbalized and given to the LLM for RAG.
Refer to caption: x2.png

### 4.1 GNN

In order to retrieve high-quality reasoning paths via Gnn-Rag, we leverage state-of-the-art GNNs for KGQA. We prefer GNNs over other KGQA methods, e.g., embedding-based methods (Saxena et al., ), due to their ability to handle complex graph interactions and answer multi-hop questions. GNNs mark themselves as good candidates for retrieval due to their architectural benefit of exploring diverse reasoning paths (Mavromatis and Karypis, ; Choi et al., ) that result in high answer recall.

When GNN reasoning is completed ($L$ GNN updates via Equation 1), all nodes in the subgraph are scored as answers vs. non-answers based on their final GNN representations ${\bm{h}}_{v}^{(L)}$, followed by the $\mathrm{softmax}(\cdot)$ operation. The GNN parameters are optimized via node classification (answers vs. non-answers) using the training question-answer pairs. During inference, the nodes with the highest probability scores, e.g., above a probability threshold, are returned as candidate answers, along with the shortest paths connecting the question entities with the candidate answers (reasoning paths). The retrieved reasoning paths are used as input for LLM-based RAG.

Different GNNs may fetch different reasoning paths for RAG. As presented in  Equation 3, GNN reasoning depends on the question-relation matching operation $\omega(q,r)$. A common implementation of $\omega(q,r)$ is $\phi({\bm{q}}^{(k)}\odot{\bm{r}})$ (He et al., ), where function $\phi$ is a neural network, and $\odot$ is the element-wise multiplication. Question representations ${\bm{q}}{(k)}$ and KG relation representations ${\bm{r}}$ are encoded via a shared pretrained LM (Jiang et al., ) as

$$ ${\bm{q}}^{(k)}=\gamma_{k}\big(\text{LM}(q)\big),\;\;{\bm{r}}=\gamma_{c}\big(\text{LM}(r)\big),$ (3) $$

where $\gamma_{k}$ are attention-based pooling neural networks so that different representations ${\bm{q}}^{(k)}$ attend to different question tokens, and $\gamma_{\text{c}}$ is the [CLS] token pooling.

In Appendix B, we develop a Theorem that shows that the GNN’s output depends on the question-relation matching operation $\omega(q,r)$ and as result, the choice of the LM in Equation 3 plays an important role regarding which answer nodes are retrieved.
Instead of trying different GNN architectures, we employ different LMs in Equation 3 that result into different output node representations. Specifically, we train two separate GNN models, one using pretrained LMs, such as SBERT (Reimers and Gurevych, ), and one using $\text{LM}_{\text{SR}}$, a pretrained LM for question-relation matching over the KG (Zhang et al., ). Our experimental results suggest that, although these GNNs retrieve different KG information, they both improve RAG-based KGQA.

### 4.2 LLM

After obtaining the reasoning paths by Gnn-Rag, we verbalize them and give them as input to a downstream LLM, such as ChatGPT or LLaMA. However, LLMs are sensitive to the input prompt template and the way that the graph information is verbalized.

To alleviate this issue, we opt to follow RAG prompt tuning (Lin et al., ; Zhang et al., ) for LLMs that have open weights and are feasible to train. A LLaMA2-Chat-7B model is fine-tuned based on the training question-answer pairs to generate a list of correct answers, given the prompt:
“Based on the reasoning paths, please answer the given question.\n Reasoning Paths: {Reasoning Paths} \n Question: {Question}”.
The reasoning paths are verbalised as “{question entity} $\rightarrow$ {relation} $\rightarrow$ {entity} $\rightarrow$ $\cdots$ $\rightarrow$ {relation} $\rightarrow$ {answer entity} \n” (see Figure 3).
During training, the reasoning paths are the shortest paths from question entities to answer entities. During inference, the reasoning paths are obtained by Gnn-Rag.

### 4.3 Retrieval Analysis: Why GNNs & Their Limitations

**Table 1: Retrieval results for WebQSP.**
| Retriever | 1-hop questions | 2-hop questions |  |  |
| --- | --- | --- | --- | --- |
| #Input Tok. | %Ans. Cov. | #Input Tok. | %Ans. Cov. |  |
| RoG (Luo et al., ) | 150 | 87.1 | 435 | 82.1 |
| GNN ($L=1$) | 112 | 83.6 | 2,582 | 79.8 |
| GNN ($L=3$) | 105 | 82.4 | 357 | 88.5 |

GNNs leverage the graph structure to retrieve relevant parts of the KG that contain multi-hop information. We provide experimental evidence on why GNNs are good retrievers for multi-hop KGQA.
We train two different GNNs, a deep one ($L=3$) and a shallow one ($L=1$), and measure their retrieval capabilities. We report the ‘Answer Coverage’ metric, which evaluates whether the retriever is able to fetch at least one correct answer for RAG. Note that ‘Answer Coverage’ does not measure downstream KGQA performance but whether the retriever fetches relevant KG information. ‘#Input Tokens’ denotes the median number of the input tokens of the retrieved KG paths.
Table 1 shows GNN retrieval results for single-hop and multi-hop questions of the WebQSP dataset compared to an LLM-based retriever (RoG; Equation 2). The results indicate that deep GNNs ($L=3$) can handle the complex graph structure and retrieve useful *multi-hop* information more effectively (%Ans. Cov.) and efficiently (#Input Tok.) than the LLM and the shallow GNN.

On the other hand, the limitation of GNNs is for simple (1-hop) questions, where accurate question-relation matching is more important than deep graph search (see our Theorem in Appendix A that states this GNN limitation). In such cases, the LLM retriever is better at selecting the right KG information due to its natural language understanding abilities (we provide an example later in Figure 5).

### 4.4 Retrieval Augmentation (RA)

Retrieval augmentation (RA) combines the retrieved KG information from different approaches to increase diversity and answer recall. Motivated by the results in Section 4.3, we present a RA technique (Gnn-Rag+RA), which complements the GNN retriever with an LLM-based retriever to combine their strengths on multi-hop and single-hop questions, respectively. Specifically, we experiment with the RoG retrieval, which is described in Equation 2. During inference, we take the union of the reasoning paths retrieved by the two retrievers.

A downside of LLM-based retrieval is that it requires multiple generations (beam-search decoding) to retrieve diverse paths, which trades efficiency for effectiveness (we provide a performance analysis in Appendix A). A cheaper alternative is to perform RA by combining the outputs of different GNNs, which are equipped with different LMs in Equation 3. Our Gnn-Rag+Ensemble takes the union of the retrieved paths of the two different GNNs (GNN+SBERT & GNN+$\text{LM}_{\text{SR}}$) as input for RAG.

## 5 Experimental Setup

KGQA Datasets.
We experiment with two widely used KGQA benchmarks: WebQuestionsSP (WebQSP) (Yih et al., ),
Complex WebQuestions 1.1 (CWQ) (Talmor and Berant, ).
WebQSP contains 4,737 natural language questions that are answerable using a subset Freebase KG (Bollacker et al., ). The questions require up to 2-hop reasoning within this KG. CWQ contains 34,699 total complex questions that require up to 4-hops of reasoning over the KG. We provide the detailed dataset statistics in Appendix C.

Implementation & Evaluation.
For subgraph retrieval, we use the linked entities and the pagerank algorithm to extract dense graph information (He et al., ). We employ ReaRev (Mavromatis and Karypis, ), which is a GNN targeting at *deep* KG reasoning (Section 4.3), for Gnn-Rag. The default implementation is to combine ReaRev with SBERT as the LM in Equation 3. In addition, we combine ReaRev with $\text{LM}_{\text{SR}}$, which is obtained by following the implementation of SR (Zhang et al., ). We employ RoG (Luo et al., ) for RAG-based prompt tuning (Section 4.2).
For evaluation, we adopt Hit, Hits@1 (H@1), and F1 metrics. Hit measures if any of the true answers is found in the generated response, which is typically employed when evaluating LLMs. H@1 is the accuracy of the top/first predicted answer. F1 takes into account the recall (number of true answers found) and the precision (number of false answers found) of the generated answers.
Further experimental setup details are provided in Appendix C.

**Table 2: Performance comparison of different methods on the two KGQA benchmarks. We denote the best and second-best method.**
| Type | Method | WebQSP | CWQ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Hit | H@1 | F1 | Hit | H@1 | F1 |  |  |
| Embedding | KV-Mem Miller et al. () | – | 46.7 | 38.6 | – | 21.1 | – |
| EmbedKGQA Saxena et al. () | – | 66.6 | – | – | – | – |  |
| TransferNet Shi et al. () | – | 71.4 | – | – | 48.6 | – |  |
| Rigel Sen et al. () | – | 73.3 | – | – | 48.7 | – |  |
| GNN | GraftNet Sun et al. () | – | 66.7 | 62.4 | – | 36.8 | 32.7 |
| PullNet Sun et al. () | – | 68.1 | – | – | 45.9 | – |  |
| NSM He et al. () | – | 68.7 | 62.8 | – | 47.6 | 42.4 |  |
| SR+NSM(+E2E) (Zhang et al., ) | – | 69.5 | 64.1 | – | 50.2 | 47.1 |  |
| NSM+h He et al. () | – | 74.3 | 67.4 | – | 48.8 | 44.0 |  |
| SQALER Atzeni et al. () | – | 76.1 | – | – | – | – |  |
| UniKGQA (Jiang et al., ) | – | 77.2 | 72.2 | – | 51.2 | 49.1 |  |
| ReaRev (Mavromatis and Karypis, ) | – | 76.4 | 70.9 | – | 52.9 | 47.8 |  |
|  | ReaRev + $\text{LM}_{\text{SR}}$ | – | 77.5 | 72.8 | – | 53.3 | 49.7 |
| LLM | Flan-T5-xl (Chung et al., ) | 31.0 | – | – | 14.7 | – | – |
| Alpaca-7B (Taori et al., ) | 51.8 | – | – | 27.4 | – | – |  |
| LLaMA2-Chat-7B (Touvron et al., ) | 64.4 | – | – | 34.6 | – | – |  |
| ChatGPT | 66.8 | – | – | 39.9 | – | – |  |
| ChatGPT+CoT | 75.6 | – | – | 48.9 | – | – |  |
| KG+LLM | KD-CoT (Wang et al., ) | 68.6 | – | 52.5 | 55.7 | – | – |
| StructGPT (Jiang et al., ) | 72.6 | – | – | – | – | – |  |
| KB-BINDER (Li et al., ) | 74.4 | – | – | – | – | – |  |
| ToG+LLaMA2-70B (Sun et al., ) | 68.9 | – | – | 57.6 | – | – |  |
| ToG+ChatGPT (Sun et al., ) | 76.2 | – | – | 58.9 | – | – |  |
| ToG+GPT-4 (Sun et al., ) | 82.6 | – | – | 69.5 | – | – |  |
| RoG (Luo et al., ) | 85.7 | 80.0 | 70.8 | 62.6 | 57.8 | 56.2 |  |
| GNN+LLM | G-Retriever (He et al., ) | – | 70.1 | – | – | – | – |
| Gnn-Rag (Ours) | 85.7 | 80.6 | 71.3 | 66.8 | 61.7 | 59.4 |  |
| Gnn-Rag+RA (Ours) | 90.7 | 82.8 | 73.5 | 68.7 | 62.8 | 60.4 |  |

Competing Methods. We compare with SOTA GNN and LLM methods for KGQA (Mavromatis and Karypis, ; Li et al., ). We also include earlier embedding-based methods (Saxena et al., ) as well as zero-shot/few-shot LLMs (Taori et al., ). We do not compare with semantic parsing methods (Yu et al., ) as they use additional training data (SPARQL annotations), which are difficult to obtain in practice. Furthermore, we compare Gnn-Rag with LLM-based retrieval approaches (Luo et al., ; Sun et al., ) in terms of efficiency and effectiveness.

## 6 Results

Main Results.
Table 2 presents performance results of different KGQA methods. Gnn-Rag is the method that performs overall the best, achieving state-of-the-art results on the two KGQA benchmarks in almost all metrics. The results show that equipping LLMs with GNN-based retrieval boosts their reasoning ability significantly (GNN+LLM vs. KG+LLM). Specifically, Gnn-Rag+RA outperforms RoG by 5.0–6.1% points at Hit, while it outperforms or matches ToG+GPT-4 performance, using an LLM with only 7B parameters and much fewer LLM calls – we estimate ToG+GPT-4 has an overall cost above $800, while Gnn-Rag can be deployed on a single 24GB GPU. Gnn-Rag+RA outperforms ToG+ChatGPT by up to 14.5% points at Hit and the best performing GNN by 5.3–9.5% points at Hits@1 and by 0.7–10.7% points at F1.

**Table 3: Performance analysis (F1) on multi-hop (hops$\geq 2$) and multi-entity (entities$\geq 2$) questions.**
| Method | WebQSP | CWQ |  |  |
| --- | --- | --- | --- | --- |
| multi-hop | multi-entity | multi-hop | multi-entity |  |
| LLM (No RAG) | 48.4 | 61.5 | 33.7 | 32.3 |
| RoG | 63.3 | 65.1 | 59.3 | 58.3 |
| Gnn-Rag | 69.8 | 82.3 | 68.2 | 64.8 |
| Gnn-Rag+RA | 71.1 | 88.8 | 69.3 | 65.6 |

Multi-Hop & Multi-Entity KGQA. Table 3 compares performance results on multi-hop questions, where answers are more than one hop away from the question entities, and multi-entity questions, which have more than one question entities. Gnn-Rag leverages GNNs to handle complex graph information and outperforms RoG (LLM-based retrieval) by 6.5–17.2% points at F1 on WebQSP and by 8.5–8.9% points at F1 on CWQ. In addition, Gnn-Rag+RA offers an additional improvement by up to 6.5% points at F1. The results show that Gnn-Rag is an effective retrieval method when deep graph search is important for successful KGQA.

**Table 4: Performance comparison (F1 at KGQA) of different retrieval augmentations (Section 4.4). ‘#LLM Calls’ are controlled by the hyperparameter $k$ (number of beams) during beam-search decoding for LLM-based retrievers, ‘#Input Tokens’ denotes the median number of tokens.**
| Retriever | KGQA Model | Input/Graph Statistics | KGQA Performance |  |
| --- | --- | --- | --- | --- |
| #LLM Calls | #Input Tokens | F1 (%) |  |  |
|  | WebQSP / CWQ | WebQSP / CWQ |  |  |
| a) Dense Subgraph | (i)  GNN + SBERT (Eq. 3) | 0 | – | 70.9 / 47.8 |
| b) Dense Subgraph | (ii) GNN + $\text{LM}_{\text{SR}}$ (Eq. 3) | 0 | 72.8 / 49.1 |  |
| c) None | LLaMA2-Chat-7B (tuned) | 0 | 59 / 70 | 49.7 / 33.8 |
| d) (iii) RoG (LLM-based; Eq. 2) | 3 | 202 / 325 | 70.8 / 56.2 |  |
| e) Gnn-Rag (default): (i) | 0 | 144 / 207 | 71.3 / 59.4 |  |
| f) Gnn-Rag: (ii) | 0 | 124 / 206 | 71.5 / 58.9 |  |
| g) Gnn-Rag+Ensemble: (i) + (ii) | LLaMA2-Chat-7B (tuned) | 0 | 156 / 281 | 71.7 / 57.5 |
| h) Gnn-Rag+RA (default): (i) + (iii) | 3 | 299 / 540 | 73.5 / 60.4 |  |
| i) Gnn-Rag+RA: (ii) + (iii) | 3 | 267 / 532 | 73.4 / 61.0 |  |
| j) Gnn-Rag+All: (i) + (ii) + (iii) | 3 | 330 / 668 | 72.3 / 59.1 |  |

Retrieval Augmentation. Table 4 compares different retrieval augmentations for Gnn-Rag. The primary metric is F1, while the other metrics assess how well the methods retrieve relevant information from the KG. Based on the results, we make the following conclusions:

- 1.
GNN-based retrieval is more efficient (#LLM Calls, #Input Tokens) and effective (F1) than LLM-based retrieval, especially for complex questions (CWQ); see rows (e-f) vs. row (d).
- 2.
Retrieval augmentation works the best (F1) when combining GNN-induced reasoning paths with LLM-induced reasoning paths as they fetch non-overlapping KG information (increased #Input Tokens) that improves retrieval for KGQA; see rows (h) & (i).
- 3.
Augmenting all retrieval approaches does not necessarily cause improved performance (F1) as the long input (#Input Tokens) may confuse the LLM; see rows (g/j) vs. rows (e/h).
- 4.
Although the two GNNs perform differently at KGQA (F1), they both improve RAG with LLMs; see rows (a-b) vs. rows (e-f). We note though that weak GNNs are not effective retrievers (see Appendix D.2).

**Table 5: Retrieval effect on performance (% Hit) using various LLMs.**
| Method | WebQSP | CWQ |
| --- | --- | --- |
| ChatGPT | 51.8 | 39.9 |
| + ToG | 76.2 | 58.9 |
| + RoG | 81.5 | 52.7 |
| + Gnn-Rag (+RA) | 85.3 (87.9) | 64.1 (65.4) |
| Alpaca-7B | 51.8 | 27.4 |
| + RoG | 73.6 | 44.0 |
| + Gnn-Rag (+RA) | 76.2 (76.5) | 54.5 (50.8) |
| LLaMA2-Chat-7B | 64.4 | 34.6 |
| + RoG | 84.8 | 56.4 |
| + Gnn-Rag (+RA) | 85.2 (88.5) | 62.7 (62.9) |
| LLaMA2-Chat-70B | 57.4 | 39.1 |
| + ToG | 68.9 | 57.6 |
| Flan-T5-xl | 31.0 | 14.7 |
| + RoG | 67.9 | 37.8 |
| + Gnn-Rag (+RA) | 74.5 (72.3) | 51.0 (41.5) |

In addition, Gnn-Rag improves the vanilla LLM by up to 176% at F1 without incurring additional LLM calls; see row (c) vs. row (e). Overall, retrieval augmentation of GNN-induced and LLM-induced paths combines their strengths and achieves the best KGQA performance.

Retrieval Effect on LLMs.
Table 5 presents performance results of various LLMs using Gnn-Rag or LLM-based retrievers (RoG and ToG). We report the Hit metric as it is difficult to extract the number of answers from LLM’s output. Gnn-Rag (+RA) is the retrieval approach that achieves the largest improvements for RAG. For instance, Gnn-Rag+RA improves ChatGPT by up to 6.5% points at Hit over RoG and ToG. Moreover, Gnn-Rag substantially improves the KGQA performance of weaker LLMs, such as Alpaca-7B and Flan-T5-xl. The improvement over RoG is up to 13.2% points at Hit, while Gnn-Rag outperforms LLaMA2-Chat-70B+ToG using a lightweight 7B LLaMA2 model. The results demonstrate that Gnn-Rag can be integrated with other LLMs to improve their KGQA reasoning without retraining.

Case Studies on Faithfulness. Figure 4 illustrates two case studies from the CWQ dataset, showing how Gnn-Rag improves LLM’s faithfulness, i.e., how well the LLM follows the question’s instructions and uses the right information from the KG. In both cases, Gnn-Rag retrieves multi-hop information, which is necessary for answering the questions correctly. In the first case, Gnn-Rag retrieves both crucial facts <Gilfoyle $\rightarrow$ characters_that_have_lived_here $\rightarrow$ Toronto> and <Toronto $\rightarrow$ province.capital $\rightarrow$ Ontario> that are required to answer the question, unlike the KG-RAG baseline (RoG) that fetches only the first fact. In the second case, the KG-RAG baseline incorrectly retrieves information about <Erin Brockovich $\rightarrow$ person> and not <Erin Brockovich $\rightarrow$ film_character> that the question refers to. Gnn-Rag uses GNNs to explore how <Erin Brockovich> and <Michael Renault Mageau> entities are related in the KG, resulting into retrieving facts about <Erin Brockovich $\rightarrow$ film_character>. The retrieved facts include important information <films_with_this_crew_job $\rightarrow$ Consultant>.

Figure 5 illustrates one case study from the WebQSP dataset, showing how RA (Section 4.4) improves Gnn-Rag. Initially, the GNN does not retrieve helpful information due to its limitation to understand natural language, i.e., that <jurisdiction.bodies> usually “make the laws”. Gnn-Rag+RA retrieves the right information, helping the LLM answer the question correctly.

Further ablation studies are provided in Appendix D. Limitations are discussed in Appendix E.

Figure: Figure 4: Two case studies that illustrate how Gnn-Rag improves the LLM’s faithfulness. In both cases, Gnn-Rag retrieves *multi-hop* information that is necessary for answering the complex questions.
Refer to caption: x3.png

Figure: Figure 5: One case study that illustrates the benefit of retrieval augmentation (RA). RA uses LLMs to fetch semantically relevant KG information, which may have been missed by the GNN.
Refer to caption: x4.png

## 7 Conclusion

We introduce Gnn-Rag, a novel method for combining the reasoning abilities of LLMs and
GNNs for RAG-based KGQA. Our contributions are the following. (1) Framework: Gnn-Rag repurposes GNNs for KGQA retrieval to enhance the reasoning
abilities of LLMs. Moreover, our retrieval analysis guides the design of a
retrieval augmentation technique to boost Gnn-Rag performance. (2) Effectiveness & Faithfulness: Gnn-Rag achieves state-of-the-art performance in two
widely used KGQA benchmarks (WebQSP and CWQ). Furthermore, Gnn-Rag is shown to retrieve multi-hop information that is necessary for faithful LLM reasoning on complex questions. (3) Efficiency: Gnn-Rag improves vanilla LLMs on KGQA performance without incurring additional LLM calls as existing RAG systems for KGQA require. In addition, Gnn-Rag outperforms or matches GPT-4 performance with a 7B tuned LLM.

## Appendix A Analysis

**Table 6: Efficiency vs. effectiveness trade-off of LLM-based retrieval.**
| Retrieval | #LLM Calls | Answer Hit (%) |
| --- | --- | --- |
| (efficiency) | (effectiveness) |  |
| RoG [Luo et al., ] | 3 | 85.7 |
| 1 | 77.2 |  |
| ToG [Sun et al., ] | up to 21 | 76.2 |
| 3 | 66.3 |  |
| Gnn-Rag | 0 | 87.2 |

In this section, we analyze the reasoning and retrieval abilities of GNN and LLMs, respectively.

###### Definition A.1 (Ground-truth Subgraph) .

Given a question $q$, we define its ground-truth reasoning subgraph ${\mathcal{G}}_{q}^{\ast}$ as the union of the ground-truth reasoning paths that lead to the correct answers $\{a\}$. Reasoning paths are defined as the KG paths that reach the answer nodes, starting from the question entities $\{e\}$, e.g., <Jamaica $\rightarrow$ language_spoken $\rightarrow$ English> for question “Which language do Jamaican people speak?”. In essence, ${\mathcal{G}}_{q}^{\ast}$ contains only the necessary entities and relations that are needed to answer $q$.

###### Definition A.2 (Effective Reasoning) .

We define that a model $M$ *reasons effectively* if its output is $\{a\}=M({\mathcal{G}}_{q}^{\ast},q)$, i.e., the model returns the correct answers given the ground-truth subgraph ${\mathcal{G}}_{q}^{\ast}$.

As KGQA methods do not use the ground-truth subgraph ${\mathcal{G}}_{q}^{\ast}$ for reasoning, but the retrieved subgraph ${\mathcal{G}}_{q}$, we identify two cases in which the reasoning model *cannot* reason effectively, i.e., $\{a\}\neq M({\mathcal{G}}_{q},q)$.

Case 1: ${\mathcal{G}}_{q}\subset{\mathcal{G}}_{q}^{\ast}$, i.e., the retrieved subgraph ${\mathcal{G}}_{q}$ does not contain all the necessary information for answering $q$. An application of this case is when we use LLMs for retrieval. As LLMs are not designed to handle complex graph information, the retrieved subgraph ${\mathcal{G}}_{q}$ may contain incomplete KG information. Existing LLM-based methods rely on employing an increased number of LLM calls (beam search decoding) to fetch diverse reasoning paths that approximate ${\mathcal{G}}_{q}^{\ast}$. Table 6 provides experimental evidence that shows how LLM-based retrieval trades computational efficiency for effectiveness. *In particular, when we switch from beam-search decoding to greedy decoding for faster LLM retrieval, the KGQA performance drops by 8.3–9.9% points at answer hit*.

Case 2: ${\mathcal{G}}_{q}^{\ast}\subset{\mathcal{G}}_{q}$ and model $M$ cannot “filter-out” irrelevant facts during reasoning. An application of this case is when we use GNNs for reasoning. GNNs cannot understand the textual semantics of KGs and natural questions the same way as LLMs do, and they reason ineffectively if they cannot tell the irrelevant KG information. We develop the following Theorem that supports this case for GNNs.

###### Theorem A.3 (Simplified) .

Under mild assumptions and due to the sum operator of GNNs in Equation 1, a GNN can reason effectively by selecting question-relevant facts and filtering-out question-irrelevant facts through $\omega(q,r)$.

We provide the full theorem and its proof in Appendix B. Theorem A.3 suggests that GNNs need to perform semantic matching via function
$\omega(q,r)$
apart from leveraging the graph information encoded in the KG. Our analysis suggests that GNNs lack reasoning abilities for KGQA if they cannot perform effective semantic matching between the KG and the question.

###### Definition A.1 (Ground-truth Subgraph) .

###### Definition A.2 (Effective Reasoning) .

###### Theorem A.3 (Simplified) .

## Appendix B Full Theorem & Proof

To analyze under which conditions GNN perform well for KGQA, we use the ground-truth subgraph $\mathcal{G}_{q}^{\ast}$ for a question $q$, as defined in Definition A.1. We compare the output representations of a GNN over the ground-truth $\mathcal{G}_{q}^{\ast}$ and another $\mathcal{G}_{q}$ to measure how close the two outputs are.

We always assume ${\mathcal{G}}_{q}^{\ast}\subseteq{\mathcal{G}}_{q}$ for a question $q$. 1-hop facts that contain $v$ are denoted as ${\mathcal{N}}_{v}^{\ast}$.

###### Definition B.1 .

Let $M$ be a GNN model for answering question $q$ over a KG $\mathcal{G}_{q}$, where the output is computed by $M(q,{\mathcal{G}}_{q})$. $M$ consists of $L$ reasoning steps (GNN layers). We assume $M$ is an effective reasoner, according to Definition A.2. Furthermore, we define the reasoning process ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}$ as the sequence of the derived node representations at each step $l$, i.e.,

$$ ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}=\Big\{\{{\bm{h}}_{v}^{(1)}:v\in{\mathcal{G}}_{q}\},\dots,\{{\bm{h}}_{v}^{(L)}:v\in{\mathcal{G}}_{q}\}\Big\}.$ (4) $$

We also define the optimal reasoning process for answering question $q$ with GNN $M$ as ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}^{\ast}}$.
We assume that zero node representations do not contribute in Equation 4.

###### Lemma B.2 .

If two subgraphs ${\mathcal{G}}_{1}$ and ${\mathcal{G}}_{2}$ have the same nodes, and a GNN outputs the same node representations for all nodes $v\in{\mathcal{G}}_{1}$ and $v\in{\mathcal{G}}_{2}$ at each step $l$, then the reasoning processes ${\mathcal{R}}_{M,q,{\mathcal{G}}_{1}}$ and ${\mathcal{R}}_{M,q,{\mathcal{G}}_{2}}$ are identical.

This is true as ${\bm{h}}_{v}^{(l)}$ with $l=1,\dots,L$ for both ${\mathcal{G}}_{1}$ and ${\mathcal{G}}_{2}$ and by using Definition B.1 to show ${\mathcal{R}}_{M,q,{\mathcal{G}}_{1}}={\mathcal{R}}_{M,q,{\mathcal{G}}_{2}}$. Note that Lemma B.2 does not make any assumptions about the actual edges of ${\mathcal{G}}_{1}$ and ${\mathcal{G}}_{2}$.

To analyze the importance of semantic matching for GNNs, we consider the following GNN update

$$ ${\bm{h}}_{v}^{(l)}=\psi\Big({\bm{h}}_{v}^{(l-1)},\sum_{v^{\prime}\in\mathcal{N}_{v}}\omega(q,r)\cdot{\bm{m}}_{vv^{\prime}}^{(l)}\Big).$ (5) $$

where $\omega(\cdot,\cdot):{\mathbb{R}}^{d}\times{\mathbb{R}}^{d}\xrightarrow[]{}\{0,1\}$ is a binary function that decides if fact $(v,r,v^{\prime})$ is relevant to question $q$ or not. Neighbor messages ${\bm{m}}_{vv^{\prime}}^{(l)}$ are aggregated by a sum-operator, which is typically employed in GNNs. Function $\psi(\cdot)$ combines representations among consecutive GNN layers. We assume ${\bm{h}}_{v}^{(0)}\in{\mathbb{R}}^{d}$ and that $\psi\Big(h_{v}^{(0)},0^{d}\Big)=0^{d}$

###### Theorem B.3 .

If $\omega(q,r)=0,\forall(v,r,v^{\prime})\notin{\mathcal{G}}_{q}^{\ast}$ and $\omega(q,r)=1,\forall(v,r,v^{\prime})\in{\mathcal{G}}_{q}^{\ast}$, then ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}$ is an optimal reasoning process of GNN $M$ for answering $q$.

###### Proof.

We show that

$$ $\sum_{v^{\prime}\in\mathcal{N}_{v}}\omega(q,r)\cdot{\bm{m}}_{vv^{\prime}}^{(l)}=\sum_{v^{\prime}\in{\mathcal{N}}^{\ast}_{v}}{\bm{m}}_{vv^{\prime}}^{(l)},$ (6) $$

which gives that ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}={\mathcal{R}}_{M,q,{\mathcal{G}}_{q}^{\ast}}$ via Lemma B.2.
This is true if

$$ $\omega(q,r)=\begin{cases}1&\text{if $(v,r,v^{\prime})\in{\mathcal{N}}^{\ast}_{v}$,}\\ 0&\text{if $(v,r,v^{\prime})\notin{\mathcal{N}}^{\ast}_{v}$,}\end{cases}$ (7) $$

which means that GNNs need to filter-out question irrelevant facts.
We consider two cases.

Case 1. Let $u$ denote a node that is present in ${\mathcal{G}}_{q}$, but not in ${\mathcal{G}}^{\ast}_{q}$. Then, all facts that contain $u$ are not present in ${\mathcal{G}}^{\ast}_{q}$. Condition $\omega(q,r)=0,\forall(v,r,v^{\prime})\notin{\mathcal{G}}_{q}^{\ast}$ of Theorem B.3 gives that

$$ $\displaystyle\omega(q,r)$ $\displaystyle=0,\forall(u,r,v^{\prime}),\text{ and }$ $\displaystyle\omega(q,r)$ $\displaystyle=0,\forall(v,r,u).$ (8) $$

as node $u\notin{\mathcal{G}}_{q}^{\ast}$.
This gives

$$ $\sum_{v^{\prime}\in\mathcal{N}(u)}\omega(q,r)\cdot\;{\bm{m}}_{uv^{\prime}}^{(l)}=0,$ (9) $$

as no edges will contribute to the GNN update. With $\psi\Big(h_{v}^{(0)},0^{d}\Big)=0^{d}$, we have

$$ $h_{u}^{(l)}=0^{d},\forall u\notin{\mathcal{G}}^{\ast}_{q}\text{ with }l=\{1,\dots,L\},$ (10) $$

which means that nodes $u\notin{\mathcal{G}}^{\ast}_{q}$ do not contribute to the reasoning process ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}$; see Definition B.1.

Case 2. Let $p$ denote a relation between two nodes $v$ and $v^{\prime}$ that is present in ${\mathcal{G}}_{q}$, but not in ${\mathcal{G}}^{\ast}_{q}$. We decompose the GNN update to

$$ $\sum_{v^{\prime}\in\mathcal{N}_{r}(v)}\omega(q,r)\cdot\;{\bm{m}}_{vv^{\prime}}^{(l)}+\sum_{v^{\prime}\in\mathcal{N}_{p}(v)}\omega(q,p)\cdot\;{\bm{m}}_{vv^{\prime}}^{(l)},$ (11) $$

where the first term includes facts $\mathcal{N}_{r}$ that are present in ${\mathcal{G}}_{q}^{\ast}$ and the second term includes facts $\mathcal{N}_{p}$ that are present in ${\mathcal{G}}_{q}$ only. Using the condition $\omega(q,r)=0,\forall(v,r,v^{\prime})\notin{\mathcal{G}}_{q}^{\ast}$ of Theorem B.3, we have

$$ $\sum_{v^{\prime}\in\mathcal{N}_{p}(v)}\omega(q,p)\cdot\;{\bm{m}}_{vv^{\prime}}^{(l)}=0.$ (12) $$

Using condition $\omega(q,r)=1,\forall(v,r,v^{\prime})\in{\mathcal{G}}_{q}^{\ast}$, we have

$$ $\sum_{v^{\prime}\in\mathcal{N}_{r}(v)}\omega(q,r)\cdot\;{\bm{m}}_{vv^{\prime}}^{(l)}=\sum_{v^{\prime}\in\mathcal{N}_{r}(v)}{\bm{m}}_{vv^{\prime}}^{(l)}.$ (13) $$

Combining the two above expression gives

$$ $\sum_{v^{\prime}\in\mathcal{N}_{v}}\omega(q,r)\cdot{\bm{m}}_{vv^{\prime}}^{(l)}=\sum_{v^{\prime}\in\mathcal{N}_{r}(v)}{\bm{m}}_{vv^{\prime}}^{(l)}=\sum_{v^{\prime}\in{\mathcal{N}}^{\ast}_{v}}{\bm{m}}_{vv^{\prime}}^{(l)}.$ (14) $$

It is straightforward to obtain ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}={\mathcal{R}}_{M,q,{\mathcal{G}}_{q}^{\ast}}$ via Lemma B.2 in this case.

Putting it altogether. Combining Case 1 and Case 2,nodes $u\notin{\mathcal{G}}_{q}^{\ast}$ do not contribute to ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}$, while for other nodes we have ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}={\mathcal{R}}_{M,q,{\mathcal{G}}_{q}^{\ast}}$. Thus, overall we have ${\mathcal{R}}_{M,q,{\mathcal{G}}_{q}}={\mathcal{R}}_{M,q,{\mathcal{G}}_{q}^{\ast}}$.
∎

###### Definition B.1 .

###### Lemma B.2 .

###### Theorem B.3 .

###### Proof.

## Appendix C Experimental Setup

KGQA Datasets.
We experiment with two widely used KGQA benchmarks: WebQuestionsSP (WebQSP) Yih et al. [],
Complex WebQuestions 1.1 (CWQ) Talmor and Berant [].
We also experiment with MetaQA-3 Zhang et al. [] dataset.
We provide the dataset statistics Table 7.
WebQSP contains 4,737 natural language questions that are answerable using a subset Freebase KG [Bollacker et al., ]. This KG contains 164.6 million facts and 24.9 million entities. The questions require up to 2-hop reasoning within this KG. Specifically, the model needs to aggregate over two KG facts for 30% of the questions, to reason over constraints for 7% of the questions, and to use a single KG fact for the rest of the questions.
CWQ is generated from WebQSP by extending the question entities or
adding constraints to answers, in order to construct more complex multi-hop questions (34,689 in total). There are
four types of questions: composition (45%), conjunction (45%), comparative (5%), and superlative
(5%). The questions require up to 4-hops of reasoning over the KG, which is the same KG as in WebQSP. MetaQA-3 consists of more than 100k 3-hop questions in the domain of movies. The questions were constructed using the KG provided by the WikiMovies Miller et al. [] dataset, with about 43k entities and 135k triples. For MetaQA-3, we use 1,000 (1%) of the training questions.

**Table 7: Datasets statistics. “avg.$|{\mathcal{V}}_{q}|$” denotes average number of entities in subgraph, and “coverage” denotes the ratio of at least one answer in subgraph.**
| Datasets | Train | Dev | Test | avg. $|{\mathcal{V}}_{q}|$ | coverage (%) |
| --- | --- | --- | --- | --- | --- |
| WebQSP | 2,848 | 250 | 1,639 | 1,429.8 | 94.9 |
| CWQ | 27,639 | 3,519 | 3,531 | 1,305.8 | 79.3 |
| MetaQA-3 | 114,196 | 14,274 | 14,274 | 497.9 | 99.0 |

Implementation.
For subgraph retrieval, we use the linked entities to the KG provided by Yih et al. [] for WebQSP, by Talmor and Berant [] for CWQ. We obtain dense subgraphs by  He et al. []. It runs the PageRank Nibble Andersen et al. [] (PRN) method starting from the linked entities to select the top-$m$ ($m=2,000$) entities to be included in the subgraph.

We employ ReaRev(^1^11https://github.com/cmavro/ReaRev_KGQA) [Mavromatis and Karypis, ] for GNN reasoning (Section 4.1) and RoG(^2^22https://github.com/RManLuo/reasoning-on-graphs) [Luo et al., ] for RAG-based prompt tuning (Section 4.2), following their official implementation codes.
In addition, we empower ReaRev with $\text{LM}_{\text{SR}}$ (Section 4.1), which is obtained by following the implementation of SR(^3^33https://github.com/RUCKBReasoning/SubgraphRetrievalKBQA) [Zhang et al., ]. For both training and inference of these methods, we use their suggested hyperparameters, without performing further hyperparameter search. Model selection is performed based on the validation data. Experiments with GNNs were performed on a Nvidia Geforce
RTX-3090 GPU over 128GB RAM
machine. Experiments with LLMs were performed on 4 A100 GPUs connected via NVLink and 512 GB of memory. The experiments are implemented with PyTorch.

For LLM prompting during retrieval (Section 4.4), we use the following prompt:

Please generate a valid relation path that can be helpful for answering the following question:
{Question}

For LLM prompting during reasoning (Section 4.2), we use the following prompt:

Based on the reasoning paths, please answer the given question. Please keep the answer as simple as possible and return all the possible answers as a list.\n
Reasoning Paths: {Reasoning Paths} \n
Question: {Question}

During GNN inference, each node in the subgraph is assigned a probability of being the correct answer, which is normalized via $\mathrm{softmax}$. To retrieve answer candidates, we sort the nodes based on the their probability scores, and select the top nodes whose cumulative probability score is below a threshold. We set the threshold to 0.95. To retrieve the shortest paths between the question entities and answer candidates for RAG, we use the NetworkX library(^4^44https://networkx.org/).

Competing Approaches.

We evaluate the following categories of methods: 1. Embedding, 2. GNN, 3. LLM, 4. KG+LMM, and 5. GNN+LLM.

- 1.
KV-Mem Miller et al. [] is a key-value memory network for KGQA. EmbedKGQA Saxena et al. [] utilizes KG pre-trained embeddings Trouillon et al. [] to improve multi-hop reasoning. TransferNet Shi et al. [] improves multi-hop reasoning over the relation set. Rigel Sen et al. [] improves reasoning with questions of multiple entities.
- 2.
GraftNet Sun et al. [] uses a convolution-based GNN Kipf and Welling []. PullNet Sun et al. [] is built on top of GraftNet, but learns which nodes to retrieve via selecting shortest paths to the answers. NSM He et al. [] is the adaptation of GNNs for KGQA. NSM+h He et al. [] improves NSM for multi-hop reasoning. SQALER Atzeni et al. [] learns which relations (facts) to retrieve during KGQA for GNN reasoning. Similarly, SR+NSM [Zhang et al., ] proposes a relation-path retrieval. UniKGQA [Jiang et al., ] unifies the graph retrieval and reasoning process with a single LM. ReaRev [Mavromatis and Karypis, ] explores diverse reasoning paths in a multi-stage manner.
- 3.
We experiment with instruction-tuned LLMs. Flan-T5 [Chung et al., ] is based on T5, while Aplaca [Taori et al., ] and LLaMA2-Chat [Touvron et al., ] are based on LLaMA. ChatGPT(^5^55https://openai.com/blog/chatgpt) is a powerful closed-source LLM that excels in many complex tasks. ChatGPT+CoT uses the chain-of-thought [Wei et al., ] prompt to improve the ChatGPT. We access ChatGPT ‘gpt-3.5-turbo’ through its API (as of May 2024).
- 4.
KD-CoT [Wang et al., ] enhances CoT prompting for LLMs with relevant knowledge from KGs. StructGPT [Jiang et al., ] retrieves KG facts for RAG. KB-BINDER [Li et al., ] enhances LLM reasoning by generating logical forms of the questions. ToG [Sun et al., ] uses a powerful LLM to select relevant facts hop-by-hop. RoG [Luo et al., ] uses the LLM to generate relation paths for better planning.
- 5.
G-Retriever [He et al., ] augments LLMs with GNN-based prompt tuning.

## Appendix D Additional Experimental Results

### D.1 Question Analysis

**Table 8: Performance analysis (F1) based on the number of maximum hops that connect question entities to answer entities.**
| Method | WebQSP | CWQ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| 1 hop | 2 hop | $\geq$3 hop | 1 hop | 2 hop | $\geq$3 hop |  |
| RoG | 73.4 | 63.3 | – | 50.4 | 60.7 | 40.0 |
| Gnn-Rag | 72.0 | 69.8 | – | 47.4 | 69.4 | 51.8 |
| Gnn-Rag +RA | 74.6 | 71.1 | – | 48.2 | 70.9 | 47.7 |

Following the case studies presented in Figure 4 and Figure 5, we provide numerical results on how Gnn-Rag improves multi-hop question answering and how retrieval augmentation (RA) enhances simple hop questions. Table 8 summarizes these results. Gnn-Rag improves performance on multi-hop questions ($\geq$2 hops) by 6.5–11.8% F1 points over RoG. Furthermore, RA improves performance on single-hop questions by 0.8–2.6% F1 points over Gnn-Rag.

**Table 9: Performance analysis (F1) based on the number of answers (#Ans).**
| Method | WebQSP | CWQ |  |  |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| #Ans=1 | 2$\leq$#Ans$\leq$4 | 5$\leq$#Ans$\leq$9 | #Ans$\geq$10 | #Ans=1 | 2$\leq$#Ans$\leq$4 | 5$\leq$#Ans$\leq$9 | #Ans$\geq$10 |  |
| RoG | 67.89 | 79.39 | 75.04 | 58.33 | 56.90 | 53.73 | 58.36 | 43.62 |
| Gnn-Rag | 71.24 | 76.30 | 74.06 | 56.28 | 60.40 | 55.52 | 61.49 | 50.08 |
| Gnn-Rag +RA | 71.16 | 82.31 | 77.78 | 57.71 | 62.09 | 56.47 | 62.87 | 50.33 |

Table 9 presents results with respect to the number of correct answers. As shown, RA enhances Gnn-Rag in almost all cases as it can fetch correct answers that might have been missed by the GNN.

### D.2 GNN Effect

**Table 10: Performance comparison of different GNN models at complex KGQA (CWQ).**
| Retriever | KGQA Model | CWQ |  |  |
| --- | --- | --- | --- | --- |
| Hit^∗ | H@1 | F1 |  |  |
| Dense Subgraph | GraftNet | – | 45.3 | 35.8 |
| Dense Subgraph | NSM | – | 47.9 | 42.0 |
| Dense Subgraph | ReaRev | – | 52.7 | 49.1 |
| RoG | LLaMA2-Chat-7B (tuned) | 62.6 | 57.8 | 56.2 |
| Gnn-Rag: GraftNet | 58.2 | 51.9 | 49.4 |  |
| Gnn-Rag: NSM | 58.5 | 52.5 | 50.1 |  |
| Gnn-Rag: ReaRev | 66.8 | 61.7 | 59.4 |  |

**Table 11: Results on MetaQA-3 dataset.**
| Method | MetaQA-3 |
| --- | --- |
|  | Hit@1 |
| RoG | 84.8 |
| RoG+pretraining | 88.9 |
| Gnn-Rag | 98.6 |

Gnn-Rag employs ReaRev [Mavromatis and Karypis, ] as its GNN retriever, which is a powerful GNN for deep KG reasoning. In this section, we ablate on the impact of the GNN used for retrieval, i.e., how strong and weak GNNs affect KGQA performance. We experiment with GraftNet [Sun et al., ] and NSM [He et al., ] GNNs, which are less powerful than ReaRev at KGQA.
The results are presented in Table 10. As shown, strong GNNs (ReaRev) are required in order to improve RAG at KGQA. Retrieval with weak GNNs (NSM and GraftNet) underperfoms retrieval with ReaRev by 9.2–9.8% and retrieval with RoG by 5.3–5.9% points at H@1.

### D.3 MetaQA-3

Table 11 presents results on the MetaQA-3 dataset, which requires 3-hop reasoning. RoG is a LLM-based retrieval which cannot handle the multi-hop KG information effectively and as a result, RoG underperforms (even when using additional pretraining data from WebQSP & CWQ datasets). On the other hand, Gnn-Rag relies on GNNs that are able to retrieve useful graph information for multi-hop questions and achieves an almost perfect performance of 98.6% at Hit@1.

### D.4 Retrieval Augmentation

**Table 12: Performance comparison of retrieval augmentation approaches (extended).**
| Retriever | KGQA Model | #LLM Calls | WebQSP | CWQ | Avg. |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| (total) | Hit^∗ | H@1 | F1 | Hit^∗ | H@1 | F1 |  |  |  |
| Dense Subgraph | (i)  ReaRev + SBERT | 0 | – | 76.4 | 70.9 | – | 52.9 | 47.8 | – |
| (ii) ReaRev + $\text{LM}_{\text{SR}}$ | 0 | – | 77.5 | 72.8 | – | 52.7 | 49.1 | – |  |
| None | LLaMA2-Chat-7B (tuned) | 1 | 65.6 | 60.4 | 49.7 | 40.1 | 36.2 | 33.8 | 47.63 |
| (iii) LLM-based | 4 | 85.7 | 80.0 | 70.8 | 62.6 | 57.8 | 56.2 | 68.85 |  |
| Gnn-Rag: (i) | 1 | 85.7 | 80.6 | 71.3 | 66.8 | 61.7 | 59.4 | 70.92 |  |
| Gnn-Rag: (ii) | 1 | 85.0 | 80.3 | 71.5 | 66.2 | 61.3 | 58.9 | 70.50 |  |
| Gnn-Rag: (i) + (ii) | LLaMA2-Chat-7B (tuned) | 1 | 87.2 | 81.0 | 71.7 | 65.5 | 59.5 | 57.5 | 70.40 |
| Gnn-Rag: (i) + (iii) | 4 | 90.7 | 82.8 | 73.5 | 68.7 | 62.8 | 60.4 | 73.15 |  |
| Gnn-Rag: (ii) + (iii) | 4 | 89.9 | 82.4 | 73.4 | 67.9 | 63.0 | 61.0 | 72.93 |  |
| Gnn-Rag: (i) + (ii) + (iii) | 4 | 90.1 | 81.7 | 72.3 | 67.3 | 61.5 | 59.1 | 72.00 |  |
| None | LLaMA2-Chat-7B | 1 | 64.4 | – | – | 34.6 | – | – | – |
| Gnn-Rag: (i) + (ii) | 1 | 86.8 | – | – | 62.9 | – | – | – |  |
| Gnn-Rag: (i) + (iii) | 4 | 88.5 | – | – | 62.1 | – | – | – |  |

Table 12 has the extended results of Table 4, showing performance results on all three metrics (Hit / H@1 / F1) with respect to the retrieval method used. Overall, Gnn-Rag improves the vanilla LLM by 149–182%, when employing the same number of LLM calls for retrieval.

### D.5 Prompt Ablation

When using RAG, LLM performance depends on the prompts used. To ablate on the prompt impact, we experiment with the following prompts:

- •
Prompt A:
Based on the reasoning paths, please answer the given question. Please keep the answer as simple as possible and return all the possible answers as a list.\n
Reasoning Paths: {Reasoning Paths} \n
Question: {Question}
- •
Prompt B:
Based on the provided knowledge, please answer the given question. Please keep the answer as simple as possible and return all the possible answers as a list.\n
Knowledge: {Reasoning Paths} \n
Question: {Question}
- •
Prompt C:
Your tasks is to use the following facts and answer the question.
Make sure that you use the information from the facts provided. Please keep the answer as simple as possible and return all the possible answers as a list.\n
The facts are the following: {Reasoning Paths} \n
Question: {Question}

**Table 13: Performance comparison (%Hit) based on different input prompts.**
|  |  | WebQSP | CWQ |
| --- | --- | --- | --- |
| Prompt A | RoG | 84.8 | 56.4 |
| Gnn-Rag | 86.8 | 62.9 |  |
| Prompt B | RoG | 84.3 | 55.2 |
| Gnn-Rag | 85.2 | 61.7 |  |
| Prompt C | RoG | 81.6 | 51.8 |
| Gnn-Rag | 84.4 | 59.4 |  |

We provide the results based on different input prompts in Table 13. As the results indicate, Gnn-Rag outperforms RoG in all cases, being robust at the prompt selection.

### D.6 Effect of Training Data

**Table 14: Performance results based on different training data.**
| Method | WebQSP | CWQ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- |
| Training Data (Retriever) | Training Data (KGQA Model) | Hit | Training Data (Retriever) | Training Data (KGQA Model) | Hit |  |
| UniKGQA | WebQSP | WebQSP | 77.2 | CWQ | CWQ | 51.2 |
| RoG | WebQSP | WebQSP | 81.5 | CWQ | CWQ | 59.1 |
| WebQSP+CWQ | None | 84.8 | WebQSP+CWQ | None | 56.4 |  |
| WebQSP+CWQ | WebQSP+CWQ | 85.7 | WebQSP+CWQ | WebQSP+CWQ | 62.6 |  |
| Gnn-Rag | WebQSP | None | 86.8 | CWQ | None | 62.9 |
| WebQSP | WebQSP+CWQ | 87.2 | CWQ | WebQSP+CWQ | 66.8 |  |

Table 14 compares performance of different methods based on the training data used for training the retriever and the KGQA model. For example, Gnn-Rag trains a GNN model for retrieval and uses a LLM for KGQA, which can be fine-tuned or not. As the results show, Gnn-Rag outperforms the competing methods (RoG and UniKGQA) by either fine-tuning the KGQA model or not, while it uses the same or less data for training its retriever.

### D.7 Graph Effect

GNNs operate on dense subgraphs, which might include noisy information. A question that arises is whether removing irrelevant information from the subgraph would improve GNN retrieval. We experiment with SR [Zhang et al., ], which learns to prune question-irrelevant facts from the KG. As shown in Table 15, although SR can improve the GNN reasoning results – see row (a) vs. (b) at CWQ –, the retrieval effectiveness deteriorates; rows (c) and (d). After examination, we found that the sparse subgraph may contain disconnected KG parts. In this case, Gnn-Rag’s extraction of the shortest paths fails, and Gnn-Rag returns empty KG information.

**Table 15: Performance comparison on different subgraphs.**
| Retriever | KGQA Model | WebQSP | CWQ |  |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- |
| Hit^∗ | H@1 | F1 | Hit^∗ | H@1 | F1 |  |  |
| a) Dense Subgraph | (A) ReaRev + $\text{LM}_{\text{SR}}$ | – | 77.5 | 72.8 | – | 52.7 | 49.1 |
| b) Sparse Subgraph [Zhang et al., ] | (B) ReaRev + $\text{LM}_{\text{SR}}$ | – | 74.2 | 69.8 | – | 53.3 | 49.7 |
| c) Gnn-Rag: (A) | LLaMA2-Chat-7B (tuned) | 85.0 | 80.3 | 71.5 | 66.2 | 61.3 | 58.9 |
| d) Gnn-Rag: (B) | 83.4 | 78.9 | 69.8 | 60.6 | 55.6 | 53.3 |  |

## Appendix E Limitations

Gnn-Rag assumes that the KG subgraph, on which the GNN reasons, contains answer nodes. However, as the subgraph is decided by tools such as entity linking and neighborhood extraction, errors in these tools, e.g., unlinked entities, can result to subgraphs that do not include any answers. For example, Table 7 shows that CWQ subgraphs contain an answer in 79.3% of the questions. In such cases, Gnn-Rag cannot retrieve the correct answers to help the LLM answer the questions faithfully. In addition, if the KG has disconnected parts, the shortest path extraction algorithm of Gnn-Rag may return empty reasoning paths (see Appendix D.7).

## Appendix F Broader Impacts

Gnn-Rag is a method that grounds the LLM generations for QA using ground-truth facts from the KG. As a result, Gnn-Rag can have positive societal impacts by using KG information to alleviate LLM hallucinations in tasks such as QA.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 5
presentation: 3
questions: 1. Why are many results missing in Table 2? In addition, expanding the experiments to include more datasets would help demonstrate the generalizability of the proposed method.
2. The model involves several hyperparameters (e.g., entity selection threshold and m), yet sensitivity analysis is absent. Including such analysis would improve the robustness of the evaluation.
3. According to the original REAREV method, H@1 on the MetaQA-3 dataset achieves 98.9, whereas performance with the LLM (i.e., GNN+RAG) in Table 3 decreases to 98.6. Could the authors clarify in which cases the integration with the LLM might lead to a performance drop?
soundness: 2
strengths: 1. The paper is clearly presented and easy to follow.
2. The GNN-RAG approach integrates seamlessly with various LLMs.
3. The proposed method shows notable improvements in the KBQA task.
summary: The paper presents GNN-RAG, a framework aimed at improving retrieval-augmented generation for KGQA. GNN-RAG leverages Graph Neural Networks for effective graph-based reasoning and retrieval, supplying candidate reasoning paths as context to the language model. Experimental results show that GNN-RAG, with a tuned 7B LLM, outperforms GPT-4 in performance.
weaknesses: 1. The contribution of the paper feels limited; the approach mainly leverages LLMs to retrieve the correct answers from REAREV, without significant novelty in methodology.
2. Experiments are conducted on only three datasets, with numerous competitor results missing, which limits comprehensive evaluation.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. In GNN-RAG framework, the role of GNN is merely to obtain high-scoring candidate entities. Can the GNN be replaced by any other model, such as a simple but effective embedding-based model? What is the necessity of having a GNN in this framework?

2. When there is more than one shortest path between the candidate entity and the question entity, how should the retrieval result be determined? How can you ensure that the shortest path accurately aligns with the semantic information of the question?

3. Is the setting of 'RA' in the method equivalent to performing an ensemble between GNN-RAG and RoG?

4. Many LLM-based methods typically report their results using the Hits@1 metric, but you place their results under the Hit metric. By doing so, larger performance improvements are observed. Is there sufficient evidence to suggest that this comparison is appropriate?
soundness: 2
strengths: 1. The paper propose to leverage GNN as a retriever to improve RAG in complex KGQA. 
2. GNN-RAG improves LLMs without incurring additional LLM calls, and achieves competitive performance.
3. The authors provide codes for their implementation.
summary: The paper proposes a GNN-RAG framework that combines GNN and LLM for KGQA task. GNN-RAG first leverages SOTA GNN model to retrieve answer candidates for a given question, then the shortest paths that connect question entities and answer candidates are extracted to represent KG reasoning paths for LLM’s final output. Experiments demonstrate that the GNN-RAG with fine-tuned LLaMA2-7B can achieve state-of-the-art performance.
weaknesses: 1. The overall framework lacks novelty and is just like a simple concatenation of existing methods.

2. The role of GNN as a retriever is only to identify candidate entities, rather than to "learn useful graph information" as stated by the author.

3. There are a large number of blank spaces in the experimental result table.

---

confidence: 3
contribution: 3
initial_rating: 8
presentation: 3
questions: 1. This paper does not provide sufficient details on the RAG WITH LLM.
2. The paper misses many related studies such as [1][2][3][4], which could provide a broader context and highlight the novelty and contribution of GNN-RGA more effectively.
[1] ARL: An adaptive reinforcement learning framework for complex question answering over knowledge base. Inf. Process. Manag. 59(3): 102933 (2022)
[2] Query Path Generation via Bidirectional Reasoning for Multihop Question Answering From Knowledge Bases. IEEE Trans. Cogn. Dev. Syst. 15(3): 1183-1195 (2023)
[3] Question-Directed Reasoning With Relation-Aware Graph Attention Network for Complex Question Answering Over Knowledge Graph. IEEE ACM Trans. Audio Speech Lang. Process. 32: 1915-1927 (2024)
soundness: 3
strengths: 1. This is a solid study with reasonable technical contributions. The large number of baseline models used in the experiments is impressive.
2. The paper is generally understandable and clearly explains the technical parts to a certain extent.
3. The figures and charts in the manuscript are exceptionally clear and well-presented.
summary: This paper present GNN-RGA, a novel graph neural method for enhancing RAG in KGQA with GNNs. GNN-RAG achieves state-of-the-art performance in two widely used KGQA benchmarks (WebQSP and CWQ).
weaknesses: 1. This paper does not provide sufficient details on the RAG WITH LLM.
2. The paper misses many related studies such as [1][2][3][4], which could provide a broader context and highlight the novelty and contribution of GNN-RGA more effectively.
[1] ARL: An adaptive reinforcement learning framework for complex question answering over knowledge base. Inf. Process. Manag. 59(3): 102933 (2022)
[2] Query Path Generation via Bidirectional Reasoning for Multihop Question Answering From Knowledge Bases. IEEE Trans. Cogn. Dev. Syst. 15(3): 1183-1195 (2023)
[3] Question-Directed Reasoning With Relation-Aware Graph Attention Network for Complex Question Answering Over Knowledge Graph. IEEE ACM Trans. Audio Speech Lang. Process. 32: 1915-1927 (2024)

---

confidence: 4
contribution: 2
initial_rating: 6
presentation: 4
questions: 1. What is the cost of model training? For example, how much data was used to fine-tune the large language model, and how long did it take to train the KBQA model?

2. How does changing the amount of training data (either fine-tuning data or KBQA model training data) affect the performance of the method?

3. In Table 6, are the results of Alpaca-7b based on the Alpaca-7b model fine-tuned for this task, or are they based on the original Alpaca-7b model? Would fine-tuning a more powerful LLM yield better results?

4. If only RA is used for knowledge retrieval, how would the performance change?

5. For the KGQA task, constructing QA training data versus constructing QA data with SPARQL queries is not significantly different since the groud truth answers need to be validated with SPARQL queries. If SOTA SP-based KGQA methods are used for obtaining candidate entities, would its performance be superior to that of the GNN-based method, especially in cases of 1-hop queries?
soundness: 3
strengths: 1. GNN-RAG uses a GNN-based KGQA algorithm to obtain candidate answers to the question, then samples the shortest path between the topic entity and candidate entities as reasoning evidence, allowing the LLM to generate the final answer. Experimental results demonstrate that the GNN-based KGQA algorithm can effectively retrieve reasoning evidence from the KG, and the fine-tuned LLM can accurately infer results from the evidence.

2. Compared to existing LLM-based KGQA methods, GNN-RAG shows better efficiency and accuracy on two KGQA datasets.

3. The paper provides thorough case studies and theoretical analysis on the strengths and weaknesses of the GNN approach.

4. The structure of the paper is clear and easy to follow.
summary: The paper introduces GNN-RAG, a KGQA (Knowledge Graph Question Answering) algorithm that combines GNN (Graph Neural Networks) with LLM reasoning. This method uses a GNN-based KGQA algorithm to retrieve reasoning evidence from a KG, allowing the LLM to generate the final answer. Experiments show that this approach effectively improves the performance of GNN-based KGQA methods and outperforms other LLM-based KGQA algorithms in terms of accuracy and efficiency.
weaknesses: 1. Using existing KGQA methods to retrieve information from a KG and then allowing a LLM to perform reasoning based on the retrieved information is a rather intuitive idea. In some LLM-based KG completion works, similar approaches have been used, where traditional KG completion models retrieve relevant triples and candidate entities from the KG, and then LLMs use their semantic understanding to perform reasoning [1][2]. This work seems to have simply applied a similar idea to the KGQA task. The retrieval stage follows existing GNN-based KGQA methods without providing sufficient theoretical analysis of the GNN's advantages in multi-hop information retrieval, and the utility of GNN-based KGQA methods is also intuitive since GNN models currently perform the best among existing IR-based KGQA methods. Moreover, the reasoning stage follows the design of RoG [3], so the innovation of this method is rather limited.

2. This approach requires training two models, namely a KBQA model and fine-tuning a LLM. While the approach has an efficiency advantage during inference, it demands a large amount of labeled training data, which is difficult to obtain in real-world scenarios. The paper does not sufficiently discuss the costs of model training, such as how much data is used to fine-tune the LLM, the time required to train the KBQA model, or how changes in training data affect the method's performance.

3. As shown in Table 5, this method is constrained by the GNN-based KGQA model, and when the performance of the GNN-based KGQA model is poor, the overall model's performance declines significantly. This point is also mentioned in the limitations section of the paper.

[1] Wei, Yanbin, et al. "KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion." Findings of the Association for Computational Linguistics: EMNLP 2023. 2023.
[2] Liu, Yang, et al. "Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion." arXiv preprint arXiv:2407.16127 (2024).
[3] LUO, LINHAO, et al. "Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning." The Twelfth International Conference on Learning Representations.

---

confidence: 5
contribution: 2
initial_rating: 6
presentation: 3
questions: 1) How does GNN-RAG initialize the entity / relation embeddings? It would be beneficial to explicitly state whether the framework uses randomly initialized or pre-trained embeddings, or whether these embeddings are optimized with GNN parameters. '
*(This is crucially important especially for the IC"LR" conference.)*

2) How to emphasize k in equation 3? Is that means, given a fixed LM, we can have k different embeddings for a specific question? Or, the question embedding is pooled based on the final layer output of each of the tokens? Please *explicitly* state the structure of the mentioned "attention-based pooling neural network" with equations.

This reviewer sincerely requests the authors to enrich the implementation details in section 4.1. The current submission is insufficient for readers to reproduce the experimental results without checking the source code. Spaces are available to add these details since the paper does not fill 10 pages. 

3) Would you mind adding a small section and/or a table to discuss the time cost of LLM fine-tuning?
soundness: 4
strengths: 1)  Key Original Contribution: This paper proposes to utilize GNN to retrieve and select proper candidate answers for the given question, which effectively alleviates the limitation of LLM-based KGQA methods. Note: The Key contribution belongs to the “Retrieval” part.

2) Compared to LLM-based baseline methods, GNN-RAG is simple and effective. It achieves state-of-the-art performance on WebQSP with Llama2-7B model. The performance on CWQ is slightly lower than ToG with GPT4 as the backbone LLM.

3) The authors of this paper conducted detailed comparisons between RoG and the proposed GNN-RAG, and examined the effectiveness of GNN-RAG, RoG, and ToG on different kinds of LLMs.
summary: This paper proposes a Retrieve-augmented generation (RAG) framework named GNN-RAG for the knowledge graph question answering~(KGQA) task. GNN-RAG first employs a GNN to find plausible candidate answers for the given question, and extract the shortest path between the topic entity and candidate answers as reasoning paths. Then, the framework finetunes and employs an LLM to answer the question based on these candidate entities and related reasoning paths. GNN-RAG achieves the SOTA performance on the WebQSP and the CWQ datasets. The paper also makes direct and detailed comparisons between the proposed method and its competitive baseline RoG.
weaknesses: 1) Although the proposed method demonstrates satisfactory performance, it is “relatively” incremental compared to ToG and RoG. The retrieval of reasoning paths with GNNs is not a brand-new idea in the QA domain [1]. Apart from the method of path retrieval, less essential differences can be found between RoG and GNN-RAG. Nevertheless, this reviewer does not believe that the paper should be rejected based solely on this reason.

2) This paper lacks a detailed explanation of the design of the GNN. Specifically, it could be beneficial to explicitly model the classification step with equation(s). This reviewer does not know how the authors perform the classification with the entity embeddings outputted by the L-th layer. He can only make an educated guess that the classification operation is conducted by an FC layer followed by a softmax / sigmoid layer. 

3) It could be beneficial to take UniKGQA (or ReaRev) and/or ToG into comparison in Table 3 and 4 since Figure 3 mentioned the landscape of three types of existing methods.

4) The paper did not discuss the time cost for LLM fine-tuning and inference. This reviewer is surprised to see that the proposed method, with a 7B model, outperforms baselines with a 70B model or GPT. However, the efficiency of the proposed method cannot be judged solely by the number of LLM calls required during the inference stage. GPU hours for LLM fine-tuning is also an important metric.

[1] Relation-aware Bidirectional Path Reasoning for Commonsense Question Answering (CoNLL
2021)

Meta-review: Summary: The paper proposes GNN-RAG, a framework combining Graph Neural Networks (GNNs) and Retrieval-Augmented Generation (RAG) for Knowledge Graph Question Answering (KGQA). The framework uses GNNs for retrieval of graph-based reasoning paths, which are then provided as context to downstream Large Language Models (LLMs). Experimental studies are conducted on two KGQA benchmarks (WebQSP and CWQ), outperforming or matching GPT-4 performance using a smaller LLM (7B parameters) and retrieval.

Strengths:
- Clear presentation with thorough explanations and illustrative examples
- Detailed ablation studies and analysis of different components

Weakness:
- Lukewarm response from all but one reviewer and the positive reviewer didn't champion the paper
- Limited Novelty: Several reviewers noted that the approach primarily combines existing GNN and RAG techniques, raising concerns about methodological originality.
- Role of GNNs: The GNN primarily retrieves high-scoring candidate entities rather than performing more complex graph reasoning, which some reviewers found underwhelming.
- Missing important baseline: Apart from the works pointed out by reviewer, there are several missing important works, e.g. Das et al "Case-Based Reasoning for Natural Language Queries over Knowledge Bases", 2021. He et al. "Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals", 2022, and inter alia.
- Incomplete empirical comparisons and stronger than substantiated claims: In light of other missing works, it is not clear if GNN-RAG achieves SoTA results

Decision: Given the lack of enthusiasm from the reviewers, incremental novelty, and missing prior works, unfortunately, the paper can't be accepted in its current form and addressing all the concerns would warrant another round of reviewing.

Decision: Reject