Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation
Authors: Shijie Wang, Wenqi Fan, Yue Feng, LIN SHANRU, Xinyu Ma, Shuaiqiang Wang, Dawei Yin
Accepted at: acl25

Knowledge Graph Retrieval-Augmented Generation for LLM-based Recommendation

Shijie Wang1, Wenqi Fan1*, Yue Feng2*, Shanru Lin3,
Xinyu Ma4, Shuaiqiang Wang4, Dawei Yin4,
1The Hong Kong Polytechnic University, 2University of Birmingham,
3City University of Hong Kong, 4Baidu Inc,
shijie.wang@connect.polyu.hk; wenqifan03@gmail.com; y.feng.6@bham.ac.uk; llam32316@gmail.com;
xinyuma2016@gmail.com; shiqiang.wang@gmail.com; yindawei@acm.org

Abstract

Recommender systems have become increasingly vital in our daily lives, helping to alleviate the problem of information overload across various user-oriented online services. The emergence of Large Language Models (LLMs) has yielded remarkable achievements, demonstrating their potential for the development of next-generation recommender systems. Despite these advancements, LLM-based recommender systems face inherent limitations stemming from their LLM backbones, particularly issues of hallucinations and the lack of up-to-date and domain-specific knowledge. Recently, Retrieval-Augmented Generation (RAG) has garnered significant attention for addressing these limitations by leveraging external knowledge sources to enhance the understanding and generation of LLMs. However, vanilla RAG methods often introduce noise and neglect structural relationships in knowledge, limiting their effectiveness in LLM-based recommendations. To address these limitations, we propose to retrieve high-quality and up-to-date structure information from the knowledge graph (KG) to augment recommendations. Specifically, our approach develops a retrieval-augmented framework, termed K-RagRec, that facilitates the recommendation generation process by incorporating structure information from the external KG. Extensive experiments have been conducted to demonstrate the effectiveness of our proposed method.

1 Introduction

Recommender systems, as techniques designed to assist people in making decisions in their daily lives, are increasingly gaining impact in various fields (Kenthapadi et al., 2017; He et al., 2020; Fan et al., 2019), such as online shopping, job matching, and social media. Recently, Large Language Models (LLMs) have achieved significant breakthroughs, which further drive developments in various domains (Fan et al., 2024b; Zhao et al., 2024; Wu et al., 2023a). Especially with the success of LLMs, recommender systems have seen rapid growth (Geng et al., 2022; Bao et al., 2023; Qu et al., 2024). By training on a wide range of data, LLMs (e.g., GPT-4 (Achiam et al., 2023) and LLaMA (Touvron et al., 2023)) are able to acquire extensive knowledge and demonstrate exceptional language understanding capability. This capability enables LLM-based recommender systems to capture user preferences through a nuanced understanding of relevant attributes (e.g., user profiles, item descriptions, historical interactions) for more accurate recommendations. As a result, LLM-based recommender systems have emerged as a new paradigm in recommendation technology (Zhao et al., 2024).

However, despite their powerful language understanding and generalization capability, LLM-based recommender systems face significant challenges, including hallucinations and lack of up-to-date and domain-specific knowledge (Luo et al., 2023). Specifically, one key issue is that LLM-

![Illustration of the issues of hallucinations and lack of domain-specific knowledge in LLM-based recommender systems and how they can be addressed by knowledge graph retrieval-augmented generation (KG RAG).](page_1012_563_388_246.png)

Figure 1: Illustration of the issues of hallucinations and lack of domain-specific knowledge in LLM-based recommender systems and how they can be addressed by knowledge graph retrieval-augmented generation (KG RAG).
based recommender systems may generate recommendations that are entirely fictional due to the inherent limitations of LLMs. For example, an LLM-based recommender system may recommend "Godmother", a non-existent film, to the user who has watched "The Godfather", as illustrated in Figure 1 (a). Additionally, LLMs usually lack up-to-date knowledge, which can prevent them from recommending the latest films or products in a timely manner. As illustrated in Figure 1 (b), the LLM-based recommender system is unable to recommend the latest films due to the training data only containing up to December 2022. Furthermore, LLMs often lack domain-specific knowledge, as recommendation-oriented corpora are very limited during the training phase of LLMs (Geng et al., 2022). Consequently, LLMs may struggle to meet the nuanced needs of recommendation tasks. To alleviate these issues, one potential solution is to frequently fine-tune the LLMs with up-to-date and domain-specific knowledge. However, the massive parameters of LLMs make this process computationally expensive and time-consuming, which severely hinders the practical application in the real world.

More recently, Retrieval-Augmented Generation (RAG) leveraging external databases to provide specific knowledge shows promise to solve these problems (Fan et al., 2024a; Gao et al., 2023). By incorporating an external knowledge base, RAG can retrieve relevant and up-to-date information to complement the LLM’s inherent knowledge, thereby mitigating the issues of hallucinations and knowledge gaps (Khandelwal et al., 2019; Min et al., 2020; Li et al., 2024). This makes RAG a promising technique for enabling LLMs to provide effective recommendations without the need for costly fine-tuning (Ram et al., 2023).

Despite this potential, vanilla RAG methods that rely on documents and paragraphs often introduce unnecessary noise and even harmful disturbance, which can negatively impact the accuracy and reliability of recommendations (He et al., 2024). In addition, the structural relationships between entities are overlooked in typical RAG, resulting in the sub-optimal reasoning capability of LLM-based recommender systems (Luo et al., 2023). To address the limitations, a prospective solution is to incorporate structured knowledge such as items’ knowledge graph (KG) to help improve recommendation performance. Specifically, KGs offer structured, factual, and editable representations of knowledge, which can provide a faithful knowledge source for recommendations. As shown in Figure 1 (c), retrieving structured knowledge from the KG can significantly enhance the recommendation capabilities of LLM-based recommender systems.

However, it is challenging to effectively retrieve KGs to enhance the recommendation capabilities of LLMs. First, KGs store rich factual knowledge in a structured format. Simply retrieving the triplets or first-order neighbors of an entity (i.e., item) with semantic information neglects the importance of these higher-order neighborhood effects among entities/items, resulting in sub-optimal recommendation performance. Second, indiscriminate retrieving for each item, regardless of whether the retrieval is necessary or the content is relevant, can degrade the performance of the recommendation while severely reducing the model’s efficiency (Asai et al., 2023; Labruna et al., 2024). Furthermore, structured data in KGs is typically encoded for LLM in the form of serialized text (Wu et al., 2023b; Sun et al., 2023), which is insufficient for fully exploiting the structural information inherent in the data (Perozzi et al., 2024; Fatemi et al., 2023). Therefore, it is crucial to explore more effective and expressive ways of representing structured data, allowing LLMs to effectively leverage the structure information of retrieved knowledge sub-graphs for recommendations.

To address the aforementioned challenges, in this paper, we propose a knowledge retrieval-augmented recommendation framework, namely K-RagRec, to provide up-to-date and reliable knowledge by retrieving relevant knowledge from item’s KGs for recommendation generation in the era of LLMs. Specifically, our proposed framework first performs knowledge sub-graph indexing on the items’ KG at a coarse and fine granularity to construct the knowledge vector database. Next, a popularity selective retrieval policy is designed to determine which items should be retrieved, followed by the retrieval of specific sub-graphs from the knowledge vector database. To refine the quality of retrieval and ensure the most relevant results are prioritized at the top of the input, we subsequently re-rank the retrieved knowledge sub-graphs. Finally, we introduce a GNN and projector to align the retrieved knowledge sub-graphs into the semantic space of the LLM for knowledge-augmented recommendation. The main contributions of this paper can be summarized as follows:
• We propose a novel framework that retrieves
faithful knowledge from KGs to augment the recommendation capability of LLM. Note that we introduce a flexible indexing method for KGs, which can provide a comprehensive view of a node’s neighborhood within KG.
• We design a popularity selective retrieval strategy to determine whether an item needs to be retrieved based on its popularity, significantly improving efficiency.
• We introduce a more expressive graph encoder for structured data inclusion in LLMs, that can facilitate the LLM to effectively leverage the structure information and avoid long context input.
• We conduct comprehensive experiments on various real-world datasets to evaluate the effectiveness of the proposed K-RagRec framework.

2 Related Work

Recently, RAG has emerged as one of the most representative technologies in the field of generative AI, combining the strengths of retrieval systems and language models (LM) to produce coherent and informative text. Early methods, such as REALM (Guu et al., 2020), RETRO (Borgeaud et al., 2022), and DPR (Karpukhin et al., 2020), typically involve retrieving relevant fragments from a large corpus to guide the LM generation. However, standard RAG methods often struggle to accurately retrieve all relevant textual chunks, due to unnecessary noise and even harmful disturbance in the documents. To address these limitations, recent studies (Baek et al., 2023; Wu et al., 2023b; He et al., 2024; Luo et al., 2023; Wang et al., 2023; Sen et al., 2023; Sun et al., 2023) focus on retrieving structured and faithful knowledge from graphs for enhancing generations. For example, Retrieve-Rewrite-Answer (Wu et al., 2023b) retrieves relevant sub-graphs from KG and converts retrieved sub-graphs into text for the generation. G-Retriever (He et al., 2024) explores retrieving sub-graphs from various types of graphs to alleviate the hallucinations of LLM.

With the explosion of LLMs in recommendations, a few works (Di Palma, 2023; Wu et al., 2024; Ang et al., 2024) make initial explorations in RAG for recommendations. For instance, the work (Di Palma, 2023) proposes leveraging knowledge from movie or book datasets to enhance recommendations. Nevertheless, retrieving faithful structured knowledge from the KGs for recommendations is under-explored and shows great potential.

To fill this gap, we propose to retrieve knowledge sub-graphs from KGs to enhance the recommendation performance of LLM. We provide more comprehensive related work on Appendix A.3.

3 Methodology

In this section, we first introduce some key notations and concepts in this work. Then, we provide the details for each component of our proposed framework K-RagRec.

3.1 Preliminary

Knowledge Graph: In this work, we propose to leverage external knowledge databases (i.e., KGs) to augment LLMs for recommendations. KGs contain abundant factual knowledge in the form of a set of triples: \( \mathcal{G} = \{(n, e, n') \mid n, n' \in N ,\ e \in E\} \), where \( N \) and \( E \) denote the set of entities and relations, respectively. For example, a triple \( Interstellar \xrightarrow{directed\_by} Nolan \) indicates that the movie "Interstellar" was directed by the director "Nolan".

LLM-based Recommendations: Let \( U = \{u_1, u_2, \ldots, u_n\} \) and \( V = \{v_1, v_2, \ldots, v_m\} \) represent the sets of users and items, where \( n \) and \( m \) are the sizes of users and items, respectively. The goal of an LLM-based recommender system is to understand users’ preferences by modeling users’ historical items interactions \( V^{u_i} = [v_1^{u_i}, v_2^{u_i}, \cdots, v_{|L_{u_i}|}^{u_i}] \) (e.g., clicks, bought, etc.), where \( |L_{u_i}| \) is interaction sequence length for user \( u_i \). Notably, item \( v_j \) side information, such as title and description, is publicly available to enhance LLM for modeling user preferences. In our setting, we consider asking the LLM to select user \( u \)'s preferred item \( v \) from the candidate set \( C = \{v_j\}_{j=1}^M \), where \( M \) is the number of candidate items. The candidate set \( C \) typically consists of one positive sample as well as \( M - 1 \) negative samples. Specifically, for a frozen LLM \( f_\delta \) with parameters \( \delta \), we denote an input-output sequence pair as \( (Q, A) \), where \( Q \) is the recommendation query/prompt, which consists of task descriptions and users’ historical items. The output \( A \) is the LLM’s prediction. Furthermore, we introduce the concept of GNNs in appendix A.2.

3.2 The Overview of Proposed Method

As shown in Figure 2, our proposed K-RagRec consists of five crucial components: Hop-Field Knowledge Sub-graphs for Semantic Indexing, Popularity Selective Retrieval Policy, Knowledge Sub-graphs
Retrieval, Knowledge Sub-graphs Re-Ranking, and Knowledge-augmented Recommendation. The model first performs indexing of hop-field knowledge sub-graphs within the KG. Following this, a popularity selective retrieval policy is implemented to determine which items should be retrieved or augmented. The model then retrieves specific sub-graphs from the knowledge vector database. Subsequently, the retrieved knowledge sub-graphs are re-ranked to refine the retrieval quality. Finally, the retrieved knowledge sub-graphs are utilized with the original prompt to generate recommendations.

3.3 Hop-Field Knowledge Sub-graphs for Semantic Indexing

Typically, retrieving knowledge from KG chunks the KG into nodes (He et al., 2024) or triplets (Luo et al., 2023) and only retrieves content locally around the target entity in the KG. However, these methods just naively retrieve the first-order neighbors of an entity (i.e., item), which makes it difficult to capture these higher-order neighborhood effects among entities/items in the recommendation process. Therefore, to effectively retrieve knowledge from the KG, we propose performing semantic indexing on hop-field knowledge sub-graphs, which can flexibly chunk KGs and provide a comprehensive view of a node’s neighborhood in KG. As illustrated in Figure 2 component 1, we first introduce a pre-trained language model (PLM), such as SentenceBert (Reimers, 2019), to capture the semantic information for node \( n_o \) as follows:

\[
z_{n_o} = \mathrm{PLM}(x_{n_o}) \in R^d,
\]

where \( d \) is the dimension of the output representation. Similarly, we also capture the semantic information for edge/relation \( e_o \) in KG:

\[
r_{e_o} = \mathrm{PLM}(x_{e_o}) \in R^d,
\]

where \( x_{n_o} \) and \( x_{e_o} \) are the text attributes (e.g., item’s title and descriptions) of node \( n_o \) and edge/relation \( e_o \), respectively.

To retrieve nuanced knowledge of both coarse and fine graph structures from KG, we introduce a GNN (i.e., GNN\(^{\text{Indexing}}_{\phi_1}\)) with parameters \( \phi_1 \) to aggregate information from neighbors for entities, where the \( l_{th} \)-hop embedding \( z_{n_o}^{(l)} \) of a central entity \( n_o \) can be defined by:

\[
z_{n_o}^{(l)} = \mathrm{GNN}_{\phi_1}^{\text{Indexing}}(\{z_{n_m}^{(l-1)}, r_{e_{<o,m>}}^{(l-1)} : n_m \in \mathcal{N}(n_o)\}),
\]

where \( \mathcal{N}(n_o) \) is the set of neighbours of node \( n_o \), and \( e_{<o,m>} \) is the edge between node \( n_o \) and \( n_m \). For each entity, its \( l \)-hop representation can be seen as a knowledge sub-graph representation containing the \( l \)-hop neighbors of itself. Therefore, we can express the knowledge sub-graph representation of \( g_o \in \mathcal{G} \) as \( z_{g_o} \), where \( \mathcal{G} \) is the set containing all the knowledge sub-graphs. For each sub-graph, we store its representation in a knowledge vector database.

3.4 Popularity Selective Retrieval Policy

Although RAG can augment the LLM for modeling user preferences with retrieved knowledge, retrieving each item can cost a significant amount of retrieval time, which can severely degrade the user experience and cause user churn. Meanwhile, most users’ online behaviours in recommender systems are following the power law distribution (Abdollahpouri et al., 2017; Celma and Herrera, 2008) in which a small proportion (e.g., less than 20%) of items (i.e., popular items) often account for a large proportion of users’ online behaviours (e.g., more than 80%), while cold-start items have a few interactions from users. Therefore, most models tend to keep rich knowledge of popular items, resulting in an inferior performance for cold-start items (Zhao et al., 2023). To this end, we design a popularity selective retrieval policy to determine whether an item needs to be augmented from KG based on its popularity (e.g., sales volume and page view). Particularly, the item is retrieved if its popularity is less than the pre-defined threshold \( p \), otherwise not. By incorporating this strategy, the retrieval time in K-RagRec can be significantly reduced to achieve more efficient retrieval.

3.5 Knowledge Sub-graphs Retrieval

Given the query for knowledge sub-graphs retrieval, we adopt the same PLM as the first indexing step to ensure that the query is in the consistent embedding space as the knowledge sub-graph representation. We define the text attribute (e.g., item’s title and descriptions) \( x_{q_j} \) of the item \( v_j \) that needs to be retrieved as the query and obtain its semantic information \( q_j \) as:

\[
q_j = \mathrm{PLM}(x_{q_j}) \in R^d.
\]

Next, we retrieve the top-\( K \) most similar sub-graphs \( G_j = \{g_1', ..., g_K'\} \) from \( \mathcal{G} \) for item \( v_j \):

\[
G_j = \operatorname{argtopK}_{g_s \in \mathcal{G}} \operatorname{sim}(q_j, z_{g_s}),
\]

where \( \operatorname{sim}(\cdot, \cdot) \) is a similarity metric for measuring the similarity between the query representation \( q_j \) and knowledge sub-graph \( g_s \)'s representation \( z_{g_s} \) in knowledge vector database. Finally, the retrieved
Figure 2: The overview of the K-RagRec. It contains five key components: Hop-Field Knowledge Sub-graphs for Semantic Indexing, Popularity Selective Retrieval Policy, Knowledge Sub-graphs Retrieval, Knowledge Sub-graphs Re-Ranking, and Knowledge-augmented Recommendation.

knowledge sub-graphs for items required to be retrieved from user \( u_i \)'s historical interactions \( V^{u_i} \) will be used to form a knowledge sub-graph set \( G \).

3.6 Knowledge Sub-graphs Re-Ranking

To refine the quality of retrieval for enhancing recommendation performance, the next crucial step is knowledge sub-graphs re-ranking. Feeding all retrieved knowledge sub-graphs \( g' \in G \) directly into LLM can lead to information overload if the user \( u_i \) has a long historical interaction list towards items. Therefore, we execute re-ranking to shorten the retrieved knowledge sub-graphs and ensure the most relevant knowledge sub-graphs are prioritized at the top of the prompt. Specifically, we adopt the recommendation prompt as a query for re-ranking, which consists of task descriptions and users’ historical items. For example, this recommendation prompt can be “What is the top recommended movie for the user who watched [Matrix, ..., Iron Man]?”. For consistency, we adopt the same PLM to capture the semantic information of the above prompt as p, and re-rank the knowledge sub-graphs in \( G \) to obtain a Top-N knowledge sub-graphs set \( \hat{G} \):
\[
\hat{G} = \operatorname{argtopN}_{g'_* \in G} \operatorname{sim}\left(\mathbf{p}, \mathbf{z}_{g'_*}\right).
\]
(6)

3.7 Knowledge-augmented Recommendation

To facilitate the LLM’s better understanding of the structure of retrieved knowledge sub-graphs and to avoid long contexts, we further integrate another GNN encoder \( \mathrm{GNN}^{\text{Encoding}}_{\phi_2} \) with parameter \( \phi_2 \) to enhance the representation learning of structural information:
\[
\mathbf{h}_{g'_*} = \mathrm{GNN}^{\text{Encoding}}_{\phi_2}(\{g'_* : g'_* \in \hat{G}\}).
\]
(7)
An MLP projector \( \mathrm{MLP}_\theta \) with parameter \( \theta \) is further introduced to shift to mapping all sub-graphs embedding in \( \hat{G} \) into the LLM embedding space:
\[
\hat{\mathbf{h}}_{\hat{G}} = \mathrm{MLP}_\theta([\mathbf{h}_{g'_1}; ...; \mathbf{h}_{g'_N}]),
\]
(8)
where \( [:;:] \) represents the concatenation operation. The extracted knowledge sub-graphs embedding \( \hat{\mathbf{h}}_{\hat{G}} \) as the soft prompt is then appended before the input token embedding in LLM.

3.8 Optimization for K-RagRec

The training process can be broadly considered as soft prompt tuning, where the retrieved knowledge sub-graphs are a series of soft graph prompts. Formally, the generation process can be expressed as follows:
\[
p_{\delta,\theta,\phi_1,\phi_2}(Y \mid \hat{G}, x_q) = \prod_{k=1}^r p_{\delta,\theta,\phi_1,\phi_2}(y_k \mid y_{<k}, \hat{\mathbf{h}}_{\hat{G}}, x_q).
\]
(9)
Therefore, instead of fine-tuning the LLM model extensively, we only learn parameters of two GNNs (i.e., \( \mathrm{GNN}^{\text{Indexing}}_{\phi_1}, \mathrm{GNN}^{\text{Encoding}}_{\phi_2} \)) and projector \( \mathrm{MLP}_\theta \), while parameters \( \delta \) of LLM backbone are frozen. We update the parameters \( \phi_1, \phi_2 \) and \( \theta \) through the Cross-Entropy Loss \( \mathcal{L}(Y, A) \), where \( Y \) is the ground-truth and \( A \) is LLM’s prediction.

4 Experiment

In this section, we evaluate the effectiveness of our proposed framework through comprehensive experiments. First, we present the experimental settings,
including details about the datasets, compared baselines, evaluation metrics, and the parameter configurations. Then, we report the main experimental results, highlighting the performance of the proposed framework compared with various baseline methods. Finally, we analyze the contributions of individual model components and the impact of parameters used in our framework. We also assess the generalizability of K-RagRec in the zero-shot setting. We present the generalizability study in Appendix A.7.

4.1 Experimental Settings

4.1.1 Datasets
To evaluate the performance of our K-RagRec framework, we adopt three real-world datasets. MovieLens-1M\footnote{https://grouplens.org/datasets/movielens/1m/} is a dataset containing approximately one million movie ratings and textual descriptions of movies (i.e., “title”). MovieLens-20M\footnote{https://grouplens.org/datasets/movielens/20m/} is a large-scale movie ratings dataset encompassing over 20 million ratings from more than 138,000 users on 27,000 movies. Amazon Book\footnote{https://jmcauley.ucsd.edu/data/amazon/} is a book recommendation dataset that records more than 10 million user ratings of books and the titles of the books. In addition, we adopt the popular knowledge graph Freebase\footnote{https://developers.google.com/freebase} and filter out the triples related to the three datasets to reconstruct the KG. The statistics of these three datasets and KG are presented in Table 3.

4.1.2 Baselines
In the realm of LLM-based recommendation research, our work pioneers the investigation of retrieving knowledge from KGs to enhance the recommendation capabilities of LLMs. Therefore, to evaluate the effectiveness, we compare our proposed framework with a series of meticulously crafted KG RAG-enhanced LLM recommendation baselines. We first include two typical inference-only methods Retrieve-Rewrite-Answer (Wu et al., 2023b) (KG-Text) and KAPING (Baek et al., 2023), where the former retrieves sub-graphs and textualizes them, and the latter retrieves triples. We exclude some knowledge reasoning path-based approaches (Luo et al., 2023), as it is difficult to retrieve faithful knowledge reasoning paths solely from the user’s interaction items. Next, we compare K-RagRec with various prompt-tuning approaches augmented by retrieval, including Prompt Tuning with KG-Text (PT w/ KG-Text), GraphToken (Perozzi et al., 2024) with retrieval (GraphToken w/ RAG) as well as G-retriever (He et al., 2024). Additionally, we evaluate our method against Lora Fine-tuning with retrieval (Lora w/ KG-Text) (Hu et al., 2021).

4.1.3 Evaluation Metrics
To evaluate the effectiveness of our K-RagRec framework, we employ two widely used evaluation metrics: Accuracy (ACC), and Recall@k (He et al., 2020). We present results for \( k \) equal to 3, and 5. Inspired by recent studies (Zhang et al., 2024; Hou et al., 2022), we adopt the leave-one-out strategy for evaluation. Specifically, for each user, we select the last item that the user interacted with as the target item and the 10 interaction items prior to the target item as the historical interactions. Then, we leverage LLM to predict the user’s preferred item from a pool of 20 candidate items (\( M = 20 \)), which contains one target item with nineteen randomly sampled items. For trained models (including prompt tuning and fine-tuning), we compute Recall@k by extracting the probability assigned to each item and evaluating the model’s ability to rank the target item within the top-k predictions. In addition, we also conduct comparison experiments with 10 candidate items (\( M = 10 \)) as shown in Appendix A.5.

4.1.4 Parameter Settings
We implement the framework on the basis of PyTorch and conduct the experiments on 2 NVIDIA A6000-48GB GPUs. We adopt the SentenceBert to encode entities, relations, and query attributes. We use the 3 layers Graph Transformer as the GNN\textsuperscript{Indexing} and GNN\textsuperscript{Encoding} for MovieLens-1M and 4 layers for MovieLens-20M and Amazon Book. The layer dimension is set to 1024, and the head number is set to 4. The popularity selective retrieval policy threshold \( p \) is set to 50%. For each item that needs to be retrieved, we retrieve the top-3 most similar sub-graphs. The re-ranking knowledge sub-graph number \( N \) is set to 5. More experiment details are shown in Appendix A.1. We also present several prompt examples in Appendix A.12.

4.2 Overall Performance Comparison
We compare the recommendation performance of K-RagRec with various baselines on three open-
Table 1: Performance comparison of different KG RAG-enhanced LLM recommendations. The best performance and the second-best performance are marked in red and blue, respectively. ACC and R@k denote Accuracy and Recall@k, respectively.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="4">MovieLens-1M</th>
    <th colspan="4">MovieLens-20M</th>
    <th colspan="4">Amazon Book</th>
  </tr>
  <tr>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td rowspan="6">LLama-2</td>
    <td>Inference-only</td>
    <td>KG-Text (Wu et al., 2023b)</td>
    <td>0.076</td>
    <td>-</td>
    <td>-</td>
    <td>0.052</td>
    <td>-</td>
    <td>-</td>
    <td>0.058</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>KAPING (Baek et al., 2023)</td>
    <td>0.079</td>
    <td>-</td>
    <td>-</td>
    <td>0.069</td>
    <td>-</td>
    <td>-</td>
    <td>0.083</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Frozen LLM w/ PT</td>
    <td>PT w/ KG-Text</td>
    <td>0.078</td>
    <td>0.191</td>
    <td>0.308</td>
    <td>0.051</td>
    <td>0.152</td>
    <td>0.250</td>
    <td>0.074</td>
    <td>0.165</td>
    <td>0.245</td>
  </tr>
  <tr>
    <td></td>
    <td>GraphToken w/ RAG (Perozzi et al., 2024)</td>
    <td>0.268</td>
    <td>0.421</td>
    <td>0.466</td>
    <td>0.186</td>
    <td>0.433</td>
    <td>0.576</td>
    <td>0.326</td>
    <td>0.515</td>
    <td>0.624</td>
  </tr>
  <tr>
    <td></td>
    <td>G-retriever (He et al., 2024)</td>
    <td>0.274</td>
    <td>0.532</td>
    <td>0.650</td>
    <td>0.342</td>
    <td>0.619</td>
    <td>0.739</td>
    <td>0.275</td>
    <td>0.487</td>
    <td>0.612</td>
  </tr>
  <tr>
    <td></td>
    <td>K-RagRec</td>
    <td>0.435</td>
    <td>0.725</td>
    <td>0.831</td>
    <td>0.600</td>
    <td>0.850</td>
    <td>0.913</td>
    <td>0.508</td>
    <td>0.690</td>
    <td>0.780</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>58.6%</td>
    <td>33.0%</td>
    <td>27.8%</td>
    <td>75.4%</td>
    <td>37.3%</td>
    <td>23.5%</td>
    <td>55.8%</td>
    <td>34.0%</td>
    <td>25.0%</td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Lora w/ KG-Text</td>
    <td>0.402</td>
    <td>0.718</td>
    <td>0.833</td>
    <td>0.609</td>
    <td>0.848</td>
    <td>0.905</td>
    <td>0.446</td>
    <td>0.648</td>
    <td>0.758</td>
  </tr>
  <tr>
    <td>Lora w/ K-RagRec</td>
    <td>0.466</td>
    <td>0.770</td>
    <td>0.863</td>
    <td>0.637</td>
    <td>0.872</td>
    <td>0.927</td>
    <td>0.516</td>
    <td>0.720</td>
    <td>0.799</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>15.9%</td>
    <td>7.2%</td>
    <td>3.6%</td>
    <td>4.5%</td>
    <td>2.7%</td>
    <td>2.4%</td>
    <td>15.7%</td>
    <td>11.1%</td>
    <td>5.4%</td>
  </tr>
  <tr>
    <td rowspan="6">LLama-3</td>
    <td>Inference-only</td>
    <td>KG-Text (Wu et al., 2023b)</td>
    <td>0.095</td>
    <td>-</td>
    <td>-</td>
    <td>0.060</td>
    <td>-</td>
    <td>-</td>
    <td>0.054</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>KAPING (Baek et al., 2023)</td>
    <td>0.084</td>
    <td>-</td>
    <td>-</td>
    <td>0.069</td>
    <td>-</td>
    <td>-</td>
    <td>0.062</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Frozen LLM w/ PT</td>
    <td>PT w/ KG-Text</td>
    <td>0.134</td>
    <td>0.294</td>
    <td>0.433</td>
    <td>0.094</td>
    <td>0.205</td>
    <td>0.296</td>
    <td>0.083</td>
    <td>0.207</td>
    <td>0.314</td>
  </tr>
  <tr>
    <td></td>
    <td>GraphToken w/ RAG (Perozzi et al., 2024)</td>
    <td>0.355</td>
    <td>0.622</td>
    <td>0.737</td>
    <td>0.473</td>
    <td>0.719</td>
    <td>0.805</td>
    <td>0.428</td>
    <td>0.567</td>
    <td>0.661</td>
  </tr>
  <tr>
    <td></td>
    <td>G-retriever (He et al., 2024)</td>
    <td>0.352</td>
    <td>0.632</td>
    <td>0.746</td>
    <td>0.502</td>
    <td>0.736</td>
    <td>0.796</td>
    <td>0.417</td>
    <td>0.584</td>
    <td>0.682</td>
  </tr>
  <tr>
    <td></td>
    <td>K-RagRec</td>
    <td>0.472</td>
    <td>0.704</td>
    <td>0.765</td>
    <td>0.634</td>
    <td>0.779</td>
    <td>0.818</td>
    <td>0.514</td>
    <td>0.662</td>
    <td>0.723</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>32.9%</td>
    <td>11.4%</td>
    <td>2.5%</td>
    <td>26.3%</td>
    <td>5.8%</td>
    <td>1.6%</td>
    <td>20.0%</td>
    <td>13.4%</td>
    <td>6.0%</td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Lora w/ KG-Text</td>
    <td>0.449</td>
    <td>0.694</td>
    <td>0.750</td>
    <td>0.648</td>
    <td>0.757</td>
    <td>0.790</td>
    <td>0.490</td>
    <td>0.638</td>
    <td>0.698</td>
  </tr>
  <tr>
    <td>Lora w/ K-RagRec</td>
    <td>0.498</td>
    <td>0.712</td>
    <td>0.771</td>
    <td>0.674</td>
    <td>0.786</td>
    <td>0.817</td>
    <td>0.546</td>
    <td>0.672</td>
    <td>0.733</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>10.9%</td>
    <td>2.6%</td>
    <td>2.8%</td>
    <td>4.0%</td>
    <td>3.8%</td>
    <td>3.4%</td>
    <td>11.4%</td>
    <td>5.3%</td>
    <td>5.0%</td>
  </tr>
  <tr>
    <td rowspan="6">Qwen2</td>
    <td>Inference-only</td>
    <td>KG-Text (Wu et al., 2023b)</td>
    <td>0.160</td>
    <td>-</td>
    <td>-</td>
    <td>0.174</td>
    <td>-</td>
    <td>-</td>
    <td>0.194</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td></td>
    <td>KAPING (Baek et al., 2023)</td>
    <td>0.196</td>
    <td>-</td>
    <td>-</td>
    <td>0.208</td>
    <td>-</td>
    <td>-</td>
    <td>0.220</td>
    <td>-</td>
    <td>-</td>
  </tr>
  <tr>
    <td>Frozen LLM w/ PT</td>
    <td>PT w/ KG-Text</td>
    <td>0.190</td>
    <td>0.371</td>
    <td>0.499</td>
    <td>0.259</td>
    <td>0.397</td>
    <td>0.494</td>
    <td>0.303</td>
    <td>0.451</td>
    <td>0.553</td>
  </tr>
  <tr>
    <td></td>
    <td>GraphToken w/ RAG (Perozzi et al., 2024)</td>
    <td>0.259</td>
    <td>0.487</td>
    <td>0.608</td>
    <td>0.370</td>
    <td>0.550</td>
    <td>0.632</td>
    <td>0.365</td>
    <td>0.568</td>
    <td>0.658</td>
  </tr>
  <tr>
    <td></td>
    <td>G-retriever (He et al., 2024)</td>
    <td>0.304</td>
    <td>0.551</td>
    <td>0.644</td>
    <td>0.389</td>
    <td>0.606</td>
    <td>0.685</td>
    <td>0.355</td>
    <td>0.552</td>
    <td>0.658</td>
  </tr>
  <tr>
    <td></td>
    <td>K-RagRec</td>
    <td>0.416</td>
    <td>0.712</td>
    <td>0.829</td>
    <td>0.586</td>
    <td>0.842</td>
    <td>0.904</td>
    <td>0.502</td>
    <td>0.686</td>
    <td>0.767</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>36.8%</td>
    <td>29.2%</td>
    <td>28.4%</td>
    <td>50.6%</td>
    <td>38.9%</td>
    <td>32.0%</td>
    <td>37.5%</td>
    <td>20.8%</td>
    <td>16.6%</td>
  </tr>
  <tr>
    <td rowspan="2">Fine-tuning</td>
    <td>Lora w/ KG-Text</td>
    <td>0.400</td>
    <td>0.701</td>
    <td>0.815</td>
    <td>0.601</td>
    <td>0.842</td>
    <td>0.906</td>
    <td>0.478</td>
    <td>0.667</td>
    <td>0.751</td>
  </tr>
  <tr>
    <td>Lora w/ K-RagRec</td>
    <td>0.466</td>
    <td>0.763</td>
    <td>0.860</td>
    <td>0.631</td>
    <td>0.868</td>
    <td>0.928</td>
    <td>0.510</td>
    <td>0.704</td>
    <td>0.780</td>
  </tr>
  <tr>
    <td></td>
    <td><b>Improvement</b></td>
    <td>16.5%</td>
    <td>8.8%</td>
    <td>5.5%</td>
    <td>5.0%</td>
    <td>3.1%</td>
    <td>2.4%</td>
    <td>6.7%</td>
    <td>5.5%</td>
    <td>3.9%</td>
  </tr>
</table>

source backbone LLMs: LLama-2-7b (Touvron et al., 2023), LLama-3-8B (Dubey et al., 2024), and Qwen2-7b (Yang et al., 2024). We present the results in Table 1. From the comparison, we have the following main observations:
• Naively retrieve KG and augment LLM with text methods (i.e., KG-Text and KAPING), have limited recommendation accuracy on the MovieLens-1M and MovieLens-20M and Amazon Book datasets.
• Compared to other prompt tuning RAG methods, K-RagRec with LLama-2-7B as the backbone LLM leads to an average of 41.6% improvement over the sub-optimal baseline across all datasets. With LLama-3-8B and Qwen2-7B as the backbone LLM, K-RagRec also brought an average of 13% to 32% improvement, highlighting the effectiveness of our proposed method in augmenting LLMs’ recommendation performance.
• Compared to the LoRA fine-tuning with the naive RAG approach, K-RagRec with prompt tuning achieves close to or better performance in most settings. Notably, K-RagRec achieves the best performance when fine-tuned with LoRA.

4.3 Ablation Study
To evaluate the impact of each component in our proposed framework, we conduct the ablation study to compare the K-RagRec with four ablated variants on MovieLens and Amazon Book datasets, using LLama-2-7B as the backbone LLM. Details of each ablation variant are provided in Appendix A.6. The results are illustrated in Figure 3. Observing the experiment results, we find that eliminating any component of the framework leads to a decrease in the overall performance of the recommendations, demonstrating the effectiveness of each module. Secondly, removing the GNN Encoder leads to a 37% decrease and a 45.9% decrease in the accuracy of the model on MovieLens and Amazon Book datasets, respectively, highlighting the significance of employing GNN to encode the structure of knowledge sub-graphs. Refer to Appendix A.6 for more details.

4.4 Efficiency Evaluation
In this sub-section, we evaluate the inference efficiency of our proposed K-RagRec framework com-
(a) ML1M ACC    (b) ML1M R@3    (c) ML1M R@5
(d) BOOK ACC    (e) BOOK R@3    (f) BOOK R@5

Figure 3: Comparison among K-RagRec and its four ablated variants on MovieLens-1M and Amazon Book datasets and LLama-2-7b across metrics Accuracy, Recall@3 and Recall@5.

Table 2: Comparison of the inference efficiency on the MovieLens-1M dataset and LLama-2-7b in seconds (s). ACC denote Accuracy.

<table>
  <tr>
    <th>Methods</th>
    <th>ACC</th>
    <th>Time (s)</th>
  </tr>
  <tr>
    <td>w/o RAG</td>
    <td>-</td>
    <td>0.92</td>
  </tr>
  <tr>
    <td>KG-Text</td>
    <td>0.076</td>
    <td>2.19</td>
  </tr>
  <tr>
    <td>KAPING</td>
    <td>0.079</td>
    <td>6.47</td>
  </tr>
  <tr>
    <td>GraphToken w/ RAG</td>
    <td>0.268</td>
    <td>3.14</td>
  </tr>
  <tr>
    <td>G-retriever</td>
    <td>0.274</td>
    <td>5.86</td>
  </tr>
  <tr>
    <td>K-RagRec</td>
    <td>0.429</td>
    <td>1.06</td>
  </tr>
</table>

pared with baselines on the MovieLens-1M dataset and LLama-2-7b. We record the time cost for one inference utilizing two NVIDIA A6000-48G GPUs. The time cost for a single inference is reported in Table 2. By observing the experimental results, we notice that various KG RAG approaches significantly increase the inference time, which is due to the large scale of the KG. In contrast, K-RagRec achieves the best computational efficiency compared to various KG RAG methods and is only about 0.1s slower than direct inference without retrieval. These findings highlight the efficiency of K-RagRec and validate the effectiveness of our popularity selective retrieval policy.

4.5 Parameter Analysis

In this section, we evaluate the impact of three main hyper parameters of K-RagRec, namely popularity selective retrieval policy threshold \( p \), retrieved knowledge sub-graph numbers \( K \), and re-ranking knowledge sub-graph numbers \( N \). In addition, we analyze the impact of various GNN encoder variants and different GNN layer numbers for the proposed framework in Appendix A.10 and A.11.

1) Impact of popularity selective retrieval policy threshold \( p \): To understand how the popularity selective retrieval policy threshold \( p \) affects K-RagRec, we conduct experiments on MovieLens-1M and LLama-2-7b across two metrics. Results are shown in Figure 4. As the threshold \( p \) increases, the recommendation performance initially improves and then decreases. When the threshold \( p \) is set to a small value, only a few items are augmented. This leads to insufficient retrieval and poor recommendation accuracy. When \( p \) is set to a larger value, more items are retrieved for augmentation. However, due to the re-ranking sub-graph numbers \( N \) being fixed, some retrieved cold-start item knowledge sub-graphs are discarded or ranked at the back of the list, resulting in sub-optimal recommendation performance. On the other hand, as observed in Figure 4 (c), the inference time increases almost linearly with threshold \( p \). Therefore, selecting an appropriate threshold \( p \) is crucial to balance the performance and inference time.

2) Impact of retrieved knowledge sub-graph numbers \( K \) and re-ranking sub-graph numbers \( N \): In this part, we analyze the impact of two key hyper parameters, which are retrieved knowledge sub-graph numbers \( K \) and re-ranking sub-graph numbers \( N \). First, to measure the impact of \( K \), we perform experiments on the MovieLens dataset and fix \( p = 50\% \), \( N = 5 \). As shown in Figure 5, some relevant knowledge sub-graphs may be overlooked when \( K = 1 \). On the other hand, larger values of \( K \) can introduce irrelevant information. Therefore, we set \( K \) equal to 3 in our experiments. Next, we evaluate the effect of \( N \) by fixing \( p = 50\% \) and \( K = 3 \), and present the results in Figure 6. We observe that setting \( N \) to between 5 and 7 results in improved performance on the Amazon Book dataset. In general, it is important to carefully choose \( K \) and \( N \) based on the scale of the dataset and the KG.

5 Conclusion

In this paper, we propose a novel framework K-RagRec to augment the recommendation capability of LLMs by retrieving reliable and up-to-date knowledge from KGs. Specifically, we first introduce a GNN and PLM to perform semantic indexing of KGs, enabling both coarse and fine-grained
retrieval for KGs. To further improve retrieval efficiency, we introduce a popularity selective retrieval policy that determines whether an item needs to be retrieved based on its popularity. Notably, K-RagRec performs more expressive graph encoding of the retrieved knowledge sub-graphs, facilitating the LLM to effectively leverage the structure information and avoid long context input. Extensive experiments conducted on three real-world datasets demonstrate the effectiveness of our proposed framework.

6 Limitations

To the best of our knowledge, this work is a pioneering study in investigating knowledge-graph RAG for LLM-based recommendations. Therefore, we realise that this work still has the following three main limitations that can be improved in the future research:

• Firstly, due to the GPU resource constraints, we were only able to evaluate our framework on 7b and 8b models. Therefore, in future work, we plan to extend our method to larger models to fully assess its effectiveness and scalability.
• Additionally, we only utilize Freebase as the external KG as it is most commonly used for recommendation tasks. Thus we also aim to adopt other KGs, such as YAGO, DBpedia, and Wikipedia, to better understand how different knowledge sources impact the performance of the proposed method.
• Lastly, designing an intelligent selective retrieval policy for LLM-based recommender systems is an important and challenging task. In this work, we propose to leverage popularity to determine which items to retrieve to improve retrieval efficiency. In the future, we will investigate more flexible mechanisms (e.g., reinforcement learning) to dynamically update policies according to changes in user interest.

7 Acknowledgements

The research described in this paper has been partly supported by General Research Funds from the Hong Kong Research Grants Council (project no. PolyU 15207322, 15200023, 15206024, and 15224524), internal research funds from The Hong Kong Polytechnic University (project no. P0042693, P0048625, P0051361, P0052406, and P0052986).

References

Himan Abdollahpouri, Robin Burke, and Bamshad Mobasher. 2017. Controlling popularity bias in learning-to-rank recommendation. In Proceedings of the eleventh ACM conference on recommender systems, pages 42–46.

Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Alemán, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774.

Yihao Ang, Yifan Bao, Qiang Huang, Anthony KH Tung, and Zhiyong Huang. 2024. Tsgassist: An interactive assistant harnessing llms and rag for time series generation recommendations and benchmarking. Proceedings of the VLDB Endowment, 17(12):4309–4312.

Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, and Hannaneh Hajishirzi. 2023. Self-rag: Learning to retrieve, generate, and critique through self-reflection. arXiv preprint arXiv:2310.11511.

Jinheon Baek, Alham Fikri Aji, and Amir Saffari. 2023. Knowledge-augmented language model prompting for zero-shot knowledge graph question answering. arXiv preprint arXiv:2306.04136.

Keqin Bao, Jizhi Zhang, Yang Zhang, Wenjie Wang, Fuli Feng, and Xiangnan He. 2023. Tallrec: An effective and efficient tuning framework to align large language model with recommendation. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1007–1014.

Sebastian Borgeaud, Arthur Mensch, Jordan Hoffmann, Trevor Cai, Eliza Rutherford, Katie Millican, George Bm Van Den Driessche, Jean-Baptiste Lespiau, Bogdan Damoc, Aidan Clark, et al. 2022. Improving language models by retrieving from trillions of tokens. In International conference on machine learning, pages 2206–2240. PMLR.

Óscar Celma and Perfecto Herrera. 2008. A new approach to evaluating novel recommendations. In Proceedings of the 2008 ACM conference on Recommender systems, pages 179–186.

Dario Di Palma. 2023. Retrieval-augmented recommender system: Enhancing recommender systems with large language models. In Proceedings of the 17th ACM Conference on Recommender Systems, pages 1369–1373.

Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. 2024. The llama 3 herd of models. arXiv preprint arXiv:2407.21783.

Wenqi Fan, Yujuan Ding, Liangbo Ning, Shijie Wang, Hengyun Li, Dawei Yin, Tat-Seng Chua, and Qing Li. 2024a. A survey on rag meeting llms: Towards
retrieval-augmented large language models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 6491–6501.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. 2019. Graph neural networks for social recommendation. In The world wide web conference, pages 417–426.

Wenqi Fan, Shijie Wang, Jiani Huang, Zhikai Chen, Yu Song, Wenzhao Tang, Haitao Mao, Hui Liu, Xiaorui Liu, Dawei Yin, et al. 2024b. Graph machine learning in the era of large language models (llms). arXiv preprint arXiv:2404.14928.

Bahare Fatemi, Jonathan Halcrow, and Bryan Perozzi. 2023. Talk like a graph: Encoding graphs for large language models. arXiv preprint arXiv:2310.04560.

Yunfan Gao, Yun Xiong, Xinyu Gao, Kangxiang Jia, Jinliu Pan, Yuxi Bi, Yi Dai, Jiawei Sun, and Haofen Wang. 2023. Retrieval-augmented generation for large language models: A survey. arXiv preprint arXiv:2312.10997.

Shijie Geng, Shuchang Liu, Zuohui Fu, Yingqiang Ge, and Yongfeng Zhang. 2022. Recommendation as language processing (rlp): A unified pretrain, personalized prompt & predict paradigm (p5). In Proceedings of the 16th ACM Conference on Recommender Systems, pages 299–315.

Kelvin Guu, Kenton Lee, Zora Tung, Panupong Pasupat, and Mingwei Chang. 2020. Retrieval augmented language model pre-training. In International conference on machine learning, pages 3929–3938. PMLR.

Will Hamilton, Zhitao Ying, and Jure Leskovec. 2017. Inductive representation learning on large graphs. Advances in neural information processing systems, 30.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. 2020. Lightgc: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval, pages 639–648.

Xiaoxin He, Yijun Tian, Yifei Sun, Nitesh V Chawla, Thomas Laurent, Yann LeCun, Xavier Bresson, and Bryan Hooi. 2024. G-retriever: Retrieval-augmented generation for textual graph understanding and question answering. arXiv preprint arXiv:2402.07630.

Yupeng Hou, Shanlei Mu, Wayne Xin Zhao, Yaliang Li, Bolin Ding, and Ji-Rong Wen. 2022. Towards universal sequence representation learning for recommender systems. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 585–593.

Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. 2021. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685.

Zhengbao Jiang, Frank F Xu, Luyu Gao, Zhijing Sun, Qian Liu, Jane Dwivedi-Yu, Yiming Yang, Jamie Callan, and Graham Neubig. 2023. Active retrieval augmented generation. arXiv preprint arXiv:2305.06983.

Vladimir Karpukhin, Barlas Oğuz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen-tau Yih. 2020. Dense passage retrieval for open-domain question answering. arXiv preprint arXiv:2004.04906.

Krishnaram Kenthapadi, Benjamin Le, and Ganesh Venkataraman. 2017. Personalized job recommendation system at linkedin: Practical challenges and lessons learned. In Proceedings of the eleventh ACM conference on recommender systems, pages 346–347.

Urvashi Khandelwal, Omer Levy, Dan Jurafsky, Luke Zettlemoyer, and Mike Lewis. 2019. Generalization through memorization: Nearest neighbor language models. arXiv preprint arXiv:1911.00172.

Thomas N Kipf and Max Welling. 2016. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907.

Tiziano Labruna, Jon Ander Campos, and Gorka Azkune. 2024. When to retrieve: Teaching llms to utilize information retrieval effectively. arXiv preprint arXiv:2404.19705.

Jiatong Li, Yunqing Liu, Wenqi Fan, Xiao-Yong Wei, Hui Liu, Jiliang Tang, and Qing Li. 2024. Empowering molecule discovery for molecule-caption translation with large language models: A chatgpt perspective. IEEE Transactions on Knowledge and Data Engineering.

Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061.

Costas Mavromatis and George Karypis. 2024. Gnnrag: Graph neural retrieval for large language model reasoning. arXiv preprint arXiv:2405.20139.

Sewon Min, Julian Michael, Hannaneh Hajishirzi, and Luke Zettlemoyer. 2020. Ambigqa: Answering ambiguous open-domain questions. arXiv preprint arXiv:2004.10645.

Bryan Perozzi, Bahare Fatemi, Dustin Zelle, Anton Tsitsulin, Mehran Kazemi, Rami Al-Rfou, and Jonathan Halcrow. 2024. Let your graph do the talking: Encoding structured data for llms. arXiv preprint arXiv:2402.05862.

Haohao Qu, Wenqi Fan, Zihuai Zhao, and Qing Li. 2024. Tokenrec: Learning to tokenize id for llm-based generative recommendation. arXiv preprint arXiv:2406.10450.
Ori Ram, Yoav Levine, Itay Dalmedigos, Dor Muhligay, Amnon Shashua, Kevin Leyton-Brown, and Yoav Shoham. 2023. In-context retrieval-augmented language models. Transactions of the Association for Computational Linguistics, 11:1316–1331.

N Reimers. 2019. Sentence-bert: Sentence embeddings using siamese bert-networks. arXiv preprint arXiv:1908.10084.

Priyanka Sen, Sandeep Mavadia, and Amir Saffari. 2023. Knowledge graph-augmented language models for complex question answering. In Proceedings of the 1st Workshop on Natural Language Reasoning and Structured Explanations (NLRSE), pages 1–8.

Yunsheng Shi, Zhengjie Huang, Shikun Feng, Hui Zhong, Wenjin Wang, and Yu Sun. 2020. Masked label prediction: Unified message passing model for semi-supervised classification. arXiv preprint arXiv:2009.03509.

Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697.

Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. 2023. Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. 2017. Graph attention networks. stat, 1050(20):10–48550.

Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledge-intensive question answering. arXiv preprint arXiv:2308.13259.

Wei Wei, Xubin Ren, Jiabin Tang, Qinyong Wang, Lixin Su, Suqi Cheng, Junfeng Wang, Dawei Yin, and Chao Huang. 2024. Llmrec: Large language models with graph augmentation for recommendation. In Proceedings of the 17th ACM International Conference on Web Search and Data Mining, pages 806–815.

Junda Wu, Cheng-Chun Chang, Tong Yu, Zhankui He, Jianing Wang, Yupeng Hou, and Julian McAuley. 2024. Coral: Collaborative retrieval-augmented large language models improve long-tail recommendation. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 3391–3401.

Shengqiong Wu, Hao Fei, Leigang Qu, Wei Ji, and Tat-Seng Chua. 2023a. Next-gpt: Any-to-any multi-modal llm. arXiv preprint arXiv:2309.05519.

Yike Wu, Nan Hu, Sheng Bi, Guilin Qi, Jie Ren, An-huan Xie, and Wei Song. 2023b. Retrieve-rewrite-answer: A kg-to-text enhanced llms framework for knowledge graph question answering. arXiv preprint arXiv:2309.11206.

Shi-Qi Yan, Jia-Chen Gu, Yun Zhu, and Zhen-Hua Ling. 2024. Corrective retrieval augmented generation. arXiv preprint arXiv:2401.15884.

An Yang, Baosong Yang, Binyuan Hui, Bo Zheng, Bowen Yu, Chang Zhou, Chengpeng Li, Chengyuan Li, Dayiheng Liu, Fei Huang, et al. 2024. Qwen2 technical report. arXiv preprint arXiv:2407.10671.

Junjie Zhang, Yupeng Hou, Ruobing Xie, Wenqi Sun, Julian McAuley, Wayne Xin Zhao, Leyu Lin, and Ji-Rong Wen. 2024. Agentcf: Collaborative learning with autonomous language agents for recommender systems. In Proceedings of the ACM on Web Conference 2024, pages 3679–3689.

Yang Zhang, Fuli Feng, Jizhi Zhang, Keqin Bao, Qifan Wang, and Xiangnan He. 2023. Collm: Integrating collaborative embeddings into large language models for recommendation. arXiv preprint arXiv:2310.19488.

Jujia Zhao, Wenjie Wang, Xinyu Lin, Leigang Qu, Jizhi Zhang, and Tat-Seng Chua. 2023. Popularity-aware distributionally robust optimization for recommendation system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4967–4973.

Zihuai Zhao, Wenqi Fan, Jiatong Li, Yunqing Liu, Xiaowei Mei, Yiqi Wang, Zhen Wen, Fei Wang, Xiangyu Zhao, Jiliang Tang, et al. 2024. Recommender systems in the era of large language models (llms). IEEE Transactions on Knowledge and Data Engineering.
A Appendix

A.1 Implementation Details
The hyper parameters used for K-RagRec and their corresponding values are shown in Table 4. The first part provides the general training setting for K-RagRec. The second part presents the details of (GNN^{Indexing}, GNN^{Encoding}). Then we list the Lora and acceleration settings. Lastly, we provide the hyper parameters for retrieval, including candidate item number \( M \), popularity selective retrieval policy threshold \( p \), retrieve knowledge sub-graphs numbers \( K \), and re-ranking knowledge sub-graphs numbers \( N \). If not specified, we run all methods three times with different random seeds and report the averaged results.

Table 3: Basic statistics of three datasets and the KG. "Items in KG" indicates the number of items that appeared in both the KG and the dataset.

<table>
  <tr>
    <th>Dataset</th>
    <th>MovieLens-1M</th>
    <th>MovieLens-20M</th>
    <th>Amazon Book</th>
  </tr>
  <tr>
    <td>User</td>
    <td>6,038</td>
    <td>138,287</td>
    <td>6,106,019</td>
  </tr>
  <tr>
    <td>Item</td>
    <td>3,533</td>
    <td>20,720</td>
    <td>1,891,460</td>
  </tr>
  <tr>
    <td>Interaction</td>
    <td>575,281</td>
    <td>9,995,410</td>
    <td>13,886,788</td>
  </tr>
  <tr>
    <td>Items in KG</td>
    <td>3,498</td>
    <td>20,139</td>
    <td>91,700</td>
  </tr>
  <tr>
    <td>Entities</td>
    <td>250,631</td>
    <td>1,278,544</td>
    <td>186,954</td>
  </tr>
  <tr>
    <td>Relations</td>
    <td>264</td>
    <td>436</td>
    <td>16</td>
  </tr>
  <tr>
    <td>KG triples</td>
    <td>348,979</td>
    <td>1,827,361</td>
    <td>259,861</td>
  </tr>
</table>

A.2 Graph Neural Networks
GNNs are a critical technique in graph machine learning and are widely employed in various graph tasks. By iteratively updating node representations through aggregating information from neighboring nodes, GNNs effectively capture the underlying topology and relational structure of graphs. Formally, a typical GNN operation can be formulated as follows:

\[
\mathbf{x}_j^{(l+1)} = \mathbf{x}_j^{(l)} \oplus \mathrm{AGG}^{(l+1)} \left( \left\{ \mathbf{x}_i^{(l)} \mid i \in \mathcal{N}(x_j) \right\} \right),
\]

where \( \mathbf{x}_j^{(l+1)} \) express node \( j \)'s feature on the \( l \)-th layer, and \( \mathcal{N}(x_j) \) is the set of neighbours of node \( j \). AGG is a aggregation function to aggregates neighbors' features, and \( \oplus \) combines neighbors' information with the node itself.

A.3 More Related Work
The remarkable breakthroughs in LLMs have led to their widespread adoption across various fields, particularly in the recommendations. Given powerful reasoning and generalization capabilities, many studies have actively attempted to harness the power of the LLM to enhance recommender systems (Geng et al., 2022; Bao et al., 2023; Qu et al., 2024; Wei et al., 2024; Zhang et al., 2023). For example, P5 (Geng et al., 2022) proposes an LLM-based recommendation model by unifying pre-training, prompting, and prediction for various recommendation tasks, such as sequential recommendation and rating predictions. Furthermore, Tallrec (Bao et al., 2023) fine-tunes LLM (i.e., LLaMA-7B) to align with recommendation data for sequential recommendations. To further capture higher-order collaborative knowledge and enhance the model's ability to generalize users and items, TokenRec (Qu et al., 2024) proposes a masked vector-quantized tokenizer to tokenize users and items in LLM-based recommendations. Despite their effectiveness, these models often face challenges, such as hallucinations and the lack of up-to-date knowledge. While fine-tuning can partially mitigate these issues, it is resource-costly and time-consuming due to the massive parameters of LLMs. To overcome these challenges, we propose a knowledge retrieval augmented recommendation framework that leverages external KGs to provide reliable and up-to-date knowledge instead of costly fine-tuning.

A.4 Comparison with Existing Methods
Existing LLM-based recommender systems usually require frequent fine-tuning on specific datasets to address the lack of knowledge and hallucinations, which is time-consuming and costly. To solve these challenges and avoid costly fine-tuning, our work

<table>
  <tr>
    <th>Item</th>
    <th>Value</th>
  </tr>
  <tr>
    <td>batch size</td>
    <td>5</td>
  </tr>
  <tr>
    <td>epochs</td>
    <td>3</td>
  </tr>
  <tr>
    <td>grad steps</td>
    <td>2</td>
  </tr>
  <tr>
    <td>learning rate</td>
    <td>1e-5</td>
  </tr>
  <tr>
    <td>Indexing layer numbers</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Indexing hidden dimension</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Encoding layer numbers</td>
    <td>4</td>
  </tr>
  <tr>
    <td>Encoding hidden dimension</td>
    <td>1024</td>
  </tr>
  <tr>
    <td>Encoding head numbers</td>
    <td>4</td>
  </tr>
  <tr>
    <td>lora_r</td>
    <td>8</td>
  </tr>
  <tr>
    <td>lora_alpha</td>
    <td>16</td>
  </tr>
  <tr>
    <td>lora_dropout</td>
    <td>0.1</td>
  </tr>
  <tr>
    <td>int8</td>
    <td>True</td>
  </tr>
  <tr>
    <td>fp16</td>
    <td>True</td>
  </tr>
  <tr>
    <td>candidate item numbers</td>
    <td>20</td>
  </tr>
  <tr>
    <td>threshold \( p \)</td>
    <td>50%</td>
  </tr>
  <tr>
    <td>top-\( K \)</td>
    <td>3</td>
  </tr>
  <tr>
    <td>top-\( N \)</td>
    <td>5</td>
  </tr>
</table>
is the first to augment the recommendation performance of LLMs by retrieving structured data (i.e., knowledge graph). Specifically, our approach differs from existing work in the following ways:
• K-RagRec introduces an indexing GNN to efficiently retrieve structured data to enhance the recommendation capability of LLMs. Although some GraphRAG approaches also introduce GNNs to capture higher-order neighbourhood information, they only apply the last layer of the GNN representation, leading to coarse retrieval (He et al., 2024; Mavromatis and Karypis, 2024). In contrast, our approach leverages the representation of each GNN layer to retrieve nuanced knowledge of both coarse and fine-grained graph structures from KG, achieving a more comprehensive and precise retrieval.
• In the recommendation domain that pursues inference speed, excessive retrieval time can seriously degrade the user experience resulting in user churn. Although many studies have explored selective retrieval for RAGs (Yan et al., 2024; Jiang et al., 2023), it is still an open question to determine whether an item needs to be retrieved and to reduce the retrieval time in the recommendation domain. We propose to use popularity to decide whether an item needs to be retrieved based on power law distribution, which greatly reduces the retrieval time. Table 2 shows that K-RagRec achieves inference times close to direct inference while maintaining high recommendation accuracy.
• Typical RAG methods usually incorporate the retrieved content into the prompt as text (Wu et al., 2023b; Baek et al., 2023). However, vanilla RAG methods rely on documents and paragraphs often introduce unnecessary noise and even harmful disturbance, which can negatively impact the accuracy and reliability of recommendations. In addition, the structural relationships between entities are overlooked in typical RAG, resulting in the sub-optimal reasoning capability of LLM-based recommender systems. We propose to incorporate the retrieved knowledge subgraphs (Knowledge-GraphRAG) into the query as a graph prompt, which facilitates LLMs to better understand the retrieved knowledge subgraphs and avoids long contexts.

A.5 Comparison with 10 Candidate Items
In this section, we conduct additional experiments to evaluate the effectiveness of K-RagRec with a candidate item number \( M = 10 \). In this setting, we randomly select nine negative samples with the target item. The results are presented in Table 5. We exclude the results of some backbone LLM models (e.g., LLama-3 and Qwen2), as similar observations as Table 1 can be found. Observing the experimental results, we can note that our proposed K-RagRec method consistently outperforms all baseline methods on MovieLens and Amazon Book datasets, further highlighting the effectiveness of our framework.

<table>
  <tr>
    <th rowspan="2">Methods</th>
    <th colspan="2">MovieLens-1M</th>
    <th colspan="2">Amazon Book</th>
  </tr>
  <tr>
    <th>ACC</th>
    <th>R@3</th>
    <th>ACC</th>
    <th>R@3</th>
  </tr>
  <tr>
    <td>KG-Text</td>
    <td>0.185</td>
    <td>-</td>
    <td>0.142</td>
    <td>-</td>
  </tr>
  <tr>
    <td>KAPING</td>
    <td>0.165</td>
    <td>-</td>
    <td>0.119</td>
    <td>-</td>
  </tr>
  <tr>
    <td>PT w/ KG-Text</td>
    <td>0.159</td>
    <td>0.493</td>
    <td>0.123</td>
    <td>0.384</td>
  </tr>
  <tr>
    <td>GraphToken w/ RAG</td>
    <td>0.512</td>
    <td>0.753</td>
    <td>0.444</td>
    <td>0.682</td>
  </tr>
  <tr>
    <td>G-retriever</td>
    <td>0.469</td>
    <td>0.721</td>
    <td>0.367</td>
    <td>0.610</td>
  </tr>
  <tr>
    <td>K-RagRec</td>
    <td><b>0.568</b></td>
    <td><b>0.779</b></td>
    <td><b>0.606</b></td>
    <td><b>0.770</b></td>
  </tr>
</table>

Table 5: Performance comparison of different KG RAG-enhanced LLM recommendations with candidate item numbers \( M = 10 \) on the MovieLens and Amazon Book dataset and LLama-2-7b across two metrics. The best performances are labeled in bold. ACC and R@3 denote Accuracy and Recall@3, respectively.

A.6 Ablation Study Setting
To assess the impact of each module in K-RagRec, we compare the framework with four ablated variants: K-RagRec (-Indexing), K-RagRec (-Popularity), K-RagRec (-Re-ranking), and K-RagRec (-Encoding). (1) K-RagRec (-Indexing) eliminates the GNN\(^{\text{Indexing}}\) and stores semantic information of PLM in the knowledge vector database. For the retrieved nodes, we extract their second-order sub-graphs as the retrieved knowledge sub-graphs. (2) K-RagRec (-Popularity) does not apply the popularity selective retrieval policy and retrieves all items from the user’s historical interactions. (3) (-Re-ranking) removes the re-ranking module and inputs the knowledge sub-graphs directly. (4) K-RagRec (-Encoding) removes the GNN\(^{\text{Encoding}}\) and replaces it with a trainable soft prompt. The retrieved knowledge sub-graphs will be added to the prompt as triples (e.g., {Moonraker, film writer film, Christopher Wood (writer)}).

A.7 Generalization Study
To evaluate the generalization capability of our proposed framework in the zero-shot setting, we trained a version of the model on the MovieLens-
Table 6: The generalization results for our K-RagRec model in a zero-shot setting. In this setting, our models are trained on MovieLens-1M dataset and evaluated on MovieLens-20M and Amazon Book datasets. ACC and R@k denote Accuracy and Recall@k, respectively.

<table>
  <tr>
    <th rowspan="2">Models</th>
    <th rowspan="2">Methods</th>
    <th colspan="3">MovieLens-20M</th>
    <th colspan="3">Amazon Book</th>
  </tr>
  <tr>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td rowspan="2">LLama2</td>
    <td>K-RagRec</td>
    <td>0.539</td>
    <td>0.740</td>
    <td>0.795</td>
    <td>0.390</td>
    <td>0.581</td>
    <td>0.671</td>
  </tr>
  <tr>
    <td>Lora w/K-RagRec</td>
    <td>0.539</td>
    <td>0.783</td>
    <td>0.863</td>
    <td>0.405</td>
    <td>0.580</td>
    <td>0.796</td>
  </tr>
  <tr>
    <td rowspan="2">LLama3</td>
    <td>K-RagRec</td>
    <td>0.597</td>
    <td>0.797</td>
    <td>0.839</td>
    <td>0.428</td>
    <td>0.628</td>
    <td>0.706</td>
  </tr>
  <tr>
    <td>Lora w/K-RagRec</td>
    <td>0.611</td>
    <td>0.775</td>
    <td>0.814</td>
    <td>0.424</td>
    <td>0.622</td>
    <td>0.732</td>
  </tr>
  <tr>
    <td rowspan="2">QWEN</td>
    <td>K-RagRec</td>
    <td>0.507</td>
    <td>0.769</td>
    <td>0.861</td>
    <td>0.418</td>
    <td>0.612</td>
    <td>0.687</td>
  </tr>
  <tr>
    <td>Lora w/K-RagRec</td>
    <td>0.546</td>
    <td>0.814</td>
    <td>0.897</td>
    <td>0.441</td>
    <td>0.652</td>
    <td>0.706</td>
  </tr>
</table>

Table 7: Quantitative comparison of hallucination on the MovieLens-1M dataset. Δ denotes the reduction in hallucinations for K-RagRec compared to Direct Inference.

<table>
  <tr>
    <th>Models</th>
    <th>Direct Inference</th>
    <th>K-RagRec</th>
    <th>Δ</th>
  </tr>
  <tr>
    <td>LLama-2</td>
    <td>39.1%</td>
    <td>2.7%</td>
    <td>93.1%</td>
  </tr>
  <tr>
    <td>QWEN2</td>
    <td>4.7%</td>
    <td>0.9%</td>
    <td>80.9%</td>
  </tr>
</table>

1M dataset and assessed it on the MovieLens-20M and Amazon Book datasets. The experiment results are shown in Table 6. We note that although the K-RagRec performance in the zero-shot setting is slightly degraded when compared to the well-trained model in Table 1, it still demonstrates 21.6% improvement over SOTA baselines on the MovieLens-20M dataset. Furthermore, despite the differences between the book recommendation and movie recommendation tasks, the model trained on MovieLens-1M delivers about 8.7% improvement in the zero-shot setting compared to prompt-tuned baselines on the Amazon Book dataset. The experimental results demonstrate that K-RagRec exhibits strong generalization capabilities and is adaptable across different domains.

A.8 Study of Hallucination

In this section, we present a qualitative analysis of hallucinations in the LLama-2-7b and QWEN2 models on the MovieLens dataset. Specifically, we include a few fictional movies in the candidate items to observe the probability of the fictional movie being recommended. We compare direct recommendations and recommendations augmented with K-RagRec, and the results are shown in Table 7. We note that K-RagRec significantly reduced hallucinations by 93.1% compared to direct inference on LLama-2. In contrast to LLama-2, QWEN2 rarely recommends fictional movies. Nevertheless, K-RagRec reduced hallucinations by 80.9%, demonstrating the effectiveness of our approach in addressing hallucinations.

A.9 Study of Cold Start Recommendation

The cold start problem is an important issue in most recommendation research. To comprehensively evaluate our approach, we particularly design a case study to evaluate the model’s recommendation performance under the cold-start setting. Specifically, we construct a separate cold-start dataset based on the MovieLens dataset that only contains these identified cold-start items as target items. We compare K-RagRec with three KG RAG-enhanced LLM recommendation methods, and the experiment results are shown in Table 8. The results demonstrate that our proposed K-RagRec still has satisfactory performance under the cold-start recommendation scenario, highlighting the effectiveness of our framework in all the cases.

Table 8: Performance comparison of different KG RAG-enhanced LLM recommendation methods on the cold-start dataset and QWEN2 across three metrics. The best performances are labeled in bold. ACC and R@k denote Accuracy and Recall@k, respectively.

<table>
  <tr>
    <th>Methods</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td>PT w/ KG-Text</td>
    <td>0.106</td>
    <td>0.239</td>
    <td>0.395</td>
  </tr>
  <tr>
    <td>GraphToken w/ RAG</td>
    <td>0.258</td>
    <td>0.473</td>
    <td>0.620</td>
  </tr>
  <tr>
    <td>G-retriever</td>
    <td>0.185</td>
    <td>0.384</td>
    <td>0.488</td>
  </tr>
  <tr>
    <td><b>K-RagRec</b></td>
    <td><b>0.406</b></td>
    <td><b>0.705</b></td>
    <td><b>0.834</b></td>
  </tr>
</table>

Table 9: Comparison of different GNN Encoders on the MovieLens-1M dataset and LLama-2-7b across three metrics. We use bold fonts to label the best performance. ACC and R@k denote Accuracy and Recall@k, respectively.

<table>
  <tr>
    <th>GNN Types</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td>GCN (Kipf and Welling, 2016)</td>
    <td>0.397</td>
    <td>0.704</td>
    <td>0.809</td>
  </tr>
  <tr>
    <td>GAT (Velickovic et al., 2017)</td>
    <td>0.420</td>
    <td>0.693</td>
    <td>0.804</td>
  </tr>
  <tr>
    <td>Graph Transformer (Shi et al., 2020)</td>
    <td><b>0.429</b></td>
    <td><b>0.711</b></td>
    <td><b>0.779</b></td>
  </tr>
  <tr>
    <td>GraphSAGE (Hamilton et al., 2017)</td>
    <td>0.418</td>
    <td>0.699</td>
    <td><b>0.823</b></td>
  </tr>
</table>

A.10 Study of Four GNN Encoders

To further understand the generality of our proposed approach, we conduct a comparative study of four variants applying different GNN encoders. Specifically, we compare GCN (Kipf and Welling, 2016), GAT (Velickovic et al., 2017), Graph Transformer (Shi et al., 2020), and GraphSAGE (Hamilton et al., 2017) as four K-RagRec variants of GNN encoder. Specifically, Graph Convolutional Net-
Table 10: Comparison of different GNN layers on the Amazon Book dataset and LLama-2-7b across three metrics. ACC and R@k denote Accuracy and Recall@k, respectively.

<table>
  <tr>
    <th>GNN Layers</th>
    <th>ACC</th>
    <th>R@3</th>
    <th>R@5</th>
  </tr>
  <tr>
    <td>3 layers</td>
    <td>0.496</td>
    <td>0.653</td>
    <td>0.736</td>
  </tr>
  <tr>
    <td>4 layers</td>
    <td>0.506</td>
    <td>0.690</td>
    <td>0.780</td>
  </tr>
  <tr>
    <td>5 layers</td>
    <td>0.498</td>
    <td>0.656</td>
    <td>0.729</td>
  </tr>
</table>

work (GCN) (Kipf and Welling, 2016) first introduces convolutional operations to graph-structured data. By aggregating features from neighboring nodes, GCN facilitates the learning of rich node representations. GraphSAGE (Hamilton et al., 2017) learns an aggregation function that samples and combines features from a node’s local neighborhood in an inductive setting, enabling the effective use of new nodes. Graph Attention Network (GAT) (Velickovic et al., 2017) further incorporates attention mechanisms, allowing the model to dynamically assign varying attention to neighboring nodes, thereby enhancing the focus on the most relevant information. Inspired by the success of the transformer, the Graph Transformer (Shi et al., 2020) adapts transformer architectures to graph data, enhancing the modeling of graphs, particularly textual graphs.

We report the experiment results on the MovieLens-1M dataset and the LLama-2-7b backbone in Table 9. It is noted that the GCN Encoder variant method performs second-best on the Recall@5 metric, although it is slightly worse than other GNN encoders on the Accuracy metric. Overall, four GNN encoder variants exhibit close performance on the MovieLens dataset, highlighting the generality and robustness of our framework across different GNN encoders.

A.11 Study of GNN Layer Numbers

In this subsection, we evaluate the impact of the number of GNN layers on model performance. We vary the numbers of the GNN layer numbers in the range of {3, 4, 5} and test on the Amazon Book dataset and LLama-2-7b across three metrics. As observed in Table 10, the model performance first improves and then decreases as the number of GNN layers increases, and the model achieves the best results across the three metrics when setting the number of GNN layers is set to four. Therefore, a smaller number of GNN layers may not have sufficient depth to capture the intricate relationships and dependencies in the graph, leading to sub-optimal performance. On the other hand, too many layers can result in indistinguishable node representations. Thus, selecting the optimal number of GNN layers is crucial for effective model training.

A.12 Used Prompt

In this part, we present the prompts designed for movie recommendations and book recommendations. We show two examples in Table 11, and set the candidate items \( M \) equal to 20. For inference, we leverage the model to make the prediction based on the user’s recent watching history and candidate items.
(a) Accuracy                (b) Recall@5                (c) Inference Time
Figure 4: Effect of popularity selective retrieval policy threshold \( p \) on MovieLens-1M and LLama-2-7b across metrics Accuracy, Recall@5 and inference time (seconds) for K-RagRec.

(a) Amazon Book: Accuracy    (b) Amazon Book: Recall@3    (c) Amazon Book: Recall@5
Figure 5: Effect of retrieved knowledge sub-graph numbers \( K \) on Amazon Book datasets and LLama-2-7b across metrics Accuracy, Recall@3 and Recall@5 for K-RagRec.

(a) Amazon Book: Accuracy    (b) Amazon Book: Recall@3    (c) Amazon Book: Recall@5
Figure 6: Effect of re-ranking knowledge sub-graph numbers \( N \) on Amazon Book datasets and LLama-2-7b across metrics Accuracy, Recall@3 and Recall@5 for K-RagRec.
Table 11: Example of the used prompt for K-RagRec. The user’s recent watching/reading history and candidate items are marked in red and blue, respectively.

<table>
  <tr>
    <th>Datasets</th>
    <th>Used Prompt</th>
  </tr>
  <tr>
    <td>Movies</td>
    <td>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Instruction: Given the user’s watching history, select a film that is most likely to interest the user from the options. Watching history: ("History of the World: Part I", "Romancing the Stone", "Fast Times at Ridgemont High", "Good Morning, Vietnam", "Working Girl", "Cocoon", "Splash", "Pretty in Pink", "Terms of Endearment", "Bull Durham"). Options: {A: "Whole Nine Yards", B: "Hearts and Minds", C: "League of Their Own", D: "Raising Arizona", E: "Happy Gilmore", F: "Brokedown Palace", G: "Man Who Knew Too Much", H: "Light of Day", I: "Tin Drum", J: "Blair Witch Project", K: "Red Sorghum", L: "Flintstones in Viva Rock Vegas", M: "Anna, N: Roger & Me" O: "Land and Freedom", P: "In Love and War", Q: "Go West", R: "Kazaam", S: "Thieves", T: "Friends & Lovers" }. Select a movie from options A to T that the user is most likely to be interested in.</td>
  </tr>
  <tr>
    <td>Books</td>
    <td>Below is an instruction that describes a task, paired with an input that provides further context. Write a response that appropriately completes the request. Instruction: Given the user’s reading history, select a book that is most likely to interest the user from the options. Watching history: ("Practice Makes Perfect: Spanish Verb Tenses", "NTC's Dictionary of Common Mistakes in Spanish", "Streetwise Spanish Dictionary/Thesaurus", "Buscalo! (Look It Up!) : A Quick Reference Guide to Spanish Grammar and Usage", "La lengua que heredamos: Curso de español para bilingües", "Vox Diccionario De Sinonimos Y Antonimos", "Schaum's Outline of Spanish Vocabulary", "Nos Comunicamos (Spanish Edition)", "The Oxford Spanish Business Dictionary", "Bilingual Dictionary of Latin American Spanish"). Options: {"A: Folk and Fairy Tales, Childcraft (Volume 3)", B: "The Waves", C: "Dead End Kids: Gang Girls and the Boys They Know", D: "Opera Stars in the Sun: Intimate Glimpses of Metropolitan Opera Personalities", E: "Five-Minute Erotica", F: "Spanish Verbs: Oxford Minireference", G: "Motorcycle Maintenance Techbook: Servicing & Minor Repairs for All Motorcycles & Scooters", H: "Father and Son: A Study of Two Temperaments (Classic, 20th-Century, Penguin)", I: "Manga Mania Fantasy Worlds: How to Draw the Amazing Worlds of Japanese Comics", J: "MCSE Designing a Windows Server 2003 Active Directory & Network Infrastructure: Exam 70-297 Study Guide and DVD Training System", K: "The Atmospheric Boundary Layer (Cambridge Atmospheric and Space Science Series)", L: "St. Augustine and St. Johns County: A pictorial history", M: "A Will to Survive: Indigenous Essays on the Politics of Culture, Language, and Identity", N: "Saved: A Guide to Success With Your Shelter Dog", O: "Mosaic (Star Trek Voyager)", P: "Fantastic Tarot: 78-Card Deck", Q: "American Sign Language-A Look at Its History, Structure and Community", R: "Warrior's Heart (Zebra Historical Romance)", S: "The New Money Management: A Framework for Asset Allocation", T: "Megabrain"}. Select a book from options A to T that the user is most likely to be interested in.</td>
  </tr>
</table>

Reviewer paper summaries: This paper introduces a graph RAG-based model, named K-RagRec, which integrates structural information from an external knowledge graph (KG) for LLM-based recommendation.

This paper proposes a Knowledge Graph Retrieval-Augmented framework for LLM-based recommendations, aiming to mitigate the hallucination problem of LLMs.

This paper proposes K-RagRec, a framework that retrieves knowledge from knowledge graphs to enhance LLM-based recommendation. It introduces a subgraph indexing method for KGs that provides a comprehensive view of a node's neighborhood within the graph. A popularity selective retrieval strategy is designed to determine whether an item should be retrieved based on its popularity. A more expressive graph encoder is introduced to incorporate structured data into LLMs. Experiments demonstrate the effectiveness of K-RagRec framework.
Summary of strengths: 1. The use of semantic similarity to retrieve subgraphs for prompt learning and the incorporation of graph embeddings into LLMs is reasonable.
2. The paper is easy to follow, with clear illustrative figures.
3. The authors tried three backbone models, both with and without LoRA, and released their code, which is good to see.

The proposed knowledge graph retrieval-augmented approach aims to address hallucinations and outdated knowledge in LLM-based recommender systems. This problem positioning is relevant and will likely attract researchers' attention in both recommender systems and LLMs.

1. The proposed method is easy to understand and implement. 

2. K-RagRec applies a popularity selective retrieval strategy to select retrieved subgraphs according to item popularity, avoiding introducing noise.

3. Experimental results show that K-RagRec outperforms other RAG-based or KG-based metho
Summary of weaknesses: 1. The method of encoding subgraph information is similar to [1] and [2] in recommendation tasks. It is essential to mention the differences.
2. Leveraging RAG to tackle hallucinations and outdated knowledge is not novel to me. It would be more convincing if there were any experiments proving that on these two issues, the model performs better than traditional Graph RAG. Otherwise, the abstract and introduction should focus on how to effective knowledge retrieval from the KG, which is uniquely yours.
3. Several details require further explanation:
- (a) How did you train your GCN retriever for Semantic Indexing to create the Knowledge Vector Database?
- (b) Does the retrieved subgraph often include the ground-truth item? If so, could a simple MLP also achieve comparable performance?
- (c) Based on the provided code, the graph is generated using all nodes and edges, not just the training data. Could this lead to an information leak?


[1] XRec: Large Language Models for Explainable Recommendation, EMNLP finding 2024

[2] Enhancing High-order Interaction Awareness in LLM-based Recommender Model, EMNLP 2024

The paper is well written; however, I have several concerns regarding its claims, experimental validation, and methodological rigor.

**1, Insufficient Justification for "High-Quality" and "Up-to-Date" Retrieval**. 
The authors repeatedly emphasize that their method retrieves high-quality and up-to-date information from knowledge graphs (KGs) to enhance recommendations. However, there is a lack of sufficient and direct mechanism design and experimental evidence to support these claims:
(1)	Up-to-date retrieval: It is unclear how the proposed method ensures up-to-date information. The retrieval process is based on a static KG (Freebase in this case), which raises the question of whether the proposed method offers any advantage over other KG-based RAG methods when all methods operate on the same KG.
(2)	High-quality retrieval: The paper does not provide direct evidence to support the claim that the retrieved information is of higher quality. While the recommendation performance improvement is presented as indirect evidence, additional visualization or qualitative analysis of retrieved information would strengthen the argument.

**2, Weak Baseline Selection.** The choice of baselines raises concerns:
(1)	Lack of comparison with traditional KG-based recommendation methods: KG-enhanced recommendation has been extensively studied. Comparing the proposed approach with classical KG-based recommendation methods (not relying on LLMs) would help establish the necessity and contribution of KG-enhanced LLM-based recommendation.
(2)	Reliance on non-peer-reviewed baselines: Nearly all baselines are from ArXiv, lacking rigorous peer review. The authors should include baselines from top-tier conferences or journals such as KDD, SIGIR, or TKDE to provide a stronger comparison.
	
**3, Issues with the "Popularity Policy":** Similar to the concerns raised by first-round reviewers, I find the "popularity policy"—which is a key contribution of this work—to be insufficiently rigorous. More importantly, this heuristic rule may introduce or exacerbate popularity bias in the recommender system, which could negatively impact its practical deployment. The authors should address this issue more carefully.
	
**4, Unclear Contribution of the Graph Encoder.** The authors claim to have introduced "a more expressive graph encoder" (Contribution 3). However, the method section lacks a detailed explanation of this proposed encoder. From my understanding, the approach primarily adopts variants of existing graph encoders (Appendix A.10). If the authors propose a novel graph encoder, they should clearly describe its unique design and advantages. Otherwise, this claim needs to be adjusted for clarity.

1. The baselines are all general RAG or KG-based methods not originally designed for recommender systems, while prevalent LLM-based recommender systems including ID-based methods and generative recommendation methods are missing in the comparisons.

2. Popularity Selective Retrieval Policy is straightforward and relies on a pre-defined threshold p.

3. This work is more related to graph based RAG and more related work should be discussed.
Comments, suggestions and typos: No.

While the paper presents an interesting direction, the above concerns significantly weaken its contributions. I encourage the authors to provide stronger empirical evidence, refine their methodological claims, and improve their baseline selection to better support the effectiveness of their approach.

Some questions and suggestions:

1. Use a pre-defined threshold in Popularity Selective Retrieval Policy, although straightforward, requires tuning during implementation. It would be better to design some learning strategies to somehow automatically decide the threshold.

2. I note that all baselines in Table 1 are not originally designed for recommender systems. For example, KG-Text and KAPING are for question answering. Besides, experiments are directly conducted on the general LLMs (e.g., LLama and QWen). This raises my concern about whether K-RagRec can work well for LLM-based recommender systems. Considering that LLM-based recommender systems are tailored to recommendation tasks and have dominated the recent research on recommender systems, it is better to quantitatively evaluate the performance of K-RagRec when it is plugged into LLM-based recommender systems. If K-RagRec cannot be easily plugged into existing  LLM-based recommender systems, then K-RagRec on LLama or QWen should be compared to original LLM-based recommendation methods as they are direct competitors. Without such comparisons, it is difficult to assess whether K-RagRec can indeed improve or exceed existing recommendation methods.

3. This work is more related to graph based RAG. Section 2 should include more discussions on graph based RAG methods like node-based, path-based, subgraph-based graph based RAG [1].

[1] Graph Retrieval-Augmented Generation: A Survey
Overall assessment scores: 3

3

3
Metareview score: 3
Metareview: This is a resubmission. It proposes a RAG framework K-RagRec, which retrieves structural information from an external KG and encodes it into soft prompts to provide high-quality information for LLM-based recommendations. This paper specifically addresses the issues of hallucinations and outdated knowledge in LLMs for recommendation tasks. Experiments on various real-world datasets demonstrate the effectiveness of the K-RagRec framework.

The reviewers agreed on the importance of the studied problem and the superiority of experimental results. They also raised questions on novelty, technical details, and missing baselines. The authors submitted responses. However, some concerns still exist, even inherited from the first-round submission.
Summary of reasons to publish: 1. This problem positioning is relevant and will attract attention in both recommender systems and LLMs.
2. Experimental results show that K-RagRec outperforms other RAG-based or KG-based methods.
3. The paper is easy to follow, with clear illustrative figures.