Knowledge Graph Tuning: Real-time Large Language Model Personalization based on Human Feedback

## Abstract

Abstract Large language models (LLMs) have demonstrated remarkable proficiency in a range of natural language processing tasks. Once deployed, LLMs encounter users with personalized factual knowledge, and such personalized knowledge is consistently reflected through users’ interactions with the LLMs. To enhance user experience, real-time model personalization is essential, allowing LLMs to adapt user-specific knowledge based on user feedback during human-LLM interactions. Existing methods mostly require back-propagation to finetune the model parameters, which incurs high computational and memory costs. In addition, these methods suffer from low interpretability, which will cause unforeseen impacts on model performance during long-term use, where the user’s personalized knowledge is accumulated extensively.
To address these challenges, we propose Knowledge Graph Tuning (KGT), a novel approach that leverages knowledge graphs (KGs) to personalize LLMs. KGT extracts personalized factual knowledge triples from users’ queries and feedback and optimizes KGs without modifying the LLM parameters. Our method improves computational and memory efficiency by avoiding back-propagation and ensures interpretability by making the KG adjustments comprehensible to humans.
Experiments with state-of-the-art LLMs, including GPT-2, Llama2, and Llama3, show that KGT significantly improves personalization performance while reducing latency and GPU memory costs. Ultimately, KGT offers a promising solution of effective, efficient, and interpretable real-time LLM personalization during user interactions with the LLMs.

## 1 Introduction

Large language models (LLM) have shown increasing power on various NLP tasks (Devlin et al., ; Raffel et al., ; Brown et al., ; Fedus et al., ; Zhang et al., ; Zeng et al., ). The development of LLMs typically encompasses several key phases (Hyperight, ): pre-training on vast corpora, alignment to ensure adherence to ethical guidelines, and domain-specific fine-tuning. While these steps are crucial, there is often an oversight in recognizing the need for further personalization during the deployment phase. Once deployed, LLMs encounter users with personalized factual knowledge (Petroni et al., ; Jiang et al., ; Roberts et al., ). These individual knowledge bases are consistently reflected through users’ feedback on various queries and the model’s responses during interactions, as shown in the interaction example in Figure 1. To enhance user experience, real-time model personalization is essential, allowing LLMs to adapt and incorporate user-specific knowledge based on persistent feedback throughout human-LLM interactions.

Figure: Figure 1: Pipeline of the development of an LLM. In the deployment phase, the model is personalized based on human feedback during the human-LLM interactions. The LLM in the figure is personalized to adapt to the knowledge that the user’s dog is vegetarian from the interaction. Then, in the later interactions, the LLM agent will recommend vegetarian dog food for the user given the same query.
Refer to caption: x1.png

Several technologies are applicable to personalize the LLM, including Parameter Efficient Finetuning (PEFT) (Ding et al., ), Knowledge Editing (KE) (Geva et al., ; Dai et al., ; Meng et al., ), and in-context learning (Brown et al., ). Nevertheless, these methodologies are often plagued by low efficiency, poor interpretability, or a combination of both drawbacks. PEFT and KE require back-propagation to optimize the model parameters. Back-propagation incurs unacceptable GPU memory and computational costs for the daily use of LLMs, especially for on-device applications where the onboard resources are limited. The high computational overhead also makes it difficult to realize real-time personalization. In addition to the low efficiency, these parameter-based methods also lack interpretability. Modifying the model parameters to satisfy the user’s personalized need for the current query may lead to corruption in model parameters, adversely affecting the responses to other queries. This adjustment fails to meet the long-term needs of users who rely on the accumulation of extensive personalized knowledge during interactions with large language models (LLMs). In-context learning has higher interpretability and does not need back-propagation, but its computational overhead, memory cost, and response latency increase drastically with the length of the reference context. Therefore, it is neither efficient nor scalable for long-term use by users.

Knowledge graphs (KGs) (Speer et al., ; Sap et al., ), which store factual knowledge in a structured format, are increasingly integrated into LLMs to enhance the inference with external knowledge (Chang et al., ; Xu et al., ; Yao et al., ; Song et al., ) and are becoming a standard component of LLM systems. In this paper, we propose a new paradigm of model personalization by tuning the KGs based on user feedback. Our proposed method, Knowledge Graph Tuning (KGT), extracts the personalized knowledge triples from the query and the user’s feedback. Instead of optimizing the LLM parameters, KGT optimizes the KGs based on the extracted personalized factual knowledge. We formulate the training objective based on the evidence lower bound (ELBO) and derive an optimization goal of achieving a high personalized knowledge retrieval probability and a high knowledge-enhanced reasoning probability. We propose a heuristic optimization algorithm to finetune the KG by adding and removing knowledge triples without touching the model parameters, significantly improving computational and memory efficiency. Additionally, the added and removed knowledge triples are comprehensible to humans, ensuring the method’s interpretability.

We conduct experiments on multiple datasets using SOTA pre-trained LLMs. The results demonstrate that KGT significantly improves personalization performance compared with baselines while reducing the latency and GPU memory cost by up to $84\%$ and $77\%$, respectively. By proposing KGT, we offer an effective, efficient, and interpretable solution for realizing real-time LLM personalization during human-LLM interactions. We summarize our contributions as follows:

- •
We present the necessity of model personalization during the human-LLM interaction and the challenges of adopting existing technologies: low efficiency and poor interpretability.
- •
We propose a method, KGT, that realizes real-time LLM personalization based on user feedback by optimizing the knowledge graph rather than the model parameters.
- •
We conduct experiments on multiple datasets with SOTA pre-trained LLMs. Compared with baselines, KGT achieves significantly better personalization performance while reducing computational and memory costs. KGT demonstrates considerable scalability as the volume of queries increases, a critical attribute for fulfilling the long-term needs of users who require accumulating extensive personalized knowledge during their interactions with LLMs.

## 2 Related Works

##### KG-enhanced LLM

Large language models (LLMs) are acclaimed for their proficiency in assimilating knowledge from expansive corpora and achieving groundbreaking results in numerous natural language processing (NLP) tasks. Nonetheless, these models are frequently critiqued for the hallucination issues (Lee et al., ; Bang et al., ) and their lack of interpretability. In response to these shortcomings, the integration of knowledge graphs (KGs) with LLMs has been proposed (Chang et al., ; Xu et al., ; Yao et al., ; Song et al., ).
KGs, by storing extensive knowledge in a structured and explicit format, enhance the knowledge capacity of LLMs. Some approaches (Zhang et al., ; Rosset et al., ) advocate for incorporating KGs during the pre-training phase of LLMs, enabling direct knowledge acquisition from these graphs. Others (Chang et al., ; Xu et al., ) suggest the use of KGs during the inference phase to improve access to domain-specific knowledge, thereby substantially enhancing the models’ performance. Furthermore, to advance the interpretability of LLMs, researchers are utilizing KGs to elucidate both the factual knowledge (Petroni et al., ) and the reasoning processes (Lin et al., ) of these models. In this paper, we focus on the setting in which KG enhances the inference phase of an LLM.

##### LLM Personalization and Knowledge Editing

Recently, Knowledge Editing (KE) approaches have been proposed to personalize LLMs by localizing and modifying the factual knowledge within transformers. Geva et al. () suggests that the MLP layers within a transformer for masked language models serve as memory units for entities and their associated information. Extending this idea, KN (Dai et al., ) developed a technique for updating facts in BERT by manipulating specific rows of the MLP matrix with the embedding of the target object. They pinpoint crucial neurons for storing knowledge through gradient-based methods. De Cao et al. () implement a hyper-network that predicts necessary weight adjustments during test time to modify facts. Their experiments involve BERT and BART (Lewis et al., ), particularly in models tailored for question-answering tasks. Mitchell et al. () introduce a hyper-network approach that adjusts gradient decomposition terms to efficiently update knowledge, showcasing scalability across large models like T5 (Raffel et al., ) and GPT-J (Wang and Komatsuzaki, ). The ROME method (Meng et al., ) employed causal tracing to pinpoint knowledge-relevant layers and then edit its FFN module, achieving superior outcomes. All these techniques employ gradient-based optimization methods, which are inefficient and unsuitable for achieving real-time personalization under constrained computational resources. Moreover, modifying model parameters lacks interpretability and may lead to unforeseen impacts on model performance. In contrast, our method solely necessitates model inference, which is significantly more efficient. By adjusting the knowledge graph (KG), our approach ensures that the personalization process is interpretable and the modifications in model performance are comprehensible.

## 3 Preliminary

##### Knowledge Graph (KG)

A knowledge graph $\mathcal{G}$ is a set of triples containing factual knowledge: $\mathcal{G}=\{(e,r,e^{\prime})|e,e^{\prime}\in\mathcal{E},r\in\mathcal{R}\}$, where $\mathcal{E}$ and $\mathcal{R}$ denote the set of entities and relations, respectively, and the triple $(e,r,e^{\prime})$ is referred to as a knowledge triple.

##### KG-enhanced LLM

A KG-enhanced LLM $f_{\theta,\mathcal{G}}$ is parameterized by both the LLM parameters $\theta$ and a KG $\mathcal{G}$. Given a natural language query $q$, the KG-enhanced LLM generates an answer $f_{\theta,\mathcal{G}}(q)$ based on both the model parameters and the retrieved knowledge triples from $\mathcal{G}$. We follow previous works (Sun et al., ; Jiang et al., ) and assume that the entities $e_{q}$ in the query and $e_{a}$ in the answer are labeled. In this paper, we focus on the setting where the LLM retrieves one-depth triples.

## 4 Method

This section introduces our method, KGT (Knowledge Graph Tuning), which enables real-time LLM personalization based on user feedback. Instead of finetuning the model parameters, we edit the user’s personalized knowledge graph, which provides the user’s personalized factual knowledge to enhance the customization capability of the LLM. KGT does not need to conduct back-propagation of the LLM, and only the inference is required, which significantly reduces the latency and computational cost. The edited knowledge triples are comprehensible, ensuring the method’s interpretability. The overview of KGT is shown in Figure 2.

Figure: Figure 2: The overview of KGT. The LLM extracts the posterior distribution of the personalized knowledge triples $Q(z|q,a)$ from the human-LLM interaction. The personalized triples are utilized to optimize the KG to achieve two goals: The model can (1) retrieve the personalized triples with high probability and (2) generate the user’s feedback with the retrieved triples in high confidence.
Refer to caption: x2.png

### 4.1 Knowledge Graph Tuning

Recently, many techniques have been developed to efficiently finetune or personalize the pre-trained LLM parameters based on the user’s data (Ding et al., ; Geva et al., ; Brown et al., ). Instead of finetuning the model parameters, we modify the user’s KG to inject personalized knowledge into the LLM.

##### Example

A user wants to let the LLM-based assistant order some food for her vegetarian dog. Instead of finetuning the LLM to remember the fact that the user’s dog is vegetarian, the user can simply delete the knowledge triple $\left(Dog,Enjoy,Meat\right)$ and add a triple $\left(Dog,Enjoy,Vegetable\right)$ in his/her personal knowledge graph. To generate an appropriate response, the LLM assistant will first retrieve the knowledge triple $\left(Dog,Enjoy,Vegetable\right)$ based on the user’s query, then recommend vegetarian dog food for the user based on the query and the retrieved personalized knowledge triple.

From the example and previous works (Xu et al., ; Chang et al., ; Dai et al., ), we conclude that there are two steps in KG-enhanced LLM reasoning: 1. knowledge retrieval and 2. knowledge-enhanced reasoning. Following this insight, we formulate the KG-enhanced LLM reasoning probability by marginalizing the knowledge triple distribution. Then, our KGT is an optimization problem that aims to maximize the probability of reasoning the answer from a KG-enhanced LLM:

$$ $P_{\theta,\mathcal{G}}(a|q)=\sum_{z\in\mathcal{G}}P_{\theta,\mathcal{G}}(a|q,z )P_{\theta,\mathcal{G}}(z|q),$ (1) $$

where $P_{\theta,\mathcal{G}}(z|q)$ is the probability of retrieving the knowledge triple $z$ given the user’s query $q$, which represents the knowledge retrieval step. Let $a$ denote the user feedback, and $P_{\theta,\mathcal{G}}(a|q,z)$ is the probability of generating the answer $a$ given the user’s query and the retrieved triple $z$, which stands for the knowledge-enhanced reasoning step. Instead of optimizing the model parameters $\theta$, we finetune the knowledge graph $\mathcal{G}$ in KGT.

### 4.2 Objective Formulation

The training objective in Equation 1 can be optimized by maximizing the evidence lower bound
(ELBO) (Jordan et al., ), which is formulated as

$$ $\log P_{\theta,\mathcal{G}}(a|q)\geq\underbrace{\mathbb{E}_{z\sim Q(z)}\left[ \log P_{\theta,\mathcal{G}}(a|q,z)\right]}_{\text{knowledge-enhanced reasoning }}\underbrace{-D_{KL}\left(Q(z)||P_{\theta,\mathcal{G}}\left(z|q\right)\right) }_{\text{knowledge retrieval}},$ (2) $$

where $Q(z)\triangleq Q(z|q,a)$ is the posterior distribution of $z$ given the user’s query and the user’s feedback $a$. This posterior distribution gives a larger probability to the triples containing personalized factual knowledge based on the user’s feedback (e.g., the example $\left(Dog,Enjoy,Vegetable\right)$ in Section 4.1), which are referred to the personalized triples. The later term minimizes the KL divergence between the posterior distribution and the prior distribution of the knowledge triple retrieved by the LLM. The former term maximizes the expectation that LLM generates correct answers based on the knowledge triples sampled from the posterior distribution.

##### Knowledge Retrieval

The goal of the knowledge retrieval term is to finetune the KG such that the LLM can retrieve personalized triples based on the user’s feedback. Given a query $q$ and the user’s feedback $a$, we can ask the user or utilize the LLM $f_{\theta,\mathcal{G}}$ itself to extract the top-$K$ appropriate relations $\{r_{k}\}_{k\in[K]}$ between $e_{q}$ and $e_{a}$ in this query’s context. We construct the personalized triple as $\{e_{q},r_{k},e_{a}\}_{k\in[K]}$ and denote this triple set as $\mathcal{H}(q,a,K)$. Then, the posterior distribution $Q(z)$ can be approximated, formally,

$$ $Q(z|q,a)\simeq\begin{cases}\frac{1}{K}&\text{if }z\in\mathcal{H}(q,a,K),\\ 0&\text{else},\end{cases}$ (3) $$

where we assume a uniform distribution over the subset of personalized triples $\mathcal{H}(q,a,K)$. Therefore, the knowledge retrieval loss term can be calculated as

$$ $\mathcal{L}_{\text{retrieve}}=D_{KL}\left(Q(z)||P_{\theta,\mathcal{G}}\left(z| q\right)\right)=-\frac{1}{K}\sum_{z\in\mathcal{H}(q,a,K)}\log P_{\theta, \mathcal{G}}\left(z|q\right).$ (4) $$

The detailed derivation can be found in Appendix A. By reducing $\mathcal{L}_{\text{retrieve}}$, we finetune the user’s KG $\mathcal{G}$ such that the LLM can retrieve the personalized triples with higher confidence.

To collect $\mathcal{H}(q,a,K)$, users can either provide feedback about the relations between $e_{q}$ and $e_{a}$ in this query’s context or extract the relations using the model $f_{\theta,\mathcal{G}}$. We design an instruction template to utilize the LLM to extract the relations $\{r_{k}\}_{k\in[K]}$ between $e_{q}$ and $e_{a}$:

Based on the provided query and answer, identify K types of relationships between the subject <subject> and the object <object>, considering the context of the query and answer. <query>: $q$ <answer>: $a$ <subject>: $e_{q}$ <object>: $e_{a}$ <relation>: [MASK](^1^11The LLMs in this paper are causal models that do not need a [MASK] token for generation. We include a [MASK] token within the templates in the paper to delineate the starting point for the model’s generation, thereby aiding reader comprehension.)

The instruction is fed into the LLM to generate $K$ relations formatted as

$r_{1}$ <sep> $r_{2}$ <sep> … <sep> $r_{K}$.

Then, we formulate $\mathcal{H}(q,a,K)$ as the set $\{e_{q},r_{k},e_{a}\}_{k\in[K]}$.

To calculate the retrieval probability $P_{\theta,\mathcal{G}}\left(z|q\right)$, we design an instruction template $\mathcal{T}_{\text{retrieve}}(\cdot)$ to instruct the LLM to predict what kind of the relationship is needed to answer this query:

To answer the query: $q$, I need information $e_{q}$ [MASK]

We calculate the probability of a specific relation $r$ as $P_{\theta}\left(r|\mathcal{T}_{\text{retrieve}}\left(q\right)\right)$. Then, the retrieval probability of a knowledge triple $z=\left(e,r,e^{\prime}\right)$ is derived as

$$ $P_{\theta,\mathcal{G}}\left(z|q\right)=P_{\theta,\mathcal{G}}\left(\left(e,r,e ^{\prime}\right)|q\right)=\begin{cases}\frac{P_{\theta}\left(r|\mathcal{T}_{ \text{retrieve}}\left(q\right)\right)}{\sum_{z\in\mathcal{G}}P_{\theta, \mathcal{G}}\left(z|q\right)}&\text{if }e=e_{q}\text{ and }z\in\mathcal{G},\\ 0&\text{else}.\end{cases}$ (5) $$

Thus, only the knowledge triples in $\mathcal{G}$ that start from $e_{q}$ have a retrieval probability larger than 0.

##### Knowledge-enhanced Reasoning

The goal of the knowledge-enhanced reasoning term is to finetune the KG such that the retrieved personalized triples can most encourage the LLM to generate the correct answer. With the approximated posterior distribution $Q(z)$ in Equation 3, the reasoning loss term can be formulated as

$$ $\mathcal{L}_{\text{reasoning}}=-\mathbb{E}_{z\sim Q(z)}\left[\log P_{\theta, \mathcal{G}}(a|q,z)\right]=-\frac{1}{K}\sum_{z\in\mathcal{H}(q,a,K)}\log P_{ \theta,\mathcal{G}}\left(a|q,z\right).$ (6) $$

We design an instruction template $\mathcal{T}_{\text{reasoning}}$ to instruct the model to predict the answer based on both the query $q$ and the knowledge triple $z$:

Answer the query considering the user’s personalized facts. <question>: $q$ <facts>: $z$ <answer>: [MASK]

The knowledge-enhanced reasoning probability is derived as $P_{\theta,\mathcal{G}}\left(a|q,z\right)=P_{\theta}\left(a|\mathcal{T}_{\text{
reasoning}}\left(q,z\right)\right)$.

The final objective of KGT is the combination of knowledge retrieval optimization and knowledge-enhanced reasoning optimization, which is formulated as

$$ $\mathcal{L}=\mathcal{L_{\text{retrieve}}}+\mathcal{L_{\text{reasoning}}}=- \frac{1}{K}\sum_{z\in\mathcal{H}(q,a,K)}\log\left[P_{\theta,\mathcal{G}}\left( a|q,z\right)P_{\theta,\mathcal{G}}\left(z|q\right)\right].$ (7) $$

### 4.3 Optimization Algorithm

Unlike finetuning the model parameters with gradient descent, KGT is achieved by adding and removing the knowledge triples to and from the KG. For a triple $z$ and a KG $\mathcal{G}$, we have two operations: 1. $\mathcal{G}\oplus z\triangleq\mathcal{G}\cup\{z\}$; 2. $\mathcal{G}\ominus z\triangleq{G}\setminus\{z\}$. Because we focus on real-time model personalization, we formulate the optimization in the online learning setting where the optimization algorithm only has access to the user’s current query $q_{t}$ at time $t$ and feedback $a_{t}$. The trivial solution to this query is that we remove all the triples in $\mathcal{G}$ that are probably retrieved based on $q_{t}$, which are the triples starting with $e_{q}$. Then we add the triple $z^{*}$ from $\mathcal{H}(q_{t},a_{t},K)$ with the highest reasoning probability $P_{\theta,\mathcal{G}}\left(a_{t}|q_{t},z^{*}\right)$ into $\mathcal{G}$. In this case, $P_{\theta,\mathcal{G}}\left(z^{*}|q_{t}\right)=1$ and the loss is minimized over $(q_{t},a_{t})$. However, this greedy solution will potentially hurt the other queries due to removing too much knowledge from the KG. Following the example in Section 4.1, the greedy solution will not only remove the triple $\left(Dog,Enjoy,Meat\right)$, but also remove other triples starting from the entity $Dog$ such as $\left(Dog,Is,Animal\right)$. The removed triples might be general or personalized factual knowledge that is essential to the other queries.

We propose a heuristic optimization algorithm based on two principles: 1. The computational cost should be low to achieve real-time personalization efficiency; 2. The KG should be less modified to preserve more knowledge. Both principles require us to modify as few triples as possible to achieve the optimization goal. The detailed algorithm is shown in Algorithm 1. In general, we evaluate the reasoning probability $P_{\theta,\mathcal{G}}\left(a_{t}|q_{t},z^{\prime}\right)$ for the triples $z^{\prime}$ in $\mathcal{H}(q,a,K)$ and triples in $\mathcal{G}$ that starting from $e_{q_{t}}$. Then, we add the triples with the highest reasoning probability (i.e., the triples most encourage the LLM to generate the correct answer) from $\mathcal{H}(q,a,K)$ to $\mathcal{G}$ and remove the triples with the lowest reasoning probability from $\mathcal{G}$ iteratively until the model generates the correct answer.

Figure: Algorithm 1 Knowledge Graph Tuning based on User Feedback

## 5 Experiments

We evaluate KGT in terms of personalization performance and efficiency compared with the baselines. The experiments are conducted on a server with eight A100 GPUs.

### 5.1 Experimental Setup

##### Datasets

To evaluate the effectiveness of KGT on personalizing the LLM, we conduct experiments in the setting where the user provides answers that conflict with the common factual knowledge that LLM learned from the pre-training dataset. We evaluate KGT on two datasets: CounterFact (Meng et al., ) dataset and CounterFactExtension dataset we create based on PARALLEL dataset (Elazar et al., ) utilizing GPT-4 (OpenAI, ). The details about our dataset can be found in Appendix B Both datasets consist of query-answer pairs involving factual knowledge that conflicts with reality. To evaluate the real-time model personalization in practice, we sequentially input query-answer pairs into the model, ensuring that each pair is accessed only once during training. Once all the pairs in the dataset have been processed by the model, we assess the personalization effectiveness across the entire dataset.

##### Baselines and Configurations

We compare KGT with fine-tunning (FT), ROME (Meng et al., ), Knowledge editing (KE) (De Cao et al., ), Knowledge neurons (KN) (Dai et al., ), and MEND (Mitchell et al., ). For all baselines, we test several specifications of layers to edit and report the best results for each baseline. For FT, we follow previous arts (Meng et al., ) to execute full fine-tuning on a single layer. For KN, we specify a subset of all layers as candidates of knowledge neurons to reduce the search space for large models. And baseline results where no knowledge is edited, are referred to as ‘no edit’. We conduct experiments on GPT2-xl (Radford et al., ), Llama2-7B (Touvron et al., ), and Llama3-8B (AI, ). We equip each model with a KG, ConceptNet (Speer et al., ), to enhance the inference. $K$ in Equation 3 is set as five for experiments.

##### Metrics

We utilize two metrics to evaluate the performance of personalization: (1) Efficacy Score (Meng et al., ), measuring the success rate of personalization using the training query-answer pairs directly. If the model generates the user’s personalized answer with a higher probability than the answer before tuning, the model is considered successful for that pair. (2) Paraphrase Score (Meng et al., ), indicating the model’s ability to accurately recall personalized knowledge in paraphrased forms. This assesses its personalization capacity while mitigating the impact of overfitting to specific contexts within the training dataset.

### 5.2 Personalization Performance

We evaluate the setting that the user only provides the answer $a$ as the feedback, and the model will extract the relations and construct $\mathcal{H}\left(q,a,K\right)$. The results on CounterFact dataset are shown in Table 1. It is shown that KGT outperforms the baselines significantly in both efficacy and paraphrase scores. Specifically, using Llama3-8B, KGT improves efficacy by more than 39%, 41%, 55%, 45%, 43%, and 61% on efficacy compared with FT, KE, KN, MEND, and no-edit, respectively. For paraphrase score, KGT outperforms by more than 36%, 32%, 46%, 37%, 33%, 34% compared with FT, KE, KN, MEND, and no-edit, respectively, on Llama3-8B. It is also observed that the results of KGT on Llama3 are better than Llama2. Our analysis is that Llama3 is more powerful in understanding and following instructions, which makes knowledge enhancement from the KG more effective. Such an improvement suggests that KGT will achieve better performance when the pre-trained LLMs are more and more powerful.

**Table 1: Results on CounterFact dataset when the user only provides the answers to the queries as feedback.**
|  | GPT2 |  | Llama2-7B |  | Llama3-8B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Efficacy | Paraphrase |  | Efficacy | Paraphrase |  | Efficacy | Paraphrase |
| FT | $58.43\%_{\pm 0.15\%}$ | $55.77\%_{\pm 0.20\%}$ |  | $62.47\%_{\pm 0.11\%}$ | $63.09\%_{\pm 0.09\%}$ |  | $54.44\%_{\pm 0.35\%}$ | $50.52\%_{\pm 0.05\%}$ |
| ROME | $49.38\%_{\pm 1.20\%}$ | $48.22\%_{\pm 1.36\%}$ |  | $49.94\%_{\pm 1.24\%}$ | $48.84\%_{\pm 1.74\%}$ |  | $51.13\%_{\pm 1.55\%}$ | $52.39\%_{\pm 1.62\%}$ |
| KE | $51.50\%_{\pm 0.32\%}$ | $51.85\%_{\pm 0.27\%}$ |  | $34.25\%_{\pm 1.63\%}$ | $30.45\%_{\pm 1.43\%}$ |  | $40.56\%_{\pm 1.21\%}$ | $41.00\%_{\pm 0.57\%}$ |
| KN | $50.66\%_{\pm 0.52\%}$ | $51.06\%_{\pm 0.11\%}$ |  | $49.41\%_{\pm 0.47\%}$ | $51.20\%_{\pm 1.38\%}$ |  | $50.52\%_{\pm 1.05\%}$ | $50.67\%_{\pm 1.10\%}$ |
| MEND | $50.41\%_{\pm 0.18\%}$ | $50.20\%_{\pm 0.02\%}$ |  | $49.35\%_{\pm 0.47\%}$ | $50.88\%_{\pm 0.30\%}$ |  | $50.29\%_{\pm 0.71\%}$ | $54.65\%_{\pm 1.12\%}$ |
| no edit | 35.87% | 29.74% |  | 30.58% | 28.21% |  | 33.52% | 52.16% |
| KGT | $91.77\%_{\pm 1.37\%}$ | $91.75\%_{\pm 1.84\%}$ |  | $91.1\%_{\pm 1.43\%}$ | $83.86\%_{\pm 1.03\%}$ |  | $94.58\%_{\pm 0.96\%}$ | $86.89\%_{\pm 1.37\%}$ |

KGT also shows significant improvement compared with the baselines on CounterFactExtension, and the results are shown in Table 2. Specifically, KGT improves efficacy by more than 31%, 34%, 51%, 42%, 48%, and 54% on efficacy when adopting Llama3-8B compared with FT, KE, KN, MEND, and no-edit, respectively. For paraphrase score on Llama3-8B, KGT outperforms by more than 27%, 35%, 41%, 36%, 44%, and 42% compared with FT, KE, KN, MEND, and no-edit, respectively.

**Table 2: Results on CounterFactExtend dataset when the user only provides the answers to the queries as feedback.**
|  | GPT2 |  | Llama2-7B |  | Llama3-8B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Efficacy | Paraphrase |  | Efficacy | Paraphrase |  | Efficacy | Paraphrase |
| FT | $58.67\%_{\pm 0.11\%}$ | $53.71\%_{\pm 0.06\%}$ |  | $59.70\%_{\pm 0.15\%}$ | $63.09\%_{\pm 0.09\%}$ |  | $62.29\%_{\pm 0.25\%}$ | $61.97\%_{\pm 0.10\%}$ |
| ROME | $57.44\%_{\pm 1.75\%}$ | $58.45\%_{\pm 1.00\%}$ |  | $47.33\%_{\pm 1.60\%}$ | $48.36\%_{\pm 0.71\%}$ |  | $59.28\%_{\pm 1.79\%}$ | $53.77\%_{\pm 1.37\%}$ |
| KE | $52.49\%_{\pm 0.28\%}$ | $52.55\%_{\pm 0.44\%}$ |  | $33.83\%_{\pm 1.91\%}$ | $44.49\%_{\pm 0.92\%}$ |  | $41.92\%_{\pm 1.16\%}$ | $47.35\%_{\pm 0.47\%}$ |
| KN | $47.40\%_{\pm 0.37\%}$ | $47.22\%_{\pm 0.04\%}$ |  | $49.74\%_{\pm 0.05\%}$ | $49.67\%_{\pm 1.36\%}$ |  | $51.51\%_{\pm 0.58\%}$ | $52.62\%_{\pm 1.91\%}$ |
| MEND | $58.30\%_{\pm 0.12\%}$ | $58.51\%_{\pm 0.07\%}$ |  | $40.73\%_{\pm 0.05\%}$ | $43.61\%_{\pm 0.04\%}$ |  | $45.62\%_{\pm 0.10\%}$ | $44.34\%_{\pm 1.75\%}$ |
| no edit | 47.14% | 51.74% |  | 30.22% | 42.93% |  | 39.26% | 47.04% |
| KGT | $82.57_{\pm 2.82\%}$ | $78.35\%_{\pm 3.26\%}$ |  | $90.68_{\pm 0.74\%}$ | $83.8_{\pm 1.20\%}$ |  | $93.80\%_{\pm 0.36\%}$ | $89.22\%_{\pm 1.17\%}$ |

The improvement in both efficacy and paraphrase rates demonstrates that KGT outperforms the baseline in personalization performance significantly.

### 5.3 Efficiency Comparison

In addition to the personalization performance, we also evaluate KGT’s efficiency in terms of latency and GPU memory cost. The method must achieve low latency and low GPU memory cost to realize real-time personalization in practice under the setting with limited computational resources. KGT can improve the time and GPU memory cost efficiency because it only requires inference, which is far more efficient than back-propagation. For latency, we test the average time that our method and the baselines require to complete the personalization for one query-answer pair. The latency results on CounterFact data are shown in Table 3. The results demonstrate that KGT achieves the shortest latency in most cases. Notably, Llama3 requires less time on several baselines than Llama2 and GPT2 because we stop training once the loss converges. The results on the GPU memory cost can also be found in Table 3. The memory cost is reduced significantly because only the inference is required for KGT. Specifically, KGT reduces 57%, 56%, 77%, 63%, and 62% GPU memory cost when adopting Llama3-8B compared with FT, KE, KN, and MEND, respectively.

**Table 3: Results of latency and GPU memory costs on CounterFact dataset when the user only provides the answers to the queries as feedback.**
|  | GPT2 |  | Llama2-7B |  | Llama3-8B |  |  |  |
| --- | --- | --- | --- | --- | --- | --- | --- | --- |
| Method | Memory | Latency |  | Memory | Latency |  | Memory | Latency |
| FT | 8516MB | 1.80s |  | 30990MB | 0.81s |  | 36968MB | 0.25s |
| ROME | 11948MB | 1.39s |  | 30452MB | 2.33s |  | 36660MB | 2.05s |
| KE | 31574MB | 2.18s |  | 33464MB | 0.30s |  | 69542MB | 0.13s |
| KN | 12832MB | 3.55s |  | 56148MB | 0.69s |  | 44000MB | 0.34s |
| MEND | 11036MB | 0.86s |  | 35166MB | 1.98s |  | 42428MB | 1.40s |
| KGT | 6686MB | 0.16s |  | 13516MB | 0.14s |  | 15904MB | 0.15s |

### 5.4 Ablation Study

#### 5.4.1 Does User Need to Provide Feedback of Relations?

We also conduct experiments under the setting where the user also provides feedback on relations $r$ between $e_{q}$ and $e_{a}$ under the query’s context. We utilize GPT-4 to mimic the user and extract the relations to construct $\mathcal{H}\left(q,a,K\right)$. We use the same instruction template in Section 4.2 for the GPT-4 to conduct relation extraction. The compared results with and without the user’s feedback of relations are shown in Figure 3. It is shown that KGT achieves similar performance if the user provides feedback on the relations in addition to the answer. Notably, utilizing the relations extracted by the LLM $\theta$ can even achieve higher performance on efficacy and paraphrase rates in most cases. Our analysis is that when extracting relations using the LLM $\theta$, KGT implicitly distills knowledge from the model to the knowledge triples, which might benefit the model inference more than human feedback on relations. Thus, in practice, the user will only need to provide the personalized answer to a query as feedback to KGT.

Figure: Figure 3: Compared results on CounterFact dataset with and without user feedback on relations.
Refer to caption: x3.png

#### 5.4.2 Effect of the Size of Query Set

We also evaluate the effect of the query set size on the personalization performance. We conduct experiments on the $CounterFact$ dataset with the Llama3-8B model and evaluate KGT and baselines with query sets of different sizes. The results are shown in Figure 4. It is shown that when the size of the query set increases, the performance of baselines degrades dramatically, while KGT can preserve high performance. This compared result illustrates that KGT can be scaled to a large amount of personalized knowledge. This scalability is crucial for meeting the long-term needs of users who require the accumulation of extensive personalized knowledge when using LLMs.

Figure: Figure 4: Compared results on CounterFact dataset using Llama3-8B with different query set sizes.
Refer to caption: x4.png

## 6 Conclusion and Limitation

We propose an approach, KGT, that personalizes models by optimizing external knowledge graph rather than model parameters. Our method and experimental results demonstrate that our approach offers benefits in terms of performance and efficiency, as supported by empirical studies. By addressing the critical challenges of efficiency and interpretability in model personalization, KGT offers a promising direction for future research and application in enhancing user interactions with LLMs, which has a positive societal impact. One limitation of this method (KGT) is that it depends on the LLM’s ability to follow instructions when calculating $P_{\theta,\mathcal{G}}(a|q,z)$, $P_{\theta,\mathcal{G}}(z|q)$, and collecting $\mathcal{H}(q,a,K)$. However, the existing state-of-the-art LLMs already meet KGT’s requirements for this capability, and future LLMs will possess even stronger abilities to follow instructions.

## Appendix A Derivation of ℒ retrieve subscript ℒ retrieve \mathcal{L}_{\text{retrieve}} caligraphic_L start_POSTSUBSCRIPT retrieve end_POSTSUBSCRIPT

We provide the derivation of the retrieve loss. By approximating the $Q(z)$ with $Q(z|q,a)$, the KL divergence is calculated as

$$ $\displaystyle\mathcal{L}_{\text{retrieve}}=D_{KL}\left(Q(z)||P_{\theta, \mathcal{G}}\left(z|q\right)\right)$ $\displaystyle=D_{KL}\left(Q(z|q,a)||P_{\theta,\mathcal{G}}\left(z|q\right)\right)$ (8) $\displaystyle=\mathbb{E}_{z\sim Q(z|q,a)}\left[\log Q(z|q,a)-\log P_{\theta, \mathcal{G}}(z|q)\right]$ $\displaystyle=-\mathbb{E}_{z\sim Q(z|q,a)}\log P_{\theta,\mathcal{G}}(z|q)+ \mathbb{E}_{z\sim Q(z|q,a)}\log Q(z|q,a)$ $\displaystyle=-\mathbb{E}_{z\sim Q(z|q,a)}\log P_{\theta,\mathcal{G}}(z|q)+ \text{CONST},$ $$

where the term $\mathbb{E}_{z\sim Q(z|q,a)}\log Q(z|q,a)$ is not related to $\mathcal{G}$ and can be treated as a constant. With $Q(z|q,a)$ formulated in Equation 3, the loss can be calculated as

$$ $\displaystyle\mathcal{L}_{\text{retrieve}}$ $\displaystyle=-\mathbb{E}_{z\sim Q(z|q,a)}\log P_{\theta,\mathcal{G}}(z|q)+ \text{CONST}$ (9) $\displaystyle=\sum_{z\in{\mathbb{Z}}}Q(z|q,a)\log P_{\theta,\mathcal{G}}\left( z|q\right)+\text{CONST}$ $\displaystyle=-\frac{1}{K}\sum_{z\in\mathcal{H}(q,a,K)}\log P_{\theta,\mathcal {G}}\left(z|q\right)+\text{CONST},$ $$

where the CONST is omitted in the optimization.

## Appendix B Details on the CounterFactExtension Dataset

CounterFactExtension is a dataset that extends the CounterFact with 27737 additional samples. Specifically, CounterFactExtension is on the task of knowledge editing for Efficacy and Paraphrase evaluation. Likewise, CounterFactExtension is crafted to identify the differences between minor adjustments in the vocabulary used by the model and substantial, widespread shifts in the foundational factual knowledge.

CounterFactExtension is built on knowledge pairs processed by Dai et al. [Dai et al., ] based on PARAREL dataset [Elazar et al., ]. Specifically, for each knowledge pair, we employ GPT-4 to extract the subject, relation, and the true target, $t=(s,r,o^{*})$, and generate a counterfactual target, $o^{c}$. Further, a generation prompt (generated by GPT-4) is also provided for the Paraphrase evaluation. Finally, we manually check each sample and the generation to ensure the correctness.

### B.1 Dataset Samples

## Peer Reviews

confidence: 5
contribution: 3
initial_rating: 6
presentation: 3
questions: Please see the weaknesses above.
soundness: 2
strengths: * The proposed approach of updating the triplets within the knowledge graph in regards to the probability of generating the correct answer with them is sound and reasonable.
* The proposed approach outperforms relevant baselines substantially, while only requiring low computational costs.
summary: The authors propose to personalize the outputs of LLMs by augmenting them with the triplets from the knowledge graph (that is constructed based on the interactions between humans and LLMs), without fine-tuning. Specifically, the authors iteratively update the knowledge graph by generating the triplet that leads to the high probability of generating the correct personalized answer while removing the one with the lowest probability, and then use this constructed knowledge graph by retrieving the relevant triplets from it in response to the given query and augmenting LLMs with them in generating the answer. The authors validate the proposed approach on two knowledge editing benchmark datasets as well as real-world interaction data between humans and LLMs collected from five people, on which it outperforms relevant baselines.
weaknesses: * The authors do not provide details about how to collect the interaction data between humans and LLMs, for example, how many samples are annotated (i.e., the number of samples seems not comprehensive enough), who are the annotators, and how to pay them. In addition to this, the authors may further demonstrate the quality of the annotated data, as currently it seems questionable.
* Besides this human-annotated data, the authors validate the proposed approach on knowledge editing benchmark datasets (whose goal is to generate the outputs that are counter intuitive compared to the knowledge that common LLMs or knowledge graphs have), which may not be appropriate to evaluate the performance of LLM personalization approaches.

---

confidence: 2
contribution: 2
initial_rating: 5
presentation: 3
questions: Please see weakness
soundness: 2
strengths: I. Efficiency: KGT reduces latency and GPU memory usage by eliminating the need for back-propagation, making it suitable for real-time applications.

II. Interpretability: By using KGs to manage user knowledge, KGT enables transparent adjustments to user-specific information, enhancing the interpretability of the personalized responses.
summary: This paper presents Knowledge Graph Tuning (KGT), a novel method for personalizing large language models (LLMs) by leveraging knowledge graphs (KGs) rather than modifying model parameters. KGT captures personalized factual knowledge from user interactions, updates knowledge graphs, and optimizes them without requiring back-propagation, thereby enhancing computational and memory efficiency.
weaknesses: I. KGT shares certain similarities with MemPrompt (https://arxiv.org/pdf/2201.06009). The authors should provide a detailed comparison to highlight the distinctive aspects of their approach.

II. Dependence on KG Quality: The effectiveness of the method relies on the accuracy and relevance of the KG. Low-quality KGs may lead to less effective personalization.

III. Complexity of Knowledge Retrieval: KGT may struggle with more complex or abstract knowledge retrieval tasks, where simple knowledge triples may not adequately represent user-specific knowledge.

IV. The quality of the presentation is below ICLR 2025 standards. For example:

 a. The format of the references should be consistent to ensure neatness and professionalism. For instance, the names of conferences should be uniformly presented either in abbreviations or full names, rather than a mixture of both.

b. Some of the referenced papers have already been accepted but are still listed as preprints on arXiv. For example, the line 661 should be "Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. ERNIE: Enhanced Language Representation with Informative Entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441–1451, Florence, Italy. Association for Computational Linguistics."

---

confidence: 5
contribution: 1
initial_rating: 1
presentation: 2
questions: N/A
soundness: 1
strengths: * This is indeed a Graph RAG paper and can be treated as a new angle which continuously updates the KG with new knowledge.
* The figures are well drawn and easy to follow.
summary: This paper proposes a personalized knowledge editing algorithm for LLMs leveraging KGs. It avoids fine-tuning the LLMs while updating the user profile for new knowledge from feedback as a graph. The pipeline would be like: query->retrieve KG->LLM->user feedback->Update KG->LLM and iteratively. The idea is valuable for rapidly changing user preferences in various downstream tasks. However, the design of the methodology lacks careful consideration and the detailed comments can be found hereunder.
weaknesses: * I recognize the merits of this paper by editing KGs for changing user preference. However, the problem of KG editing is not rocket science, especially each time there will only be very limited triples involved to be added or removed. It could be far more concisely and effectively solved with engineering techniques.
* The word 'tuning' is not suitable which may cause misunderstanding, especially when it only refers to changing the structure of the KG without any parameter learning.
* The code is very important to figure out how the structure of KGs is indeed edited, as well as for the probability calculation. 
* **The paper heavily refers to ROG [1] for the entire methodology which involves plagiarism problem. It changes the storyline by introducing the concept of knowledge editing with no new methods/components.**  Therefore, there is no comment on the remaining of methodology.

[1] Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning, ICLR' 24.

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 3
questions: (1) Can the proposed method generalize to other alignment tasks and compare the performance with other efficient alignment methods?
(2) Can you compare it with other LLM personalized methods instead of just knowledge editing ones?
soundness: 2
strengths: (1) The paper tackles a significant issue in large language models: real-time model personalization based on user feedback. The proposed method presents a promising solution to improve user experience by tailoring LLMs to individual knowledge.

(2) The incorporation of knowledge graphs (KGs) in the proposed method is both innovative and practical. The author offers a theoretical analysis of the method's design.

(3) The experimental evaluation is thorough, featuring tests on multiple datasets with state-of-the-art LLMs. The results highlight the effectiveness and efficiency of the proposed KGT method compared to various baselines.
summary: This paper proposes a novel approach called Knowledge Graph Tuning (KGT) for real-time large language model personalization based on user feedback. The paper addresses the challenges of computational efficiency and interpretability in model personalization by leveraging knowledge graphs (KGs) to optimize the KGs instead of the model parameters. The proposed method is evaluated on state-of-the-art LLMs and compared with several baselines, showing significant improvements in personalization performance while reducing latency and GPU memory costs.
weaknesses: (1) The novelty of the proposed method is limited. The proposed framework is a typical retrieval-based method that relies on including retrieved facts in the context to tailor the output of LLMs. Similar ideas have been proposed by previous studies [1] and even have been applied in ChatGPT today with the proposed "memory module" (similar open-source version here: https://github.com/mem0ai/mem0).

(2) The proposed method relies on explicit human feedback to "optimize" the retrieval process, which largely limits its practicability. Moreover, it relies on structured KGs, which are hard to construct. Nowadays, ChatGPT and Mem0 can unsupervised and automatically extract memory from conversational contexts for personalized outputs, which is more flexible. 

Although the author proposes a theoretical analysis to explain the "optimization of retrieval". The analysis is trivial, which is widely adopted in the RAG such as [2]

(3) The effectiveness of the proposed method is only evaluated on the fact editing tasks. However, other aspects of alignment are not discussed in the paper, such as HarmfulQ, and compared with other efficient alignment methods, such as Deal [3].

[1] Baek, J., Chandrasekaran, N., Cucerzan, S., Herring, A., & Jauhar, S. K. (2024, May). Knowledge-augmented large language models for personalized contextual query suggestion. In Proceedings of the ACM on Web Conference 2024 (pp. 3355-3366).    
[2] LUO, L., Li, Y. F., Haf, R., & Pan, S. Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoning. In The Twelfth International Conference on Learning Representations.   
[3] Huang, J. Y., Sengupta, S., Bonadiman, D., Lai, Y. A., Gupta, A., Pappas, N., ... & Roth, D. (2024). Deal: Decoding-time alignment for large language models. ICML 2024.

---

confidence: 3
contribution: 2
initial_rating: 6
presentation: 3
questions: Can the authors elaborate on how KGT handles contradictory user feedback over time, especially when a user’s personalized knowledge evolves and conflicts with previously stored information in the knowledge graph?
soundness: 3
strengths: The strengths of the paper lie primarily in its innovative approach to personalization for LLMs. By avoiding the use of back-propagation, KGT significantly reduces computational overhead, leading to faster processing times and lower resource consumption. This makes it particularly suitable for real-time applications, where latency can be a critical factor. Additionally, KGT offers a notable advantage in interpretability. By managing user-specific knowledge through adjustments to external knowledge graphs rather than internal parameters, the process becomes more transparent. This makes it easier to trace how user interactions influence changes, allowing for clearer insights into the personalization mechanics. The scalability of KGT is also a strong point, as the method handles an increasing volume of queries effectively, which is essential for users who accumulate extensive personalized knowledge over time. Experimental results back up these claims, showing that KGT performs better than existing methods in personalization benchmarks.
summary: This paper introduces a novel method called Knowledge Graph Tuning (KGT) to personalize large language models (LLMs) in real time. Unlike traditional methods that rely on computationally expensive and memory-intensive back-propagation, KGT leverages external knowledge graphs (KGs) to manage user-specific information. This enables the model to adjust without altering its core parameters, improving efficiency and interpretability. Experimental results demonstrate that KGT significantly enhances personalization performance while lowering latency and resource requirements, showing promise in delivering efficient, effective, and interpretable real-time personalization.
weaknesses: The success of KGT heavily depends on the quality and completeness of the knowledge graphs it uses. If these graphs contain inaccuracies or gaps, the effectiveness of personalization can be compromised, limiting the model's ability to provide relevant responses. Another limitation lies in its reliance on the LLM’s ability to follow instructions accurately when interpreting the knowledge graph. This could lead to inconsistent results if deployed on language models with varied instruction-following capabilities. Furthermore, while KGT aims to reduce user effort by handling knowledge updates through interactions, there are cases where explicit user feedback might still be required. This adds an element of manual intervention, potentially diminishing the appeal of fully automated personalization.

Meta-review: The paper proposes a method to enhance user experience by personalizing large language models (LLMs) in real-time using Knowledge Graphs (KGs). The authors introduce Knowledge Graph Tuning (KGT), which extracts personalized knowledge from user interactions and optimizes KGs without modifying LLM parameters, aiming to improve computational and memory efficiency, ensure interpretability, and reduce latency.

There is discussion highlighting the paper’s strengths, including its efficiency, interpretability, scalability, and sound experimental validation. However, concerns were raised about the dependence on KG quality, limited novelty, and handling contradictory user feedback. Ethical concerns were also mentioned due to similarities with a previous paper (RoG). The authors defended their work by clarifying differences from the RoG paper, emphasizing unique contributions, and addressing concerns about KG quality and user feedback. 

The AC checked both papers and did see the differences between them. However, a careful discussion of this previous work would be beneficial in a revised version. I suggest the authors take this discussion and other concerns (including novelty and generalization) into consideration to improve their paper and resubmit it to another venue.

Decision: Reject