KG-Adapter: Enabling Knowledge Graph Integration in Large Language Models through Parameter-Efficient Fine-Tuning (2024)

Abstract: Although large language models (LLMs) show remarkable capabilities and generalizability across various tasks, they are criticized for lack of expertise. One promising solution is to combine knowledge graphs (KGs) with LLMs, and recent studies focus on integrating KGs into LLMs through prompt-based methods. However, these approaches fail to use the structural information of the KGs, suffer from the problem of knowledge conflict, and over-reliance on super LLMs. To address these challenges, we propose KG-Adapter, a parameter-level KG integration method based on parameterefficient fine-tuning (PEFT). Specifically, we introduce a novel adapter structure designed for decoder-only LLMs, which can encode KGs from both node-centered and relation-centered perspectives, and then perform joint reasoning with LLMs to generate responses end-toend. Experiments with diverse models on four datasets for two different tasks all demonstrate significant improvements. With only 28M parameters trained, we make the 7B-parameter LLM outperform the previous full-parameter fine-tuned state-of-the-art method and comparable to the prompt-based ChatGPT methods.

heading: 1 Introduction
text: Large language models (LLMs), such as Llama (Touvron et al., 2023), Mistral (Jiang et al., 2023a), ChatGPT and GPT4 (OpenAI et al., 2023), exhibit remarkable abilities in multiple natural language processing (NLP) tasks (Bang et al., 2023; Ye et al., 2023), and are even considered as "the sparks of Artificial General Intelligence" (Bubeck et al., 2023). However, LLMs are not the elixir nowadays, they are argued for lack of up-to-date knowledge (Peng et al., 2023) and domain-specific expertise (Schick et al., 2023); the problem of hallucination (Ji et al.,
*Corresponding authors.
2023; Li et al., 2023a); and the lack of interpretability (Zhang et al., 2023).
To address the above issues, a feasible approach is to combine knowledge graphs (KGs) with LLMs (Pan et al., 2023). KGs explicitly contain massive accurate factual knowledge and domain expertise in a structured format (Ji et al., 2021; Abu-Salih, 2021). They are easy to modify, can update new knowledge readily (Mitchell et al., 2018), and provide human-readable reasoning paths (Zhang et al., 2021).
Recent works (Jiang et al., 2023b; Baek et al., 2023a; Wang et al., 2023a; Li et al., 2023b; Sun et al., 2023) focus on combining KGs with LLMs through prompt-based methods or LLMs-as-Agent (Xi et al., 2023) approach. These methods first
3813
extract triplets from KGs by using LLMs through well-designed prompts (or other retrieval models), then linearize these triplets to textual form and concatenate them into the input. Although these methods achieve success, they have flaws (Figure 1 as an example): (1) LLMs cannot process structured KGs directly and the linearization may lose some underlying information (Li et al., 2023c; Guo et al., 2023; Chen et al., 2023); (2) Incorporating KGs via prompt-based methods suffers from the Knowledge Conflict problem (Longpre et al., 2021; Zhou et al., 2023), namely, the LLMs may completely ignore the knowledge provided in the context; (3) Overreliance on super LLMs (e.g., ChatGPT), which makes these methods achieve limited improvement on small-scale LLMs.
Different from the above methods, we endeavor to integrate structured KGs directly into the LLMs at the parameter level to solve these problems. While some early works (Lin et al., 2019a; Yasunaga et al., 2021; Liu et al., 2020; Sun et al., 2020) have also attempted to inject KGs into pre-trained models at the parameter level, these approaches usually need pre-training or fullparameter fine-tuning, requiring inaccessible KGtext-aligned data and high training costs, or are based on encoder architectures, which are incompatible with current LLMs.
Specifically, inspired by the parameter-efficient fine-tuning (PEFT) (Lialin et al., 2023), we propose a novel adapter structure, named KG-Adapter, which contains a Sub-word to Entity Hybrid Initialization (SEHI) and multiple KG-Adapter layers, enabling LLMs to access structured KGs directly. The SEHI initializes the KG representations by fusing the entity-level KG representations (from pre-trained KG embeddings) with the sub-wordlevel KG representations (from the word embedding of LLM). Between each LLM layer, we insert a KG-Adapter layer that encodes KG from both node-centric and relation-centric perspectives and performs joint reasoning via bi-directional crossattention to inject KG information into the LLM.
Experimental results demonstrate the effectiveness of our approach on four datasets across two tasks. With only 28M trainable parameters, we enable the 7B model to outperform the full-parameter fine-tuned SOTA methods and achieve comparable performance to the prompt-based ChatGPT methods.
Our contributions are summarized below:
• For the first time, we propose a PEFT-based method that integrates KGs directly into LLMs with just 28M trainable parameters and can be trained on a single A40 GPU within a few hours.
• We propose a novel adapter structure, KGAdapter, which is compatible with the decoder-only LLMs and can encode KG from both node-centric and relation-centric perspectives to fully utilize the structure information.
• The experimental results demonstrate our approach is applicable to LLMs of different types and sizes, achieving significant improvements across four datasets, allowing the 7Bparameter model to outperform prior SOTA methods of full-parameter fine-tuning and is comparable to the prompt-based ChatGPT methods.

heading: 2 Related Work

heading: 2.1 Integration of KG by Prompt
text: Recent LLM-related studies are almost based on the RAG framework, where relevant KG subgraphs are obtained through retrieval methods, then linearized and inserted into the input context directly with well-designed prompts. CoK (Li et al., 2023b) retrieves relevant knowledge from KG after dividing the original question into multiple subquestions through a question generation model and uses the retrieval results to guide the answer generation. KAPING (Baek et al., 2023a) retrieves facts relevant to the question from the KG based on semantic similarity and fills in a special prompt as the input of LLMs. RoG (Luo et al., 2023) proposes a planning-retrieval-reasoning framework, which first generates relation paths grounded by KGs and uses them to retrieve valid reasoning paths from the KGs for LLMs to conduct reasoning. StructGPT (Jiang et al., 2023b) adopts a toollearning approach to design an iterative readingthen-reasoning framework, which contains specialized interfaces that LLMs can call to access structured data and acquire valuable knowledge to help generate answers. However, these prompt-based methods suffer from the problems of loss of structural information, knowledge conflicts, and overreliance on super LLMs.

heading: 2.2 Integration of KG by Pre-training
text: Some previous studies incorporate KG knowledge through specialized pre-training tasks. For example, ERNIE (Zhang et al., 2019) designs a token-entity alignment mask task, WKLM(Xiong et al., 2019) introduces an entity replacement discrimination task, GLM (Shen et al., 2020) proposes a KG-guided entity masking scheme to conduct the entity-level masked language modeling task, and Deterministic-LLM (Li et al., 2022) focuses on deterministic knowledge thus masks only deterministic entities, and additionally introduces the clue contrastive learning and the clue classification tasks. These pre-training tasks require large amounts of KG-text-aligned data and long training time, which are prohibitively expensive and impractical for LLMs.

heading: 2.3 Integration of KG by Fine-tuning
text: Before the era of LLMs, there was a series of studies (Lin et al., 2019a; Sun et al., 2021; Yasunaga et al., 2021; Zhang et al., 2022b; Wang et al., 2023d; Park et al., 2023) focusing on combining pre-trained models with KGs. These methods usually design a special KG encoder to integrate the KG information and perform full-parameter finetuning on specific tasks. However, there are two long-standing problems of heterogeneous representation (Lin et al., 2019a; Yasunaga et al., 2021; Sun et al., 2021; Zhang et al., 2022b) and knowledge noise (Liu et al., 2020; Sun et al., 2020) for this type of methods, which limit their effectiveness. Furthermore, these methods are incompatible with current LLMs because they rely on the encoder architecture and the full-parameter fine-tuning is also costly for LLMs. For this reason, our KG-Adapter is designed for decoder-only LLMs and is based on PEFT to reduce costs.

heading: 3 Methodology: KG-Adapter

heading: 3.1 Overview
text: To incorporate KGs into LLM at the parameter level, we first have to address the problem of heterogeneous representations, i.e., the pre-trained KG embeddings are very different from the word embeddings of the LLM since they exist in two different vector spaces. Thus, we propose the Subword to Entity Hybrid Initialization (SEHI) module to bridge the gap. Then, to fully use the information in KG, we design the KG-Adapter layer to encode KG from two perspectives and perform joint rea-
soning. Finally, we insert the KG-Adapter layer between each of the LLM blocks, ensuring that the internal structure and parameters of the original LLM are not modified.
As shown in Figure 2, we first fuse the entitylevel KG representations (from the pre-trained KG embedding) with the sub-word-level KG representations (from the word embedding of LLM) by the SEHI and obtain the final KG representations (3.2). Then, the KG representations will be input into the KG-Adapter layer, where the KG representations will be updated in a node-centric way through the graph neural network (GNN) (3.3.1). After that, we reconstruct the KG representations into a relation-centric form by MLPtrip (3.3.2) and fuse them with the textual representations through bi-directional cross-attention to achieve joint reasoning (3.3.3). Finally, the updated textual representations will be passed back to the LLM, while the updated KG representations will be passed to the next layer.

heading: 3.2 Sub-word to Entity Hybrid Initialization
text: Since the KG representations and the textual representations exist in two different vector spaces, this gap will cause the heterogeneous representation problem. To alleviate that, we propose the SEHI that fuses the two types of representations at the initialization stage.
In pre-trained KG representations, nodes and edges are all at the entity level. For example, "knowledge graph" is an entity (multi-word), and it consists of two words "knowledge" and "graph". In contrast, word embedding is at the sub-word level since LLMs commonly use BPE (Shibata et al., 1999) or SentencePiece (Kudo and Richardson, 2018) encoding.
A multi-word (i.e., nodes and edges in KG) contains multiple words and a word contains multiple sub-words. To align them, we first input each node into the pre-trained word embedding of LLM to get their sub-word level representations:
{hsub1n , hsub2n , ..., hsubkn } = Emb(n) (1)
where Emb is the pre-trained word embedding in LLM, n is a node in KG, hsubin ∈ Rd ′ is a subword level hidden vector of node n and d′ is the hidden size of the word embedding. We then sum these hidden vectors to obtain the multi-word level representations and pass them through a MLP to downscale the hidden size to the same as the pre-
trained KG embedding (from 4096 to 1024):
ḣmwn = MLP1( k∑
i=1
hsubin ) (2)
where ḣmwn ∈ Rd ′′
is the multi-word level representations from LLM word embedding and d′′ is the hidden size of KG embedding. Then, we update the pre-trained KG representation hmwn with another MLP:
ḧmwn = MLP2(h mw n ) (3)
and fused ḣmwn with ḧ mw n by gated sum and pass through a MLP to further downscale the hidden size (from 1024 to 64):
hn = MLP3(σ(ḣ mw n ) + (1− σ)ḧmwn ) (4)
where σ is a learnable factor, hn ∈ Rd is the final representation of node n and d is the hidden size of KG-Adapter layer. Using the same way, we get the hybrid initialization representations for all nodes and edges in KG:
Hn = {hn1 , ..., hnN } (5) Hr = {hr1 , ..., hrM } (6)
where N and M are the number of nodes and edges in KG, Hn is the representation of all nodes and Hr is the representation of all edges in KG.
These initial representations contain information from both LLM and KG, thus alleviating the heterogeneous representation problem and making it more understandable for LLM.

heading: 3.3 KG-Adapter Layer

heading: 3.3.1 GNN layer
text: GNNs learn node features through a node-centric message-passing scheme. Thus, we first use RGAT (Wang et al., 2020) to update the KG representations, where each node is updated with the information from its neighboring nodes and corresponding edges, to obtain the node-centric KG representations:
H ln = {hln1 , ..., hlnN } = RGAT (H l−1 n , H l−1 r )
(7) where H l−1n = {hl−1n1 , ..., hl−1nN } and H l−1r = {hl−1r1 , ..., hl−1rM } are node and edge representations from previous layer, hlni is a updated node representation, computed by the following equations:
αji = Softmax( (hl−1ni Wq)(h l−1 nj Wk + h l−1 rji ) T
√ d
)
ĥl−1ni = ∑
j∈neighbor(i) αji(h
l−1 nj Wv + h l−1 rji )
hlni = LN(h l−1 ni + ĥ l−1 ni Wo) (8)
where Wq,Wk,Wv,Wo are trainable linear projections, d is the hidden size of KG-Adapter layer, neighbor(i) is all neighbor nodes of node i, LN is layer norm.
3.3.2 MLPtrip Previous research (Park et al., 2023) found that node-centered updating using only GNNs is insufficient to encode KG, and they argued that a relation-centered representation is more suitable. Inspired by that, we introduce a triple projection MLP - MLPtrip - which merges the node and edge representations into relation-centered triple representations. Since each triple consists of two nodes and an edge, we first concatenate them together and fuse their representations via MLPtrip:
htij = [hni ;hrij ;hnj ]
ĥtij = MLPtrip(htij ) (9)
where htij ∈ R3d is a triple representation concatenated of nodes and edge representations, ĥtij ∈ Rd is the fused triple representation obtained from MLPtrip and ";" is the concatenation operation of vectors.
Now, we have the relation-centered triple representations that contain information about the KG structure.

heading: 3.3.3 Joint Reasoning Module
text: The next step is to perform joint reasoning between the representations of KG and text to integrate these two types of information thoroughly. We first align the hidden size of the textual representations H ′text from LLM to the same as KG representations through a downscale MLP (from 4096 to 64):
Htext = MLPdown(H ′ text) (10)
and concatenate all triple representations into a matrix as the new KG representations:
Hkg = [ĥt1 ; ...; ĥtM ] (11)
then use a bi-directional cross-attention to conduct joint reasoning:
Qkg,Kkg, Vkg = HkgW̃Q, HkgW̃K , HkgW̃V
Qtext,Ktext, Vtext = HtextW̃Q, HtextW̃K , HtextW̃V
Ĥkg = Softmax(QkgK T text/ √ d)Vtext
Ĥtext = Softmax(QtextK T kg/ √ d)Vkg (12)
where W̃Q, W̃K , W̃V are three attention matrices, Ĥkg is the KG representations updated by textual information, Ĥtext is the text representations updated by KG information. We then upscale the hidden size of Ĥtext (from 64 to 4096) and sum it with H ′text:
Ĥ ′ text = MLPup(Ĥtext) +H ′ text (13)
where Ĥ ′ text is the final textual representation to be passed into the LLM layer. Note that we omit all residuals (the ⊗ in Figure 2) in equations, which fuses the two values by gated summation.

heading: 4 Experimental Setup

heading: 4.1 Datasets
text: Datasets used. We conduct experiments on two types of tasks: Multiple-Choice Question Answering (MCQA) and Knowledge Graph-based Question Answering (KGQA) (see A.1 for the formal definitions). For the MCQA task, we evaluate our method on two commonly used datasets, OpenBookQA (OBQA) (Mihaylov et al., 2018) and CommonsenseQA (CSQA) (Saha et al., 2018). For the KGQA task, we use WebQuestionsSP (WQSP) (Yih et al., 2016) and ComplexWebQuestions (CWQ) (Talmor and Berant, 2018) datasets.
Datasets split. For CSQA, we follow Lin et al. (2019b) to split the data into in-house development (IHdev) and in-house test (IHtest) sets. For other datasets, we use their official split. Dataset statistics are in A.2.
KG used and pre-processing. For OBQA and CSQA, we use ConceptNet (Speer et al., 2017) as the source KG and directly use the processed KG subgraphs from Park et al. (2023), where each KG subgraph is retrieved from the entire KG based on the entities in the question sentence. For WQSP and CWQ, we use FreeBase (Bollacker et al., 2008) as the source KG and use the processed data by Xie et al. (2022b). See A.3 for more details about the KGs pre-processing.

heading: 4.2 Implementation Details
text: For training, we choose Llama2-7B-base 1 and Zephyr-7B-alpha 2 as the base models. Different instructions and templates are used for various tasks and base models (See A.4). We perform up to 10 epochs instruction-tuning with early-stop, use
1https://huggingface.co/meta-llama/Llama-2-7b 2https://huggingface.co/HuggingFaceH4/
zephyr-7b-alpha
the autoregressive objective and calculate the loss on answer tokens only. For testing, we select the best model based on the performance of the development set and use greedy decoding to generate responses with a zero-shot context. We use one NVIDIA A40 48GB GPU for training and testing. For hyperparameters, refer to A.5.

heading: 4.3 Baselines
text: To comprehensively evaluate our method, we select a total of 26 methods from three categories for comparison, containing (1) KG+Full-Parameter Fine-Tune LMs, which design specialized KG encoders and combine them with pre-trained language models for full-parameter fine-tuning; (2) LLM Baselines, which are original LLMs after pre-training or supervised fine-tuning (SFT); (3) KG+Prompt Enhanced LLMs, which are enhanced LLMs with retrieval and prompt-based methods.
KG+Full-Parameter Fine-Tune LM. This type of approach usually designs specialized KG encoding modules and combines them with pretrained language models for full-parameter finetuning. For OBQA and CSQA datasets, we choose GreaseLM (Zhang et al., 2022b), JointLK (Sun et al., 2021), GSC (Wang et al., 2021), DHLK (Wang et al., 2023d) and QAT (Park et al., 2023), where QAT and DHLK are the recent SOTA methods. For WQSP and CWQ, we compare with the embedding-based methods, including EmbedKGQA (Saxena et al., 2020), NSM (He et al., 2021), KGT5 (Saxena et al., 2022), and retrievalaugmented methods, including PullNet (Sun et al., 2019), SR (Zhang et al., 2022a), UniKGQA (Jiang et al., 2023d).
LLM Baselines. The original LLMs or the LLMs after SFT, including Llama2 (Touvron et al., 2023), Zephyr (a SFT version of Mistral) (Jiang et al., 2023a) , Flan-T5 (Chung et al., 2022), Alpaca (Taori et al., 2023), GPT-3 (Brown et al., 2020), ChatGPT and GPT4 (OpenAI et al., 2023).
KG+Prompt Enhanced LLMs. Includes prompt engineering and RAG methods for LLMs. For OBQA and CSQA, we compare with Few-shot (Brown et al., 2020), CoT (Wei et al., 2022), AutoCoT (Zhang et al., 2022c), KSL (Feng et al., 2023), KAPING (Baek et al., 2023b) and CoK (Wang et al., 2023b). For WQSP and CWQ, we compare with KD-CoT (Wang et al., 2023c), StructGPT (Jiang et al., 2023c), KAPING (Baek et al., 2023b).

heading: 4.4 Evaluation Metrics
text: For the MCQA task, we follow previous work (Park et al., 2023; Wang et al., 2023d) using Accuracy to assess the correctness of the prediction. For the KGQA task, we follow the same way as Jiang et al. 2023c to calculate Hits@1 which assesses whether the top-1 predicted answer is correct or not, since we use the greedy decoding strategy that generates only one answer.

heading: 5 Results and Analysis

heading: 5.1 Main Results
text: MCQA task. In Table 1, we report our KGAdapter and baseline performances in the MCQA task, including OBQA and CSQA datasets. Our KG-Adapter exhibits an average improvement of 40.8 and 47.5 points over the base models Llama27B and Zephyr-7B on two datasets. With only 27.9M parameters trained, our KG-Adapter outperforms all LLM baselines and KG+Full-Finetune methods, including previous SOTA method QAT, demonstrating the effectiveness of our method and the potential of LLM with well-designed PEFT architectures for KG-related tasks. In comparison to KG+Prompt-enhanced LLMs, KG-Adapter outperforms all methods even though their base model (ChatGPT) is much stronger than ours (Zephyr7B). Furthermore, these methods over-rely on super LLMs and achieve limited enhancements on small-scale LLMs, whereas our method is effective on the 7B models as well.
KGQA task. Unlike the MCQA task, the KGQA task is open-ended QA with no options, so the KG subgraphs may contain extensive invalid information unrelated to the correct answers, to address that, many previous methods use a twostage retrieval-then-generation pipeline. However, retrieval is not the focus of our study, the KG subgraphs we used contain more noise than the work focus on retrieval (marked with † in Table 2), putting our approach at a disadvantage in comparisons.
We present the results of both WQSP and CWQ datasets in Table 2. Compared to our base models Llama2-7B and Zephyr-7B, our method achieves an average of 32.8 and 16.7 points improvement over the two datasets. Compared with KG+FullFinetune methods, KG-Adapter outperforms all methods that don’t use retrieval and achieves comparable results to methods using retrieval. Except for UniKGQA, which first pre-trains a model and
then fine-tunes it to get both a retrieval model and a generation model for retrieval-then-generation, whereas our method only performs end-to-end training with 25.7M learnable parameters and no pre-training or retrieval is used. For the LLM-based approaches, our KG-Adapter outperforms all LLM baselines and achieves a close performance to the
KG+Prompt-enhanced LLMs, even though all of them have stronger base models (e.g., ChatGPT) than ours and all use retrieval.

heading: 5.2 Ablation Studies
text: Impact of KG-Adapter components. We perform ablation experiments to analyze the effec-
tiveness of each module, including (1) w/o SEHI, where we remove the SEHI and use the LLM’s word embedding as the KG representations; (2) w/o GNN, where we remove the GNN which updates the KG representations from node-centered; (3) w/o MLPtrip, where we remove the MLPtrip which is used to get the relation-centered KG representations.
As shown in Table 3, removing any module from KG-Adapter hurts performance, illustrating the positive effects of each module. KG-Adapter w/o MLPtrip and w/o GNN cause the greatest decrease in performance on Zephyr-7B and Llama2-7B, respectively, suggesting that both node-centered and relation-centered KG encoding helps the models to utilize KG information better. When w/o SEHI, the decrease in performance is minimal, suggesting that the word embedding of LLM contains most of the information in the KG embedding, with the size and pre-trained corpus of LLM growing, training a KG embedding may be unnecessary. In addition, we observe that the effectiveness of different modules varies across different base models, suggesting that the kind of base model affects the effectiveness of our approach because they have different capabilities and parameter knowledge and therefore may have different mechanisms for understanding KG.
Impact of using KG. To test the effect of integrating KG, we also experiment with (1) w/o KG, where we input an empty KG containing two PAD nodes and a PAD edge; (2) w/ LoRa, where we train a LoRa (Hu et al., 2021) using the same number of learnable parameters and data as KGAdapter.
The results are shown in Table 3. Without using KG, our performance is similar to the w/ LoRa, suggesting that our method can also serve as a general PEFT approach beyond injecting KG. Compared to KG-Adapter (w/ KG), the use of KG leads to 1.6%
↑ on average across two models, indicating the effectiveness of injecting KG. Surprisingly, a simple LoRa can also yield impressive improvement, showing the potential of LLM with well-designed PEFT methods, since the LLMs may have acquired most of the knowledge in KG as parameterized knowledge during pre-training, and may generate correct answers without external knowledge due to their strong reasoning ability.
Impact of the number of KG-Adapter layers. We verify the effect of the number of KG-Adapter layers. As shown in Figure 3 (a), we find that increasing the number of KG-Adapter layers results in a consistent improvement. This can be attributed to the increase of trainable parameters and more stacked layers can help encode the KG better.
Impact of the hidden size of KG-Adapter. In Figure 3 (b), we examine different hidden sizes of KG-Adapter. We observe a notable improvement when the hidden size increases from 16 to 64. Instead, the effect decreases slightly as the hide size continues to increase to 128. This indicates that a hidden size of 64 is sufficient to represent the main information of KG and a larger hidden size leads to slower convergence during training and may contain more noise.

heading: 5.3 Analysis
text: Applicability to LLMs with different sizes. We evaluate the generalizability of our KG-Adapter by applying it to LLMs with different numbers of parameters. As shown in Figure 4 (a), our KG-Adapter improves the performance of models ranging from 3B to 13B in size, proving our KGAdapter can apply to LLMs with different sizes and can inherit the capabilities of the base models that achieve better performance on a larger base model.
Applicability to LLMs at different stages. To further assess the generalizability of KG-Adapter, we select Llama-base, Llama-chat, Mistral-base, and Mistral-inst as the base models to test whether
our approach can be applied to LLMs at different stages (after SFT or not). As illustrated in Figure 4 (b), our method demonstrates applicability to different types of LLMs, regardless of whether they have undergone SFT or not. Furthermore, experimental results show that our approach does not lead to catastrophic forgetting even when the SFT models are trained three times (i.e., pre-training, SFT, fine-tuning).
Case Study. We conduct a case study to analyze how our KG-Adapter leverages the structure information in KG and mitigates the knowledge conflict problem. We select some representative nodes and their corresponding edges from the whole KG subgraph for visualization, and the weights on edges are obtained by summing the attention weights in each KG adapter layer and performing normalization. We compare our KG-Adapter with two types of vanilla approaches: (1) baseline, which is the base model of our KG-Adapter, and (2) base-
line+KG, in which the linearized KG is added in the context as model input.
As shown in the upper of Figure 5, our approach starts from the key entity "Overpopulation" in the question, and through 2-hop reasoning, successfully finds the correct entity "lower life", which has the highest weight on edge. Conversely, the baseline methods generate an incorrect answer, regardless of the addition of KG, which illustrates incorporating KG through prompts may lead to knowledge conflicts and the lack of KG structure leads to underutilization of KG information. Similarly, in the figure below, our approach reaches the correct answer entity “Cello” through the path with the highest weight from the two question entities, showing the interpretability of our method. However, the baseline method response is “No Correct Answer" due to the lack of additional knowledge, and the baseline+KG gives an answer but is wrong, suggesting that simply inputting a linearized KG is difficult to understand by the model and loses structural information in the KG.

heading: 6 Conclusion
text: To enable LLMs to directly access structured KGs and overcome the limitations of current prompt-based approaches, i.e., the inability to utilize the structural information of KGs, the problem of knowledge conflicts, and the over-reliance on super LLMs. We propose a novel PEFT-based adapter structure, named KG-Adapter, which is a plug-andplay adapter module applicable to various LLMs and requires only 28M parameters to train. Benefiting from the internal GNN layer and MLPtrip, KGAdapter can encode KGs from both node-centered and relation-centered perspectives to fully utilize the structure information in KGs. Experiments on four datasets across two tasks show that our KGAdapter achieves superior performance over previous full-parameter fine-tuned SOTA methods and is comparable to the recent prompt-based ChatGPT methods.
Limitations
Due to the limitation of computational resources, our main experiments were performed only on the models with 7B parameters, and based on the experiments in Section 5.3, we believe that our method can achieve better performance on larger models (e.g., 70B). For the same reason, we do not compare with the full-parameter fine-tuned LLMs.
Many studies have indicated that a good KG retrieval method can effectively improve the quality of the final generated responses (as shown in Table 2, the methods with retrieval usually get better results). However, retrieval is not the focus of our work, so we do not use complex retrieval methods, which means that our approach can be further improved by combining with good retrieval methods.
We find that existing KGQA datasets are not entirely suitable for LLMs; they are either too easy for LLMs, which can generate answers well without relying on KG knowledge, or too dependent on retrieval, which makes the input KG contain much noise, making it hard to independently test the ability of LLMs to comprehend and utilize structured knowledge.
Ethical Considerations
Our study focuses on integrating KGs into LLMs through parameter-efficient fine-tuning, aiming to alleviate the issues of losing KG structure information in prompt-based methods. All models, datasets and KGs used in this paper are open-source and publicly available. Based on this, we believe that our research does not compromise data security or personal privacy. Furthermore, our study is beneficial in reducing the generation of false information by LLMs. We are confident that our research not only avoids causing harm to society but also contributes to societal well-being.

heading: Acknowledgements
text: We would like to thank anonymous reviewers for their suggestions and comments sincerely. The work was partially supported by the National Natural Science Foundation of China (NSFC62076032).

heading: 2022 Conference on Empirical Methods in Natural
text: Language Processing, pages 11118–11131.
Xingxuan Li, Ruochen Zhao, Yew Ken Chia, Bosheng Ding, Lidong Bing, Shafiq Joty, and Soujanya Poria. 2023b. Chain of knowledge: A framework for grounding large language models with structured knowledge bases. arXiv preprint arXiv:2305.13269.
Yuhan Li, Zhixun Li, Peisong Wang, Jia Li, Xiangguo Sun, Hong Cheng, and Jeffrey Xu Yu. 2023c. A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311.12399.
Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303.15647.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019a. Kagnet: Knowledge-aware graph networks for commonsense reasoning. arXiv preprint arXiv:1909.02151.
Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019b. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829–2839.
Biyang Liu, Huimin Yu, and Guodong Qi. 2022. Graftnet: Towards domain generalized stereo matching with a broad-spectrum and task-oriented feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13012– 13021.
Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2901–2908.
Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics.
Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310.01061.
Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391.
Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018. Never-ending learning. Communications of the ACM, 61(5):103–115.
OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov,
Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report.
Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2023. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306.08302.
Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeonjin Park, Ji-Hoon Kim, Jisu Jeong, Kyungmin Kim, and Hyunwoo Kim. 2023. Relation-aware languagegraph transformer for question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13457–13464.
Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302.12813.
Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequence knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2814–2828.
Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over
knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4498– 4507.
Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302.04761.
Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020. Exploiting structured knowledge in text via graph-guided representation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8980–8994.
Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching.
Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5.5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31.
Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380–2390.
Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307.07697.
Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge embedding. arXiv preprint arXiv:2010.00309.
Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. 2021. Jointlk: Joint reasoning with language models and knowledge graphs for commonsense question answering. arXiv preprint arXiv:2112.02732.
Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991–13005.
Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651.
Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https:// github.com/tatsu-lab/stanford_alpaca.
Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V Chawla, and Panpan Xu. 2024. Graph neural prompting with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19080–19088.
Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307.09288.
Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023a. Boosting language models reasoning with chain-of-knowledge prompting. arXiv preprint arXiv:2306.06427.
Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023b. Boosting language models reasoning with chain-of-knowledge prompting.
Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational graph attention network for aspect-based sentiment analysis. arXiv preprint arXiv:2004.12362.
Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023c. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledgeintensive question answering. arXiv preprint arXiv:2308.13259.
Kuan Wang, Yuyu Zhang, Diyi Yang, Le Song, and Tao Qin. 2021. Gnn is a counter? revisiting gnn for question answering. In International Conference on Learning Representations.
Yujie Wang, Hu Zhang, Jiye Liang, and Ru Li. 2023d. Dynamic heterogeneous-graph reasoning with language models and knowledge representation learning for commonsense question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14048–14063.
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837.
Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309.07864.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022a. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models.
Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022b. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 602–631.
Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In International Conference on Learning Representations.
Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qagnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2104.06378.
Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3.5 series models. arXiv preprint arXiv:2303.10420.
Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206.
Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1– 37.
Jing Zhang, Bo Chen, Lingxi Zhang, Xirui Ke, and Haipeng Ding. 2021. Neural, symbolic and neuralsymbolic reasoning on knowledge graphs. AI Open, 2:14–35.
Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022a. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5773– 5784.
Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning,
and Jure Leskovec. 2022b. Greaselm: Graph reasoning enhanced language models for question answering. arXiv preprint arXiv:2201.08860.
Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441– 1451.
Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022c. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations.
Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544–14556, Singapore. Association for Computational Linguistics.

heading: A Appendix
text: A.1 Preliminary In this section, we introduce the definition of the knowledge graph and formally define the problem of multiple-choice question answering (MCQA) and knowledge graph-based question answering (KGQA).
Definition 1. Knowledge Graph. A knowledge graph is defined as a set of triples G = {T1, T2, ..., Tn}, each T is a triple T = (h, r, t), where h and t are the head entity and the tail entity, and r is a directed edge from h to t.
Problem 1. Multiple-Choice Question Answering. Given a question Q, a set of answer options A = {a1, a2, ..., ak}, and an optional context C depending on open-book or close-book, the task is to select the best answer. We additionally use knowledge graphs to provide external knowledge, therefore the task can be defined as a∗ = argmaxθ fθ(Q,C,G,A), where a∗ ∈ A is the correct option and fθ is the model with parameter θ.
Problem 2. Knowledge Graph-Based Question Answering. Given a question Q and a knowledge graph G, the target is to generate the expected result to answer the question Q based on the knowledge from G, which is defined as a∗ = argmaxθ fθ(Q,G), where a∗ is an open answer.
A.2 Statistics of Datasets A.3 Detail of KG Processing
Subgraph Retrieval. In the MCQA and KGQA tasks, the general process is first to retrieve a sub-
graph related to the question from the full KG, and then input that subgraph to the model with the question. Specifically, the retrieval process is to (1) find all paths from each question entity to each option entity (if have) within n-hop; (2) then calculate the relevant scores of each node (entity) and question by a pre-trained LM, such as Sentence-BERT; (3) only retain the top-k scoring nodes and build a subgraph.
In our work, we do not focus on the retrieval process, so we directly use the retrieved subgraphs from previous works. Specifically, for the MCQA task, we use the processed KG data by Park et al. (2023), which provides a sub-graph of KG for each sample. It merges the edge types in ConceptNet from 31 to 17, and introduces two additional edges of global nodes "G → q" and "G → a", and finally, we add the reverse edge to each edge to obtain 38 edge types (relations). For the KGQA task, we use the processed KG-QA pair data by Xie et al. (2022a) that can be downloaded here 3, where contain 772 and 801 relation types for WQSP and CWQ datasets, respectively.
KG Embeddings. KG embeddings are the same as word embeddings which provide a vector representation for each node/entity in the KG. A simple way to get KG embeddings is first using templates to convert the triples into sentences and feed them into some LMs (such as BERT-Large), obtaining a sequence of token embeddings from the last layer and performing mean pooling to get the embedding for an entity. Another approach is using special algorithms (e.g., TransE) to train KG embeddings on KGs.
For the MCQA task, we use the KG embeddings from Park et al. (2023). For the KGQA task, we do not use pre-trained KG embeddings since the FreeBase is a super large KG (over 100GB), so there are no open-source KG embeddings.
3https://drive.google.com/drive/folders/ 1GXigUv3MU-Sh4XiY6Wz3xVeNT_s0SuON
A.4 Detail of Training We use the same way to initialize the parameters in KG-Adapter as Sung et al. (2022), which first calculates the importance score of each weight vector in LLMs and chooses the n rows with top-n importance scores as the initial parameters of new modules.
The inputs and outputs examples during training are shown in Table 6. For different LLMs, we use different templates. For different tasks, we design two types of system instructions (upper for MCQA task and below for KGQA task). We only calculate the loss on answer tokens, such as "(B) fecal matter
" in Table 6.
Since the dataset for the KGQA task (WQSP and CWQ) does not have pre-trained KG embedding, the implementation of the Sub-word to Entity Hybrid Initialization on the KGQA task is different in which it only contains the KG representations initialized from the word embedding of LLM.
A.5 Detail of Hyperparameter We list all hyperparameters of different datasets in Table 5. We do not change any hyperparameters of the LLM.

heading: Future Work
text: A survey of graph meets large language model: Progress and future directions. arXiv preprint arXiv:2311. 12399. Vladislav Lialin, Vijeta Deshpande, and Anna Rumshisky. 2023. Scaling down to scale up: A guide to parameter-efficient fine-tuning. arXiv preprint arXiv:2303. 15647. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019a. Kagnet: Knowledge-aware graph networks for commonsense reasoning. arXiv preprint arXiv:1909. 02151. Bill Yuchen Lin, Xinyue Chen, Jamin Chen, and Xiang Ren. 2019b. Kagnet: Knowledge-aware graph networks for commonsense reasoning. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2829–2839. Biyang Liu, Huimin Yu, and Guodong Qi. 2022. Graftnet: Towards domain generalized stereo matching with a broad-spectrum and task-oriented feature. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 13012– 13021. Weijie Liu, Peng Zhou, Zhe Zhao, Zhiruo Wang, Qi Ju, Haotang Deng, and Ping Wang. 2020. K-bert: Enabling language representation with knowledge graph. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 2901–2908. Shayne Longpre, Kartik Perisetla, Anthony Chen, Nikhil Ramesh, Chris DuBois, and Sameer Singh. 2021. Entity-based knowledge conflicts in question answering. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, pages 7052–7063, Online and Punta Cana, Dominican Republic. Association for Computational Linguistics. Linhao Luo, Yuan-Fang Li, Gholamreza Haffari, and Shirui Pan. 2023. Reasoning on graphs: Faithful and interpretable large language model reasoning. arXiv preprint arXiv:2310. 01061. Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. 2018. Can a suit of armor conduct electricity? a new dataset for open book question answering. In Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pages 2381–2391. Tom Mitchell, William Cohen, Estevam Hruschka, Partha Talukdar, Bishan Yang, Justin Betteridge, Andrew Carlson, Bhavana Dalvi, Matt Gardner, Bryan Kisiel, et al. 2018. Never-ending learning. Communications of the ACM, 61(5):103–115. OpenAI, :, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, Red Avila, Igor Babuschkin, Suchir Balaji, Valerie Balcom, Paul Baltescu, Haiming Bao, Mo Bavarian, Jeff Belgum, Irwan Bello, Jake Berdine, Gabriel Bernadett-Shapiro, Christopher Berner, Lenny Bogdonoff, Oleg Boiko, Madelaine Boyd, Anna-Luisa Brakman, Greg Brockman, Tim Brooks, Miles Brundage, Kevin Button, Trevor Cai, Rosie Campbell, Andrew Cann, Brittany Carey, Chelsea Carlson, Rory Carmichael, Brooke Chan, Che Chang, Fotis Chantzis, Derek Chen, Sully Chen, Ruby Chen, Jason Chen, Mark Chen, Ben Chess, Chester Cho, Casey Chu, Hyung Won Chung, Dave Cummings, Jeremiah Currier, Yunxing Dai, Cory Decareaux, Thomas Degry, Noah Deutsch, Damien Deville, Arka Dhar, David Dohan, Steve Dowling, Sheila Dunning, Adrien Ecoffet, Atty Eleti, Tyna Eloundou, David Farhi, Liam Fedus, Niko Felix, Simón Posada Fishman, Juston Forte, Isabella Fulford, Leo Gao, Elie Georges, Christian Gibson, Vik Goel, Tarun Gogineni, Gabriel Goh, Rapha Gontijo-Lopes, Jonathan Gordon, Morgan Grafstein, Scott Gray, Ryan Greene, Joshua Gross, Shixiang Shane Gu, Yufei Guo, Chris Hallacy, Jesse Han, Jeff Harris, Yuchen He, Mike Heaton, Johannes Heidecke, Chris Hesse, Alan Hickey, Wade Hickey, Peter Hoeschele, Brandon Houghton, Kenny Hsu, Shengli Hu, Xin Hu, Joost Huizinga, Shantanu Jain, Shawn Jain, Joanne Jang, Angela Jiang, Roger Jiang, Haozhun Jin, Denny Jin, Shino Jomoto, Billie Jonn, Heewoo Jun, Tomer Kaftan, Łukasz Kaiser, Ali Kamali, Ingmar Kanitscheider, Nitish Shirish Keskar, Tabarak Khan, Logan Kilpatrick, Jong Wook Kim, Christina Kim, Yongjik Kim, Hendrik Kirchner, Jamie Kiros, Matt Knight, Daniel Kokotajlo, Łukasz Kondraciuk, Andrew Kondrich, Aris Konstantinidis, Kyle Kosic, Gretchen Krueger, Vishal Kuo, Michael Lampe, Ikai Lan, Teddy Lee, Jan Leike, Jade Leung, Daniel Levy, Chak Ming Li, Rachel Lim, Molly Lin, Stephanie Lin, Mateusz Litwin, Theresa Lopez, Ryan Lowe, Patricia Lue, Anna Makanju, Kim Malfacini, Sam Manning, Todor Markov, Yaniv Markovski, Bianca Martin, Katie Mayer, Andrew Mayne, Bob McGrew, Scott Mayer McKinney, Christine McLeavey, Paul McMillan, Jake McNeil, David Medina, Aalok Mehta, Jacob Menick, Luke Metz, Andrey Mishchenko, Pamela Mishkin, Vinnie Monaco, Evan Morikawa, Daniel Mossing, Tong Mu, Mira Murati, Oleg Murk, David Mély, Ashvin Nair, Reiichiro Nakano, Rajeev Nayak, Arvind Neelakantan, Richard Ngo, Hyeonwoo Noh, Long Ouyang, Cullen O’Keefe, Jakub Pachocki, Alex Paino, Joe Palermo, Ashley Pantuliano, Giambattista Parascandolo, Joel Parish, Emy Parparita, Alex Passos, Mikhail Pavlov, Andrew Peng, Adam Perelman, Filipe de Avila Belbute Peres, Michael Petrov,
Henrique Ponde de Oliveira Pinto, Michael, Pokorny, Michelle Pokrass, Vitchyr Pong, Tolly Powell, Alethea Power, Boris Power, Elizabeth Proehl, Raul Puri, Alec Radford, Jack Rae, Aditya Ramesh, Cameron Raymond, Francis Real, Kendra Rimbach, Carl Ross, Bob Rotsted, Henri Roussez, Nick Ryder, Mario Saltarelli, Ted Sanders, Shibani Santurkar, Girish Sastry, Heather Schmidt, David Schnurr, John Schulman, Daniel Selsam, Kyla Sheppard, Toki Sherbakov, Jessica Shieh, Sarah Shoker, Pranav Shyam, Szymon Sidor, Eric Sigler, Maddie Simens, Jordan Sitkin, Katarina Slama, Ian Sohl, Benjamin Sokolowsky, Yang Song, Natalie Staudacher, Felipe Petroski Such, Natalie Summers, Ilya Sutskever, Jie Tang, Nikolas Tezak, Madeleine Thompson, Phil Tillet, Amin Tootoonchian, Elizabeth Tseng, Preston Tuggle, Nick Turley, Jerry Tworek, Juan Felipe Cerón Uribe, Andrea Vallone, Arun Vijayvergiya, Chelsea Voss, Carroll Wainwright, Justin Jay Wang, Alvin Wang, Ben Wang, Jonathan Ward, Jason Wei, CJ Weinmann, Akila Welihinda, Peter Welinder, Jiayi Weng, Lilian Weng, Matt Wiethoff, Dave Willner, Clemens Winter, Samuel Wolrich, Hannah Wong, Lauren Workman, Sherwin Wu, Jeff Wu, Michael Wu, Kai Xiao, Tao Xu, Sarah Yoo, Kevin Yu, Qiming Yuan, Wojciech Zaremba, Rowan Zellers, Chong Zhang, Marvin Zhang, Shengjia Zhao, Tianhao Zheng, Juntang Zhuang, William Zhuk, and Barret Zoph. 2023. Gpt-4 technical report. Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. 2023. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306. 08302. Jinyoung Park, Hyeong Kyu Choi, Juyeon Ko, Hyeonjin Park, Ji-Hoon Kim, Jisu Jeong, Kyungmin Kim, and Hyunwoo Kim. 2023. Relation-aware languagegraph transformer for question answering. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 37, pages 13457–13464. Baolin Peng, Michel Galley, Pengcheng He, Hao Cheng, Yujia Xie, Yu Hu, Qiuyuan Huang, Lars Liden, Zhou Yu, Weizhu Chen, et al. 2023. Check your facts and try again: Improving large language models with external knowledge and automated feedback. arXiv preprint arXiv:2302. 12813. Amrita Saha, Vardaan Pahuja, Mitesh Khapra, Karthik Sankaranarayanan, and Sarath Chandar. 2018. Complex sequential question answering: Towards learning to converse over linked question answer pairs with a knowledge graph. In Proceedings of the AAAI conference on artificial intelligence, volume 32. Apoorv Saxena, Adrian Kochsiek, and Rainer Gemulla. 2022. Sequence-to-sequence knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2814–2828. Apoorv Saxena, Aditay Tripathi, and Partha Talukdar. 2020. Improving multi-hop question answering over
knowledge graphs using knowledge base embeddings. In Proceedings of the 58th annual meeting of the association for computational linguistics, pages 4498– 4507. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. Toolformer: Language models can teach themselves to use tools. arXiv preprint arXiv:2302. 04761. Tao Shen, Yi Mao, Pengcheng He, Guodong Long, Adam Trischler, and Weizhu Chen. 2020. Exploiting structured knowledge in text via graph-guided representation learning. In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 8980–8994. Yusuxke Shibata, Takuya Kida, Shuichi Fukamachi, Masayuki Takeda, Ayumi Shinohara, Takeshi Shinohara, and Setsuo Arikawa. 1999. Byte pair encoding: A text compression scheme that accelerates pattern matching. Robyn Speer, Joshua Chin, and Catherine Havasi. 2017. Conceptnet 5. 5: An open multilingual graph of general knowledge. In Proceedings of the AAAI conference on artificial intelligence, volume 31. Haitian Sun, Tania Bedrax-Weiss, and William Cohen. 2019. Pullnet: Open domain question answering with iterative retrieval on knowledge bases and text. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCNLP), pages 2380–2390. Jiashuo Sun, Chengjin Xu, Lumingyuan Tang, Saizhuo Wang, Chen Lin, Yeyun Gong, Heung-Yeung Shum, and Jian Guo. 2023. Think-on-graph: Deep and responsible reasoning of large language model with knowledge graph. arXiv preprint arXiv:2307. 07697. Tianxiang Sun, Yunfan Shao, Xipeng Qiu, Qipeng Guo, Yaru Hu, Xuanjing Huang, and Zheng Zhang. 2020. Colake: Contextualized language and knowledge embedding. arXiv preprint arXiv:2010. 00309. Yueqing Sun, Qi Shi, Le Qi, and Yu Zhang. 2021. Jointlk: Joint reasoning with language models and knowledge graphs for commonsense question answering. arXiv preprint arXiv:2112. 02732. Yi-Lin Sung, Jaemin Cho, and Mohit Bansal. 2022. Lst: Ladder side-tuning for parameter and memory efficient transfer learning. Advances in Neural Information Processing Systems, 35:12991–13005. Alon Talmor and Jonathan Berant. 2018. The web as a knowledge-base for answering complex questions. In Proceedings of the 2018 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long Papers), pages 641–651. Rohan Taori, Ishaan Gulrajani, Tianyi Zhang, Yann Dubois, Xuechen Li, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. 2023. Stanford alpaca: An instruction-following llama model. https:// github. com/tatsu-lab/stanford_alpaca. Yijun Tian, Huan Song, Zichen Wang, Haozhu Wang, Ziqing Hu, Fang Wang, Nitesh V Chawla, and Panpan Xu. 2024. Graph neural prompting with large language models. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 19080–19088. Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. 2023. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint arXiv:2307. 09288. Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023a. Boosting language models reasoning with chain-of-knowledge prompting. arXiv preprint arXiv:2306. 06427. Jianing Wang, Qiushi Sun, Nuo Chen, Xiang Li, and Ming Gao. 2023b. Boosting language models reasoning with chain-of-knowledge prompting. Kai Wang, Weizhou Shen, Yunyi Yang, Xiaojun Quan, and Rui Wang. 2020. Relational graph attention network for aspect-based sentiment analysis. arXiv preprint arXiv:2004. 12362. Keheng Wang, Feiyu Duan, Sirui Wang, Peiguang Li, Yunsen Xian, Chuantao Yin, Wenge Rong, and Zhang Xiong. 2023c. Knowledge-driven cot: Exploring faithful reasoning in llms for knowledgeintensive question answering. arXiv preprint arXiv:2308. 13259. Kuan Wang, Yuyu Zhang, Diyi Yang, Le Song, and Tao Qin. 2021. Gnn is a counter? revisiting gnn for question answering. In International Conference on Learning Representations. Yujie Wang, Hu Zhang, Jiye Liang, and Ru Li. 2023d. Dynamic heterogeneous-graph reasoning with language models and knowledge representation learning for commonsense question answering. In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14048–14063. Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Fei Xia, Ed Chi, Quoc V Le, Denny Zhou, et al. 2022. Chain-of-thought prompting elicits reasoning in large language models. Advances in Neural Information Processing Systems, 35:24824–24837. Zhiheng Xi, Wenxiang Chen, Xin Guo, Wei He, Yiwen Ding, Boyang Hong, Ming Zhang, Junzhe Wang, Senjie Jin, Enyu Zhou, et al. 2023. The rise and potential of large language model based agents: A survey. arXiv preprint arXiv:2309. 07864. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I. Wang, Victor Zhong, Bailin Wang, Chengzu Li, Connor Boyle, Ansong Ni, Ziyu Yao, Dragomir Radev, Caiming Xiong, Lingpeng Kong, Rui Zhang, Noah A. Smith, Luke Zettlemoyer, and Tao Yu. 2022a. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. Tianbao Xie, Chen Henry Wu, Peng Shi, Ruiqi Zhong, Torsten Scholak, Michihiro Yasunaga, Chien-Sheng Wu, Ming Zhong, Pengcheng Yin, Sida I Wang, et al. 2022b. Unifiedskg: Unifying and multi-tasking structured knowledge grounding with text-to-text language models. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 602–631. Wenhan Xiong, Jingfei Du, William Yang Wang, and Veselin Stoyanov. 2019. Pretrained encyclopedia: Weakly supervised knowledge-pretrained language model. In International Conference on Learning Representations. Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. 2021. Qagnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2104. 06378. Junjie Ye, Xuanting Chen, Nuo Xu, Can Zu, Zekai Shao, Shichun Liu, Yuhan Cui, Zeyang Zhou, Chao Gong, Yang Shen, et al. 2023. A comprehensive capability analysis of gpt-3 and gpt-3. 5 series models. arXiv preprint arXiv:2303. 10420. Wen-tau Yih, Matthew Richardson, Christopher Meek, Ming-Wei Chang, and Jina Suh. 2016. The value of semantic parse labeling for knowledge base question answering. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 201–206. Hanqing Zhang, Haolin Song, Shaoyu Li, Ming Zhou, and Dawei Song. 2023. A survey of controllable text generation using transformer-based pre-trained language models. ACM Computing Surveys, 56(3):1– 37. Jing Zhang, Bo Chen, Lingxi Zhang, Xirui Ke, and Haipeng Ding. 2021. Neural, symbolic and neuralsymbolic reasoning on knowledge graphs. AI Open, 2:14–35. Jing Zhang, Xiaokang Zhang, Jifan Yu, Jian Tang, Jie Tang, Cuiping Li, and Hong Chen. 2022a. Subgraph retrieval enhanced model for multi-hop knowledge base question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 5773– 5784. Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D Manning,
and Jure Leskovec. 2022b. Greaselm: Graph reasoning enhanced language models for question answering. arXiv preprint arXiv:2201. 08860. Zhengyan Zhang, Xu Han, Zhiyuan Liu, Xin Jiang, Maosong Sun, and Qun Liu. 2019. Ernie: Enhanced language representation with informative entities. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics, pages 1441– 1451. Zhuosheng Zhang, Aston Zhang, Mu Li, and Alex Smola. 2022c. Automatic chain of thought prompting in large language models. In The Eleventh International Conference on Learning Representations. Wenxuan Zhou, Sheng Zhang, Hoifung Poon, and Muhao Chen. 2023. Context-faithful prompting for large language models. In Findings of the Association for Computational Linguistics: EMNLP 2023, pages 14544–14556, Singapore. Association for Computational Linguistics