Making Large Language Models Perform Better in Knowledge Graph Completion

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-07 15:03:15

Abstract: Large language model (LLM) based knowledge graph completion (KGC) aims to predict the missing triples in the KGs with LLMs. However, research about LLM-based KGC fails to sufficiently harness LLMs' inference proficiencies, overlooking critical structural information integral to KGs. In this paper, we explore methods to incorporate structural information into the LLMs, with the overarching goal of facilitating structure-aware reasoning. We first discuss on the existing LLM paradigms like in-context learning and instruction tuning, proposing basic structural information injection approaches. Then we propose a Kno}wledge Prefix Adapter (KoPA) to fulfill this stated goal. The KoPA uses a structural pre-training phase to comprehend the intricate entities and relations within KGs, representing them as structural embeddings. Then KoPA communicates such **cross-modal structural information understanding to the LLMs** through a knowledge prefix adapter which projects the structural embeddings into the textual space and obtains virtual knowledge tokens positioned as a prefix of the input prompt. We conduct comprehensive experiments and provide incisive analysis concerning how the introduction of cross-modal structural information would be better for LLM's factual knowledge reasoning ability. Our code and data are available at https://anonymous.4open.science/r/KoPA-3415.

Main paper: 
 related works 2.1 knowledge graph completion: 
 Knowledge graph completion (KGC) [37] is an important topic in the KG community, aiming to mine unobserved triples in a given KG. KGC contains several sub-tasks such as triple classification [3] , entity prediction [3] . The common point among KGC tasks is to establish an effective mechanism to measure the plausibility of the triples. The mainstream KGC methods can be divided into two categories: embedding-based and PLM-based methods. Embeddingbased methods [3, 31, 34, 38] are designed to embed the entities and relations of KGs into continuous representation spaces. These approaches make full use of structural information from the KGs to model triple plausibility with a well-designed score function and learn the entity/relation embeddings in a self-supervised manner. Moreover, PLM-based methods consider KGC as text-based tasks by fine-tuning pre-trained language models [8] . The short textual descriptions are organized as an input sequence and encoded by the PLMs. KG-BERT [40] is the first PLM-based method that models KGC as a binary text classification task. Subsequent works like MTL-KGC [16] and StAR [35] have further improved KG-BERT by introducing more training tasks such as relation classification and triple ranking and more complex triple encoding strategy. PKGC [21] utilizes manual prompt templates to capture the triple semantic.Other methods like KGT5 [5, 29] make a step on the generative KGC [43] in a sequence-to-sequence paradigm with encoder-decoder PLMs like T5 [27] . PLM-based methods leverage the power of PLM but make the training process into text-based learning, which is difficult to capture complex structure information in the KGs.In recent years, large language models (LLMs) [23, 33, 46] have made rapid progress and demonstrated powerful capabilities in a considerable number of text-related tasks [49] . LLMs are usually pre-trained in an auto-regressive manner with next word prediction task [4] and demonstrate strong capability on text comprehension and generation. Among the research topics of LLM, integrating LLM and KG [25] is a popular and important one. On the one hand, hallucination [39, 48] is widespread in LLMs which means LLMs are lack factual knowledge and not interpretable. KGs that store structured knowledge can mitigate such a phenomenon [10, 15, 26] by introducing factual knowledge into LLMs. On the other hand, LLMs can benefit KG-related tasks such as KGC [51, 52] , entity alignment [47] , and KGQA [1] by its powerful generation capability. KGs for LLMs (KG4LLM) and LLMs for KGs (LLM4KG) are both important research topics. We focus on applying LLMs in the KGC task (LLM4KGC), which has not been carefully studied yet. KGLLaMA [41] made the first step by vanilla instruction tuning approach but it lacks in-depth and systematic exploration about how to unleash the power of KGs themselves to make structureaware reasoning in LLMs and achieve better KGC performance. In this paper, we will dive into this problem from a more systematic perspective with the knowledge graph completion task.As LLMs demonstrate generalizable capabilities on text generation, many other works attempt to incorporate non-textual modality such as images [19, 50] , audio [22] , and video [22] , which are also called multi-modal LLMs [45] . These methods tend to encode non-textual information through the modality encoders and then process it as virtual text tokens. The non-textual tokens are aligned with the word tokens by instruction tuning on multi-modal datasets.The multi-modal LLM mentioned above usually excludes graph, which is another important data modality. There are also some works talking about how to incorporate graph data into LLMs. Drug-Chat [17] proposes to encode the drug molecule graphs with graph encoders and fine-tune the LLM to predict drug interactions. Other works [11, 18, 36, 44] explore how to solve graph learning tasks like node classification and graph classification by convert the graph structure information into LLMs.Our research is relative to this topic as KGs also have complex graph structures on top of the text descriptions. In this paper, we will explore how to incorporate complex structural information in the KGs into the LLMs to achieve better reasoning capabilities on knowledge graph completion. 
basic settings for llm-based kgc 3.1 notations and preliminaries: 
 A KG can be denoted as G = (E, R, T , D) where E, R are the entity set, relation set respectively. T = {(ℎ, 𝑟, 𝑡) | ℎ, 𝑡 ∈ E, 𝑟 ∈ R} is the triple set and D is the description set of each entity and relation. We denote D (𝑒), D (𝑟 ) as the short textual description of each entity 𝑒 ∈ E and each relation 𝑟 ∈ R. For example, the text description of the entity '/m/0ctzf1' is D('/m/0ctzf1')='The Transformers'. When applying LLMs to KGC tasks, we denote a LLM as M that serves as a text decoder. The input textual sequence S of the model M consists of several parts: the instruction prompt I, the triple prompt X, and the optional auxiliary demonstration prompt U. The instruction prompt I is the manually prepared instruction to guide the LLM M to execute the KGC task. The triple prompt X contains the textual information about the triples that need to be processed, which can be denoted as X(ℎ, 𝑟, 𝑡) = D (ℎ) ⊕ D (𝑟 ) ⊕ D (𝑡), where (ℎ, 𝑟, 𝑡) ∈ T is a triple and ⊕ denotes the textual token concatenation operation. In other words, the short descriptions of ℎ, 𝑟, 𝑡 would be applied as the input information. The auxiliary demonstration prompt U is an optional prompt for different settings. In the following, we will follow this set of notations.Meanwhile, we use triple classification as an entry point to investigate how to utilize LLM to accomplish the KGC task. Triple classification is a basic KGC task aiming to conduct binary classification tasks on the given triples. Whereas in the LLM paradigm, all tasks are converted into the form of text generation. Therefore, we desire the model M to answer true or false given the textual sequence input S = I ⊕ U ⊕ X.Triple classification is different from vanilla text classification because the entities and the relations in the prompt have complex semantic information defined by the given KG. Without knowledge of this type of information, the model response is unreliable and unstable. Despite the vast amount of commonsense knowledge that exists in the LLMs [48] , research has shown that large models are numb to fine-grained factual knowledge and will fall into a hallucination. Thus, incorporating the KG information into the prompt to provide more auxiliary information and guide the LLM to make structure-aware reasoning is the key to achieving excellent LLM-based KGC.In this section, we first discuss how to solve the KGC task with existing mainstream LLM paradigms called training-free reasoning approaches and instruction-tuning approaches.Training-free reasoning approaches prompt the LLMs to get direct answers without training. Common training-free methods consist of zero-shot reasoning (ZSR) and in-context learning (ICL). For ZSR, we directly utilize the sequence S 𝑧𝑠𝑟 = I ⊕ X as the input to get the prediction results. The decoding process of the LLM M can be formulated as:EQUATIONwhere A is the generated answer of the model M and I 𝑧𝑠𝑟 is the instruction template for ZSR. In the setting of ZSR, no KG information is added to the input sequence S 𝑧𝑠𝑟 . The determinative information in the ZSR prompt is only the textual descriptions of the test triple. ZSR is unable to incorporate KG information due to its setting limitations, otherwise, it cannot be called zero-shot.As another training-free paradigm, in-context learning (ICL) [9] allows the model M to add auxiliary demonstration U to the input S and accomplish the task in the form of analogical reasoning, which can be denoted as:EQUATIONAs for the triple classification task, the demonstration U should be some triples and their labels in the form of {(X 𝑖 , 𝑦 𝑖 ), 1 ≤ 𝑖 ≤ 𝑘 }, where X 𝑖 is the demonstration triple and 𝑦 𝑖 is the label. We denote the ICL with 𝑘 demonstrations as 𝑘-shot ICL.The demonstration triples can be randomly sampled from the existing training KG. However, to further incorporate the relative KG information of the test triple (ℎ, 𝑟, 𝑡), we propose to sample triples that are in the local structure of ℎ and 𝑡, which means one of the entities in each sampled triple should be ℎ or 𝑡. Besides, as existing KG only consists of positive triples, we employ negative sampling [21] to sample negative triples for demonstration. The number of positive and negative triples are the same for balanced predictions. In the demonstration prompt, the positive triples are labeled as true and the negative triples are labeled as false.By doing this, we incorporate the local structural information into the demonstration prompt U with both positive and negative samples. Such a structure-aware demonstration could better enhance the analogical reasoning process of the model M.Instruction tuning approaches fine-tune the LLMs with instruction template to activate the instruction following ability of LLMs. Vanilla instruction tuning leverages the input S 𝑖𝑡 to fine-tune LLMs. The instruction prompt I 𝑖𝑡 will describe the details of completing the triple classification task and the triple prompt X consists of the input triple. No other auxiliary demonstrations are included in the input template. To train the model M, the input sequence is organized as S 𝑖𝑡 = I 𝑖𝑡 ⊕ X ⊕ A 𝑖𝑡 , where A 𝑖𝑡 is the predicted answer of the training data. The model M is fine-tuned with the next word prediction task [49] which is a universal approach to training LLMs. The training objective can be formulated as:EQUATIONwhere negative sampling [21] is also applied to generate negative data samples as training KG only consists of positve triples.𝑠 𝑖 (𝑖 = 1,To incorporate semantic-rich KG information into LLMs, we also propose a structure-aware instruction tuning approach by adding the one-hop neighborhood structure information in the input prompt to inform the LLM with the local structural information. As mentioned before, the structural information of KG plays a significant role in the KGC tasks [37] . To incorporate such KG information during the fine-tuning stage, we achieve this goal by adding the neighborhood descriptions of the input triple. Specifically, we can sample the neighborhoods of the head ℎ and tail 𝑡 and put the textual descriptions of neighborhood triples in the demonstration prompt U 𝑖𝑡 . In this way, the input training sequence is enhanced asS 𝑖𝑡 = I 𝑖𝑡 ⊕ U 𝑖𝑡 ⊕ X ⊕ A 𝑖𝑡 .Therefore, we provide a detailed discussion of how the existing LLM paradigms can introduce local structural information about KGs to further enhance the model performance. However, though these approaches can work to some extent, they have obvious drawbacks. This textbf fundamental approaches to incorporate KG structural information focus on adding the neighborhood information to the input prompt in the text form. However, representing the KG structural information in text is not a good choice, which may bring in more invalid or redundant information to the prompt. It's not scalable and effective to increase prompt length indefinitely because a long context will lead to both a decline in model capability and high computational consumption. Besides, we also have difficulty finding the structural information in the KGs that is decisive for triple discrimination. These two problems put us in a dilemma. 
methodlogy: 
 To solve such issues, we propose the Knowledge Prefix Adapter (KoPA for short) to incorporate the KG structural information into LLM for KGC. Figure 2 presents an intuitive view of KoPA. Firstly we extract the structural information of entities and relations from the KG through structural embedding pre-training, and then we inform this structural information to LLM through a structural prefix adapter into the input sequence S. The LLM M is further fine-tuned with the structural-enhanced text sequence. We will discuss the details in the next few sections about our design.Instead of adding text about the neighborhood information into the input sequence, KoPA extracts the structural information of the entities and relations by self-supervised structural embedding pre-training. For each entity 𝑒 ∈ E and each relation 𝑟 ∈ R, we learn a structural embedding 𝒆 ∈ R 𝑑 𝑒 , 𝒓 ∈ R 𝑑 𝑟 respectively, where 𝑑 𝑒 , 𝑑 𝑟 are the embedding dimensions. We encode the KG structural information in the embeddings and further adapt them into the textual representation space of LLMs. Referring to the existing embedding-based KGC paradigm, we define a score function F (ℎ, 𝑟, 𝑡) to measure the plausibility of the triple (ℎ, 𝑟, 𝑡). We adopt the self-supervised pre-training objective by negative sampling [3] :EQUATIONwhere 𝛾 is the margin, 𝜎 is the sigmoid activation function and (ℎ ′ 𝑖 , 𝑟 ′ 𝑖 , 𝑡 ′ 𝑖 )(𝑖 = 1, 2, . . . , 𝐾) are 𝐾 negative samples [3] of (ℎ, 𝑟, 𝑡). The weight 𝑝 𝑖 is the self-adversarial weights proposed in [31] .By minimizing such a pre-training loss, the structural embeddings of each entity and relation are optimized to fit all its relative triples thus the KG structural information such as subgraph structure and relational patterns is captured in the embeddings. Such an approach has been proven effective in many embedding-based KGC methods [3, 31] to capture classic structural information like relational patterns and distributed entity representations [13] in the earliest days. After structural embedding pre-training, we could obtain the structural embeddings (𝒉, 𝒓, 𝒕) of a triple (ℎ, 𝑟, 𝑡) where the KG structural information is encoded in. However, the structural embeddings are learned in a different representation space against the textual token representation space of the LLM M, which means M can not directly understand these embeddings. Thus we apply a knowledge prefix adapter P to project them into the textual token representation space of M. Specifically speaking, the structural embeddings are converted to several virtual knowledge tokens K by P:EQUATIONIn practice, the adapter P would be a simple projection layer [50] . Then we put K in the front of the original input sequence S serving as a prefix of the instruction and triple prompt S 𝑘𝑝𝑎 = K ⊕ I 𝑖𝑡 ⊕ X. This way, all the following text tokens can be seen with the prefix K due to the unidirectional attention in decoderonly LLMs. By doing this, the textual tokens can pay unidirectional attention to the structural embeddings of the input triple. Such a structure-aware prompt will be employed during fine-tuning and inference. During training, we froze the pre-trained structural embeddings. The adapter is optimized to learn the mapping from structural knowledge toward textual representation and will have the generalization to new triples in the inference stage, which will benefit the textual description and provide the triple information from another perspective to make enhanced predictions.After proposing KoPA, we make a comparison among LLM-based KGC methods to demonstrate the advantages of KoPA, which is shown in Table 1 . Compared with the basic paradigms (ZSR/ICL/IT), KoPA incorporates the KG structural embeddings into LLM to combine the textual and structural information. Meanwhile, KoPA makes the length of the prompt more refined as the length of virtual tokens generated by the structural prefix adapter is fixed to 3 for head/relation/tail respectively. In contrast, the prompt length of structure-aware IT (enhanced IT in the table) is linearly related to the number of neighborhood triples 𝑘. In contrast to methods that incorporate structural information based on textual descriptions, KoPA achieves this goal by fixed-length virtual knowledge tokens generated by the adapter. In our experiments, we use three public KG benchmarks UMLS [40] , CoDeX-S [28] , and FB15K-237N [21] to evaluate the proposed LLM-based KGC methods. The detailed split information of the datasets is shown in Table 2 . 
baseline methods.: 
 In our experiments, we provide a comprehensive comparison with three broad classes of baselines on triple classification, which is an important subtask of KGC. The KGC baselines can be divided into three parts: embedding-based methods [3, 31, 34, 38] , PLM-based methods [21, 40] , and LLM-based methods [41] . Besides, we further divide the LLM-based methods into two categories: training-free methods and fine-tuning methods. Training-free methods consist of ZSR and ICL while fine-tuning methods consist of vanilla IT and structure-aware IT (enhanced IT). The specific models used for these baselines are listed below:(1). Embedding-based KGC methods. We select four traditional embedding-based KGC methods for comparisons, namely TransE [], DistMult [38] , ComplEx [34] , and RotatE [31] . These methods predict the triple plausibility by the learned structural embeddings and the score functions defined in the model.(2). PLM-based KGC methods. We select KG-BERT [40] and PKGC [21] as PLM-based KGC baselines, which are classic methods focusing on the triple classification task. These methods treat triple classification as a binary text classification task.(3). LLM-based KGC methods. LLM-based KGC research is still at an early stage. There are only KGLLaMA [41] to be the LLM-based KGC baseline. In addition to KGLLaMA, the methods proposed in Section 3 by us including ZSR, ICL, IT, and structureaware IT (enhanced IT) will also serve as baselines.Settings. We reproduce the baseline results and implement the KoPA proposed by us.For embedding-based KGC methods, we reproduce the results with OpenKE we set the embedding dimension 𝑑 𝑒 = 𝑑 𝑟 = 512 and sample 𝐾 = 32 negative samples during training. The margin 𝛾 is tuned among {0, 4, 6, 8, 12}. After training KGC models, we search for the best classification score threshold on the validation set for test data following the traditional setting [3] .For PLM-based methods, the backbone model for PLM-based KGC methods is BERT [8] . We fine-tune the KG-BERT according to the official code implementation. Since PKGC requires a lot of manual work to annotate each relation with a prompt, we only report the results of FB15K-237N shown in the original paper.For zero-shot reasoning, in addition to measuring with the same backbone Alpaca, we also test the performance of the GPT-3.5-turbor Table 3 : The main experiment results of triple classification. We report the accuracy (ACC), precision (P), recall (R), and F1-score (F1) results for each method on the three datasets. "-" means the result are missing because the specificity of PKGC makes it difficult to reproduce. The best Acc / F1 results in baselines are marked with underline, and we highlight our results with bold when we achieve new SOTA. which has 175B parameters. For the in-context learning method, we sample k-shot (k=1,2,4,8) structure-aware demonstrations. Besides, we sample 4 neighborhood triples for each triple to conduct structure-aware instruction tuning. For KoPA, we employ RotatE [31] and the score function of structural embedding pre-training and the embedding dimension is set to 512 and the adapter is a 512×4096 linear projection layer.For KoPA, we employ Alpaca-7B [32] as the LLM backbone. Alpaca is a famous extended version of LLaMA [33] model fine-tuned on instruction-following data. We reproduce the triple classification results of KGLLaMA [41] over two backbones (LLaMA and Alpaca) to avoid the effect of backbone choice on the results. We name the two baseline models KGLLaMA and KGAlpaca respectively. For all the fine-tuning methods (instruction tuning, structure-aware instruction tuning, and KoPA), we fine-tune Alpaca using LoRA [14] with rank 64. The number of epochs is searched in {3, 4, 5} and the learning rate is tuned in {1𝑒 -4 , 3𝑒 -4 , 5𝑒 -4 }. We use the AdamW optimizer [20] with a fixed batch size of 12. We conducted all the experiments with Nvidia A800 GPUs. The structural embedding pre-training process is efficient and only takes several minutes to finish. Therefore, the main time cost is caused by the LLM finetuning, which takes several hours for different datasets. (1 hour for UMLS, 4 hours for CoDeX-S, and 8 hours for FB15K-237N in our experimental environments).We evaluate the methods with the triple classification task [3] , which is essentially binary classification and all the test datasets are label-balanced. Therefore, we use accuracy, precision, recall, and F1-score as the evaluation metrics.The main experiment results of triple classification are shown in Table 3. Since precision and recall alone do not give a good response to the model's performance on the classification task, we focus on accuracy and F1-score. However, to provide a comprehensive analysis of different models, we also report the precision and recall results in the table. Overall, we can find that KoPA achieves outperforming accuracy and F1 results compared with the existing 16 baseline models on all three datasets. Taking CoDeX-S as an example, KoPA achieves 1.81% improvement in accuracy and 1.85% improvement on F1. As we use the pre-trained RotatE embeddings in KoPA, we can observe that KoPA significantly outperforms the original embedding-based RotatE method, especially on larger and more challenging datasets like CoDeX-S and FB15K-237N.Meanwhile, compared with all LLM-based approaches, we can see that the LLMs cannot understand the KG structural information well without fine-tuning. The zero-shot LLMs perform very poorly in the triple classification task even though GPT-3.5-turbo (175B parameters) has excellent capability. Though the demonstrations provided by ICL can incorporate the KG information, the performance gain is limited. Besides, the prediction results of training-free methods are biased and easy to slip into the extremes of all-right or all-wrong, as the recall of them is either very high or very low but the F1 scores are relatively low all the time.However, fine-tuning LLMs can introduce the KG information into LLMs as the overall performance makes obvious improvements. Meanwhile, though structure-aware IT enhances the input prompt with neighborhood information of triples, its performance is also limited compared with KoPA. This suggests that the structural embeddings consist of more semantic-rich information compared with text-based auxiliary prompts, which can also be understood by the LLM through the prefix adapter. Combining the analysis in Section 4.3 and the experimental results, KoPA achieves better results on top of shorter prompts.The results in the main experiments have shown the effectiveness of KoPA. To further validate the generality and the transferability of KoPA, we conduct a new transferability experiment. In this experiment, we will demonstrate that the knowledge prefix adapter will learn to transfer from structural embeddings to textual token representations and provide semantic-rich auxiliary information to enhance the decoding process of LLM inference. We demonstrate this point by testing the influence of KoPA for entities that do not appear in the training phase, which is also called inductive setting in other KGC works [6] . We split the KG dataset into an inductive setting with a defined inductive rate (IR), which refers to the ratio of unseen entities during training. For example, if IR=10%, we will randomly select 10% entities as the inductive entity set. Any triple in the training set whose head or tail is in the inductive set will be removed during training. Besides, the triples in the test set will be divided into two parts: the seen (S) part and the unseen (U) part. If the head or tail in a triple is in the inductive entity set, it will be regarded as unseen. We fine-tune the LLM with In this setting, a set of entities will not participate in the training process and the LLM does not see their textual descriptions, which will make the test process more challenging. We report the accuracy and F1 score for seen (S), unseen (U), and all (A) test triples, which is shown in Figure 3 for three fine-tuning methods: KoPA, vanilla IT, and structure-aware IT (enhanced IT in the figure).From the radio charts, we can observe that KoPA outperforms the other methods for unseen triples and has less performance degradation when IR increases. The performance of structure-aware IT (enhanced IT) with neighborhood triples in the textual form is more unstable. These phenomena suggest that the knowledge prefix adapter can learn a good mapping from the structural embeddings to the textual representation, which is transferable even if the entities are unseen during training. The structural embeddings captured from KG play a more significant role in informing the LLM with useful structural information.To verify the effectiveness of the KoPA design, we conduct a twopart ablation study. The first part is designed to verify the effectiveness of structural embedding and the second part is designed to verify the effectiveness of prefix adapter. As shown in Table 4 , we can find that removing the structural embeddings or replacing them with random initialized embeddings both lead to performance decline. Also, we find that the model is compatible with different types of structural embeddings. However, the performance gain depends on whether the embedding was originally powerful in the triple classification task or not. Refer to Tables 3, TransE [3] and Ro-tatE [31] are better embedding-based KGC models compared with DistMult [38] and ComplEx [34] . This demonstrates that semanticrich structural information is the key to performance improvement and KoPA takes full advantage of it.Meanwhile, putting the virtual knowledge tokens generated by the adapter in the middle (infix) or in the last (suffix) of the input sequence will also decrease the performance. We believe the reason is that putting tokens in the front of the sequence will make all the text pay attention to them as LLMs are usually decoder-only architectures with unidirectional self-attention. Then the LLM can make a better decision with the structural embeddings that fully interact with the text. Combining these two parts of the ablation study, we believe that our design of KoPA is effective and reasonable. To make a more intuitive view of KoPA, we conduct a case study in this section from both macro and micro perspectives. From a macro perspective, we count the prediction overlap of several models and plot a Venn diagram shown in Figure 4 .From the diagram we can find that KoPA has a significant portion of the proper predictions that do not intersect with several other models, which means that KoPA makes the right prediction on some test data that many other models predict incorrectly. This suggests that the structural information incorporated in KoPA has a significant role in making correct predictions. For a micro example, a test triple (John Landis, film director film, Coming to America) is predicted as wrong by the RotatE model and vanilla instruction tuning LLM. With retrieved neighborhood triples (Coming to America, locations, New York City), (John Landis, nationality, USA ), (Coming to America, genre, romantic comedy), (Comedy, common netflix titles, Coming to America), the structure-aware fine-tuned LLM still makes a wrong prediction because the neighborhood information is of little use in the judgment of the current prediction though they are the correct factual. The structural embeddings applied in KoPA contain more information than structural information in the form of text and are easier for us to extract by a structural pre-training process. Thus, KoPA outperforms other models in the triple classification task.To delve into the preservation of generic capabilities in LLMs, we conducted another experiment to assess the overall proficiency of LLMs both before and after fine-tuning. We apply the MMLU [12] benchmark for this problem. MMLU is the most popular benchmark to evaluate the general abilities of LLMs in different domains such as Humanities, Social Sciences, STEM, and others. The overall evaluation results on different datasets are shown in Figure 5 :From the results, it can be noticed that after KoPA training, there were discernible alterations in the generalized abilities of LLMs. In most instances, there was a decrease, but notably, STEM proficiency exhibited improvement on the UMLS dataset. We attribute this phenomenon to the UMLS being a medical KG, encompassing substantial knowledge in medicine, biology, and chemistry, and training on this dataset allows the model to acquire more STEM knowledge. Consequently, when facing natural language inputs differing from the training task, the model adeptly leverages the acquired knowledge from KGC task fine-tuning to get enhanced results. We have listed several subjects in MMLUs that showed improvement after training with UMLS. These subjects are highly relevant and close to the knowledge domain encapsulated in the UMLS in Table 5 . The LLMs trained with the KGC task also achieved significant improvements across different input prompts, marking a compelling observation.  
conclusion: 
 In this paper, we systematically explore how to incorporate structural understanding ability into LLMs to make structure-aware reasoning for KGC tasks. We extend the original LLM paradigms and propose structure-aware ICL and IT methods to incorporate the structural information by text. We further propose KoPA, a knowledge prefix adapter to incorporate the pre-trained structural embeddings into the LLMs. We conduct triple classification experiments to make comprehensive comparisons among the structure-aware methods and demonstrate the outperforming results achieved by KoPA. In the future, we plan to dive deep into LLM-based KGC and think about a more unified framework to accomplish all the KGC tasks with LLMs. Besides, we will also explore flexibly adapting KGs into LLM-based downstream applications to make the LLMs knowledgeable, reliable, and human-friendly. 
: 
 resulting in an incomplete representation riddled with unobserved or missing triples. This phenomenon motivates knowledge graph completion (KGC), which aims to predict the missing triples and further enhance the given KG.Existing KGC approaches can be divided into two categories: methods based on embeddings [3] and pre-train language models (PLM) [40] . Recently, as large language models (LLMs) [23, 46] show outperforming capabilities [24] , this field has recently been revolutionized by LLMs. Some works [41] make the first step towards LLM-based KGC, employing existing paradigms like zero-shot reasoning (ZSR) [4] and instruction tuning (IT) [24] to accomplish the KGC task. However, such approaches transform the KGC task into a text-based prediction of individual triples, leading to specific fundamental problems. LLMs lack the depth and precision of factual knowledge which always results in the hallucination [48] problem of LLMs. Besides, the structural intricacies of KGs such as subgraph structure, relational patterns, and relative entities/relations are often overlooked. This richly non-textual structured information, if properly incorporated, can significantly enhance the LLM's understanding and representation of KGs. Figure 1 presents an intuitive view of the importance of structural information for LLM reasoning. However, this is neglected by vanilla ZSR and IT approaches [41] because each input typically only includes a single input triple, leading to potential wastage of the structural information inherent in the KG. Such an approach fails to equip the LLMs with the awareness of the KG structure.To address these issues, we take a strategic step to LLM-based KGC, aiming to explore how to incorporate the KG structural information into the LLMs and enable structure-aware reasoning. Our initial focus involves transferring the existing LLM paradigms such as in-context learning (ICL) [9] and instruction tuning (IT) [24] to a structure-aware context. We propose a structure-aware ICL method and a structure-aware IT method as the base models, focusing on integrating the KG structural information into LLM through text form. Such an approach benefits from the fact that specific textual information exists about entities and relationships in KG so that we can use the text to represent this knowledge as complementary background information, expecting that LLMs can learn the local structural information in KG through textual prompts. But they also have the obvious disadvantage that there is a clear semantic divide between structural and textual information. The textual descriptions in the expanded prompt still fail to fully exploit the structural information in the complex KG.Additionally, we propose a novel Knowledge Prefix Adapter (KoPA) approach to make LLMs a better knowledge reasoner, leveraging structural embedding pre-training to capture the KG structural information. Then KoPA transforms the structural embeddings into textual embedding space by a knowledge prefix adapter and obtains several virtual knowledge tokens. These tokens, acting as prefixes in the input prompt sequence, direct the instruction-tuning process, providing valuable supplementary input triple information. This mapping of structural embeddings to textual form provides auxiliary information to input triples. Besides, we conduct comprehensive analysis and experiments, highlighting the remarkable performance and transferability of KoPA. In summary, our contribution is three-folded:• Extending the existing LLM paradigms. We are the first extensive investigation of LLM-based KGC methods, specifically by incorporating KG structural information to enhance the reasoning ability of LLMs. We discuss the pipeline to adapt the existing LLM paradigms like ICL and IT to a structure-aware setting for KGC using addtional textual prompts. • Designing new cross-modal LLM paradigm. We further propose a knowledge prefix adapter (KoPA) that effectively integrates pre-trained KG structural embeddings with LLMs. KoPA fosters comprehensive cross-modal interactions between textual embeddings from LLMs and structural embeddings sourced from KGs to enhance LLM's reasoning ability. • Comprehensive evaluation. We conduct extensive experiments on three public benchmarks and evaluate the KGC performance of all the structure-aware methods proposed by us with adequate baseline comparison with further exploration of the transfer ability and knowledge retention degree. 
 
 Appendix:  


Decision: The paper presents a novel approach, KoPA, integrating pre-trained KG structural embeddings with LLMs, enhancing cross-modal interactions for KGC tasks. Reviewers have praised the comprehensive experimental validation, clear writing, and sound methodology. However, concerns were raised regarding the detailed description of the negative sample strategy, the efficiency analysis, and the oversimplification of the KGC problem as a binary classification. Additionally, the necessity of four metrics over conventional ones like MRR and Hit@K, as well as the practical applicability of the approach in multimedia contexts, were questioned.

Reviews: review: paper_summary: paper_summary: The paper proposes a novel KoPA framework, which incorporates the pre-trained structural embeddings and knowledge prefix adapter into the LLMs, thereby facilitating LLM’s factual knowledge reasoning ability. Experiments and analysis demonstrate the outperforming results for KGC tasks. Furthermore, the paper evaluates the proposed KoPA's generic capability preservation via the MMLU benchmark to make LLMs knowledgeable, reliable, and human-friendly.
strength_weakness: strength_weakness: 1. The paper designs a novel KoPA, which effectively integrates pre-trained KG structural embeddings with LLMs and aligns cross-modal interactions between textual embeddings from LLMs and structural embeddings sourced from KGs.
2. The paper clearly and detailedly introduces the existing LLM paradigms (i.e., training-free reasoning and instruction tuning approaches) and elaborates on their shortcomings.
3. The paper conducts comprehensive experiments to evaluate the proposed KoPA’s factual knowledge reasoning ability, transferability, and common ability retention.
4. The paper is well-written and can be easy to follow. Moreover, the proposed solution is clear and technically sound.
limitations: limitations: 1. The adopted negative sample strategy and score function $\mathcal{F}$ in Section 4.1 need to be further described. In Figure 1, we can observe that compared with useless structural information, useful structural information facilitates LLM for KGC tasks. Hence, I wonder if you design some negative sample strategies to learn structural embeddings.
2. The efficiency analysis of KoPA should be discussed. Although KoPA improves the accuracy of KGC tasks, the training time and inference time of LLMs still incur a very high computational cost and time.
3. Please explain the necessity of four metrics (i.e., Accuracy, Precision, Recall, and F1-score) instead of MRR and Hit@K which are commonly used in these works for KGC [1-4].
4. There are some typos in the article. For example, in Line 552, "TransE [], DistMult" should be revised as "TransE [3], DistMult".

**Reference**   
[1] KC-GenRe: A Knowledge-constrained Generative Re-ranking Method Based on Large Language Models for Knowledge Graph Completion, LREC-COLING, 2024.  
[2] Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models, LREC-COLING, 2024.  
[3] KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion, EMNLP, 2023.  
[4] Contextualization Distillation from Large Language Model for Knowledge Graph Completion, EACL, 2024.
paper_summary: paper_summary: This paper proposes a novel framework knowledge prefix adapter (KoPA) to combine structural knowledge graphs (KG) and large language models (LLMs) together to achieve cross-modal structure-aware reasoning in the textual representation space. KoPA first pre-trains on the knowledge graphs and inject the structural knowledge into LLMs with a prefix adapter.
strength_weakness: strength_weakness: - This paper focuses on the combination of KGs and LLMs, which is a very important topic nowadays. The authors have devised a concise and effective approach to cross-modal knowledge transfer using the prefix adapter method.
- The workload of the experiment is sufficient, not only for KGC but also for more in-depth investigations such as experiments on the transferability of methods.
- I think the final test on the common ability retention is very interesting. It explores the extent to which the LLM retained knowledge in generalized domains after cross-modal knowledge injection. The conclusions obtained from these experiments are interesting and instructive.
limitations: limitations: - I have concerns about the experiment settings. The author simplifies the KGC problem as a binary classification, which is not very convincing. Existing studies in the KGC usually use ranking metrics like Hit@k, and recall. The binary classification is an over-simplified KGC setting, which might not be practical in reality.
- For structural-aware instruction tuning in the base LLMs paradigms proposed as baselines in the paper, some more detailed examples are missing to show how structural information is injected into the text.
- There are some typos in the paper. For example, GPT-3.5-turbor --> GPT-3.5-turbo in line 579. Missing reference for TransE at Sec 5.1.2
paper_summary: paper_summary: The paper presents a framework for knowledge graph completion that utilizes large language models and prefix tuning to incorporate both structural and semantic information. The authors conduct extensive experiments on multiple datasets, demonstrating that their proposed method achieves state-of-the-art performance. The ablation study and detailed analysis further support the effectiveness of the framework.
strength_weakness: strength_weakness: 1. The motivation behind the research is clear and the proposed method is sound.
2. The paper includes abundant experiments with excellent results, showcasing the effectiveness of the proposed approach.
3. The writing is clear and the paper is easy to follow, aided by well-designed figures.
limitations: limitations: There are no significant disadvantages in this paper.

This paper mainly focuses on text-attributed knowledge graphs and large language models. In this situation, the reference to ``Multimedia'' topic seems somewhat far-fetched.