MKGL: Mastery of a Three-Word Language

Authors: lingbing  guo
zhongpu  bo
zhuo  chen
yichi  zhang
jiaoyan  chen
yarong  lan
mengshu  sun
zhiqiang  zhang
yangyifei  luo
qian  li
qiang  zhang
wen  zhang
huajun  chen
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 08:58:24

Abstract: Large language models (LLMs) have significantly advanced performance across a spectrum of natural language processing (NLP) tasks. Yet, their application to knowledge graphs (KGs), which describe facts in the form of triplets and allow minimal hallucinations, remains an underexplored frontier. In this paper, we investigate the integration of LLMs with KGs by introducing a specialized KG Language (KGL), where a sentence precisely consists of an entity noun, a relation verb, and ends with another entity noun. Despite KGL's unfamiliar vocabulary to the LLM, we facilitate its learning through a tailored dictionary and illustrative sentences, and enhance context understanding via real-time KG context retrieval and KGL token embedding augmentation. Our results reveal that LLMs can achieve fluency in KGL, drastically reducing errors compared to conventional KG embedding methods on KG completion. Furthermore, our enhanced LLM shows exceptional competence in generating accurate three-word sentences from an initial entity and interpreting new unseen terms out of KGs.

Main paper: 
 introduction: 
 Knowledge graphs (KGs) are important resources for many data-driven applications, offering structured repositories of factual information that empower a variety of intelligent tasks [1, 2] . Yet, the strides made through the rapid advancement of large language models (LLMs) have challenged the conventional reliance on KGs. Nonetheless, LLMs are often critiqued for their susceptibility to generating factually incorrect or nonsensical outputs-a phenomenon known as the "hallucination problem" [3, 4] . Many recent studies propose to resort KGs to mitigate this problem [5] [6] [7] [8] .In this paper, we investigate the capacity of LLMs to assimilate and generate knowledge graph facts proficiently. For example, the natural language sentence, "Wendee Lee is an actor in Mighty Morphin Power Rangers," translates into a KG triplet format as (Wendee Lee, actor of, Mighty Morphin Power Rangers). It is worth noting that, English names such as Wendee Lee and Mighty Morphin Power Rangers, while can serve as identifiers for entities, are perceived as atomic elements within the KG framework. They are indivisible and distinct from their constituent words or characters.When the LLMs interpret these text identifiers as mere sequences of tokens, they risk producing output that misrepresents entities or relations, therefore compromising the integrity of KG-based tasks. Consequently, existing research that integrates LLMs with KGs tends to limit its scope to relatively straightforward tasks. Examples of these limitations include validating the correctness of fully-formed triplets [9] , or picking an appropriate entity from a limited set of options [10] . Given the sheer volume of entities in a KG, such narrow applications fall short in addressing more complicated tasks like KG completion, wherein a model predicts missing components of a provided incomplete triplet, e.g., identifying the unknown tail entities in (Wendee Lee, actor of, ?) against thousands of candidates. While these methods may lean on pretrained KG embedding models to narrow down possible candidates, the process remains inefficient.To transcend the limitations on predictive scope, we propose a novel approach, named MKGL, to instruct an LLM in the lexicon of the unique KG language (KGL). KGL sentences are strictly threeword sentences, starting with an entity noun, followed by a relation verb, and ending with another entity noun. The vocabulary of KGL does not immediately resonate with the machines. A common triplet like (Wendee Lee, actor of, Mighty Morphin Power Rangers) is encoded abstractly as e i r k e j , with e i , e j symbolizing the entity nouns and r j denoting the relation verb. For an LLM such as Llama-2 [11] , these symbols are entirely alien, absent from its pretraining corpus. Our investigation thus centers on how an LLM can navigate and master this specialized, atomic language of KGs.As illustrated in Figure 1 , to bridge this comprehension gap, we introduce an English-KGL dictionary, and the LLM is supposed to assemble new KG sentences using the provided linguistic building blocks. The basic elements of KGL, while different from our natural language, are familiar to the LLM as they are constructed from the pretrained token embeddings. We leverage a context retriever to retrieve the text information and relational information of a KGL token, which transforms the sequential token embeddings of its name into an embedding vector. Subsequently, we update the LLM token embedding layer with new KGL token embeddings. In the scoring layer, we also employ a KG score retriever to supplement the LLM with extra KG relational information for prediction.Instructing an LLM in KGL offers three main advantages over prompt-based methods [10, 13] or conventional KG embedding methods [14, 15] : (1) Broadened applicability. KGL tokens originate from textual tokens of an LLM, thus our method does not mandate that all entities be observed during training.(2) End-to-end framework. Unlike recent LLM-based methods that necessitate pre-sorted results from conventional KG embedding methods, our approach can rapidly rank all candidate entities at one-step. (3) High efficiency. The representations of KG tokens are derived from pretrained token embeddings rather than learned from scratch. The proposed KGL context and score retrievers also leverage a LoRA-like adaption. Using Llama-2-7b [11] as the base LLM, the number of training parameters is less than 0.3%.However, instructing an LLM in KGL also has its limitations, as it demands more computational resources compared with conventional methods. For instance, fine-tuning MKGL (Llama-2-7b) on the FB15k-237 dataset [16] to outperform most conventional methods requires only 1 epoch. Nevertheless, with 8 A100 GPUs, it still takes half an hour, which is comparable to training a TransE model from scratch with a single GPU. 
related works: 
 We category the related works into two groups:Knowledge Graph Completion KG completion can be regarded as a classification problem like many NLP tasks [17] [18] [19] [20] [21] , such as node classification and semantic role labeling. However, its label space is significantly larger than most NLP tasks. For example, the WN18RR [22] dataset contains over 40,000 different entities, making it impractical to simply feed them all as possible results and let the LLM select one as output. Most conventional KG completion methods are embedding-based methods, including the triplet-based methods [23] [24] [25] [26] [27] , e.g, TransE [23] , ComplEx [25] , RotatE [26] ; the GNN-based methods [15, [28] [29] [30] [31] [32] , e.g., DAN [15] , CompGCN [28] , CoKE [29] ; and other neuralbased methods [22, [33] [34] [35] , e.g., ConvE [22] and RSN [34] . Despite differences in their neural methods and input forms, all these methods focus on relational information and are not good at utilizing other types of information such as textual attributes.Pretrained Language Models for Knowledge Graphs Leveraging Pretrained language models for KG completion has been explored for many years [36] . Some works treat BERT as a GNN model to encode graph features [30, 37] , while others consider the textual information of KGs and use pretrained BERT to encode the textual labels of entities and relations [14, [38] [39] [40] . The resulting outputs are regarded as the entity and relation embeddings or their concatenations.With the rapid advancements in leveraging LLMs for KG completion, recent works have begun designing prompts or instructions to guide LLMs in this task. Initially, the results were not promising [41] , as it appeared that even state-of-the-art LLMs without context information could not outperform basic KG embedding models like TransE. However, subsequent works such as KGLlama [42] and KoPA [13] discovered that LLMs might perform better in triplet classification, i.e., estimating the correctness of a given triplet.More recently, KICGPT [10] has proposed leveraging in-context learning [43, 44] to provide explicit instructions and guide the behavior of LLMs. This involves a triplet-based KG embedding model to generate the initial rankings of the top-k entities, followed by a multi-round interaction with the LLM, providing textual information and triplet demonstrations for the query entity and relation.The LLM should then re-rank the initial list. KICGPT has achieved state-of-the-art results on KG completion tasks. However, its performance not only depends on the LLM and the instructions but also on the pretrained KG embedding model. Additionally, KICGPT cannot be deployed offline due to the demand of commercial LLMs [45] . It also cannot provide embeddings for downstream tasks.In contrast, the proposed MKGL has an embedding module based on the LLM token embeddings and KG relational information, which overcomes the weaknesses of existing KG embedding methods that cannot provide embeddings for unseen entities. The context information is implicitly encoded into the KGL token embeddings and efficiently captured by the LLM during fine-tuning. 
mastery of kg language: 
 In this section, we discuss the details of MKGL. We first introduce the general architecture of an LLM and how to convert a KG triplet into a fine-tuning instruction. Then, we present the details of constructing KGL token embeddings and scores. Finally, we illustrate how to train an MKGL and analyze its complexity.We start by a brief introduction to KGs and LLMs.Knowledge Graph and Knowledge Graph Language Knowledge graphs are conceptualized as directed, multi-relational graphs. We describe a knowledge graph by G = (T , E, R), where T , E, R are the sets of triplets, entities, and relations, respectively. KG language (KGL) is construed as a rigorously defined three-word construct, mirroring the structure of a simple sentence. Specifically, a KGL sentence e i r k e j invariably commences with an entity noun e i ∈ E, proceeds with a relation verb r k ∈ R, and culminates with another entity noun e j ∈ E. Analogous to the syntactic conventions in Chinese, KGL sentences eschew the use of spaces or commas to demarcate KGL terms.Knowledge Graph Completion KG completion is one of the most important tasks in the KG area. The target of KG completion is to predict the head entity e i given the relation and tail entity (?, r j , e k ), or predict the tail entity e k given (e i , r j , ?). In the scenario of KGL, this task is equivalent to completing the KG sentence ?r j e k or e k r j ?.The inductive KG completion focus on completing an unobserved KG G ind = (T ind , E ind , R ind ). Specifically, the relation set R ind is identical to the original set R, but the inductive entity set E ind shares no elements with E, i.e., E ind ∩ E = ∅. The triplet set T ind is further split into the fact set T ind-fact and test set T ind-test . We train a model on the original triplet set T and use the fact set T ind-fact as context to evaluate it on the test set T ind-test .Large Language Models As depicted on the left side of Figure 1 , the architecture of a typical LLM can be divided into four main components:• Tokenizer, which breaks down the input sequence of words w 0 , w 1 , ..., w m into tokens t 0 , t 1 , ..., t n ;• Token embedding, which maps the input tokens t 0 , t 1 , ..., t n to a sequence of low-dimensional vectors t 0 , t 1 , ..., t n ;• Transformer M, the core of the LLM, which consists of multiple attention-based blocks that process the input token embeddings into hidden states:EQUATION• Score layer, which features a weight matrix S ∈ R N ×d with an identical shape to the token embedding matrix T ∈ R N ×d , where N , d denote the vocabulary size and hidden size, respectively. The score layer projects the output of Transformer at the n-th step to a probability distribution p n+1 for predicting the next token t n+1 :EQUATIONRecent studies reveal that LLMs harbor the potential to acquire unfamiliar natural languages [46, 47] . Given this premise, it is of particular interest to investigate how LLMs might interpret and operate within our KGL. We first design a prototype instructional text for this purpose. For a given triplet (Wendee Lee, actor of, Mighty Morphin Power Rangers), suppose that the task is to predict the tail entity Mighty Morphin Power Rangers, the instructional text is formatted as follows: Instruction 3.1. Supposed that you are a linguist versed in an esoteric three-word knowledge graph language. Given the following dictionary comprising two words from this language, please kindly Here, <kgl: Wendee Lee> denotes the definitive KGL token (corresponding to e i in previous sections and Figure 1 ) assigned to the entity Wendee Lee. We enrich the tokenizer's vocabulary with all pertinent KGL tokens, thereby enabling it to translate these KGL tokens into token IDs, which append sequentially to the LLM's original vocabulary range. It is worth noting that we only provide at most one example KGL sentence for each KGL word. Our intention is to introduce the schematics of KGL sentences to the LLM, rather than leveraging augmented KG data for in-context learning. To mitigate potential biases, the example sentences are sampled randomly. The practice of incorporating supplementary context information alongside instructional prompts, known as in-context learning (ICL), has proven effective in enhancing performance across many NLP tasks [43, 44] . However, the concatenation of retrieved context on KGs with the input text can easily exceed the input length constraints of LLMs. Processing such long input sequences remains computationally intensive even with truncation. To address these constraints, we propose an alternative approach to encode context information into compact vector representations. Our experiments in Section 4.6 also demonstrate its superiority in terms of both efficiency and performance.We propose the low-rank adaption (LoRA)-based KGL context Retriever R context to effectively aggregate textual and KG information into KGL token embeddings. Typically, the vocabulary scope of a KG (comprising both entities and relations) usually surpasses that of an LLM. For instance, WN18RR is a KG completion dataset set sampled from WordNet [48] . It has over 40,000 unique entities, while the vocabulary size of Llama-2-7b is 32,000. Therefore, initializing new token embeddings for each KG elements and optimizing them from scratch would be prohibitively resource-intensive.Moreover, the dynamic nature of real-world KGs consistently introduces new entities. This is analogous to the evolution of human language, where new words are often synthesized or derived from existing ones. Drawing inspiration from this linguistic adaptability, we propose leveraging existing textual tokens to generate new KGL tokens, thereby avoiding the computational burden of learning unique embeddings for every KG element.Scale Down As illustrated in Figure 2 , the first step is to reduce the dimensionality of LLM token embeddings to lower computational demands during text and KG context aggregation. Inspired by LoRA [12] , we leverage a projection matrix W T ∈ R d×r to transform the token embedding matrix T ∈ R N ×d into a reduced space R N ×r :EQUATIONwhere T r ∈ R N ×r represents the compact token embedding matrix.We leverage a text encoder to encode the textual token embeddings of each KGL token into a unified vector. For example, the entity name "Mighty Morphin Power Rangers" would be converted into individual token embeddings t ei,0 , t ei,1 , ..., t ei,n , which are then aggregated into a single vector for the entity e i :EQUATIONwhere t ei is the textual token embedding for e i . The choice of the encoder E text is free. In this paper, we leverage principal neighbourhood aggregation (PNA) [49] , which can be roughly understood as applying multiple pooling operations (including max, min, mean, std etc.) on the token embedding sequences. A detailed introduction to PNA can be found in Appendix C.We employ a multi-layered PNA encoder E kg to aggregate the KG information of e i and its adjacent entities, which can be formulated as:EQUATIONwhere N (e i ) denotes the neighboring entities to e i . The adoption of PNA for encoding both textual and relational data of KGL tokens is due to its parameter efficiency and superior performance compared to attention-based alternatives like GAT [50] . An empirical comparison of different encoders can be found in Appendix F.To finalize, we adjust the dimensionality of the output embeddings to align with the LLM input requirements:EQUATIONFor the sake of clarity, we will continue to use t ei to represent the KGL token embedding in subsequent discussions. For efficiency, we retrieve the KG information only for entities. This operation also make the embeddings of entities and relations distinguishable.While recent studies have adapted LLMs to various tasks by either restricting the output space or reformulating tasks into multiple-choice questions [9, 10, [51] [52] [53] , such strategies pose challenges for KG completion. Specifically, the existing methods are inapplicable to entity constrastive learning as their main objective is optimized against text tokens instead of entities. Also, they incur significantly slow inference times, as the LLM must traverse to the output tree's leaf nodes to generate predictions. Even then, the generation of top-k results, dependent on beam search parameters, may not accurately reflect the true likelihoods of entities.In contrast, in this paper we propose a new approach to reconstruct the KGL scores through LLM's score layer and hidden states, providing a one-shot probability distribution for all candidates. Our method seamlessly integrates with contrastive loss and negative sampling techniques [54] , making it highly compatible with prevalent KG completion frameworks. This compatibility also ensures that MKGL has the potential of being applied for downstream KG embedding tasks [32, 55] .We propose a LoRA-based KGL score retriever R score to produce the probability distribution of KGL tokens, which can be formulated as follows:EQUATIONEQUATIONEQUATIONThe score retriever also starts from a down-scaling layer to reduce the dimensionality of the score matrix S ∈ R N ×d to S R ∈ R N ×r with W S , and similarly scales down the LLM's output hidden vector h n with W H . Subsequently, the text information (i.e., the token score vectors s ′ j,0 , s ′ j,1 , ..., s j,n ) associated with target entity e j is fed to the score text encoder S text to construct the KGL score vector s ′ j . It is then concatenated with the LLM hidden state h ′ n to obtain the conditioned input [h ′ n , s ′ j ]. Upon gathering the neighboring information of the target entities via a multi-layered PNA S kg , an output matrixW O ∈ R r×1 is employed to map the result s ′′ ej |ei,r k ∈ R r to the 1-d probability estimate p ej |ei,r k ∈ R.Table 2 : The KG completion results on FB15k-237 and WN18RR. The best and second-best results are boldfaced and underlined, respectively. ↑: higher is better; ↓: lower is better. -: unavailable entry.FB15k-237 WN18RR MRR↑ Hits@1↑ Hits@3↑ Hits@10↑ MRR↑ Hits@1↑ Hits@3↑ Hits@10↑ TransE [23] . Optimization With the above score retriever, estimating the probability for any candidate entity becomes straightforward at a single step. To refine MKGL, we consider a contrastive loss leveraged in most existing KG embedding methods [22, 23, 28, 34] , expressed as:EQUATIONwhere N neg (e j ) = {e neg |e neg ̸ = e j , e neg ∈ E} is the sampled negative entity set for the target entity e j . The loss function L is in a form of a binary cross-entropy, contrasting the likelihood of correctly predicting the relation p ej |ei,r k as positive example, against the probabilities of erroneously predicting relations (e i , r k , e neg ) as negative examples. We also present an algorithm to demonstrate the step-by-step fine-tuning process, please refer to Appendix D for details.It is clear that the primary computational cost for MKGL lies in the LLM. By employing LoRA-based KGL retrievers to retrieve context vectors instead of texts, we can significantly reduce the major expenditure. For instance, our retrievers can reduce the average input lengths from 811.2 to 91.4 on the FB15k-237 dataset, compared to using one-hop neighbors for in-context learning. All operations within the LoRA-based retrievers are performed under low dimensionality. Furthermore, the token embeddings and score matrix of the LLM are frozen during fine-tuning, thus ignoring their gradient computation. In the worst case, the complexity of text information retrieval is O(N kgl L kgl r), where N kgl , L kgl , r are the number of KGL tokens, maximum text token lengths of KGL tokens, and the reduced dimensionality, respectively. Subsequently, the complexity of KG information retrieval in the worst case is linear to the number of triplets, i.e., O(|T |N layer r), where |T |, N layer denote the number of triplets in the KG and the number of PNA layer, respectively. 
experiments: 
 In this section, we evaluate the performance of the proposed MKGL through extensive experiments, comparing it against both LLM-based and KG embedding methods. The source code and datasets are available at github.com/zjukg/MKGL.We evaluate MKGL on the FB15k-237 and WN18RR datasets, which are widely used by most KG completion methods [22, 23, 26, 28, 34, 56, 57] . We also evaluate MKGL on the inductive version of these two datasets [58] . We follow REDGNN [59] to evaluate MKGL on all entities rather than 50 sampled candidates. Please refer to Appendix E for dataset statistics.For our experiments, we employ Llama-2-7b [11] as the base LLM and train MKGL using 8 A100 GPUs. A standard LoRA adaptation is applied to the query and value layers of the LLM. Full hyper-parameter details are available in Appendix D. We evaluate performance using MRR (mean reciprocal rank of target entities) and Hits@k (percentage of target entities ranked in the top k).Our baselines include conventional KG embedding methods such as TransE [23] , RotatE [26] , and TuckER [56] ; GNN-based methods like CompGCN [28] , DAN [15] , and CoKE [29] ; methods that integrate language models including KG-BERT [14] , StAR [38] , KGLM [40] , FTL-LM [39] , and DET [30] ; and LLM-based methods: KG-Llama [42] , GPT 3.5 [41] , and KICGPT [10] . In the inductive scenario, we compare against rule-based reasoning methods such as RuleN [60] ,NeuralLP [33] , DRUM [61] , GraIL [58] and RED-GNN [59] , acknowledging that standard methods fail to predict relations without entity embeddings.The knowledge graph completion results are presented in Table 2 . MKGL outperforms other baselines in nearly all metrics. Notably, MKGL and KICGPT significantly surpass other LLM-based methods, demonstrating the importance of KG relational information. Contrarily, many BERT-based methods fall short against GNN-based methods, suggesting that merely incorporating text information may not yield the anticipated benefits. In summary, the proposed MKGL clearly outshines its counterparts, particularly those founded on commercial LLMs.To our knowledge, existing LLM-based methods have not addressed the inductive KG completion challenge. We benchmark MKGL against the state-of-the-art inductive methods. Although we can not assess KICGPT [10] due to unavailable source code, it is worth mentioning that our MKGL could potentially augment KICGPT by supplying a candidate list, facilitating seamless integration between the two methods. We present the results in Table 3 . MKGL significantly outperforms all the baseline methods across metrics. As most inductive reasoning methods do not have an embedding module for entities, the proposed MKGL represents the first embedding-based method to deliver state-of-the-art results in inductive KG completion.Beyond its capability as a KG completion tool, MKGL also serves as a language model for KG languages. To evaluate its proficiency in generating KGL sentences, we employ a sequence generation loss and remove the relation context from the input prompts. We leverage the second-to-last output of the LLM for relation prediction.The results are shown in Figure 3 . The left section contrasts the sequence prediction results against standard KG completion, revealing only a modest loss in performance. MKGL still outperforms many conventional methods, especially on WN18RR dataset. The right panel displays sample sentences generated by MKGL, illustrating its potential to discover legitimate KGL sentences absent from the existing KG. We observe that WN18RR is more difficult than anticipated as it contains many plausible entities that challenge even an LLM's discernment.We conduct ablation studies to assess the importance of each module, as detailed in Table 4 . The unmarked cells indicate that we either substitute the text retrieval module with a learnable embedding module or remove the KG retrieval module. Clearly, the method with complete features achieves best results, while replacing or removing either module significantly impacts performance. Notably, removing the KG retrieval module yields more performance loss on WN18RR, as many entities in this dataset have similar names. For example, there are 14 different entities named "call". In this case, incorporating with KG information becomes necessary.We examine the computational efficiency of our method (MKGL) relative to "in-context" baselines. Specifically, we develop several variants: LLM randomly-initialized new entity token embeddings (NewToken), LLM with KGL context from 1-hop neighbors (NewToken (1-hop)), LLM with KGL context from 2-hop neighbors (NewToken (2-hop)), and MKGL without score retriever (MKGL w/o SR). The results are shown in Figure 4 . MKGL surpasses all alternatives in performance. NewToken variants slightly lag behind MKGL w/o SR, but notably, our proposed methods demand fewer trainable parameters than NewToken variants. By encoding all context information within KGL token embeddings, the average input length is significantly reduced, which decreases training time considerably. Moreover, MKGL supports larger batch sizes during both training and inference phases, enhancing computational efficiency. 
conclusion and future work: 
 In this paper, we propose MKGL to instruct the LLM in the language of KGs. MKGL employs a context retriever that efficiently provides LLMs with pertinent textual and relational context, markedly reducing input lengths relative to in-context-learning and supervised fine-tuning methods. Meanwhile, MKGL also leverages a score retriever to supply score information and aid in KGL inference. Extensive experiments confirm the superiority of MKGL in terms of both performance and computational efficiency. The proposed context and score retrievers point out a new direction in incorporating LLMs with semantic data, such as question answering and entity linking. They may also shed lights on a more broaden area where the input cannot be precisely represented by text, e.g., node classification and protein representation learning. Furthermore, the construction of KGL vocabulary enables contrastive learning not only limited on tokens, which may provide insights on general machine learning. Therefore, there are plenty of future directions. We would like to pretrain LLM using the mixture corpora of KG and natural languages, such that the LLM could understand and create responses with linked data. 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [Yes] Justification: The paper discusses both potential positive societal impacts and negative societal impacts in appendices.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? 
 
 Appendix: We would like to discuss the potential limitations of our method from the following three aspects:Efficiency. As MKGL is an LLM-based fine-tuning method, it inevitably demands more computational resources. In the main experiments, MKGL significantly outperforms all the conventional and LLM-based methods. The later analysis also reveal that the trainable parameters and runtime of MKGL are less than general fine-tuning framework. Therefore, we believe that MKGL is still an efficient LLM-based method.Robustness. MKGL leverage multiple retrievers to retrieve text and KG information for constructing both input embeddings and score estimations, which may accumulate more errors during fine-tuning. Even though, most modules are learnable with back-propagation. To avoid biased evaluation and occasional results, we also report the averaged results of multiple runs with variance statistics. Thus, we believe MKGL is a robust method.Generality. The advances in LLMs have revolutionized many NLP tasks, and it is important for an LLM-based method where the LLM makes use of the proposed modules and whether the performance can continually improves as the LLM get promotion. We have conducted experiments to visualize the KGL embeddings and compare the performance with different base LLMs. The results empirically demonstrate the generality and potential of MKGL.Our work focuses on the integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) through the introduction of a specialized KG Language (KGL), has substantial broader impacts spanning technological advancements, societal implications, and educational benefits. Here, we outline the diverse and far-reaching impacts of our research.Technological Advancements Our research contributes to the cutting-edge of artificial intelligence, pushing the boundaries of what LLMs can achieve when combined with the structured knowledge represented by KGs. This can potentially unlock new capabilities in AI, ranging from more accurate context-aware natural language understanding to enhanced machine reasoning across diverse domains such as healthcare, finance, and legal systems. Additionally, our approach of using a specialized language (KGL) and the method of contextual embedding augmentation can inspire novel AI architectures and learning paradigms that bridge the gap between unstructured and structured forms of knowledge.The enhancement of AI systems with a more profound understanding of structured knowledge has broad societal implications. For one, it could lead to the development of AI assistants that provide more accurate, consistent, and reliable information, thus improving decisionmaking in critical sectors. In healthcare, for instance, AI systems equipped with our technology could offer more precise recommendations by thoroughly understanding medical knowledge graphs. Moreover, by reducing the propensity for errors and hallucinations, our approach could foster greater trust in AI technologies among the general public, paving the way for wider acceptance and integration into daily life.Ethical Considerations As with any advancement in AI, our work prompts important ethical considerations. Ensuring our technology is used responsibly involves critical discussions around privacy, bias, and transparency, especially as AI systems become more adept at interpreting and generating human-like text. We advocate for the continued examination of these aspects in tandem with technological development to ensure AI benefits society equitably and ethically. Our work, by facilitating error reduction in AI outputs, also contributes to the broader effort of minimizing harm and bias in AI-generated content.Educational Benefits Our integration of LLMs with KGs presents a novel avenue for educational tools and applications. AI tutors or educational platforms powered by our enhanced LLMs can offer students personalized and accurate learning experiences. Such systems could better understand and integrate complex academic content structured within knowledge graphs, from history timelines to scientific concepts, thereby improving the quality of automated tutoring services and opening up new methods for engaging with educational material.Future Directions This research opens up exciting avenues for future exploration, including refining the KGL for broader applicative scopes, exploring the ethical considerations of nuanced AI-generated content, and expanding our understanding of AI's potential when it deeply integrates diverse forms of knowledge. The cross-disciplinary nature of this work invites collaboration among computer scientists, ethicists, domain experts, and policymakers to harness the full potential of AI for societal benefit.In conclusion, the integration of LLMs with KGs not only represents a significant step forward in AI capabilities but also poses thoughtful considerations for societal impact, ethical use, and educational applications. Our work underscores the importance of continuous exploration, responsible innovation, and cross-disciplinary collaboration to harness the transformative potential of AI technologies.Graph Neural Networks (GNNs) have emerged as a powerful family of neural models for learning on graph-structured data [17, 18, 62] . Among the recent advances is the principal neighborhood aggregation (PNA) mechanism [49] , which enhances the representational capacity of GNNs by diversifying the aggregation functions applied to neighboring nodes.PNA leverages a combination of multiple aggregators such as sum, mean, and standard deviation, together with a scalable degree-specific weighting scheme. This approach is designed to address the shortcomings associated with simple aggregation functions that may fail to capture the complexity and diversity of neighborhood structures in graphs.The key component of PNA is its aggregation scheme, which is formally defined as follows:EQUATION)Please note that the symbols used in describing PNA are independent to the main paper for clarity.Here, a (l+1) vis the aggregated information for node v at layer l + 1, N (v) denotes the set of neighbors of v, h (l) u represents the hidden features of neighbor nodes at layer l, is a concatenation operator over all aggregators in the set R, AGG ρ is an aggregation function (e.g., sum, mean, max), δ is a nonlinear activation function such as ReLU, and W (l) is a learnable weight matrix at layer l.EQUATIONPNA's distinctive blend of multiple aggregation functions and degree-aware weighting significantly enhances the expressive power of GNNs, allowing for more complex feature representations and, consequently, improved performance on downstream tasks. We also follow the KG embedding methods [15, 28, 63] to incorporate the relation embeddings into PNA as relational PNA.We introduce Algorithm 1 to demonstrate the fine-tuning process of MKGL for KG completion. We first construct input instructions following Instruction 3.1 and tokenize them into IDs. For those in the score of original LLM vocabulary, their embeddings can be looked up from T, while those of out of scope will be retrieved by our context retriever R context . After assembling the input embeddings, we feed them to the LLM to obtain output hidden states and then obtain the scores from the score retriever R score . Finally, we optimize MKGL by minimizing the constrastive loss L. The main hyper-parameter settings are summarized in Table 5 .Algorithm 1 MKGL for KG Completion 1: Input: the training KG G, the language model M, the token embedding matrix T, the original vocabulary of the LLM V llm , the KGL context retriever R context , the KGL score retriever R score ; 2: for each batched data in the training set do Tokenize the input instructions with the tokenizer; 5:for each input token sequence {t 0 , t 1 , t 2 , ...} do 6:for each token t k do 7:if t k ∈ V llm then 8:t k ← T k ; // look up embedding from the token matrix Compute and minimize the constrastive loss L (Equation ( 11)); 17: end for We use the following benchmark datasets to evaluate the performance of MKGL, and summarize the statistics in Table 6 :• FB15k-237: This dataset is a subset of the original FB15k dataset [23] and is created by removing inverse relations that may lead to test set leakage.• WN18RR: This dataset is a subset of the original WN18 dataset [23] and is created by removing inverse relations that may lead to test set leakage.• FB15k-237-ind: The 'ind' suffix denotes the inductive setting adopted in FB15k-237 [58] .It includes new entities in the validation and test sets that are not present during training, thus requiring models to generalize beyond the transductive assumptions of previously seen entities.• WN18RR-ind: Similarly to FB15k-237-ind, the WN18RR-ind dataset is adapted for inductive KG completion on the WordNet [48] .These datasets have been instrumental in the development and benchmarking of advanced KG completion models, enabling comparison of different approaches and understanding of their effectiveness in both conventional and inductive settings. We present detailed results on all inductive KG completion benchmarks in Table 7 , where MKGL consistently and significantly outperforms all state-of-the-art baselines. We conduct experiments to analyze the influence of layer numbers in the KGL retrievers. The results are illustrated in Figure 6 . Clearly, increasing the number of layers enhances performance across all datasets and metrics. Additionally, we observe that a small number of layers (i.e., 2) significantly impairs performance. We conduct experiments to explore the impact of different encoders in the retrievers. The results are depicted in Figure 7 . We find that the MKGL is not highly sensitive to the choice of encoders. The performance when using GAT [50] is slightly lower than when using PNA.Answer Justification: The paper provides sufficient information on the computer resources.Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Justification: There is no such risk for this paper.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: All assets are fully licensed.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.* Correspondence to: {zhang.wen, huajunsir}@zju.edu.cn 


Decision: The authors introduce a method for LLMs to carry out knowledge graph completion.  The system outperforms a number of established baselines.  Reviewers agree that that the method is sound, novel and impactful, if only in applications.  No substantial weaknesses remain after the rebuttal/discussion periods.

Reviews: review: paper_summary: paper_summary: The paper proposes MKGL, a novel approach to integrate LLMs with KGs by instructing them in a specialized KG Language (KGL). KGL is a three-word language that mirrors the structure of KG triplets. The authors introduce a KGL context retriever and a score retriever, both based on LoRA, to efficiently encode textual and relational information into KGL token embeddings. MKGL outperforms existing KG completion methods, including LLM-based and conventional approaches, on both standard and inductive KG completion tasks. The paper also demonstrates MKGL's ability to generate valid KGL sentences and its computational efficiency compared to in-context learning methods.
strength_weakness: strength_weakness: * There are only limited details on certain aspects of the methodology, for example I couldn't find details about the actual implementation of the multi-layered PNA for KG information retrieval.
* While the authors claim that the proposed "three-word language" parsing of natural sentences is novel, it boils down to semantic-role labeling (SRL), a well-established NLP task. I believe that the paper should include a clear comparison to past SRL methods.
* The results in Table 2 should contain a column with computational cost (or at least the number of parameters of each method), to make it clear if it compares apples to apples.
* The computation runtime of the proposed method and the baselines is another thing that is lacking.
questions: questions: * What exactly are the trainable parameters of the in-context-learning baseline in Section 4.6 and Figure 4? Isn't the point of ICL to not do any parameter updates at all and rely only on the contextual prompt?
* How does MKGL scale to much larger knowledge graphs? And how is it compared to other KG completion methods?
limitations: limitations: No issues found.
paper_summary: paper_summary: The authors introduce a SOTA method for allowing LLMs to incorporate information from knowledge graphs, relying on Knowledge Graph Language token embeddings to retrieve context, and then score it using a retriever that helps form a distribution over the possible entities to be incorporated.
strength_weakness: strength_weakness: Could use a longer and more detailed discussion section; ends a little too abruptly.
questions: questions: None.
limitations: limitations: I have no concerns about this paper being published.
paper_summary: paper_summary: This paper proposes what seems to be an elaborate GNN+LLM+GNN  sandwich of a model for doing knowledge base completion, having a GNN pipeline to form KB-informed token embeddings, passing those to a LLM (lllama-2 ) into a knowledge base completion prompt template, and then passing that output into another GNN-like ("PNA") set of layers.  That whole set-up is then used to train representations of the data optimized with contrastive loss for the entity prediction task in knowledge base completion tasks (FBK and Wordnet)   (I'll admit that I found the model explanation quite hard to follow, and so it's entirely possible I'm slipping on some details -- some of this had to be inferred by glancing at their code.)  It seems to outperform existing methods on these two tasks.
strength_weakness: strength_weakness: - I found the model extremely rather hard to follow, and had particular trouble discerning why this collection of model assumptions would result in a meaningful improvement over prior work.  Since the work is so complicated, it may be both useful to focus on very clear graphs and progressively introducing parts of the model.  I'll admit that the "Retriever" framing felt very confusing as well, as this models "Retrievers" don't seem to do any retrieving. 
- For understanding the model, the "three world language" framing seems quite separate from the meat of what this model seems to be doing.    It implies that the somewhat simple "template" setup they use is important, but it seems to be only formed to teach a model what triplet completion is, which is something that one would think would be easily addressed in fine-tuning; there is not experimental exploration showing the value of those prompts.  
- I'll admit to feeling suspicious at the high performance here (even after ablating nearly everything, they outperform most models?).
questions: questions: -Looking at the code, it was unclear whether the  model is calculating the metrics (MRR, Hits@1, Hits@10) by ranking solely within a small batch containing the correct answer, or actually predicting the highest ranked entity from a full space of candidates. Could the authors clarify which one is being done? 
- Could the authors clarify what is removed in the second , "text" ablation reow
limitations: limitations: yes, a broader impacts section is included and no major limitations are missing.
paper_summary: paper_summary: This paper proposes a method to leverage LLMs as knowledge graph completion systems. New tokens that correspond to (potentially multi-word) concepts and relations are introduced to the model’s vocabulary, and then the LLM’s embeddings for the tokens composing those concepts/relations are aggregated and upscaled to estimate embeddings for these new tokens. Given these new tokens, the goal of the system is to complete the knowledge graph triplet given two of these embeddings by retrieving the correct third token from the KG vocabulary.

In a series of experiments, it is observed that the proposed system outperforms a variety of previous baselines employing diverse methods. It also excels at inductive KG completion. An ablation study reveals that each proposed part of the pipeline is necessary for achieving the best performance.
strength_weakness: strength_weakness: 1. Given how many moving parts there are, reproducibility seems difficult. It would be nice to see the variance in performance of the method across multiple random seeds, where each random seed entails running the entire pipeline of optimizations from scratch.
2. Relatedly, it is unclear whether the approach will scale as LMs continue to improve (and presumably to become better bases for approaches like this). Having a comparison with other base models would be a nice way to hedge against this.
3. No detailed discussion of limitations. The checklist says it is discussed, but there are only brief comments distributed throughout the paper (which, in my opinion, do not address limitations thoroughly enough). A dedicated section would be helpful.
4. It is unclear whether the new KG embeddings encode relevant concepts to the target token, or whether they are picking up on certain spurious correlations that happen to be helpful (but may not generalize robustly). It would be nice to have an analysis where the new embeddings are directly decoded into vocabulary space, such that we can observe what concepts are included in these new representations.
questions: questions: 1. Would it be possible to base this approach on other LMs as well? The various Llama scales would be ideal, but if scale is an issue, even just comparing Mistral, Llama 2 (7B), Llama 3 (8B), and ideally some smaller models would be nice. This is quite compute-intensive, so I wouldn’t expect it for the camera-ready, but it would definitely be nice to have.
2. Do you have any hypotheses as to why the proposed method is better at inductive KG completions than prior methods? In other words, is there a particular aspect of your pipeline that you believe makes it better for handling novel triplets than past approaches?

Typos:
* L226: “clear that The” -> “clear that the”
limitations: limitations: I do not believe limitations have been sufficiently addressed. There is no "Limitations" section, nor is there a dedicated space in any part of the paper that directly addresses the drawbacks of the proposed method and experiments. For example, there are many moving parts; there are multiple stages of optimization that could lead to cascading errors; only one LLM base was considered; etc.

Comments: comment: We sincerely appreciate your increased rating and recognition of the efforts we put into addressing your concerns! Your contribution to our work is highly valued and greatly appreciated.
I thank the authors for their detailed response! With the authors clarifications, I take back some of my initial concerns, and  I've revised my scores up from 4 to 6 .
Thank you very much for your prompt feedback and unwavering support!
Thank you for the additional information!
We are truly grateful for your increased rating and insightful comments. We completely agree with your point about listing the number of full parameters, which is important to provide a comprehensive comparison. While we are unable to revise "rebuttal.pdf" at this stage, we are fully committed to updating Table 2 to include the statistics of full parameters. 

In most methods, the number of trainable parameters aligns with the number of full parameters. However, there are two exceptions: KG-Llama and our method MKGL, both of which employ LoRA.

| Methods | # Trainable Parameters (M) |# Full Parameters (M) |
|---|:---:|:---:|
| KG-Llama  | 13 | 6,755 |
| **MKGL**  | 20 | 6,762 |

MKGL incorporates additional neural layers for aggregating text and KG information, thus necessitating a greater parameter count. Once again, we sincerely thank you for your valuable suggestions.
Thank you very much for your response! I still believe that you should publish the (training & inference) computational cost and full parameter count, and compare it against the other methods. I don't see the significantly increased number of parameters as a negative, as long as it's transparent to the reader. Informing only about the training parameters seems somewhat misleading. I'm happy to increase the final score to 7 if you consider this.
We sincerely appreciate your increased rating and recognition of the efforts we put into addressing your concerns. A more straightforward comparison between two types of embeddings is indeed helpful. We believe  the Wasserstein metric (also known as Earth mover’s distance, the cost of an optimal transport plan to convert one distribution into another) is appropriate for estimating the similarity between two distributions. We sampled 1,000 tokens/entities to estimate the Wasserstein distance and present the results in the following table:

| (X, Y) | Wasserstein distance |
|---|:---:|
| (Token, Token)  | 1.4418 | 
| (Token, Entity)  | 2.0019 |

The Wasserstein distance between the sampled entity and token distributions is slightly larger than that of the two sampled token distributions. Therefore, it is reasonable to conclude that our method has successfully encoded the new KGL tokens into the original token embedding space. We will update the corresponding paragraph to discuss the Wasserstein distance results. Thank you again for your prompt and kind response.
Thanks for the thorough response. These low variances are nice to see, and help me trust the robustness of the results.  Thanks also for running the experiments on various Llama versions and sizes.

The new embeddings visualization is nice to see, but it presents only a couple examples, and does not directly compare to the original representations. I feel that a more systematic quantitative comparison would better address Weakness 4; this could, for example, be based on the average distance between entity or relation tokens that co-occur in queries in the original vs. new space.

That said, I consider the other weaknesses to be well-addressed, even if preliminarily. I'm therefore raising my score.