MKGL: Mastery of a Three-Word Language
Authors: Lingbing Guo
Zhongpu Bo
Zhuo Chen
Yichi Zhang
Jiaoyan Chen
Lan Yarong
Mengshu Sun
Zhiqiang Zhang
Yangyifei Luo
Qian Li
Qiang Zhang
Wen Zhang
Huajun Chen
Venue: NeurIPS 2024 spotlight (2024)
Keywords: Knowledge Graph
Large Language Model
Knowledge Graph Completion
Knowledge Graph Embedding
Low-Rank Adaption
Primary area: natural_language_processing
TL;DR: instruct an LLM in the language of knowledge graphs

Abstract: Large language models (LLMs) have significantly advanced performance across a spectrum of natural language processing (NLP) tasks. Yet, their application to knowledge graphs (KGs), which describe facts in the form of triplets and allow minimal hallucinations, remains an underexplored frontier. In this paper, we investigate the integration of LLMs with KGs by introducing a specialized KG Language (KGL), where a sentence precisely consists of an entity noun, a relation verb, and ends with another entity noun. Despite KGL's unfamiliar vocabulary to the LLM, we facilitate its learning through a tailored dictionary and illustrative sentences, and enhance context understanding via real-time KG context retrieval and KGL token embedding augmentation. Our results reveal that LLMs can achieve fluency in KGL, drastically reducing errors compared to conventional KG embedding methods on KG completion. Furthermore, our enhanced LLM shows exceptional competence in generating accurate three-word sentences from an initial entity and interpreting new unseen terms out of KGs.

MKGL: Mastery of a Three- Three-Word Language
Lingbing Guol Zhongpu Bo Zhuo Chen Yichi Zhang Jiaoyan Chen
Yarong Lan 1.2 Mengshu Sun Zhiqiang Zhang', Yangyifei Luo Qian Li", Qiang Zhang !2, Wen Zhang 7. and Huajun Chen 1.2+
College of Computer Science and Technology. Zhejiang University
27JU-Ant Group Joint Lab of Knowledge Graph
Ant Group
4 Department of Computer Science, The University of Manchester School of Computer Science and Engineering, Beihang University
School of Computer Science, Beijing University of Posts and Telecommunication School of Software Technology, Zhejiang University
Abstract
Large language models (LLMs) have significantly advanced performance across a spectrum natural language processing (NLP) tasks. Yet, their application to knowledge graphs (KGs), which describe facts in the form of triplets and allow minimal hallucinations, remains an underexplored frontier. In this paper, we investigate the integration of LLMs with KGs by introducing specialized KG Language (KGL), where a sentence precisely consists of an entity noun, a relation verb, and ends with another entity noun. Despite KGL's unfamiliar vocabulary the LLM, we facilitate its learning through a tailored dictionary and illustrative sentences, and enhance context understanding via real-time KG context retrieval and KGL token embedding augmentation. Our results reveal that LLMs can achieve fluency in KGL, drastically reducing errors compared to conventional KG embedding methods on KG completion. Furthermore, our enhanced LLM shows exceptional competence in generating accurate three-word sentences from initial entity and interpreting new unseen terms out of KGs.
Introduction
Knowledge graphs (KGs) are important resources for many data-driven applications, offering struc- tured repositories of factual information that empower variety of intelligent tasks 2 Yet, the strides made through the rapid advancement of large language models (LLMs) have challenged the conventional reliance KGs. Nonetheless, LLMs often critiqued for their susceptibility to generating factually incorrect or nonsensical outputs-a phenomenon known as the "hallucination problem"
Many recent studies propose resort KGs mitigate this problem (58).
In this paper, we investigate the capacity of LLMs to assimilate and generate knowledge graph facts proficiently. For example, the natural language sentence, "Wendee Lee is an actor in Mighty Morphin Power Rangers," translates into KG triplet format as (Wendee Lee, actor of, Mighty Morphin Power Rangers). It is worth noting that, English names such as Wendee Lee and Mighty Morphin Power Rangers, while can serve as identifiers for entities, are perceived as atomic elements within the KG framework. They are indivisible and distinct from their constituent words or characters.
When the L.MS interpret these text identifiers as mere sequences of tokens, they risk producing output that misrepresents entities relations, therefore compromising the integrity of KG-based tasks. Consequently, existing research that integrates L.LMs with KGs tends to limit its scope to relatively straightforward tasks. Examples of these limitations include validating the correctness of
"Correspondence to: (zhang.we) wen, huajunsir) @ @zju.edu.cn
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

KG Vocabulary
Laptimbattion agaier KGl rocabulrr
Score Estimation
Conditionnd KG Score Retrieval
Layre
Information Retrieval
Down Scaling
Transfemer
WIT1 PAOp
Blodks
Micrphis Poweer Redgenx
Score
Resembled
KGL Score Retriever
Embadding
retriove catimation
Token
Embodding
c.srsio ambording
Up- Scallng
Tokuniser
Suppased
Supposrd.... giwen the blllwinngketionrryoonng
Token Emsbodding
Text Infeemationn Retrieval
two words from this language:
Down Scaling
Textual
Word
Type
Translatton
Ebssmple
collertter
Instruction
Q.
entity ioun
Wendee
relation varts actor of
plesen kindly complete the followingrnttonn
KGL Context Retriever
retriene vector
Figure I: A workflow of MKGL (from bottom to top). The instruction to the LLM includes dictionary exemplifying the entity e, and relation rk. The task is construct new KG sentences initialized with e The tokenizer first tokenizes the input text, where the entities and relations are represented as special tokens out of the original vocabulary. (a) To process these special tokens, MKGL. collects the embeddings of their constituting text tokens; (b) Then, retriever performs 4-step process to aggregate textual and relational information into KGL token embeddings. The first and the last steps are LoRA-like down-scaling and up-scaling operations [2]; (c) The output assigned as the embeddings of these special KGL tokens; (d) Similar to the context retriever, we design score retriever to retriever the score information. (f) The output is in a form of probability distribution among candidate entities.
fully-formed triplets [9] picking appropriate entity from limited set of options 10] Given the sheer volume of entities in KG, such narrow applications fall short in addressing more complicated tasks like KG completion, wherein a model predicts missing components of a provided incomplete triplet, c.g., identifying the unknown tail entities in (Wendee Lee, actor of. ?) against thousands of candidates. While these methods may lean pretrained KG embedding models to narrow down possible candidates, the process remains inefficient.
To transcend the limitations predictive scope, we propose a novel approach, named MKGL, to instruct an LLM the lexicon of the unique KG language (KGL). KGL sentences are strictly three- word sentences, starting with an entity noun, followed by a relation verb, and ending with another entity noun. The vocabulary of KGL does not immediately resonate with the machines. A common triplet like (Wendee Lee, actor of, Mighty Morphin Power Rangers) is encoded abstractly as e:Taej with ei, e, symbolizing the entity nouns and rj denoting the relation verb. For an LLM such as Llama-2 [T]] these symbols are entirely alien, absent from its pretraining corpus. Our investigation thus centers on how an LLM can navigate and master this specialized, atomic language of KGs.
As illustrated in Figure I to bridge this comprehension gap, we introduce an English- dictionary, and the LLM supposed to assemble new KG sentences using the provided linguistic building blocks. The basic elements of KGL, while different from our natural language, are familiar to the LLM as they are constructed from the pretrained token embeddings. We leverage a context retriever to retrieve the text information and relational information of a KGL token, which transforms the sequential token embeddings of its name into an embedding vector. Subsequently, we update the LLM token embedding layer with new KGL token embeddings. In the scoring layer, we also employ a KG score retriever to supplement the LLM with extra KG relational information for prediction.
Instructing an LLM in KGL offers three main advantages over prompt-based methods [10 [3] or conventional KG embedding methods 14 15]: : (1) Broadened applicability. KGL tokens originate from textual tokens of an LLM, thus our method does not mandate that all entities be observed during training. (2) End-to-end framework. Unlike recent LLM- based methods that necessitate pre-sorted results from conventional KG embedding methods, our approach can rapidly rank all candidate entities one-step. (3) High efficiency. The representations of KG tokens are derived from pretrained token embeddings rather than learned from scratch. The proposed KGL. context and score retrievers also leverage LoR Alike adaption. Using Llama-2-7b [T] as the base L.M,, the number of training parameters is less than 0.3%.

---

However, instructing an LLM in KGL also has its limitations, as it demands more computational resources compared with conventional methods. For instance, fine-tuning MKGL (Llama-2-7b) on the 2B335-237 dataset [6] to outperform most conventional methods requires only epoch. Nevertheless, with A100 GPUs, it still takes half an hour, which is comparable training a TransE model from scratch with single GPU.
2 Related Works
We category the related works into two groups:
Knowledge Graph Completion KG completion can be regarded as a classification problem like many NLP tasks 21, such as node classification and semantic role labeling. However, its label space is significantly larger than most NLP tasks. For example, the WN18RR [22] dataset contains over 40.000 different entities, making impractical to simply feed them all as possible results and let the LLM select one as output. Most conventional KG completion methods are embedding-
methods, including the triplet-based methods 223-27 e.g. TransE (23, ComplEx
[25) RotatE [26]: the ENN-based methods 15 32 e.g., DAN 15 CompGCN (28. CoKE [29]: and other neural-
based methods [22 33 e.g.. ConvE 22 and RSN (4). Despite differences in their neural
methods and input forms, all these methods focus relational information and are not good at utilizing other types information such as textual attributes.
Pretrained Language Models for Knowledge Graphs Leveraging Pretrained language models for KG completion has been explored for many years [36]. Some works treat BERT GNN model to encode graph features [30 37], while others consider the textual information of KGs and use pretrained BERT to encode the textual labels of entities and relations [14] 38 40
The resulting
outputs are regarded as the entity and relation embeddings or their concatenations.
With the rapid advancements in leveraging LLMs for KG completion, recent works have begun design- ing prompts or instructions to guide LLMS in this task. Initially, the results were not promising [41], as it appeared that even state-of-the-art LLMs without context information could not outperform basic KG embedding models like TransE. However, subsequent works such as KGLlama [42] and KoPA [3] discovered that LLMs might perform better in triplet classification, i.e., estimating the correctness of a given triplet.
provide explicit instructions and guide the behavior of L.Mss. This involves a triplet-based KG embedding model to generate the initial rankings of the top-k entities, followed by a multi-round interaction with the LLM, providing textual information and triplet demonstrations for the query entity and relation. The LLM should then re-rank the initial list. KICGPT has achieved state-of-the-art results on KG completion tasks. However, its performance not only depends on the LLM and the instructions but also on the pretrained KG embedding model. Additionally, KICGPT cannot be deployed offline due to the demand of commercial LLMs
More recently, KICGPT [10 has proposed leveraging in-context learning 44 44
45
It also cannot provide embeddings for downstream tasks.
In contrast, the proposed MKGL has an embedding module based on the LLM token embeddings and KG relational information, which overcomes the weaknesses of existing KG embedding methods that cannot provide embeddings for unseen entities. The context information is implicitly encoded into the KGL token embeddings and efficiently captured by the L.M during fine-tuning.
3 Mastery of KG Language
In this section, discuss the details of MKGL. We first introduce the general architecture of an LLM and how to convert a KG triplet into fine-tuning instruction. Then, we present the details of constructing KGL token embeddings and scores. Finally, we illustrate how train an MKGL and analyze its complexity.
3.1 Preliminaries
We start by brief introduction to KGs and LLMs.
3

---

Knowledge Graph and Knowledge Graph Language Knowledge graphs are conceptualized as directed, nulti-relational graphs. We describe knowledge graph by 9 = (T. R), where T, E. RR are the sets of triplets, entities, and relations, respectively. KG language (KGL) is construed as rigorously defined three-word construct, mirroring the structure of a simple sentence. Specifically, a sentence e,TRe; invariably commences with an entity noun e, E &, proceeds with a relation verb The E R, and culminates with another entity noun ej E &. Analogous to the syntactic conventions in Chinese, KGL sentences eschew the use of spaces or commas to demarcate KGL terms.
Knowledge Graph Completion KG completion is one of the most important tasks in the KG area. The target of KG completion is to predict the head entity given the relation and tail entity (2, (.ryyek) ex), or predict the tail entity ex given (et, . In the scenario of KGL, this task is equivalent to completing the KG sentence ?rye; or exr,?.
The inductive KG completion focus completing an unobserved KG Giad (Tind- Eind, Rind)- Specifically, the relation set Rand is identical to the original set R. but the inductive entity set Eind shares elements with E, i.e., Emd n 3 Ø. The triplet set Tind is further split into the fact set Taud-iact and test set Tind-test- We train a model on the original triplet set T and use the fact set Tind-fact as context evaluate it on the test set Tind-test-
Large Language Models As depicted on the left side of Figure the architecture of a typical I.I can be divided into four main components:
Tokenizer, which breaks down the input sequence of words wo, W1, ... Wm into tokens ty. t1, .... tnj Token embedding, which maps the input tokens to f,,,.., tn to a sequence of low-dimensional vectors to: t1,...,tu. tn:
Transformer M. the core of the LLM, which consists of multiple attention-based blocks that process the input token embeddings into hidden states:
ho, hi.,, h, = M(to, t,,...,,,.)..
Score layer, which features weight matrix S Nxd with an identical shape to the token embedding matrix xdx xd, where N, d denote the vocabulary size and hidden size, respectively, The score layer projects the output of Transformer at the n2-h step to a probability distribution Px+1 for predicting the next token 1n+1:
Pa+1 = S,,,,
(2)
3.2 Instruct an LLM in KG Language
Recent studies reveal that LLMs harbor the potential to acquire unfamiliar natural languages 46 47
Given this premise, is of particular interest to investigate how LLMs might interpret and operate within our KGL. We first design prototype instructional text for this purpose. For a given triplet (Wendee Lee, actor of. Mighty Morphin Power Rangers), suppose that the task is predict the tail entity Mighty Morphin Power Rangers, the instructional text is formatted as follows:
Instruction 3.1. Supposed that you are linguist versed in esoteric three-word knowledge graph language. Given the following dictionary comprising two words from this language, please kindly
Word
Type
Translation
Example
<kgl:: Wendee Lee'>
entity noun relation verb
Wendee Lee actor of
<kgll Wendee Lee <kgl: profession < <kkl: Actor'> <kgl: Peter O Toole"> <kgl: actor <kgl: Gulliver's is Travels
<kgli actor of>
Table I: An illustrative KGL-to-English dictionary.
complete the following sentence: <kgl: Wendee Lee"> <kgl: actor of>
Here, <kgl: Wendee Lee>> denotes the definitive KGL token (corresponding to e; in previous sections and Figure assigned to the entity Wendee Lee. We enrich the tokenizer's vocabulary with all pertinent KGL tokens, thereby enabling it to translate these KGL tokens into token IDs, which append sequentially to the LLM's original vocabulary range. It is worth noting that we only provide at most one example KGL sentence for each KGL word. Our intention is to introduce the schematics of KGI, sentences to the LLM, rather than leveraging augmented KG data for in-context learning. To mitigate potential biases, the example sentences are sampled randomly.

---

LLM Tuken
Scaled Embedding
Sequeace of Tokens
Retrieved Test
Retrieved Relational
RGL. Token Embedding
Assembled Input Embeddingg
Embedding
Suppoasd that Inguie...
<hihh Weedee
ITAOR SIE TEaR
2 Aeturals
compltnethe sentence
PNA
mults-layend PKA
high Latg-ckggaaaeeeeee
a.scale
information
scale up
input embeddingg
Figure 2: Illustration of LoRA-baat A-bsed KGL Context Retriever. (a) The token embeddings are first scaled down to power-dimensiome vectors; (b) For each input KGL token, their constituting textual token embeddings are aggregated by PNA encoder; (c) The output embeddings are further aggregated by multi-layered PNA encoders retrieve neighboring information within KG; (e) The final embeddings are assigned to the KGL tokens.
3.3 In-Context Learning versus Special Token Embedding
The practice of incorporating supplementary context information alongside instructional prompts, known as in-context learning (ICI.), has proven effective enhancing performance across many NLP tasks 43 However, the concatenation of retrieved context on KGs with the input text can easily exceed the input length constraints of LLMs. Processing such long input sequences remains com- putationally intensive even with truncation. To address these constraints, we propose alternative approach to encode context information into compact vector representations. Our experiments in Section 4.6 also demonstrate its superiority in terms of both efficiency and performance.
3.4 LoRA-based KGL Context Retriever
We propose the low-rank adaption (LoRA)-based KGL context Retriever Reontext to effectively aggregate textual and KG information into KGL token embeddings. Typically, the vocabulary scope of a KG (comprising both entities and relations) usually surpasses that of an LLM. For instance, WN18RR unique entities, while the vocabulary size of Llama-2-7b is 32,000. Therefore, initializing new token embeddings for each KG elements and optimizing them from scratch would be prohibitively resource-intensive.
KG completion dataset set sampled from WordNet [48). It has over 40,000
Moreover, the dynamic nature of real-world KGs consistently introduces entities. This is analogous to the evolution of human language, where new words are often synthesized or derived from existing ones. Drawing inspiration from this linguistic adaptability, propose leveraging existing textual tokens to generate new KGL tokens, thereby avoiding the computational burden of learning unique embeddings for every KG element.
Scale Down As illustrated in Figure 2 the first step is to reduce the dimensionality of LLM token embeddings to lower computational demands during text and KG context aggregation. Inspired by LoRA [12, we leverage projection matrix Wy E Rdxr to transform the token embedding matrix T R Xxd into reduced space IR Nx.;
T TWT.
where T, E Nxr represents the compact token embedding matrix.
leverage a text encoder to encode the textual token embeddings of each KGL token into unified vector. For example, the entity name "Mighty Morphin Power Rangers" would converted into individual token embeddings tog.0s te,, beg which are then aggregated into single vector for the entity e;:
Retrieve Text Information
te: Erexr(tes. ter, .... tern))
(4)
5

---

where te, is the textual token embedding for é;. The choice of the encoder Euxt is free. In this paper, we leverage principal neighbourhood aggregation (PNA) 49) which can be roughly understood as applying multiple pooling operations (including max, min, mean, std etc.) on the token embedding sequences. detailed introduction to PNA can be found in Appendix C
Retrieve KG Information We employ a multi-layered PNA encoder Egg to aggregate the KG information of e, and its adjacent entities, which can be formulated
= Et
(5)
where (e;, denotes the neighboring entities to eç. The adoption of PNA for encoding both textual and relational data of KGL tokens is due to its parameter efficiency and superior performance compared to attention-based alternatives like GAT 50 An empirical comparison of different encoders can found Appendix
Scale Up To finalize, we adjust the dimensionality of the output embeddings to align with the LLM input requirements:
t" = W
For the sake of clarity, we will continue to use tez represent the KGL token embedding in subsequent discussions. For efficiency, we retrieve the KG information only for entities. This operation also make the embeddings of entities and relations distinguishable.
3.5 Reconstructing Vocabulary or Constraining the Output Space
While recent studies have adapted LLMs various tasks by either restricting the output space or reformulating tasks into multiplc-choice questions ( I0 53153), such strategies pose challenges for KG completion. Specifically, the existing methods are inapplicable to entity constrastive learning as their main objective is optimized against text tokens instead of entities. Also, they incur significantly slow inference times, as the LLM must traverse the output tree's leaf nodes generate predictions. Even then, the generation of top-k results, dependent on beam search parameters, may not accurately reflect the true likelihoods of entities.
In contrast, in this paper propose new approach to reconstruct the KGL scores through LLM's score layer and hidden states, providing a one-shot probability distribution for all candidates. Our method seamlessly integrates with contrastive loss and negative sampling techniques [54), making it highly compatible with prevalent KG completion frameworks. This compatibility also ensures that MKGL has the potential of being applied for downstream KG embedding tasks [32 55
3.6 LoRA-based KGL Score Retriever
We propose LoRA-based KGL score retriever Rycore to produce the probability distribution of KGL tokens, which can be formulated as follows:
s' SWs, h' hn WH
(Down Scaling)
(7)
si Sreul'', 8 s..1.... Szn);
(Text Information Retrieval)
(8)
5 Syleur Szellh' stT,N(e)))
(Conditioned Retrieval)
(9)
Peylesrr
5 Sle Wo
(Score Estimation)
(10)
The score retriever also starts from down-scaling layer reduce the dimensionality of the score matrix E xd to Sg E R with W and similarly scales down the LLM's output hidden vec- tor h, with WH. H. Subsequently, the text information (i.c., the token score vectors s'.n, s.. s'.......S.... Sy.n) associated with target entity e, is fed to the score text encoder S to construct the KGL score vector s'. It is then concatenated with the LLM hidden state h', to obtain the conditioned input [h,, sy]. Upon gathering the neighboring information of the target entities via a multi-layered PNA Sig. an output matrix Wo E [Rrx1 is employed to map the result s" Syle,,r E RT to the 1-d probability estimate Peyler.rs R.

---

Table 2: The KG completion results on FB15K-237 and WN18RR. The best and second- results are boldfaced and underlined, respectively. 1: higher is better; 4: lower is better. -: unavailable entry.
FB15k-237
WNISRR
Model
MRR
Hits@11
Hits@3
Hits@ 101
MRR1
Hits@1
Hiss
Hits@ 101
TransE
23
.310
.218
.345
.495
232
.061
.366
.522
RotatE: 26
.338
.241
.355
.533
476
.288
.492
.571
TuckKR
56
.358
.266
.344
.544
470
.443
.526
.566
CompGCN
.355
.264
.300
.535
479
.43
.94
.546
DAN
458
.422
CoKE
.364
.272
.400
.549
.844
450
.466
.553
KG-BHKI 14
.420
.216
.041
.302
.524
StAR 38
.296
.205
.322
.482
401
.243
.411
.709
KGLM [40
.289
.200
.314
.468
.467
.330
741
FTL-LM
.348
.253
.386
.521
.543
.452
.637
.773
DET 30
.376
.281
.560
.507
.665
.585
KG-Lluma-7b
42
242
GPT 3.5 Turbo
.267
.122
KICGPT 101
.412
.327
.554
.549
.61
MKGL
.115
.325
.454
.591
.552
.500
.577
.656
Optimization With the above score retriever, estimating the probability for any candidate entity becomes straightforward at single step. To refine MKGL, we consider a contrastive loss leveraged most existing KG embedding methods [22
28
34 expressed as:
L
Σ
[-log(Pe,jewe (enrr)) +
log(1 - Peaaleerre)l (11)
[ (c,)) (nng
(es,
where No (ez) = {enegleneg # ejz Oneg E E} the sampled negative entity set for the target entity eg. The loss function L. is in form of a binary cross-entropy, contrasting the likelihood of correctly predicting the relation Peylec.r, as positive example, against the probabilities erroneously predicting relations (e;, Tk, Eneg) as negative examples. We also present an algorithm to demonstrate the step-by-step fine-tuning process, please refer to Appendix for details.
3.7 Complexity
is clear that the primary computational cost for MKGL lies in the LLM. By employing LoRA-based KGL retrievers to retrieve context vectors instead of texts, we can significantly reduce the major expenditure. For instance, our retrievers can reduce the average input lengths from 1.2 91.4 on the FB15k-237 dataset, compared to using one-hop neighbors for in-context learning. All operations within the LoRA-basd Abased retrievers are performed under low dimensionality. Furthermore, the token embeddings and score matrix of the LLM are frozen during fine-tuning, thus ignoring their gradient computation. the worst case, the complexity of text information retrieval O (NNigI where Nigl. Tagi, are the number of KGL tokens, maximum text token lengths of KGL tokens, and the reduced dimensionality, respectively. Subsequently, the complexity of KG information retricval the worst case is linear to the number of triplets, i.c., [(T]]Nnyer" [Nnyerr), where [T), Ninyr denote the number of triplets in the KG and the number PNA layer, respectively.
Experiments
In this section, we evaluate the performance of the proposed MKGL through extensive experiments, comparing against both LLM--bssed and KG embedding methods. The source code and datasets are available at Ctthhbb.coo//jukg/MKGLLL..
4.1 Datasets
We evaluate MKGL. on the FB15k-237 and WN18RR datasets, which are widely used by most KG completion methods 22 23
26
28
34
56, 57]. We also evaluate MKGL on the inductive version of
7

---

Table 3: The inductive KG completion results FB15k-237-mm and WN18RR-ind (v1). The results on other subsets can be found in Appendix F]
FB15k-237-ind
WNISRR-in ind
Model
MRR1
Hits@11
Hits@ 10
MRR1
Hits@ !1
Hits@ 10
RuleN 60
.363
.309
446
.668
.355
.730
Neurall.F P 33
.325
.243
468
.69
.52
.772
DRUM 61
.333
.247
.444
.666
.613
.777
GraiL
58
.279
.205
.499
.627
.554
.760
RED-GNN
366
.302
483
.701
.653
779
CharGPT 3,5 Turbo 41
.288
.279
MKGL
.475
.400
.595
.746
.700
.822
Head entiyy
Relation
Tall eatiyy
FB15K-237
<Priusetime Emmy Awards>
Chominated tor>
<Templ Grancioo (timm)>
MRR
0.360
10.047
<Backstalrs the White House>
Hiss@
0..061
<John Adams
V
<Gulliver's Travels (miniseries)s>
HI
0.508
0.063
Coward winnerr
<kar
conrermany'
<60th Primetime Erumy Awardss>
WN18RR
<play, movee
<verb group?>
<play, put or piece) into play duringa game> <utilize, make work or employ for particular purpose >
0,490
0.056
Hits@h
0.431
0.069
<play, participate
sport>> <play, employ game spectfic position>>
H1
agalisst opponent in sport, game, or bottle>
Figure Illustration of KGL modeling. The left shows the performance degradation (in lighter shades) from consecutive predictions of relations and entities. The right presents sentences generated by MKGL, with deeper hues indicating higher probabilities. In the final column, colors grey, green, yellow, and red represent existing, valid, valid but not within the KG, and invalid, respectively.
these two datasets 158
We follow REDGNN 59 to evaluate MKGL on all entities rather than 50
sampled candidates. Please refer Appendix F for dataset statistics.
4.2 Settings
For our experiments, we employ Llama-2-7b 11 as the base L.M and train MKGL using A100 GPUs. A standard LoRA adaptation is applied to the query and value layers of the LLM. Full hyper-parameter details are available in Appendix [ We evaluate performance using MRR (mean reciprocal rank of target entities) and Hits @ (percentage of target entities ranked in the top k).
Our baselines include conventional KG embedding methods such TransE 23 RotatE 26], and
TuckER [56]: GNN-based methods like CompGCN (28), DAN [15]. and COKE 22]; methods
that integrate language models including KG-BERT [1]. StAR 38 KGLM 40) FTL-LM [39].
and DET 30;; and M.Le-b methods: KG-Llama [4). GPT 3.5 [41, and KICGPT [T0] In
the inductive scenario, we compare against rule-based reasoning methods such as RuleN (60)
NeuralI.P (33, DRUM (61), Grall. (58) and RED-GNN [5), acknowledging that standard methods fail to predict relations without entity embeddings.
4.3 Knowledge Graph Completion
The knowledge graph completion results are presented Table MKGL outperforms other baselines in nearly all metrics. Notably, MKGL and KICGPT significantly surpass other LLM-based methods, demonstrating the importance of KG relational information. Contrarily, many BERT-basee methods fall short against GNN-based methods, suggesting that merely incorporating text information may not yield the anticipated benefits. summary, the proposed MKGL clearly outshines its counterparts, particularly those founded commercial LLMs.
To our knowledge, existing M--ed methods have not addressed the inductive KG completion challenge. We benchmark MKGL against the state-of-the-an inductive methods. Although we can

---

Table 4: Ablation studies FB15k-237 and WNI8RR.
Score Retriever
Context Retriever
FB15K-237
WNISRR
Text
KG
Text
KG
MRR
Hits If
Hits 101
MRRT
Hits@ 11
Hits 101
A155
.325
.591
.552
.500
.656
.382
.294
.556
.541
.482
.649
.365
.272
.550
.512
.470
.622
.359
.260
.546
.492
.437
.615
.335
.247
.535
.466
.376
.574
Performance
Trainable Parameter
Avg. Input Length
Train Time
Max Batch-Size
0.1
million
800
GPU hour
3000
8.3
SYED
100
PI6
F1G
PT6
150
6.0
0.2
NewToken
NewToken (1-hop)
NewToken (2-hop)
+ MMKGL w/o SR
MKGL
entity token embeddings (denoted by NewToken) and MKGL. on FB15k-237. 1-hop and 2-hop are the versions leveraging the 1-hop and 2-hop KG neighboring information. MKGL w/o SR denotes MKGL without the score retriever.
Figure 4: comprehensive comparison between the methods with randonty-initialized
not assess KICGPT [0] due unavailable source code, it worth mentioning that our MKGL could potentially augment KICGPT by supplying a candidate list, facilitating seamless integration between the two methods. present the results in Table MKGL significantly outperforms all the baseline methods across metrics. As most inductive reasoning methods do not have an embedding module for entities, the proposed MKGL represents the first embedding- method to deliver state-of-the-art results in inductive KG completion.
4.4 Knowledge Graph Language Modeling
Beyond its capability as a KG completion tool, MKGL also serves as a language model for KG languages. To evaluate its proficiency in generating KGL sentences, we employ sequence generation loss and remove the relation context from the input prompts. We leverage the second-to-last output of the L.LM for relation prediction.
The results are shown Figure The left section contrasts the sequence prediction results against standard KG completion, revealing only modest loss in performance. MKGL still outperforms many conventional methods, especially on WNI dataset. The right panel displays sample sentences generated by MKGL, illustrating its potential to discover legitimate KGL sentences absent from the existing KG. We observe that WN18RR is more difficult than anticipated as contains many plausible entities that challenge even LLM's discernment.
4.5 Ablation Study
We conduct ablation studies to assess the importance each module, as detailed in Table The unmarked cells indicate that either substitute the text retrieval module with learnable embedding module or remove the KG retrieval module. Clearly, the method with complete features achieves best results, while replacing removing either module significantly impacts performance. Notably, removing the KG retrieval module yields more performance loss on WN IRRR, many entities in this dataset have similar names. For example, there 14 different entities named "call". In this case, incorporating with KG information becomes necessary.
4.6 Computational Cost
examine the computational efficiency of our method (MKGL) relative to "in-context" baselines. Specifically, we develop several variants: LLM randomly-initialized new entity token embeddings (NewTokeen). LI.M with KGL. context from 1-hop neighbors (NewToken (1-hop)), L.M with KGL. context from 2- h-hop neighbors (NewToken (2-hop)), and MKGL without score retriever (MKGL w/o

---

SR). The results are shown in Figure 9 MKGL surpasses all alternatives in performance. NewToken variants slightly lag behind MKGL w/o SR, but notably, our proposed methods demand fewer trainable parameters than NewToken variants. By encoding all context information within KGL token embeddings, the average input length is significantly reduced, which decreases training time considerably. Moreover, MKGL supports larger batch sizes during both training and inference phases, enhancing computational efficiency.
5 Conclusion and Future Work
In this paper, we propose MKGL to instruct the LLM in the language of KGs. MKGL employs context retriever that efficiently provides LLMs with pertinent textual and relational context, markedly reducing input lengths relative to in-context-learning and supervised fine-tuning methods. Meanwhile, MKGL also leverages inference. Extensive experiments confirm the superiority of MKGL in terms of both performance and computational efficiency. The proposed context and score retrievers point out a new direction in incorporating LLMs with semantic data, such as question answering and entity linking. They may also shed lights a more broaden area where the input cannot be precisely represented by text, e.g., node classification and protein representation learning. Furthermore, the construction of KGL vocabulary enables contrastive learning not only limited on tokens, which may provide insights on general machine learning. Therefore, there are plenty of future directions. We would like to pretrain LLM using the mixture corpora of KG and natural languages, such that the LLM could understand and create responses with linked data.
retriever to supply score information and aid in KGL
Acknowledgement
This work is founded by National Natural Science Foundation of China 200088/66SS22222225SSSSC22222220
Zhejiang Provin- cial Natural Science Foundation of China (No. LQ23F020017), Yongjiang Talent Introduction Programme (2022A-238-G) and Fundamental Research Funds for the Central Universities
(226-2033000138). This work was supported by AntGroup.
References
[1] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition and applications. CoRR, abs/2002.003888, 2020.
[2] Andrea Rossi, Donatella Firmani. Antonio Matinata, Paolo Merialdo, and Denilson Bar- bosa. Knowledge graph embedding for link prediction: comparative analysis. CoRR. abs/2002.008 19, 2020.
[3] Ziwei Ji, Nayeon Lec, Rita Fricske, Tiezheng Yu, Dan Su. Yan Xu, Etsuko Ishii, Ye Jin Bang, Andrea Madotto, and Pascale Fung. Survey of hallucination in natural language generation. ACM Computing Surveys, 55(12):1-38, 2023.
[4] Vipula Rawte, Amit Sheth, and Amitava Das. A survey of hallucination in large foundation models, arXiv preprint arXiv:2309.055. 05922, 2023.
[5] Linyao Yang, Hongyang Chen, Zhao Li, Xiao Ding, and Xindong Wu. Chatgpt not enough: Enhancing large language models with knowledge graphs for fact-aware language modeling. arXiv preprint arXiv:2306.1 1489, 2023.
[6] Xinyan Guan, Yanjiang Liu, Hongyu Lin, Yaojie Lu, Ben He, Xianpei Han, and Le Sun. Miti- gating large language model hallucinations via autonomous knowledge graph-based retrofitting. In AAAI, pages 18126-188 22-11814, 2024.
[7] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: roadmap. IEEE Transactions on Knowledge and Data Engineering, 2024.
10

---

[8] Jeff Z Pan, Simon Razniewski, Jan- Christhtitoph Kalo, Sneha Singhania, Jiaoyan Chen, Stefan Di- etze, Hajira Jabeen, Janna Omeliyanenko, Wen Zhang, Matteo Lissandrini. et al. Large language models and knowledge graphs: Opportunities and challenges. arXiv preprint arXiv: urXiv:3388.06374 2023.
[9] Yichi Zhang, Zhuo Chen, Wen Zhang, and Huajun Chen. Making large language models perform better in knowledge graph completion. arXiv preprint arXiv:23 2023.
[10] Yanbin Wei, Qiushi Huang, James T Kwok, and Yu Zhang. Kiegpt: Large language model with knowledge in context for knowledge graph completion. arXiv preprint arXiv:2402.02389 2024.
[11] Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, et al. Llama 2: Open foundation and fine-tuned chat models. arXiv preprint wwx.i:2307.09222 2023.
[12] Edward Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2 2021.
[13] Yichi Zhang, Zhuo Chen, Wen Zhang, and Huajun Chen. Making large language models perform better in knowledge graph completion. arXiv preprint arXiv:2310.066711 2023.
[14] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv: 1909.0319,, 2019.
[15] Lingbing Guo, Zhuo Chen, Jiaoyan Chen, Yichi Zhang, Zequn Sun, Zhongpu Bo, Yin Fang, Xiaoze Liu, Huajun Chen, and Wen Zhang. Distributed representations of entities in open- porldworld knowledge graphs. Knowledge-Basee Systems, page 111582, 2024.
[16] Kurt D. Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: A collaboratively created graph database for structuring human knowledge. In SIGMOD, pages 1247-1250, 2008.
[17] Aditya Grover and Jure Leskovec. node2vec: Scalable feature learning for networks. In KDD, pages 855-864, 2016.
[18] William L. Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In NIPS, pages 1024-1034, 2017.
[19] Chaoyi Ai and Kewei Tu. Frame semantic role labeling using arbitrary-order conditional random fields. In AAAI, pages 17638- 17646. AAAI Press, 2024.
[20] Roberto Navigli, Marco Pinto, Pasquale Silvestri, Dennis Rotondi, Simone Ciciliano, and Alessandro Scirè. Nounatlas: Filling the gap in nominal semantic role labeling. In ACL(1), pages 16245-1666 16258. Association for Computational Linguistics, 2024.
[21] Jinan Zou, Maihao Guo, Yu Tian, Yuhao Lin, Haiyao Cao, Lingqiao Liu, Ehsan Abbasne- jad, and Javen Qinfeng Shi. Semantic role labeling guided out-o--distribution detection. In LREC/COLING, pages 14641-14651. ELRA and ICCL, 2024.
[22] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2D knowledge graph embeddings. In AAAI, pages 181 1-1818, 2018.
[23] Antoine Bordes, Nicolas Usunier, Alberto Garcia-Durán, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In NIPS, pages 2787-2795, 2013.
[24] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. In ICLR, 2015. http
//arxiv. www.orgggbbb.... abs/aas//111 6575
[25] Théo Trouillon, Johannes Welbi, Sehastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In ICML, pages 2071-2080 2016.
11

---

[26] Zhiqing Sun, Zhi-Hong Deng. Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embed- ding by relational rotation in complex space. In ICLR, 2019. /ttpps/////pppeeeviii
[27] Zequn Sun, Jiacheng Huang, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu. Transedge: Translating
embeddings for knowledge graphs. In ISWC, volume 11778 of Lecture Notes in Computer Science, pages 612-629. Springer, 2019.
[28] Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha P. Talukdar. Composition-based multi-relational graph convolutional networks. In ICLR, 2020. URL httptttt/ppppeeeiiiiiiiee net//forum/id-ByyAA.com C4tPr
[29] Quan Wang, Pingping Huang, Haifeng Wang. Songtai Dai, Wenbin Jiang, Jing Liu, Yajuan Lyu, Yong Zhu, and Hua Wu. Coke: Contextualized knowledge graph embedding. arXiv preprint arXiv:1911.02166. 2019.
[30] Lingbing Guo, Qiang Zhang, and Huajun Chen. Unleashing the power of transformer for graphs. CoRR, abs/2202. 10581, 2022.
[31] Yichi Zhang, Zhuo Chen, Linghing Guo, Yajing Xu, Binbin Hu, Ziqi Liu, Wen Zhang, and Huajun Chen. Native: Multi-modal knowledge graph completion in the wild. In SIGIR, pages 91-101. ACM, 2024.
[32] Zhuo Chen, Jiaoyan Chen, Wen Zhang, Lingbing Guo. Yin Fang. Yufeng Huang, Yuxia Geng, Jeff Z Pan, Wenting Song. and Huajun Chen. Meaformer: Multi-modal entity alignment transformer for meta modality hybrid. arXiv preprint arXiv:2212.14454, 2022.
[33] Fan Yang, Zhilin Yang, and William Cohen. Differentiable learning of logical rules for knowledge base reasoning. In NIPS, pages 2319-2328, 2017.
[34] Lingbing Guo. Zequn Sun, and Wei Hu. Learning exploit long-term relational dependencies in knowledge graphs. In ICML, pages 2505-2511 2019.
[35] Mingyang Chen, Wen Zhang, Wei Zhang, Qiang Chen, and Huajun Chen. Meta relational learning for few-shot link prediction knowledge graphs. In EMNLP, pages 4216-4225, 2019.
[36] Jacob Devlin, Ming-Was Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. In NAACL-HLI, pages 4171-4186, 2019.
[37] Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofei Zhang, and Yangfeng Ji. Hitter: Hicrarchical transformers for knowledge graph embeddings. In EMNLP, pages 10395-10407, 2021.
[38] Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. Structure- augmented text representation learning for efficient knowledge graph completion. In The Web Conference, pages 1737-1748, 2021.
[39] Qika Lin, Rui Mao, Jun Liu, Fangzhi Xu, and Erik Cambria. Fusing topology contexts and logical rules in language models for knowledge graph completion. Information Fusion, 90: 253-264, 2023.
[40] Jason Youn and Ilias Tagkopoulos. Kglm: Integrating knowledge graph structure in language models for link prediction. arXiv preprint arXiv:2211.0274. 2022.
[41] Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. Llms for knowledge graph construction and reasoning: Recent capabilities and future opportunities. arXiv preprint trXiv:2305.13166 2023.
[42] Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308. / 13916, 2023.
[43] Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiyong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. A survey on in-context learning. arXiv preprint 2rXiv:2301.00234 2022.
12

---

[44] Noam Wies, Yoav Levine, and Amnon Shashua. The learnability of in-context learning. NeurIPS, 36, 2024.
[45] Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Florencia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report. arXiv preprint arXiv:2303.08774, 2023.
[46] Zoltan Csaki, Bo Li, Jonathan Li, Qiantong Xu, Pian Pawakapan, Leon Zhang, Yun Du, Hengyu Zhao, Changran Hu, and Urmish Thakker. Sambalingo: Teaching large language models new languages. arXiv preprint
2024.
[47] Xuan-Phi Nguyen, Wenxuan Zhang, Xin Li, Mahani Aljunied, Qingyu Tan, Liying Cheng, Guanzheng Chen, Yue Deng, Sen Yang. Chaoqun Liu, et al. Seallms-large language models for southeast asia. arXiv preprint arXiv:2312.007388 2023.
[48] George A. Miller. WordNet: An electronic lexical database. Communications of the ACM, 38, 1995.
[49] Gabriele Corso, Luca Cavalleri, Dominique Beaini, Pietro Lió, and Petar Veličković. Principal neighbourhood aggregation for graph nets. NeurIPS, 33:13260-13271, 202.0
[50] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Lid, and Yoshua Bengio. Graph attention networks. In ICLR, 2018. URL https://openreview htw.:ppprrrevieee.eefe/rrrrrr
JX=rXKM
[51] Senbao Shi, Zhenran Xu, Baotian Hu, and Min Zhang. Generative multimodal entity linking. arXiv preprint arXiv:2306.12725, 2023.
[52] Nicola De Cao, Gautier Izacard, Sebastian Riedel, and Fabio Petroni. Autoregressive entity retrieval, In ICLR. OpenReview.net, 2021. URL https://openreview.ncem wtw.:ppenwweeeeeeeteeeoorruuaaaa net/forum?id 5K8F6UU39V
[53] Nicola De Cao, Ledell Wu, Kashyap Popat, Mikel Artetxe, Naman Goyal, Mikhail Plekhanov, Luke Zettlemoyer, Nicola Cancedda, Sebastian Riedel, and Fabio Petroni. Multilingual au- toregressive entity linking. Transactions of the Association for Computational Linguistics, 10: 274-290, 2022.
[54] Michael Gutmann and Aapo Hyvärinen. Noise-contrasiive estimation: A new estimation principle for unnormalized statistical models. In AISTAS, pages 297-304, 2010.
[55] Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Nitish Shirish Keskar, and Caiming Xiong. Modeling multi-hop question answering as single sequence prediction. ACL, pages 974-990, 2022.
[56] Ivana Balazevic, Carl Allen, and Timothy M. Hospedales. Tucker: Tensor factorization for knowledge graph completion. In EMNLP-IJCNLP, pages 5184-5193, 2019.
[57] Farahnaz Akrami, Lingbing Guo, Wei Hu, and Chengkai Li. Re-evaluating embedding- based knowledge graph completion methods. In CIKM, pages 1779-1782. ACM, 2018
(58) Komal Teru, Etienne Denis, and Will Hamilton. Inductive relation prediction by subgraph reasoning. In ICML., pages 9448-99 9451-457744 2020.
[59] Yongqi Zhang and Quamming Yao. Knowledge graph reasoning with relational digraph. In Proceedings of the ACM web conference 2022, pages 912-924, 2022.
[60] Christian Meilicke, Manuel Fink, Yanjic Wang. Daniel Ruffinelli, Rainer Gemulla, and Heiner Stuckenschmidt. Fine-graiee aene-grained evaluation of rule-and embedding basedd systems for knowledge graph completion. In ISWC, pages 3-20, 2018.
[61] Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. Drum: End-to-end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems, 32, 2019.
13

---

[62] Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. In ICLR, 2017. httpps///pppprrrrieeeoooororrrro
[63] Michael Sejr Schlichtkrull, Thomas Kipf. Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional networks. In ESWC, pages 593-607, 2018.
14

---

A Limitations
We would like to discuss the potential limitations of our method from the following three aspects:
Efficiency. As MKGL is an LLM-based fine-tuning method, it inevitably demands more computa- tional resources. In the main experiments, MKGL significantly outperforms all the conventional and LLM- based methods. The later analysis also reveal that the trainable parameters and runtime of MKGL are less than general fine-tuning framework. Therefore, we believe that MKGL. still an efficient M--aesed method.
Robustness. MKGL leverage multiple retrievers retrieve and KG information for constructing both input embeddings and score estimations, which may accumulate more errors during fine-tuning, Even though, most modules are learnable with pack-propagation. To avoid biased evaluation and occasional results, we also report the averaged results of multiple runs with variance statistics. Thus, we believe MKGL is a robust method.
Generality. The advances in LLMs have revolutionized many NLP tasks, and it is important for an LLM-based method where the L.M makes of the proposed modules and whether the performance can continually improves as the L.LM get promotion. We have conducted experiments to visualize the KGL embeddings and compare the performance with different base L.MMs. The results empirically demonstrate the generality and potential of MKGL.
B Broader Impacts
Our work focuses on the integration of Large Language Models (LLMs) with Knowledge Graphs (KGs) through the introduction specialized KG Language (KGL), has substantial broader impacts spanning technological advancements, societal implications, and educational benefits. Here, we outline the diverse and far-reaching impacts of our research.
Technological Advancements Our research contributes the cutting-edge of artificial intelligence, pushing the boundaries of what LLMs can achieve when combined with the structured knowledge represented by KGs. This can potentially unlock new capabilities in AI, ranging from more accurate context-aware natural language understanding to enhanced machine reasoning across diverse domains such as healthcare, finance, and legal systems. Additionally, our approach of using a specialized language (KGL) and the method of contextual embedding augmentation can inspire novel AI archi- tectures and learning paradigms that bridge the gap between unstructured and structured forms of knowledge.
Societal Implications The enhancement of Al systems with a more profound understanding of structured knowledge has broad societal implications. For one, could lead to the development of AI assistants that provide more accurate, consistent, and reliable information, thus improving decision- making in critical sectors. In healthcare, for instance, AI systems equipped with our technology could offer more precise recommendations by thoroughly understanding medical knowledge graphs. Moreover, by reducing the propensity for errors and hallucinations, our approach could foster greater trust in AI technologies among the general public, paving the way for wider acceptance and integration into daily life.
Ethical Considerations As with any advancement in AI, our work prompts important ethical considerations. Ensuring our technology is used responsibly involves critical discussions around privacy, bias, and transparency, especially as Al systems become more adept at interpreting and generating human-like text. We advocate for the continued examination of these aspects in tandem with technological development to ensure AI benefits society equitably and ethically. Our work, by facilitating error reduction in AI outputs, also contributes to the broader effort of minimizing harm and bias in Al Al-generated content.
Educational Benefits
Our integration of LLMs with KGs presents a novel avenue for educational tools and applications. Al tutors or educational platforms powered by our enhanced LLMs can offer students personalized and accurate learning experiences. Such systems could better understand and integrate complex academic content structured within knowledge graphs, from history timelines to
15

---

scientific concepts, thereby improving the quality of automated tutoring services and opening up new methods for engaging with educational material.
Future Directions This research opens up exciting avenues for future exploration, including refining the KGL for broader applicative scopes, exploring the ethical considerations of nuanced Al-generated content, and expanding our understanding of Al's potential when it deeply integrates diverse forms of knowledge. The cross cooss-iscipplinryy nature of this work invites collaboration among computer scientists, ethicists, domain experts, and policymakers harness the full potential of Al for societal benefit.
In conclusion, the integration of LLMs with KGs not only represents significant step forward in Al capabilities but also poses thoughtful considerations for societal impact, ethical use, and educational applications. Our work underscores the importance of continuous exploration, responsible innovation, and cross-disciplinary collaboration to harness the transformative potential of Al technologies.
C Principal Neighborhood Aggregation
Graph Neural Networks (GNNs) have emerged as a powerful family of neural models for learning Among the recent advances the principal neighborhood which enhances the representational capacity of GNNs by
on graph-structured data [7
I8
aggregation (PNA) mechanism
49
diversifying the aggregation functions applied to neighboring nodes.
PNA leverages combination of multiple aggregators such as sum, mean, and standard deviation, together with a scalable degree-specific weighting scheme. This approach is designed to address the shortcomings associated with simple aggregation functions that may fail to capture the complexity and diversity of neighborhood structures graphs.
The key component of PNA is its aggregation scheme, which is formally defined as follows:
(1+1) 0
=
AGG Vu e ((o))) w() PER
(12)
Please note that the symbols used in describing PNA are independent to the main paper for clarity. Here, Og ((+1) is the aggregated information for node at layer 1+ (u) denotes the set of neighbors of v, (1) represents the hidden features of neighbor nodes at layer + is concatenation operator over all aggregators in the R. AGG,, an aggregation function (e.g., sum, mean, max), ò is nonlinear activation function such as ReLU, and W()) is learnable weight matrix at layer
(1+1) 8 a
+ AGG @ 6 (e,r,u) TJ), w())
(13)
PNA's distinctive blend of multiple aggregation functions and degree-aware weighting significantly enhances the expressive power of GNNs, allowing for more complex feature representations and, consequently, improved performance downstream tasks. We also follow the KG embedding methods 15 to incorporate the relation embeddings into PNA as relational PNA.
D Implementation Details
We introduce Algorithm I to demonstrate the fine-tuning process of MKGL for KG completion. We first construct input instructions following Instruction and tokenize them into IDs. For those the score of original LLM vocabulary, their embeddings can be looked up from T, while those of out of scope will be retrieved by our context retriever Reontext After assembling the input embeddings. we feed them to the LLM to obtain output hidden states and then obtain the scores from the score retriever Rycare. Finally, we optimize MKGL by minimizing the constrastive loss The main hyper-parameter settings are summarized in Table
16

---

Algorithm MKGL for KG Completion
1: Input: the training KG G, the language model M, the token embedding matrix T, the original vocabulary of the LLM Vilm. the KGL context retriever Reentext, the KGL score retriever Ryore
2: for each batched data in the training set do
3: Construct input instructions according to Instruction 3.1
4: Tokenize the input instructions with the tokenizer,
5: for each input token sequence {lo, 11,2,,2,1 l2.... do
6: for each token do
7:
if tR E Vımm then
8:
tz Tx: look up embedding from the token matrix
9:
else
tz Reomtex (t&) (Equations 3 66)
11:
end if
end for
13:
end for
Compute the batched output hidden states of the LLM MM (Equation T):
15:
Compute the batched scores with Recore (Equations 7 10
Compute and minimize the constrastive loss L (Equation
17: end for
Table 5: Hyper-parameter settings in the main experiments.
LoRA
LoRA dropout
LoRA
train batch per device
loss criterion
gradient accumulation sleps
target modules
optimizer
FB15K-237 WNISRR
Jlamme2-77b-cha Llama-2-7b-chat
query, value
BCE BCE
Adam
32
0.05
query, value
16
Adam 8bit
content retriever
score text encoder layer
learning rate (Lon/MKGL))
# epoch
text encoder
KG encoder
retriever
KG encoder
layer
layer
layer
FB15k-225 237
5
32
6
32
0.0005/0. 00005/00005
WNISRR
E
Dataset Details
We use the following benchmark datasets to evaluate the performance of MKGL, and summarize the statistics in T
FB15k-237: This dataset is a subset of the original FB 15k dataset 23 and is created by removing inverse relations that may lead to test set leakage.
This dataset is subset of the original WN18 dataset 23 and is created by removing inverse relations that may lead test leakage.
WN18RR:
FB155-233-ind: The 'ind' suffix denotes the inductive setting adopted in FB15k-237 58 It includes new entities in the validation and test sets that are not present during training, thus requiring models to generalize beyond the transductive assumptions previously seen entities.
Similarly to FB 1B15k-237-ind the WN18RR-ind dataset is adapted for inductive KG completion on the WordNet (48
WN18RR-ind:
These datasets have been instrumental in the development and benchmarking of advanced KG com- pletion models, enabling comparison different approaches and understanding their effectiveness in both conventional and inductive settings.
17

---

Table 6: Dataset statistics.
Dataset
Relation
Train
Valid
Test
Entity
Triplet
Entity
Evaluation
Fact
Eaiity
Evaluation
# Fact
FB15K-257
237
14,54]
272.115
17,535
20,466
40,945
86.815
FB15k-257-ind-v/
180
1,594
4,245
1,594
489
4,245
1,093
205
1,993
2B115k-237-ind-V
200
2.608
9,739
2,608
1,166
9,739
1,660
478
4.145
28155-257-imd-yy
215
3,668
17,986
3,668
2,194
17,986
2,501
865
7,406
FBI5k-237-ind-4
219
4,707
27,203
4,707
3,352
27,203
3,051
1,424
11,714
WN1SRR-ind -in--vl
2,746
5,410
2,746
630
5,410
922
188
1,618
WN1SRR-ind-v2
10
6,954
15,262
6,954
1,838
15,262
2,757
441
4,011
WN11RR-inn-v2
12,078
25,901
12,078
3,097
25,901
5,084
605
6,327
WN13RRRind-v4
3,861
7,940
3,861
934
7,940
7,084
1,429
12,334
Table 7: The detailed inductive KG completion results, where vl-v4 represent four different subsets.
FB15k-237-ind-
FB15k-237-ind-
Method
MRR1
Hits@ 11
Hits 10
MRRI
Hits@11 11
Hits@ 10
Grall. S8
.279
.205
.429
.276
.202
.444
DRUM 61
.333
.247
.444
395
.284
.595
RED-GNN 59
.369
.302
.48
.469
.381
.629
MKGL
.400
.508
.417
.681
FB15k-237-ind-v3
FB 1B1--2337-dddvv
MRR
Hits@ 1+
MRR1
Hits@
Hits@ 10
GralL.
58
.251
.165
.424
.227
.143
.389
NeuralLP
.400
.309
.571
.396
.289
.593
DRUM 61
.402
.308
.571
.410
.309
.593
RED-GNN
MKGL
.486
.392
.643
.471
.374
.645
WNNISRR-ind-1
WN1SRR-ind-v2
MRRI
Hits@1 11
Hits @ 10
MRR1
Hits@11
Hits@ 10
GralL
.627
.776
NeuralLP
33
.649
.592
.772
.635
.575
.749
DRUM 61
.666
.613
.777
.646
.595
.747
RED-GNN 59
.011
.653
.799
.690
.633
.780
MKGI.
.712
.799
WN18RR v3
WNISRR ind-v4
MRR1
11
Hits
MRR1
Hitx#11
Hitsé 10
Grall.
58
.323
.278
.409
.533
443
.687
Neural
33
.361
.304
.476
.688
.83
.706
DRUM
.380
.477
.627
.702
RED-GNN
39
.427
.368
.524
.651
.606
.721
MKGL
.456
.406
.559
.664
.620
.741
F Additional Experiment Results
E1
More Examples Knowledge Graph Language Modeling
We present additional examples of KGL modeling in Table which demonstrates that MKGL can not only generate KGL sentences seen during training but also produce previously unseen triplets within the testing set.
F.2 Details Results on Inductive Knowledge Graph Completion
present detailed results on all inductive KG completion benchmarks Table 7 where MKGI. consistently and significantly outperforms all state-of-the-arr baselines.
18

---

Head entity
Relation
Tail entity
<Melancholia>>
<film release region
<Sweden>>
<South Korea>
<Singapore>>
<France>
<Romania>>
<Argentina>>
<country>
<France>
<Sweden>>
<Sampler (musical instrument)>
<rogular performance mie>>
<Hammond organ>
<Stephen Cannell>
<Maraca>>
<French horn>
<laiee>
<Percussion>>
<lead guitarists>
instrumentalists
<The Neptunes>
<just
Head entity
Relation
Tall entity
<pilot, operate an airplane>>
<verh group>
<fly, transport by aeroplane
<fly, travel over (an area of land sea) in alreraft>
<fly, travel in airplane>>
<aviation, travel via aircraft>>
<navigational>
<aviation>
<derivationally
<flying an instance of traveling by air> <aviation, travel via aircraft>
form>
<inform, impart knowledge some fact>>
<derivationally related form>
<information, a collection of facts from which conclusions may be drawn>>
<making known, a speech act that conveys Information>>
<source, person who supplies information
<information, a message received and understood>>
<information, knowledge acquired through study or experience or instruction>>
<instructive, serving to instruct or enlighten or inform>>
<teaching the activities of educating or instructing>
hypernym
<interoommunicat,, transmit thoughts or feelings>
Figure 5: More examples on KGL modeling.
E33 Different Layer Numbers
We conduct experiments to analyze the influence layer numbers the KGL retrievers. The results are illustrated in Figure Clearly, increasing the number of layers enhances performance across all datasets and metries. Additionally, we observe that small number of layers (i.e., 2) significantly impairs performance.
19

---

FRI
PH115-2577
FRI1562377
033
0.8
0,88
0664
0,656
MRX
0.6
0.533
0.552
[itb@81
0.6
0.5
Hit:010
0.6
0.181
0.363
0.308
0,243
0.32
0.0
0.0
0.00
Layer Number
Layer Number
Layer Number
Figure 6: The performance of MKGL on FB15k-237 and WNI8RR, with respect to the layer number of the retrievers.
GAT
Mean
D PNA
GAT
Mean
OPN
0A54 0.656
F815k-237
0..6
0.584
0.591
WN18RR
0.6
0.522 0.523
0.499
0,37
0.307 0.99 03255
0.2
0.2
MRR
Hits@1
H1ts@10
MRR
Hits@1
H1
Figure 7: The performance of MKGL on FB 15k-237 and WNISRR, with respect to different encoders in the retrievers.
Table 8:
Detailed results of Table The KG completion results on FB15k-237 and WN18RR. The best and second-best results boldfaced and underlined, respectively. higher is better; 4: lower better. +: unavailable entry.
Tr. Purams
MRRT
H
Hiss 101
T:: Parani
MBRT
His 11
Hisee"
His-l 10?
310
332
55
CompCRCN
10M
3355
264
.30
535
12M
.79
.443
.414
.546
DAN [5
354
261
544
458
.422
.577
.354
.000
.884
495
KG BERT
14
110M
A20
110M
216
302
.524
SIAR [S
355M
.266
.305
.322
482
355M
.011
.243
.911
2009
KGL 500
355M
280
2200
.314
468
355M
467
.300
538
.741
FTL-LM 59
125M
.348
225
.366
521
125M
545
.452
.337
.773
DET 10
16M
3376
251
560
24M
507
.455
.585
GPT Turbo +1]
KICGPT
.12
.377
448
554
549
.474
.955
.611
325
.454+-00
59111001
5552 MC
557-
.55+h
F.4 Different Encoders
conduct experiments to explore the impact of different encoders in the retrievers. The results are depicted in Figure 7 We find that the MKGL is not highly sensitive the choice of encoders. The performance when using GAT [50] is slightly lower than when using PNA.
20

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification:
The main claims made in the abstract and introduction accurately reflect the paper's contributions and scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The paper discusses the limitations of the work.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
21

---

Justification:
This paper does not include theoretical result.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: The paper fully discloses the information needed to reproduce the main experimental results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
22

---

Answer: [Yes]
Justification:
The source code is uploaded.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ ccpppp.ccppulli//uuuii wwwiicccguie//eeeeeeeeeis CoodeSubeiiieeee ssi wswonnolli onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are included in appendices.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: The main results are based on the average of multiple runs.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
23

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The paper provides sufficient information on the computer resources.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question:
Does the research conducted in the paper conform, in every respect, with the http://wwwwp..coooooo///ccacaaaa
NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research fully conforms with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: The paper discusses both potential positive societal impacts and negative societal impacts in appendices.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: There is no such risk for this paper.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: All assets are fully licensed.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
25

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: All new assets well documented.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
No crowdsourcing experiment has been involved.
Guidelines:
The answer NA means that the paper does involve crowdsoureing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification:
This research has no such risk.
Guidelines:
The answer NA means that the paper does involve crowdsourcing research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
26


Reviews:
[{"review_id": "VDfrSrhbdJ", "reviewer": "Reviewer_XXej", "summary": "The paper proposes MKGL, a novel approach to integrate LLMs with KGs by instructing them in a specialized KG Language (KGL). KGL is a three-word language that mirrors the structure of KG triplets. The authors introduce a KGL context retriever and a score retriever, both based on LoRA, to efficiently encode textual and relational information into KGL token embeddings. MKGL outperforms existing KG completion methods, including LLM-based and conventional approaches, on both standard and inductive KG completion tasks. The paper also demonstrates MKGL's ability to generate valid KGL sentences and its computational efficiency compared to in-context learning methods.", "questions": "* What exactly are the trainable parameters of the in-context-learning baseline in Section 4.6 and Figure 4? Isn't the point of ICL to not do any parameter updates at all and rely only on the contextual prompt?\n* How does MKGL scale to much larger knowledge graphs? And how is it compared to other KG completion methods?", "limitations": "No issues found.", "rating": 7, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "* The authors present a novel approach to LLM-KG integration using a completion of entity-relation-entity triplets.\n* The performance seems to be strong and the method outperforms the previous work on KG completion.\n* At least compared to naive in-context learning, MKGL is more efficient and also achieves better scores.", "weaknesses": "* There are only limited details on certain aspects of the methodology, for example I couldn't find details about the actual implementation of the multi-layered PNA for KG information retrieval.\n* While the authors claim that the proposed \"three-word language\" parsing of natural sentences is novel, it boils down to semantic-role labeling (SRL), a well-established NLP task. I believe that the paper should include a clear comparison to past SRL methods.\n* The results in Table 2 should contain a column with computational cost (or at least the number of parameters of each method), to make it clear if it compares apples to apples.\n* The computation runtime of the proposed method and the baselines is another thing that is lacking.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your helpful feedback and constructive suggestions. We have carefully integrated them into our paper.\n\n### Weaknesses:\n\n- **Include more details of the methodology, e.g., the implementation of the multi-layered PNA for KG information retrieval.**\n\n    Thank you for your suggestion. The implementation of the multi-layered PNA closely follows the original paper, with the key difference lying in the input feature. As this isn’t a core contribution of MKGL, we have relocated it to Appendix B. In our revised version, we have elaborated on the encoding of KG information. Recall Equation (12) where the original multi-layer PNA can be written as:\n\n    $a_v^{(l+1)} = \\delta \\left( \\bigoplus_{\\rho \\in \\mathcal{P}} AGG_{\\rho}(\\{h_u^{(l)}, \\forall u \\in \\mathcal{N}(v)\\}), W^{(l)} \\right)$\n\n    Here, $a_v^{(l+1)}$ represents the aggregated information for node $v$ at layer $l+1$, $\\mathcal{N}(v)$ denotes the set of neighbors of $v$, $h_u^{(l)}$ signifies the hidden state of neighbor node $u$ at layer $l$, $\\bigoplus$ is a concatenation operator, $AGG_{\\rho}$ is an aggregation function (e.g., sum, mean, max), $\\delta$ is a nonlinear activation function.\n\n    For knowledge graphs, there exists a relation (or edge type) $r$  between $u$ and $v$, which also needs to be encoded into the hidden states. To address this, we modify the above equation as:\n\n    $a_v^{(l+1)} = \\delta \\left( \\bigoplus_{\\rho \\in \\mathcal{P}} AGG_{\\rho}(\\{h_u^{(l)}\\otimes h_r^{(l)}, \\forall (v,r,u) \\in \\mathcal{T}\\}), W^{(l)} \\right),$\n\n    where $h_r^{(l)}$ denotes the hidden state of relation $r$ at layer $l$, and $\\mathcal{T}$ denotes the triplet set. $\\otimes$ is the operator used to combine the relation $r$ and neighboring node $u$, typically set as point-wise multiplication. Importantly, the hidden states at the first layer are not randomly initialized. They are the output of the text information retrieval, as illustrated in Figure 2c.\n\n\n- **Include a clear comparison to semantic-role labeling (SRL) methods.**\n\n    We have revised the related work section to compare KG completion with Semantic Role Labeling (SRL). Both tasks can be viewed as classification problems. However, the label spaces differ significantly. Most NLP tasks involve a smaller number of classes, usually less than 1,000, whereas for KG completion, the label space can exceed the vocabulary of LLMs. For example, the WN18RR dataset contains over 40,000 different entities, making it impractical to simply feed them all as possible results and let the LLM select one as output.\n\n- **Better to add a column with computational cost/parameter in Table 2.**\n\n    Many thanks. The number of parameters for MKGL is evidently greater than most conventional methods, a limitations we have discussed in the paper. A more comprehensive comparision metric may be the number of trainable parameters. Below are the results for FB15k-237 (the full table is available in the newly uploaded rebuttal.pdf), with some results sourced from [1], and others (marked by *) evaluated using the official repositories.\n    \n    | Methods  | # FB15k-237 Trainable Parameters (M) | FB15k-237 MRR | # WN18RR Trainable Parameters (M) | WN18RR MRR | \n    |---|:---:|:---:|:---:|:---:|\n    | TransE  | 2 | .310 | 21 | .232 |\n    | RotatE  | 15 | .338 | 20 | .476 |\n    | TuckER  | 11* | .358 | 9* | .470 |\n    | CompGCN |  10* | .355 |  12* | .479 |\n    | CoKE | 10 | .364 | 17 | .484 |\n    | KG-BERT | 110* | - | 110* | .216 |\n    | StAR | 355* | .296 | 355* | .401 |\n    | KGLM | 355* | .289 | 355* | .467 |\n    | DET | 16 | .376 | 24 | .507 |\n    | **MKGL** | 20 | .415 | 20 | .552 |\n\n    It is evident that the number of trainable parameters for MKGL is comparable to that of conventional methods, and this gap narrows as the knowledge graph gets larger (WN18RR). Some language-model-based methods (e.g., KG-BERT) leverage a full-parameter-fine-tuning strategy, employing significantly more trainable parameters.\n\n- **The runtime of the proposed method and the baselines is not included.**\n\n   The runtime of LLM-based and conventional methods may be incomparable, but we compare our method with a vanilla supervised fine-tuning LLM-based method in Figure 4. The results demonstrate the high efficiency of our method.\n\n### Questions:\n\n- **The name of ICL (1-hop)/ICL (2-hop) in Section 4.6 and Figure 4 may be confusing.**\n\n    Sorry for the confusion. The current names are inappropriate and misleading. Raw/ICL (1-hop)/ICL (2-hop) are methods involving new random-initialized token embeddings and score layers to represent every entity. In essence, each entity is considered a new token for the LLM, and is added to the vocabulary (accomplished using the native “resize_token_embeddings” function to add new tokens to LLM). The new embeddings do require training.\n    \n    This experiment aims to verify whether the proposed text retriever and KG retriever are superior to initializing new tokens and directly incorporating the KG context information in the input, respectively. It’s crucial to emphasize the necessity of including such new token embeddings to estimate the probabilities of all entities. This explains why these variants have more trainable parameters than MKGL.\n    \n    In the revised version (Figure 1 in rebuttal.pdf), we have renamed them and provided more detailed explanations: Raw has been renamed to NewToken, ICL (1-hop) is now NewToken (1-hop), and ICL (2-hop) is now NewToken(2-hop).\n\n\n- **How does MKGL scale to much larger knowledge graphs? And how is it compared to other KG completion methods?**\n\n    MKGL can easily scale to large KGs, as the number of trainable parameters remains independent of KG size. All KGL tokens stem from constant LLM token embeddings. This characteristic potentially positions MKGL as advantageous compared to conventional methods that initialize an embedding for each KG entity.\n\n[1] HittER: Hierarchical Transformers for Knowledge Graph Embeddings. EMNLP, 2021.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "F5MzM1TuFX", "reviewer": "Reviewer_4rXf", "summary": "The authors introduce a SOTA method for allowing LLMs to incorporate information from knowledge graphs, relying on Knowledge Graph Language token embeddings to retrieve context, and then score it using a retriever that helps form a distribution over the possible entities to be incorporated.", "questions": "None.", "limitations": "I have no concerns about this paper being published.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "Creative, interesting paper. Introduces a novel approach in a domain that could be of great use to data analysts (LLMs for knowledge graphs). \nWell-supported empirically with strong evaluation on major benchmarks. Outperforms alternative methods mostly across the board.\nSimple and effective illustrations. Good use of formatting in the paper itself to facilitate understanding (appreciated the use of color, especially).\nMethod may have notable benefits for reducing problems like hallucination and distribution drift, contributing to a solution for major outstanding issues with LLMs.", "weaknesses": "Could use a longer and more detailed discussion section; ends a little too abruptly.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We are grateful for your encouraging comments and insightful suggestion. We hope the following response address your concern:\n\n### Weaknesses:\n\n- **Could use a longer and more detailed discussion section; ends a little too abruptly.**\n\n    Thank you for your insightful suggestion. We also agree that a more detailed discussion could enhance understanding of our method and its implications across broader research areas. We have updated the conclusion section to include more comprehensive discussions on methodology, results, and potential value for other research fields:\n\n    The proposed context and score retrievers point out a new direction in incorporating LLMs with KGs for various tasks, such as question answering and entity linking. They also have implications in broader areas where the input cannot be precisely represented by text, e.g., node classification and protein representation learning. Furthermore, the construction of KGL vocabulary enables contrastive learning beyond tokens, offering insights into general machine learning. Hence, there are also numerous future directions. We plan to pretrain LLMs using a mixed corpus of KG and natural languages, enabling the LLM to comprehend and generate responses with linked data.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZXKpPkPjHG", "reviewer": "Reviewer_dtBb", "summary": "This paper proposes what seems to be an elaborate GNN+LLM+GNN  sandwich of a model for doing knowledge base completion, having a GNN pipeline to form KB-informed token embeddings, passing those to a LLM (lllama-2 ) into a knowledge base completion prompt template, and then passing that output into another GNN-like (\"PNA\") set of layers.  That whole set-up is then used to train representations of the data optimized with contrastive loss for the entity prediction task in knowledge base completion tasks (FBK and Wordnet)   (I'll admit that I found the model explanation quite hard to follow, and so it's entirely possible I'm slipping on some details -- some of this had to be inferred by glancing at their code.)  It seems to outperform existing methods on these two tasks.", "questions": "-Looking at the code, it was unclear whether the  model is calculating the metrics (MRR, Hits@1, Hits@10) by ranking solely within a small batch containing the correct answer, or actually predicting the highest ranked entity from a full space of candidates. Could the authors clarify which one is being done? \n- Could the authors clarify what is removed in the second , \"text\" ablation reow", "limitations": "yes, a broader impacts section is included and no major limitations are missing.", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "If the authors work and evaluation are sound, their method outperforms other methods at two commonly used knowledge base completion tasks.", "weaknesses": "- I found the model extremely rather hard to follow, and had particular trouble discerning why this collection of model assumptions would result in a meaningful improvement over prior work.  Since the work is so complicated, it may be both useful to focus on very clear graphs and progressively introducing parts of the model.  I'll admit that the \"Retriever\" framing felt very confusing as well, as this models \"Retrievers\" don't seem to do any retrieving. \n- For understanding the model, the \"three world language\" framing seems quite separate from the meat of what this model seems to be doing.    It implies that the somewhat simple \"template\" setup they use is important, but it seems to be only formed to teach a model what triplet completion is, which is something that one would think would be easily addressed in fine-tuning; there is not experimental exploration showing the value of those prompts.  \n- I'll admit to feeling suspicious at the high performance here (even after ablating nearly everything, they outperform most models?).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your constructive and detailed comments. We appreciate the opportunity to provide further clarifications.\n\n\n### Weaknesses:\n\n- **Why is the proposed model better than the prior works? It is better to illustate each modules step-by-step with clear figures. The name of \"Retriever\" is confusing: what it retreves?**\n\n    Thank you for your insightful comments. We believe the reasons for the improved performance are threefold: (1) the power of LLMs; (2) the additional text information, where methods leveraging text features generally perform better; (3) the proposed MKGL effectively retrieves text and KG information for the LLM.\n\n    We indeed have a more detailed implementation section, including a step-by-step algorithm, which is currently in Appendix C due to space limitations. If the paper could be accepted, we plan to reintroduce this content into the main paper in camera-ready version (with an additional page allowance). We are willing to reintroduce the step-by-step process (Figure 1) of MKGL: Firstly, we construct input instructions and tokenize them into IDs. For out-of-vocabulary KGL tokens, their embeddings will be retrieved by our context retriever. Specifically (Figure 2), we aggregate the text tokens of each entity as its embedding, then use this embedding as the entity feature for aggregating KG information as the final KGL token embedding. Finally, we assemble the (KGL and LLM) token embeddings as the input sequence and feed them to the LLM to obtain output hidden states.\n\n    The term \"retriever\" simply implies that it can retrieve information from external resources that are helpful for inference. Since the KG information is not originally included in the input triplet, we designed a module to retrieve and process the information for the LLM. We have updated the introduction section in the revision to include an explanation for the name \"retriever\". Thank you again for your comment.\n\n- **The prompts used to fine-tune MKGL seems important, but it just teaches the model what triplet completion is. This can be easily addressed in fine-tuning.**\n\n    We also agree that the prompt templates may not be essential for fine-tuning an LLM on KG completion. In fact, the presentation of Instruction 3.1 in the main paper is not to demonstrate its novelty and importance. We simply aim to show how the input instruction is structured. Although we organize the context as a table, using sentences or other formats would not cause a significant performance loss.\n\n\n- **Lacking an explanation of the high performance (even after ablating nearly everything, they outperform most models?).**\n\n    Thank you for the comment. We do not ablate everything; the text retriever is actually replaced with a randomly initialized embedding module, as stated in Lines 285-287. We have highlighted this point in the caption of Table 4 and discussed the reasons in the revision. It is still a supervised fine-tuning LLM-based model, and it is expected to outperform most (not all) conventional models.\n\n### Questions:\n\n- **It was unclear whether the model is calculating the metrics (MRR, Hits@1, Hits@10) by ranking solely within a small batch containing the correct answer, or a full space of candidates. Could the authors clarify which one is being done from the source code?**\n\n    Thank you for your detailed comments. To incorporate LLMs into KG completion, we developed a new framework based on the Transformers package provided by Hugging Face. The training and evaluation procedures also follow the corresponding suggestions. Although we have provided extensive comments in the uploaded code, it may not be as familiar as previous KG completion source code to the reviewer.\n\n    We do rank the target entities against all candidates, and we have highlighted this point in our paper in the revision. In the \"predict\" function of \"llm.py\" (Lines 299-325), we construct the candidate list according to the status of \"self.training\". \n\n    ```python\n    all_index = torch.arange(graph.num_node, device=device)\n    if self.training:\n        # train, do negative sampling\n        neg_index = self._strict_negative(\n         pos_h_index, pos_t_index, pos_r_index)\n        ...\n    else:\n        # test, constrcut testing examples (h,r,?) and (?,r,t)\n        h_index, t_index = torch.meshgrid(pos_h_index, all_index) \n        it_index, ih_index = torch.meshgrid(pos_t_index, all_index)\n        ...\n    ```\n\n    To ensure the evaluation function is correct, we have also tested the performance of a classical KG completion method, TuckER, within our framework, which achieves similar performance compared to its paper. We cannot provide an external link in the rebuttal, but TuckER is a very simple model. The reviewer may simply replace our model with its model (only 40 lines) and keep everything else unchanged to reproduce its results. Additionally, we adopt a strict ranking strategy where the target will be ranked below the entities with the same probability (Lines 278-286 in llm.py). \n\n\n- **Could the authors clarify what is removed in the second \"Text\" in the ablation results (Table 4)**\n\n    As stated in Line 286, toggling off \"Text Retriever\" does not mean we remove it, but we replace it with a learnable embedding module. This is equivalent to adding many new tokens into the LLM. The new tokens are randomly initialized and will be learned during fine-tuning. We have updated the caption of Table 4 to include this explanation.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "J8GBi8xeJ6", "reviewer": "Reviewer_NCjt", "summary": "This paper proposes a method to leverage LLMs as knowledge graph completion systems. New tokens that correspond to (potentially multi-word) concepts and relations are introduced to the model’s vocabulary, and then the LLM’s embeddings for the tokens composing those concepts/relations are aggregated and upscaled to estimate embeddings for these new tokens. Given these new tokens, the goal of the system is to complete the knowledge graph triplet given two of these embeddings by retrieving the correct third token from the KG vocabulary.\n\nIn a series of experiments, it is observed that the proposed system outperforms a variety of previous baselines employing diverse methods. It also excels at inductive KG completion. An ablation study reveals that each proposed part of the pipeline is necessary for achieving the best performance.", "questions": "1. Would it be possible to base this approach on other LMs as well? The various Llama scales would be ideal, but if scale is an issue, even just comparing Mistral, Llama 2 (7B), Llama 3 (8B), and ideally some smaller models would be nice. This is quite compute-intensive, so I wouldn’t expect it for the camera-ready, but it would definitely be nice to have.\n2. Do you have any hypotheses as to why the proposed method is better at inductive KG completions than prior methods? In other words, is there a particular aspect of your pipeline that you believe makes it better for handling novel triplets than past approaches?\n\nTypos:\n* L226: “clear that The” -> “clear that the”", "limitations": "I do not believe limitations have been sufficiently addressed. There is no \"Limitations\" section, nor is there a dedicated space in any part of the paper that directly addresses the drawbacks of the proposed method and experiments. For example, there are many moving parts; there are multiple stages of optimization that could lead to cascading errors; only one LLM base was considered; etc.", "rating": 7, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "* The proposed approach outperforms a wide variety of prior methods with respect to accuracy. It also achieves a better trade-off between accuracy and compute-efficiency.\n* A wide variety of baselines employing diverse methods are compared.\n* Informative ablation study.", "weaknesses": "1. Given how many moving parts there are, reproducibility seems difficult. It would be nice to see the variance in performance of the method across multiple random seeds, where each random seed entails running the entire pipeline of optimizations from scratch.\n2. Relatedly, it is unclear whether the approach will scale as LMs continue to improve (and presumably to become better bases for approaches like this). Having a comparison with other base models would be a nice way to hedge against this.\n3. No detailed discussion of limitations. The checklist says it is discussed, but there are only brief comments distributed throughout the paper (which, in my opinion, do not address limitations thoroughly enough). A dedicated section would be helpful.\n4. It is unclear whether the new KG embeddings encode relevant concepts to the target token, or whether they are picking up on certain spurious correlations that happen to be helpful (but may not generalize robustly). It would be nice to have an analysis where the new embeddings are directly decoded into vocabulary space, such that we can observe what concepts are included in these new representations.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your detailed and insightful comments. We have addressed your concerns below and hope our responses provide clarity:\n\n### Weaknesses:\n\n1. **Reproducibility: can the authors provide the variance in performance of the method?**\n\n    Thanks for your suggestion. We follow the existing methods to report the average results of 5 runs in the main tables. Most KG completion methods are not very sensitive to the initialization seed [1], as the large label space and testing sets make the result quite stable. This may be the reason why they did not provide the variance statistics. Re-producing all baseline results is difficult at this phase, but we are willing to update the tables with the variance statistics of our method:\n\n    | Methods  | FB15k-237 MRR | FB15k-237 Hits@1 | FB15k-237 Hits@3 | FB15k-237 Hits@10 | WN18RR MRR | WN18RR Hits@1 | WN18RR Hits@3 | WN18RR Hits@10 |\n    |---|:---:|:---:|:---:|:---:|:---:|:---:|:---:|:---:|\n    | MKGL  | .415 $\\pm$ .002 | .325 $\\pm$ .004 | .454 $\\pm$ .001 | .591 $\\pm$ .001 | .552 $\\pm$ .002 | .500 $\\pm$ .005  | .577 $\\pm$ .003  | .656 $\\pm$ .002 |\n\n\n2. **How does MKGL perform as LMs continue to improve?**\n\n    Thanks very much for your suggestion. We have conducted a new experirment investigating the impact of the base model, where we consider Mistral (7B), Llama 2 (7B), Llama 3 (8B), and LLama 2 (13B) for comparison. The results are shown the following table:\n\n    | Methods  | FB15k-237 MRR | FB15k-237 Hits@1 | FB15k-237 Hits@3 | FB15k-237 Hits@10 | \n    |---|:---:|:---:|:---:|:---:|\n    | Llama-2-7B  | .415 | .325 | .454 | .591 |\n    | Llama-2-13B  | .421 | .331 | .455 | .590 |\n    | Llama-3-8B  | .429 | .337 | .459 | .593 |\n    | Mistral-7B |  .424 | .334 |  .457 | .591 |\n\n    From the results, we can find the improvement from 7B model to 13B model is limited, possibly because 13B is not significantly larger than 7B. However, more advanced base models do contribute to better performance, which can be empirically verified from the results of Llama-3-8B and Mistral-7B. The results are significantly better than the Llama-2-7B version, especially on Hits@1. They also outperform the best baseline method KICGPT (.327) in Table 2. We have added a subsection to include the above results and discussion in the revision.\n\n3. **A dedicated limitation section would be helpful.**\n\n    Thank you for the detailed comments. We have incorporated a dedicated limitations section in the revised version, as outlined below:\n\n    We would like to discuss the potential limitations of our method from the following three aspects:\n\n    Efficiency: As MKGL is an LLM-based fine-tuning method, it inherently requires more computational resources. In our primary experiments, MKGL significantly outperformed all conventional and LLM-based methods. Subsequent analyses also revealed that the trainable parameters and runtime of MKGL are fewer than those of a general fine-tuning framework. Hence, we believe that MKGL remains an efficient LLM-based method.\n\n    Robustness: MKGL leverages multiple retrievers to gather text and KG information for constructing both input embeddings and score estimations, which may introduce more errors during fine-tuning. Nevertheless, most modules are learnable through back-propagation. To mitigate biased evaluation and sporadic results, we also present the averaged results of multiple runs alongside variance statistics. Consequently, we consider MKGL to be a robust method.\n\n    Generality: The advancements in LLMs have revolutionized many NLP tasks, and it is important for an LLM-based method whether the proposed module is effective and the performance can continually improve as the LLM gets promoted. We have conducted experiments to visualize the KGL embeddings and compare the performance with different base LLMs. The results empirically demonstrate the generality and potential of MKGL.\n\n\n\n4. **It would be nice to have an analysis about the new KGL embeddings and original token embeddings.**\n\n    We have visualized the embeddings of the original LLM tokens and KGL tokens in Figure 2 of the newly uploaded rebuttal.pdf. In Figure 2a, we observe that two types of embeddings are (nearly) uniformly distributed in the space. The KGL tokens have been successfully encoded into the original token space. In Figure 2b, we present a sample from the center, where the entity *Canal+* (a sports TV channel) is closely encoded to tokens like *TV* and *_Team*. In Figure 2c, we present a sample from the corner, which also demonstrates high semantic correlations.\n\n    We have included the figure and discussion in the revision. Additionally, we believe that the inductive KG completion experiments could further demonstrate the robustness and generality of MKGL, as the entities in the testing set are unseen and unknown during fine-tuning.\n\n### Questions:\n\n1. **Would it be possible to base this approach on other LMs as well? Even just comparing Mistral, Llama 2 (7B), Llama 3 (8B) would be nice.**\n\n    Please refer to our response to the weaknesses.\n\n\n2. **Why is MKGL better at inductive KG completions than prior methods?**\n\n    We believe there are two possible reasons: Firstly, the incorporation of additional text information. Previous works primarily rely on structural similarities for inductive KG completion, while our method additionally leverages text information. Although the entities in the testing set are new, their text token features are not new to MKGL. Secondly, the utilization of LLM. Without loss of generality, we believe that LLM possesses knowledge helpful in KG completion and offers better inference ability compared to smaller models.\n\n\n- **Typos: L226, “clear that The”**\n\n    Many thanks. We have fixed it.\n\n- **Limitations**\n\n    Thank you once again for the detailed and constructive comments. We hope our response has sufficiently addressed these limitations.\n\n[1]  A Re-evaluation of Knowledge Graph Completion Methods. ACL, 2020.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear all reviewers:

We sincerely appreciate the time and effort you have dedicated to reviewing our paper. 

We would like to express our gratitude to Reviewers XXej for suggesting the inclusion of more details in the related work and methodology sections. We have incorporated these suggestions in the revision.

We are also grateful to Reviewers 4rXf for providing suggestions on the enrichment of the conclusion section. We have expanded our discussions not only limited to KG but also their potential impact on other areas.

We genuinely thank Reviewer dtBb for recommanding more details about the implementation and experimental settings. We have updated the corresponding paragraphs to cover more specific settings in the revision.

Furthermore, we extend our special thanks to Reviewer NCjt for pointing out the underlying limitations and providing  insightful solutions. We have added a standalone section to discuss the limitations and conducted experiments to analyze them.

Thanks again to all reviewers. Your comments are invaluable in helping us enhance the quality of our paper.

Best Regards,

Authors

Meta-review: The authors introduce a method for LLMs to carry out knowledge graph completion.  The system outperforms a number of established baselines.  Reviewers agree that that the method is sound, novel and impactful, if only in applications.  No substantial weaknesses remain after the rebuttal/discussion periods.
Decision: Accept (spotlight)