KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
Authors: Yanbin Wei
Qiushi Huang
Yu Zhang
James Kwok
Venue: EMNLP 2023 Findings (2023)
Keywords: Knowledge Graph Completion
Large Language Model

Abstract: Knowledge Graph Completion (KGC) is crucial for addressing knowledge graph incompleteness and supporting downstream applications. Many models have been proposed for KGC and they can be categorized into two main classes, including triple-based and test-based approaches. Triple-based methods struggle with long-tail entities due to limited structural information and imbalanced distributions of entities. Text-based methods alleviate this issue but require costly training for language models and specific finetuning for knowledge graphs, which limits their efficiency. To alleviate the limitations in the two approaches, in this paper, we propose KICGPT, a framework that integrates a large language model (LLM) and a triple-based KGC retriever, to alleviate the long-tail problem without incurring additional training overhead. In the proposed KICGPT model, we propose an in-context learning strategy called Knowledge Prompt, which encodes structural knowledge into demonstrations to guide LLM. Empirical results on benchmark datasets demonstrate the effectiveness of the proposed KICGPT model with lighter training overhead and no finetuning.

KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion
Yanbin Wei 1,2, Qiushi Huang!, Yu Zhangl, James T. Kwok? Southern University of Science and Technology
Hong Kong University of Science and Technology
3 University of Surrey
Abstract
main classes: triple-based methods and text-based methods (Wang et al., 2022a). Triple-hased meth- ods (c.g., TransE (Bordes et al., 2013), R-GCN (Schlichtkrull et al., 2018), and HitER (Chen et al., 2021)) utilize the structure of the knowledge graph as the only source of information for KGC. Given the typical imbalance of entities in KGs, long-tail entities are prevalent and the KGC task has limited structural information about them. Consequently, due to information scarcity, the performance of triple-based methods tends to degrade when pro- cessing long-tail entities (Wang et al., 2022a). Text- based methods (such as KG-BERT (Yao et al., 2019)) mitigate this information scarcity problem by encoding textual description as an extra infor- mation source (Wang et al., 2022a). Besides the use of pre-trained language models (PLMs), fine- tuning remains necessary for text-based methods handle diverse knowledge graphs. However, this is resource- resource-intensive and requires task-specific fine- tuning for each downstream task.
Knowledge Graph Completion (KGC) is cru- cial for addressing knowledge graph incom- pleteness and supporting downstream applica- tions. Many models have been proposed for KGC. They can be categorized into two main classes: triple-based based and text-based approaches. Triple-based methods struggle with long-tail entities due to limited structural information and imbalanced entity distributions. bost- methods alleviate this issue but require costly training for language models and specific fine- tuning for knowledge graphs. which limits their efficiency. To alleviate these limitations, in this paper, propose KICGPT, framework that integrates a large language model (LLM) and triple-based KGC retriever. It allevi- ates the long-tail problem without incurring additional training overhead. KICGPT uses an in-context learning strategy called Knowledge Prompt, which encodes structural knowledge into demonstrations to guide the LLM. Empiri- cal results benchmark datasets demonstrate the effectiveness of KICGPT with smaller train- ing overhead and no finetuning.
Large language models (LLMss, such as Chat- GPT and GPT-4 (OpenAI, 2023), have extensive internal knowledge repositories from their vast pre- training corpora, which can be used as an extra knowledge base to alleviate information scarcity for long-tail entities. Tay et al. (2022), Ouyang et al. (2022) demonstrated the potential for LLMs to handle wide range of tasks by specially- designed prompts without requiring training or fine- tuning. Given the appealing properties LLMs, a straightforward approach is to directly apply LLMs to KGC tasks. However, empirical evidence (Zhu et al., 2023) indicates that pure LLMs still can- not achieve state-of-the-an performance in KGC tasks such as link prediction. Additionally, there are several challenges that hinder the application of LLM KGC tasks. First, the LLM outputs can be unconstrained and may fall outside the scope of entities in the KGs. Second, LLMs impose length limits on the input tokens, and the limits
1 Introduction
Knowledge Graphs (KGs) are powerful representa- tions of real-world knowledge. The relationships among entities are captured by triples of the form <head entity, relation, tail entity>, KGs serve as the foundation for various applications such ommendation systems, question-answeringg and knowledge discovery. Knowledge Graph Comple- tion (KGC) plays a crucial role for KGs by com- pleting incomplete triples and hence addressing the inherent incompleteness of KGs. This paper fo- cuses on the link prediction task in KGC, which is to predict the missing entity in an incomplete triple.
Based the source of information used, ex- isting KGC methods can be categorized into two
"Corresponding author

---

are far from sufficient for describing a complete KGC task. Lastly, there is no effective in-context learning prompt design for LLM on KGC tasks. To alleviate the aforementioned limitations, we propose in this paper a Knowledge In Context with GPT (KICGPT) framework. This integrates LLMs with traditional structure-aware KG model (which is called retriever). Specifically, for each query 7- (h, r, ?) or q = (2,r,t) t) in the link pre- diction task, where "?" denotes the missing tail or head entity to be predicted, the retriever first processes the query independently and gener- ates an ordered candidate entity list Ryetriever that ranks all entities in the KG based on their retrieval scores. The LLM then performs re-ranking on the top entities returned by Reetriever, and replaces these m entities with re-ranked ones returned by the LLM as the final result Rk ICGPP. To achieve the LLM re-ranking, we propose Knowledge Prompt, a strategy of in-context learning (ICL) (Wei et al., 2022), to encode the KG knowledge into demon- strations in prompts. Note that in KICGPT we only need to train the retriever. Compared with existing triple-based methods, KICGPT enables si- multancous utilization of knowledge sources (i.e., KG and LLM's knowledge base) and facilitates their alignment and enrichment to alleviate infor- mation scarcity for long-tal -til entities. Moreover, dif- ferent from cxisting text-based models, KICGPT leverages a much larger semantic knowledge base without incurring additional training overhead. Un- like directly applying the LLM to the KGC task, which may generate undesired outputs, the pro- posed KICGPT constrains its output by formalizing link prediction re-ranking task for the given sequence Rretrieser. Moreover, the standard link prediction task requires ranking for all the entities, which not feasible for LLM due to the length limit on input tokens. To overcome this limitation, KICGPT utilizes the retriever to obtain the top-m entities in Ryatricwer, and only allows the LLM to perform re-ranking on these entities.
Extensive experiments on benchmark datasets demonstrate that KICGPT achieves state-of- the-art performance with low training over- head.
2 Related Work
Triple-based KGC Most existing KGC meth- ods are triple-based methods, which complete the knowledge graph solely based on the triple infor- mation. Early shallow knowledge graph embed- ding (KGE) methods represent entities and rela- tionships as low-dimensional embedding vectors in a continuous embedding space. Based on the scoring function, these methods can be further cat- egorized (Wang et al., 2017) translation- based (e.g., TransE (Bordes et al.. 2013)) and semantic matching models (e.g., RESCAL (Nickel et al., 201 1) and DistMult (Yang et al., 2014)).
However, they suffer from limited expressive power due to the use of shallow network structures, In recent years, more powerful network structures are integrated to solve KGC tasks. Examples in- clude the Graph Neural Networks (Schlichtkrull et al., 2018), Convolutional Neural Networks (Dettmers et al., 2018), and Transformer (Chen et al., 2021). Most of these aggregate local struc- ture context into node embeddings and achieve much improved performance. However, they are still limited by the imbalanced distribution of knowledge graph structure with insufficient knowl- edge about the toil- entities. Meta-learning (Xiong et al., 2018 8; Chen al., 2019) and logical rules (Sadeghian et al., 2019) can mitigate the long- tail problem in KG. The main difference between them and our work that they handle long-tail entities by extracting and summarizing common structural patterns or rules from the limited infor- mation in KG, while KICGPT combines vast external knowledge base inside the LLM with the structural information in KGs, which can help alle- viate information scarcity.
In summary, our contributions are follows.
Text-based KGC In light of the success of Natu- ral Language Processing (NLP), text-based knowl- edge graph completion is gaining more attention. As another mode of knowledge different from the structured KG, text can provide rich semantic in- formation. DKRL (Xie al., 2016) first introduces textual descriptions into entity embeddings pro- duced by a convolutional neural network. Subse- quent works (such as KG-BERT (Yao et al., 2019). KEPLER (Wang et al., 2021b), and Pretrain-KGF
We propose a novel cost-effective framework KICGPT for KGC tasks. To the best of our knowledge, this is the first work that combines LLMs with triple-based KGC methods, offer- ing unique solution to address the task.
We propose a novel in-context learning strat- egy, Knowledge Prompt, specifically de- signed for KGC.

---

(Zhang et al.. 2020b)) use a pre-trained language model (PLM) to encode the text descriptions. More recently, LMKE (Wang et al., 2022a) proposed a contrastive learning framework that uses the PLM to obtain entity and relation embeddings in the same space as word tokens, and demonstrated its effectiveness on long-tail problem These methods generally rely the language models to process text descriptions and require finetuning for different knowledge graphs. The proposed KICGPT, which uses LLM directly, is more efficient because is training-free and requires no finetuning.
structures for precise generation (Ouyang et al., 2022). However, the success of in-context learning depends heavily on the quality of the prompt, and crafting suitable prompts can be delicate (Wang et al., 2022b). While there has been work explor- ing ICL on different tasks (Dong et al., 2022), to the best of our knowledge, no such work has been done for KGC. Unlike existing works, the proposed ICL strategy, Knowledge Prompt, considers the characteristics of KGC tasks and demonstrates its effectiveness on KGC tasks.
3 Methodology
LLMs for KGs Some recent works also explore the ability LLM on KG tasks. StruetGPT (Jiang et al., 2023) proposed general framework for im- proving zero-shot reasoning ability of LLMs on structured data. It leverages the LLM to perform reasoning on the KG Question Answering (KGQA) task with the help of auxiliary interfaces that fetch the needed information from KG. Though both StructGPT and our work utilize KG and LLM, StructGPT aims to help the LLM to handle struc- tured data and explore KGQA tasks, while ours uti- lizes the knowledge base inside the LLM to handle KGC's long-tai tail problem. Moreover, StructGPT performs multi-step reasoning directly based on the KG structure, while the proposed KICGPT utilizes the KG information in different way. First, use the whole KG generate preliminary results before using the L.M. Moreover, we incorporate a portion of the KG triples into ICL demonstra- tions to help the LLM conduct reasoning. Zhu et al. (2023) directly evaluated the performance of LLM on KG reasoning. Our work design an ICL strategy to guide the LLM to perform reasoning. Moreover, their experiments the link prediction task only involve 25 sampled instances from the FB1 FB1 dataset (which has 20,466 test triples). Instead, we re-run their setting on ChatGPT and report the new results as a baseline in our experi- ments.
In this section, we introduce the proposed KICGPT model. The complete algorithm is in appendix A.I.
3.1 Problem Setting
A knowledge graph can be represented as a set of triples G = {(h, r, t)}, where E and R denote the set of entities and relations in G, respectively, h E E is head entity, E E is tail entity, and
R represents the relation between them. Link prediction an important task in KGC. Given an incomplete triple (h, r, or (2, (2,r, rt) as query, link prediction aims to predict the missing entity (denoted as ?). A link prediction model usually needs to score all plausible entities as missing entity and then rank all entities in descending order. For simplicity of presentation, we focus on queries with missing tail entities (i.e., (h, r, ?)). Queries with missing head entities (i.e., (?, r, r)) can be handled analogously.
3.2 Overview
Figure 1 illustrates the KICGPT framework. There two components: triple-based KGC re- triever and LLM. For each query triple (h, ?). the retriever first generates the score of (h, r)e) for each entity e E. The ranking (in de- scending score) of all the entities is denoted The
ICL for LLMs A typical way of approach- ing LLM is through in-context learning (Brown et al., 2020), by providing explicit instructions and demonstrations to guide the model's behavior. This approach has been effective in various language un- derstanding and generation tasks (Rae al., 2021; Wei al., 2022). In-context learning exposes the model to specific contextual information, allowing it to grasp and reproduce necessary patterns and
Ryetricver = [C1st: C2nd, C3rd, LLM then performs a re-ranking of the top-m entities based on its knowledge and demonstra- tions in the proposed Knowledge Prompt. The re-ranking RI..M [cistr ééna: Card...., emain is permutation of [eists 62nd, E3rd, ..., emth). By replacing the m leading entities in Ryetricver by RLLM, the KICGPT outputs RKIGGPT = [cists Gena cards..., Cymth Emp 1th, Ith....,

---

Reorddered candidate entisies REAM
LLM
OUTPUT: Raseerr Fie Cress foc-tth..... ssa)
top m entities lestt....
Vemonstaaiion
instructions
text
Knowledge Prompt
R Restrieve Claty Ca
Prompt Engineering
Unified Prompe Template: Convert Triplet to Text (optional: text self-aligmment)
KGC Retriever
Krowlodge Graph
Diversily-laeed ordered L,
BM25 ordering b,
INPUT: Query Triplol (Milim, moath to travel." ?
analogy pool Dr
suppietaent pook
Figure I: An illustration of the KICGPT framework.
3.3 Knowledge Prompt
that promotes diversity. Specifically, first we set a zero counter for each entity. triple from Da is randomly selected as demonstration, and the coun- ters associated with its entities are increased by 1. We iteratively choose as demonstration the triple whose entities' associated counters have the small- sum (tie is resolved randomly). The associated counters are then increased by This repeated until all triples in Da are used, and the resultant demonstration list obtained denoted La-
In this section, we introduce Knowledge Prompt, an in-context learning strategy specially designed for KGC tasks. By encoding part of the KG into demonstrations, Knowledge Prompt boosts the per- formance of LLM link prediction.
3.3.1 Demonstration Pool
For each query (h, r, ?), we construct two pools of triples, Da and D,. from the KG as demonstrations.
For the supplement pool D,, as serves to pro- vide supplementary information on the query's head entity h, we prefer related demonstrations to the query. Specifically, we rank all triples in D, according their BM25 scores (Robertson and Walker, 1994). This score is used evaluate the correlation between texts in each demonstration and query. The resultant ranking list, denoted L., contains all triples in D..
Analogy Pool The analogy pool Da contains triples that help the LLM to better understand the semantics of the query by analogy. Let Girain and Gvalid be the sets of KG triples for training and validation, respectively. Da {ee',, rr,") E Gtrain U Geatiale', e" E E} includes triples with the same relation as the query (h, r, ?).
Supplement Pool The supplement pool D. con- tains triples which provide supplementary infor- mation about the query's head entity h. Specifi- cally, Dj includes all triples with h as the head tail entity the training and validation parts: [h, r', e') Gtrain Gealial'' R E E} U {(e r', h) E Gtrain U Gratialr' R, E E}.
Queries with a specific relation or head entity use shared analogy or supplement pool. To prevent double counting, demonstration pools are created and ordered in data pre-processing. During infer- ence, the corresponding pool is used based the query.
3.3.2 Demonstration Ordering
3.3.3 Prompt Engineering
As the demonstration order affects performance (Ye et al., 2023), we propose different ordering strategies for the analogy and supplement pools. For the analogy pool Da. all its triples are simi- lar to the query because they share the same rela- tion. As diversity of demonstrations is important so that the LLM can learn from various analogy demonstrations, we propose an ordering strategy
Prompt engineering is an important in ICL (Zhao et al., 2021). In this section, we show the whole interaction workflow with the LLM, and also siderations of the prompt design.
The demonstrations and query take the form of triples. However, I.LMs require natural lan- guage input. To remedy this gap, KICGPT uses a unified prompt template to convert the query and

---

demonstrations plain text with the same format. We then perform multi-round interactions with the LLM, so as to guide to perform re-ranking, where these texts and some instructions are organized as prompt inputs. Figure illustrates the workflow of a multi-round interaction with the LLM. For each link prediction query rh,r,?), KICGPT cre- ates an independent conversational prompt. The whole multi-round interaction process includes four stages: responsibility description, question and demonstration description, multiple demon- strations, and final query.
Reggosibllity Deserition
Yeu axcetant mokling candidate srenor fer -uaetive baad plusibilty youo have know exppal "You"
Decerptiinn
quexire is (ankk] by jueeety
Auc skan and mining polenial rchaivealip then.
potcatil urssing
of cortas expioenentrry information
Clay, saalesteed
Pleae poeside for cxauples.
Responsibility description is illustrated in the first part of Figure 2. We tell the LLM that its role is an assistant to ranking candidate answers for question based on plausibility. We then check its feedback to ensure that knows the task.
Mehiph Dommiiration
icampl und Analogs: "poudic the entry [MASK] trtred
is
Aprll.
The question and demonstration description stage is shown in the second part of Figure 2. We the query's
(MASK] infortation "wodic" gf Abbln Conter The anoser Milan.
th
input the question (corresponding text) and tell it that two types of examples are be provided and should treated differently, one for analogy and one containing supplementary infor- mation.
Obay, I wwll leey tlisking
Final
quorice Prodict
'voel) inclade (Separbeer, March,
Mile"" The eastilate
Next, illustrated in the third part of Figure provide the LLM with a batch of demonstrations (La and L2) from the analogy pool and supple- ment pool. To include more KG information and demonstrations, we repeat this step as many times as possible subject to the input token length limit. Finally, restate the query text, and ask the LLM re-rank the top-m candidate entities. The feedback is parsed into an ordered list R.LM. which then replaces the top-m entities in Ryetriever as the final answer.
New, hasd
prsveca caampler and you know loder diaking the to caaliiatee raawers whick mree peeclte be the asoser quection
Rearnnged Ciderr (Jayy |.- (Septarbar] March -..
Figure 2: Illustration of multi-round interaction with the LLM. Stage 3 is repeated many times to provide more demonstrations.
However, this description may still be hard for the LLM to comprehend, and may lead to incorrect responses.
3.4 Text Self-Alignment
In this section, we propose Text Self-alignment for KG text cleaning. It transforms the raw text in the KG to more understandable descriptions by the LLM.
To address the above issue, the LLM summarize natural text descriptions from the given demonstrations that use raw descrip- tions. Specifically, for a relation r, we use the analogy pool Da for r (Section 3.3.1) as demon- strations and order them to get the ordered list La (Section 3.3.2). These demonstrations are fed as prompt to the LLM (Appendix A.2), which is asked summarize the semantics of the relation into sentence (the self-aligned text description of r). For the above example with raw relation text hhtht__pnngram/couutty... _oriigii"""" the LLM generates the much clearer and natural self-aligned text description "[T] is the country where the TV
ICL and let
Raw and obscure text descriptions generally exist in the KG. For example, raw rela- tion description in the FB15K-237 dataset is http/v_pprggrm/coountt of oriigin.. Most existing methods (Yao et al., 2019; Zhang al., 2020b; Wang et al., 2021b, 2022a) convert it cleaner description by removing the symbols (c.g.. "Iv program country of origin"). By default, KICGPT uses "of" to organize such hicrarchical relation, leading to "country of origin of tv program of tv".

---

Datasct
FB15k-237
WNI8RR
Metric
MRR
Hits @ I
Hits@r
Hits TO
MRR
0i I
Hits@
Hits@ 10
Triple-based methods
RESCALLNNicel al., 2011)< +
0.356
0.266
0.390
0.535
0.467
0.439
0.478
0,516
TransF. (Rordes al., 2013)
0.279
0.198
0.376
0.441
0.243
0.043
0.441
0.532
DistMult (Yang et al., 2014)
0.241
0.155
0,263
0.419
0.430
0.390
0.440
0,490
ComplEx (Trouillon al., 2016)
0.247
0.158
0.275
0.428
0.440
0.410
0.460
0.510
RotatE (Sun et al., 2019)
0.338
0.241
0.375
0.533
0.476
0.428
0.492
0.571
TuckER (Balazević et al., 2019)
0.358
0.266
0.394
0.544
0,470
0,443
0,482
0,526
HAKE (Zhang et al., 2020a)
0.346
0.250
0.381
0,542
0.497
0.452
0.516
0.582
CompGCN (Vashishth ct al., 2019)
0.355
0.264
0.390
0.535
0.479
0.443
0.494
0,546
Hihe al., 2021)
0.344
0.246
0.380
0.535
0.496
0.449
0.514
0.586
Text-based methods
Pretrain-KGon (Zhang et al., 2020b)
0.332
0.529
0.235
0,557
KG-BERT (aa al., 2019)
0.420
0.216
0.041
0.302
0.524
StAR (Wang ct al., 2021a)
0,263
0.171
0.287
0.452
0.364
0.222
0.436
0.647
MEM-KGC (w/o (Choi al., 2021)
0.339
0.249
0.372
0.522
0.533
0.473
0.570
0.636
MEM-KGG (w/ EP) (Choi et al., 2021)
0.346
0.253
0.381
0.531
0.557
0.475
0.604
0.704
HLM-baser methods
ChatGPT shon (Zhu al., 2023)
0.237
0.190
ChatGPT (Zhu et al., 2023)
0.267
0.212
KICGPT
0.412
0.327
0.448
0,554
0.549
0.474
0.585
0.641
KICGPT
0.410
0.321
0.430
0.581
0.564
0.478
0.612
0.677
Table Comparison between the proposed methods and baseline methods. The best result in terms of each metric is shown in bold and the second best one is underlined. A indicates that results are copied from (Wang et al., 2022a). implies that results are copied from (Chen et al., 2021), o means that the results are running on entire data according to the settings in (Zhu et al., 2023), and other results are taken from their original papers. EP denotes the entity prediction task in the MEM-KGC model.
program [H] originated from.", where [7] and [H] are placeholders for the tail and head entities, re- spectively.
and WN18RR datasets remove redundant inverse relations in case of information leakage. Com- pared with FB15k-237, the knowledge graph of WNI8RR is much sparser. The statistics of the two datascts are shown in Table 2.
Data pre-processing uses text self-alignment to create clear and aligned descriptions for each rela- tion, which can be directly used as relation text in KICGPT for link prediction. This variant is called KICGPTisa tsa- Since the texts are derived from the LLM, they conform to its presentation conventions, making them easier understand and improving performance.
Dataset WNI&RR
# Ent
Rel
train
valid
# test
40,943
86,835
3,034 17,535
3,134
FB15K-237
14.541
272,115
20,466
Table 2: Statistics of the henchmark datasets.
4
Experiment
Baselines. We compare the proposed KICGPT with a number of triple-based, text-based and LLM- based baselines. The triple-based baselines include RESCAL (Nickel et al., 2011), TransE (Bordes et al., 2013), DistMult (Yang et al., 2014), Com- plEx (Trouillon et al., 2016), RotatE (Sun et al., 2019). TuckER (Balažević et al., 2019), HAKE (Zhang et al., 2020a). CompGCN (Vashishth et al., 2019), and HitER (Chen et al., 2021). The text- based basclines! include Pretrain-KGE (Zhang et al., 2020b), KG-BERT (Yao et al., 2019), StAR (Wang et al., 2021a), and MEM-KEE (Choi et al.,
this section, we empirically evaluate KICGPT.
4.1 Setup
Datasets. We evaluate the proposed methods on the FB15k-237 (Toutanova et al., 2015) and WN18RR (Dettmers et al., 2018), which are widely- used benchmark for link prediction. FB 137 is subset of the freebase (Bollacker et al., 2008) knowledge graph, which includes commonsense knowledge about topics such as movies, sports. awards, and traveling. WNI8RR is a subset of WordNet (Miller, 1995), which contains knowledge about English morphology. Both the FB 155k237
We did not compare our method with Wang et al. (2022a) due to a data leakage issue on the entity degrees.

---

2021). The LLM-based baselines are based on ChatGPT. Zhu et al. (2023) reported zero-shot and one-shot link prediction results on 25 instances sampled from FB15k-237, We run their codes on the whole dataset and report the updated results as ser and ChatGPT onetGPTome-shot- one-shot-
improve the Hits @ performance but degrades the MRR and Hits [1.3] performance. This may be because the text self-alignment texts are derived from the demonstrations, which are also used in the LLM inference process along with the text self- alignment texts. While the mutually supportive text self-alignment texts and demonstrations enhance the LLM's understanding of correct semantics, they also make incorrect interpretations of relational semantics more persistent.
zero-shot
Implementation details. Though the proposed method can be used with various retrievers and LLMs, we prefer lightweight models for efficiency considerations. Specifically, our KICGPT imple- mentation uses RotatE (Sun et al., 2019) the retriever and ChatGPT (gpt-3.5-turbo) as the LLM. Hyper-parameters for RotatE are set as in (Sun et al., 2019). The value of m is selected from (10, 20, 30, 40, 50), and the batch size from (4, 8, 16. 32) } on a small random subset with 200 in- stances from the validation set. For the ChatGPT API, we set temperature, presence_penaiee "rquency_pennal to 0, and top_p to 1 avoid randomness. The detailed prompts are shown in Appendix A.2.
WNI8RR also_ see
FB15k-237
/tv/tv program/countryy of origin
hypernym
/ccation//ocaiio//prriillly_ contains
/commonthpicc/webpage. lcommotwebpaggeectegggry
has_part
Table 3: Example relations from the WNI8RR and FB15k-237 datasets.
and
On the WNISRR dataset, KICGPT achieves state-of- site-of-thee the-art performance on all metrics except Hits @ 10. Unlike the FB15k-237 dataset, text self-alignment improves all metrics. This may be partly because the higher average number of triples per relation in WNI8RR (8454.8) com- pared to FB FB15k-237 (1308.5), which facilitates the generation of more precise text descriptions due the increased availability of information for each relation. Besides, as illustrated in Table 3, WNI relations have more concise and direct semantics than those in FB 15k-237, making text self self-alignment for WN18RR easier than for FB 15k- 237. These differences cause the LLM to sum- marize more concise and precise descriptions for relations from WNISRR, which boosts the perfor- mance with higher-quality aligned text.
Metries. Link prediction outputs ranked list of all KG entities. We report Mean Reciprocal Rank (MRR), and Hits 1, 3, of the ranked list under the "filtered" setting. The "filtered" setting (Bordes et al., 2013) is a common practice that filters out valid entities other than ground truth target entities from the ranked list.
4.2 Experimental Results
Results are shown in Table 1.2 As can be seen, the proposed KICGPT and KICGPT achieve state-of-the-art performance on both FB15k-237 and WN18RR in most metrics.
Specifically, on the Fb15k-237 dataset, both variants of KICGPT surpass all the baselines in terms of all the evaluation metrics. The text self- alignment method helps improve the Hits@ 10 per- formance but degrades the MRR and Hits @ (1,3) performance. This may be because these demon- strations and aligned text are mutually supportive. which enhances the confidence of LLM to under- stand correct semantics, but for relations that LLM does not understand well, the aligned texts may not convey the true semantics.
The proposed methods show better performance over the triple-based and text-based baselines. This demonstrates the usefulness of the knowledge base inside the LLM. Besides, compared with the LLM- based baselines (i.e., ChatGPT baselines"), the pro- posed methods significantly outperform on both datasets, which shows the superiority of integrat- ing KG with LLMs for KGC. The performance
'The results we report for ChatGPT baselines are worse than those reported in (Zhu et al., 2023). This is because they experimented on only 25 triples on the FB15k 133k-2337 dataset while we test on the whole 20,466 testing triples. Besides, they manually restated descriptions in KG as clean and nat- ural expressions for the 25 triple, but doing this for all the testing triples is labor-intensive and time-costly. In our exper- iment, we use the text generated by text self-alignment as substitution.
On the Fb15k-237 dataset, both KICGPT and KICGPT surpass all the basclines in terms of all evaluation metrics. Text self-alignment helps
Wee only report @its@0 I for the I.I. J..M-basce baselines be- cause the LLM only outputs an answer but not a list.

---

improvement mainly comes from the injection of information contained in the knowledge graph.
ordering (Section 3.3.2). The observed degraded performance implies that the specially designed prompts can make use the properties of the KGC task to boost performance.
The demonstration ordering process takes 50.13 and 30.29 minutes for the FB15k-237 and WN18RR datasets, respectively. In terms of train- ing efficiency, KICGPT takes only 28.36 and 20.63 minutes to train on FB15k-237 and WN18RR, re- spectively, making more efficient compared to existing text-based methods that can take hours to train. This is because KICGPT does not require fine-tuning of the LLM.
MRR
Hits01
Hita@3
Hits 10
KICGPT
0.412
0,448
0.554
shuffle candidares
0.401
0.312
0.433
0.521
wro demonstration ordering
0.368
0,283
0.417
0.497
random demonstrations
0.349
0,271
0,387
0.481
who ICL
0.342
0.241
0.403
0.481
promppt engineering
0.432
0.548
Table 4: Ablation results on FB15k-237 (averaged over 4 random runs).
4.3 Ablation Study
this experiment, using the FB 15k-2237 dataset, we perform ablation studies to demonstrate useful- ness of each component in KICGPT. Table 4 shows the results.
4.4 Analysis Long-Tail Entities
To show the effectiveness of KICGPT and KICGPT to handle long-tail entities, we follow (Wang et al., 2022a) and group entities by their logarithm degrees in the knowledge graph. Entities with lower degrees (and hence more likely to be long-tal tail entities) assigned groups with lower indexes. A triple rh, r) is considered relevant to group d if belongs d.
In the first ablation study, we shuffle the top-m entities in Rretriever before feeding into the LLM. As can be seen, this causes a slight performance degradation. This shows that the order offered by the retriever is important and can reduce the dif- ficulty of re-ranking. Recently, Sun et al. (2023) also noted the significance of the initial order in search re-ranking.
Figure 3 displays the Hits@ and Hits@ 10 per- formance averages of various models on the FB 15k- 237 dataset, categorized by the logarithm of entity degrees. Text-based methods demonstrate slightly better performance than triple-based methods on long-tail entries. However, this improvement is not significant since it only applies small portion of long-tail entities (specifically, groups 0, 2). Compared with these baselines, the proposed mod- els achieve performance improvement on almost all groups and perform significantly better on long-tail entities, which confirms the benefits of combining the LLM with KG.
In the second study, we randomize the demon- stration orders from the analogy and supplement pools. Again, the performance degrades, demon- strating effectiveness of the proposed demonstra- tion ordering in Section 3.3.2.
In the third study, we retrieve random triples from the KG as demonstrations. The resultant per- formance degradation shows usefulness of the con- struction of demonstration pools in Section 3. In the fourth study, we exclude all demonstra- tions, while still providing the top-m candidates to constrain the ChatGPT output. As can be seen, a significant performance gap with the full model is observed, demonstrating the necessity of the pro- posed ICL strategy. Note also that @it@3 without ICL is superior Hit@3 3 with random demonstra- tions. This may be attributed the vast number of triples in the KG, rendering the random demon- strations to have insignificant semantic relevance to the query. The use irrelevant triples as demon- strations to LLM can introduce noise, potentially resulting in misleading outcomes.
WCOFT
SAC
HENOSC
94g
Figure 3: Average performance of different models (in terms of Hits@1 and Hits@ 10) grouped by the loga- rithm of entity degrees on the FB 151k--237 dataset.
Conclusion
In the last ablation study (Appendix A.2), use a prompt without the prompt engineering com- ponent (Section 3.3.3). This variant still uses the proposed demonstration pools (Section 3.3.1) and
In this paper, we propose KICGPT, an effective framework to integrate LLM and traditional KGC methods for link prediction, and a new ICL strategy

---

called Knowledge Prompt. KICGPT utilizes the LLM as an extra knowledge base. Compared with text-based methods, KICGPT utilizes the training- free property of LLM to significantly reduce train- ing overhead, and does not require finetuning for different KGs. Experimental results show that KICGPT achieves state-of-the-art performance on the link prediction benchmarks and is effective in the handling of long-tail entities.
Winter, Chris Hesse, Mark Chen, Eric Sigler, Ma- teusz Litwin, Scott Gray, Benjamin Chess, Jack Clark, Christopher Berner, Sam McCandlish, Alec Radford, Ilya Sutskever, and Dario Amodei. 2020, Language models are few-shot learners. In Ad- vances in Neural Information Processing Systems, volume 33, pages 1877-1901. Curran Associates, Inc.
Mingyang Chen, Wen Zhang. Wei Zhang, Qiang Chen, and Huajun Chen. 2019. Meta relational learning for few-shot link prediction in knowledge graphs. arXiv preprint arXiv: arXiv:190.0051..
Acknowledgements
This work is supported by NSFC general grant 62076118 18 and Shenzhen fundamental research pro- gram JCY12021033210500000
Sanxing Chen, Xiaodong Liu, Jianfeng Gao, Jian Jiao, Ruofci Zhang, and Yangfeng Ji. 2021. Hitter: Hicrar- chical transformers for knowledge graph embeddings. In Proceedings of the 2021 Conference Empirical Methods in Natural Language Processing (EMNLP). Association for Computational Linguisties.
Limitations
The performance improvement of the proposed ap- proach KGC is largely due to the large and ex- tensive knowledge base within the LLM. However, for some KGs (such as personal preference data for users in an E-commerce platform), the LLM may not contain enough relevant knowledge. Therefore, the proposed method works mostly for common- sense KGs. Besides, because of the limited token length in the LLM, we cannot inject all relevant facts from the KG as prompts.
Bonggeun Choi, Daesik Jang, and Youngjoong Ko. 2021. Mem-kge: Masked entity model for knowl- edge graph completion with pre-trained language model. IEEE Access, 9: 1312055--33203..
Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. 2018. Convolutional 2d knowl- edge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32.
Qingxiu Dong, Lei Li, Damai Dai, Ce Zheng, Zhiy- ong Wu, Baobao Chang, Xu Sun, Jingjing Xu, and Zhifang Sui. 2022. A survey for in-context learning. arXiv preprint arXiv:2301.002.
Ethics Statement
There is no ethical problem in our study.
Jinhao Jiang, Kun Zhou, Zican Dong, Keming Ye. Wayne Xin Zhao, and Ji- Ji-Rong Wen. 2023. Struct- gpt: A general framework for large language model to reason structured data. arXiv preprint arXiv:2305.09645
References
Ivana Balažević, Carl Allen, and Timothy M Hospedales. 2019. Tucker: Tensor factorization for knowledge graph completion. arXiv preprint arXiv:1901.09590.
George A Miller. 1995, Wordnet: lexical database for english. Communications of the ACM, 38(11):39-44
Maximilian Nickel, Volker Tresp, Hans-Peter Kriegel, et al. 2011. A three-way way model for collective learning multi-relational data. In Ieml, volume 11, pages 3104482-310433
Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. 2008. Freebase: a collabo- ratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIG- MOD international conference on Management of data, pages 1247-1250.
OpenAI. 2023. Gpt-4 technical report. CoRR.
Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida. Carroll Wainwright, Pamela Mishkin. Chong Zhang. Sandhini Agarwal, Katarina Slama, Alex Ray, et al. 2022. Training language models to follow instruc- tions with human feedback. Advances in Neural Information Processing Systems, 35:27730-27744,
Antoine Bordes, Nicolas Usunier, Alberto Garcia- Duran, Jason Weston, and Oksana Yakhnenko. 2013. Translating embeddings for modeling multi- relational data. Advances in neural information pro- cessing systems, 26.
Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Von Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens
Jack W Rae, Sebastian Borgeaud, Trevor Cai, Katic Millican, Jordan Hoffmann, Francis Song, John Aslanides, Sarah Henderson, Roman Ring, Susan- nah Young, et al. 2021. Scaling language models: Methods, analysis insights from training gopher. arXiv preprint arXiv:21 1112. 12.11444..

---

Stephen E Robertson and Steve Walker. 1994. Some simple effective approximations to the 2-poisson model for probabilistic weighted retrieval. In SI- GIR 94: Proceedings of the Seventeenth Annual In- lernational ACM-SIGIR Conference on Research and Development in Information Retrieval, organised by Dublin City University, pages 232-241. Springer.
Xiaozhi Wang. Tiayyu Gao, Zhaocheng Zhu, Zhengyan Zhang, Zhiyuan Liu, Juanzi Li, and Jian Tang. 2021b. Kepler: A unified model for knowledge embedding and pre-trained language representation. Transac- tions of the Association for Computational Linguis- tics, 9: 174-194.
Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. 2022a. Language models as knowledge embed- dings. I/CAI.
Ali Sadeghian, Mohammadreza Armandpour, Patrick Ding, and Daisy Zhe Wang. 2019. Drum: End-to- end differentiable rule mining on knowledge graphs. Advances in Neural Information Processing Systems, 32.
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoe Le, Ed Chi, and Denny Zhou. 2022b. Rationale- augmented ensembles in language models. arXiv preprint arxiv:2207.00747.
Michael Schlichtkrull, Thomas N Kipf, Peter Bloem, Rianne Van Den Berg, Ivan Titov, and Max Welling. 2018. Modeling relational data with graph convolu- tional networks. In The Semantic Web: 15th Inter- national Conference, ESWC 2018, Heraklion, Crete, Greece, June 3-7, 2018, Proceedings 15, pages 593- 607. Springer.
Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, Ed H. Chi, Tatsunori Hashimoto, Oriol Vinyals, Percy Liang, Jeff Dean, and William Fedus. 2022. Emer- gent abilities of large language models. Transactions on Machine Learning Research. Survey Certifica- tion.
Weiwei Sun, Lingyong Yan, Xinyu Ma. Pengjie Ren, Dawei Yin, and Zhaochun Ren. 2023. Is chatgpt good at search? investigating large lan- guage models as re-ranking agent. arXiv preprint arXiv:2304.09542
Ruobing Xie, Zhiyuan Liu, Jia Jia, Huanbo Luan, and Maosong Sun. 2016. Representation learning of knowledge graphs with entity descriptions. In Pro- ceedings of the AAAI Conference on Artificial Intelli- gence, volume 30.
Zhiqing Sun, Zhi-Hong Deng, Jun Nie, and Jian Tang. 2019. Rotate: Knowledge graph embedding by relational rotation in complex space. ICLR.
Wenhan Xiong, Mo Yu, Shiyu Chang, Xiaoxiao Guo, and William Yang Wang. 2018. One-shot rela- tional learning for knowledge graphs. arXiv preprint arXiv:1808.0004
Yi Tay, Jason Wei, Hyung Won Chung, Vinh Q Tran, David R So. Siamak Shakeri, Xavier Gar- cia, Huaixiu Steven Zheng. Jinfeng Rao, Aakanksha Chowdhery, et al. 2022. Transcending scaling laws with 0.1% extra compute. arXiv preprint 2XXi:2210.11399
Bishan Yang. Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng, 2014. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv: 1472.655.
Kristina Toutanova, Danqi Chen, Patrick Pantel, Hoi- fung Poon, Pallavi Choudhury, and Michael Gamon. 2015. Representing text for joint embedding of text and knowledge bases. In Proceedings of the 2015 conference empirical methods in natural language processing, pages 1499-1509.
Liang Yao, Chengsheng Mao, and Yuan Luo. 2019. Kg- bert: Bert for knowledge graph completion. arXiv preprint arXiv: 1909.03793
Jiacheng Ye, Zhiyong Wu, Jiangtao Feng, Tao Yu. and Lingpeng Kong. 2023. Compositional ex- emplars for in-context learning. arXiv preprint arXiv:2302.05698.
Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. 2016. Complex embeddings for simple link prediction. In Interna- tional conference on machine learning. pages 2071- 2080. PMLR.
Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. 2020a. Learning hierarchy- knowledge graph embeddings for link prediction. In Proceed- ings of the AAAI conference on artificial intelligence. volume 34, pages 3065-3072
Shikhar Vashishth, Soumya Sanyal, Vikram Nitin, and Partha Talukdar. 2019. Composition-bass multi- relational graph convolutional networks. ICLR.
Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. 2021a. Structure-auune augmented text representation learning for efficient knowledge graph completion. In Proceedings of the Web Confer- ence 2021, pages 1737-1748.
Zhiyuan Zhang, Xiaoqian Liu, Yi Zhang, Qi Su, Xu Sun, and Bin He. 2020b. Pretrain-kge: learning knowl- edge representation from pretrained language models. In Findings of the Association for Computational Lin- guistics: EMNLP 2020, pages 259-266.
Quan Wang, Zhendong Mao, Bin Wang, and Li Guo. 2017. Knowledge graph embedding: A survey of approaches and applications. IEEE Transactions on Knowledge and Data Engineering, 29(12):2724 2743.
Zihao Zhao, Eric Wallace, Shi Feng, Dan Klein, and Sameer Singh. 2021. Calibrate before use: Improv- ing few-shot performance of language models. In In- ternational Conference on Machine Learning, pages 12697-12706. PMLR

---

Yugi Zhu, Xiaohan Wang. Jing Chen, Shuofei Qiao. Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, and Ningyu Zhang. 2023. Llms for knowledge graph construction and reasoning: Recent capa- bilities and future opportunities. arXiv preprint arXiv:2305.134 13168.

---

A Appendix
A.1 KICGPT Algorithm
We provide the algorithm of KICGPT here.
Algorithm Demonstration pools from KG
Input:
KG (h, r, t), E,, and R denote the entity set and the relation set of G, Girain and Gealid sets of triples in G for training and validation link prediction query q rh
Output: analogy pool Da
supplement pool D
Da = [(e',,,,") E Gtrain U Gvalidl é,, e" E E}
2 D. ((u,r'ee) E Gtrain U Gratielr' E R E E E Girain U G E R. E
F)
Algorithm 2: Diversity- ordering for analogy pool
Input: Analogy demonstration pool: Da, entity set of KG Output: ordered list La foreach element E E do initialize zero counter Ce 0 for
Randomly select triple (h', r',t)) t') from Da, append L. and increase the counter oj and ce by
3
4 repeat
Find a triple that has the minimum sum of ce and c where e' and e" are the head and tail entity in triple (If there are multiple triples with minimum sum of entity counters, randomly select one of them as a)
Append to Ja and remove from Da increase the counter of entities ce, Cer in triple by
until D2 is empty

---

Algorithm 3: BM25 score-based ordering for supplement pool
Input: Supplement demonstration pool: DN link prediction query !
Output: ordered list L.
foreach triple in D3 do
calculate the BM25 score between text descriptions of and q
Lx O-Odeering triples in D3 according the BM25 score in descending order
Algorithm 4: Algorithm for KICGPT
For presentation simplicity, we represent the algorithm on missing tail queries. missing head queries are handled in a similar way.
Input:
KG = (h, r, t), E, and R denote the entity set and the relation set of G, Girain and Gealid are sets of triples in G for training and validation
link prediction query = rh
Output: Ranked list RKIGGPT contains all e E
Ranked list Bretriever contains all E +- Handling by triple-based KGC retriever Ryetriever [Eist, E2nd, E3rds.... e [[E)
2 Analogy Pool D., Supplement Pool D, + Fetch Demonstration Pools from based on
= rh,
La + Ordering triples in Da based on diversity
L <- Ordering triples in D2 based on BM25 score
Use unified prompt template to convert the query q and triples in La and La to text
With some instructions, demonstrations arranged by order in L,, and L, organized to ICL
prompts
7 Given the ICL prompts and query, LLM asked to perform re-ranking for first m entities in
Byetriever
RLLM < LLM (first m entities in Ryctrieser: ICL prompts, text of ()
RLLM = 1é e'znds
Emthh is permutation of first m entities in Reetriever
*RKIGGPT =
(esst 22nd: 3rdd ... mith: 1mt .., e E enl, where first m entities is RELM and remaining come from corresponding location of Beetriever

---

A.2 Prompt Formatting
We list the prompts used this paper as follows.
Datasets
Prompt Template
ChatGPT
FB15k-237
You are a good assistant to roading, understanding and sumarizing. Ecuador is the partially_contains of location location of thing partially contains of Pacific Ocean
If the example shows some-
location of location of of some- thing B3 means A located partially within boundarics
Appalachian Mountains is the partially.contains of location of location Massachusetts Moldavia is the partial ly.ontaiissns of loration of location of of B.
Moldova
Ecuador is the partially_contains location of location of South America
Adirondack Mountains is the partially.contains location of location of Warren County
In above examples, What do you think "partially_contain of location location of mean? Sunnarize and descript neaning using the format: "If the exappe shows something A is pertially_contaite of location of location of of something B, it the
[nass] B." be short as possible.
WNISRR
You good assistant to reading, understanding and sumarizzing If the example shows thing be member of domain usage of something A term word that be- longs to the cutegory or do- main B' usage.
red indian be menber of donain usage of dispargemmnnt compass be donain usage archaism
penicillin potassium be member of domain usage of trade name nanna be nember of domain usage of united kingdon of great britain nor thenn ireland
nerves be nember of donain usage of plural form zyloprin be menber of donain usage of trademark examples, What you of" mean? Summarize and descript its meaning using the format: the example shows something be menber of domain usage of sonething B, statonent should be as short possible.
nember
usage
[mask] of B. Fill the mask and
Table 5: Examples of prompts for text self self-alignment

---

Prompt Formatting
Prompt Template
ChatGPT
Trivial Prompt
predict the tail entity [MASK] from the Blanc, type_of_union of marriage of people of spouse_s of person Dumestic partnership Civil of people of [MASK] by completing the sentence "what the union Official Website HBO persan
(Mel The final order: [Marriage
type_of_union of marriage of people of spouse_s people of Mel Blanc? The answer is The answer is Harriage, so Basanti Judaism-GB I Cru- the [MASK] Marriage, predict the tail entity [MASK] fron the sades).
Male Teleevsiio Rang De
given (Screen Actors Guild Life Achievenent Award, awnrd_winner award_honor honor of award of winners of award_categary of award of
[MASK]) by completing the sentence "what is the award_winner award_honor of award winners award_category of award of Screen Actors Guild Life Achievement Award? The answer is The answer Stan Laurel, so the [HASK] is Stan Laurel.
The list of candidate Civil union, Official Website, Rang De Basanti, HBO, Hale, Television, Judaisn-CB, Crusades]. And question is predict the tail entity [MSS] from the given (Stan Laurel, Lype_of_untt marriage of people of spouse_s of person of people of [MASK]) by completing the sentence "what the type_of ofinno marriage people of spouse_s of
is [Marriage, Donestic partnership,
of people of Stan Laurel? The answer is New, based on the previous examples and your own knowledge and thinking, sort the list let the candidate answers which are possible be the true the sorted order of candidate answers using the format "[most possible answer | second possible answer ... | least possible answer]" and please start your response with "The final order:"
the question prior. Output
Do not output anything except the final order. Note your output sorted
contain all candidates the list but not add new answers to it.
Table 6: Trivial Prompts on the FB15k-237 dataset.

---

Steps
Prompt Template
ChatGPT
Responsibiiity Description
You are good assistant to perform link prediction and sorting. Yes. list of candidate answers this question. You need to order these candi answers in the list to let candidate answers which are more possible to be the answer to the question prior. If you respond "Yes". Otherwise, respond "No". Da not output anything except "Yes" and "No".
Given goal question
your responsibility,
Question and Demonstration Description
The goal question is: predict the tail entity [MASK] from the given Okay, understand. I will wait (Stan Laurel, sypreo. sype_of_union person of people of, [MASK]) by completing the sentence "what tions.
marriage of people of spouse_s of for your examples and instrue-
is the type_af_union of the narriage of people of spouse_s of the person people of Stan Laurel? The answer is To sort the candidate other examples that may be similar to or related to the question. Part of the given examples are similar to the goal question, you should analogy then to understand the potential meaning the goal question. Another part of the given facts contains supplementary information, keep capturing this extra information and mining potential relationships among then to help the sarting. Please carefully read, real ize, and think about these examples. Sumnarize the way of thinking in these examples and menorize the information you think maybe help your sorting task. During I give examples please keep silent until let you output.
typically you would need to refer
Multiple Demonstrations
Examples used Analogy: "predict the tail entity [MASK] from the Okay. will keep thinking and given (Mel Blanc, slanc,type_of type_of__nioon of narriage of people of spouse_s analyzing the given examples of person of people is the type_ef_ of__nion of narriage of people of spouse_s of person ships and patterss that can help of people of Mel Blanc? The answer is * The answer is Marriage, with the sorting task. so the [HASK] is Marriage."
[MASK]) by completing the sentence "what identify potential relation-
Examples give supplement information: "predict the tail entity [MASK]
given (Screen Actors Guild Life Achievenent of award_honor of award of winners of category of award of [MASK] by completing the sentence "what. award_winner
award_hond of award__ategory of award of Screen Actors Guild Life Achievement Award? The answer is ". The answer is Stan Laurel, so the [MASK] is Stan Laurel. Keep thinking
winners
not output.
Final Query
The list of candidate answers is [Marriage, Donestic partnership, The final order: [Marriage Civil union, Official Website, Rang De Basanti, HBO, Male, Domestic partnership Civil Television, Judaisn-GB, Crusades]. And the question is predict the union I Official Website HBO tail entity [MAS] from the given (Stan Laurel, marriage people spouse_s person of people
of Male I Teleevision I Rang De [MASK]] by Basanti Judaism-GB completing the sentence "what the type_of type_of_union of narriage of sades).
people of spouse_s of person of people of Stan Laurel? The answer is New, based the previous examples and your own knowledge and thinking, sort the list to let the candidate answers which are more possible to be the true answer to the question prior. Output the sorted order possible answer I second possible answer answer]" and please start your response with "The final order:" ". Do not output anything except final order, Note your output sorted order should contain all the candidates in the list but not add new answers to it.
candidate answers using the fornat "[most least possible
Table 7: Examples of prompts KICGPT on the FB15k-237 dataset.

---

Steps
NOT Prompt Template
ChatGPT
Responsiblity Description
you're linguist English lexicons. will be first Yes. given some examples. Then use these examples as references and your own knowledge to score for some statements. If you have known your responsibility, respond "Yes". Otherwise, respond "No". not output anything except "Yes" and "No".
Question and Demonstration Description
nember of domain usage trade name. nkae. trade name name given to product ar service. Part. of the given examples are similar to the statements, you should analogy then understand scored. Another part of the given exaplees contains supplementary information, keep capturing this extra information and mining potential relationships amang then to help the scoring. Please carefully read, realize ize and think about these examples. Sumariize the way of thinking in these examples and menorize the information you think maybe help. NOT give any feedback.
The goal statements
potential meaning of statements to
Multiple Demonstrations
Examples used to Analogy: trade name : name given product Okay. or service. vinblastine periwinkle plant derivative used an antineoplastic drug (trade name Velban) that disrupts cell division. vinblastine be menber domain usage of trade nane vernacular characteristic language of particular group (as among thieves); "they don't speak our lingo". chink narrow opening as e.g. between planks a wall. chink be nember donain usage of vernacular.
Examples give supplement information: trade nane a name given to product or service. cortone acetate : corticosteroid hormone (trade name Cortone Acetate) normally produced by the adrenal cortex; is converted to hydrocortisone. cortone acetate be nember of donain usage trade nane trade name
name given product service. phenelzine monoamine oxidase inhibitor (trade nane Nardil) used to treat clinical depression. phenelzine be menber donsin usage give any feedback.
name. Keep thinking
Final Query
trade name : name given to a product or service. verapanil 90.
drug (trade parenteral calcium blacker in cases of hypertension
Calan and Isoptin) used
oral
congestive 100
heart failure or angina or migraine, verapanil be nember of domain
out of 100 for the 50 statement and DO NOT output any other thing
usage of trade
Directly give
given to product service. nitrostat trade nanes for nitroglycerin used as a coronary lator in the treatment of angina pectoris. nitrostat be nember of domain Directly give score out of 188 for the statement and DO NOT output any other thing
trade
usage of trade
given to product or service. hydantoin any of a group of anticonvulsant drugs used in treating epilepsy. hydantoin momber donain usage trade name .. Directly give score out of 180 for the statement and DO NOT output any other thing.
trade nane
Table 8: EExamples of prompts for KICGPT on the WNISRR dataset.


Reviews:
[{"review_id": "Z2rk7Kc9d2", "reviewer": "Reviewer_QYzB", "summary": "", "questions": "1. I don't quite agree with the long-tail problem of entity distribution mentioned in the article that the use of entity description information can solve the problem. In papers such as KG-BERT, there is no mention of the description that PLM can solve the long-tail problem. Is this motivation valid?\n2. From the experimental results of WN18RR, the Text Self-Alignment part is relatively important, but the author's introduction to this part is too simple. How to obtain a fluent expression that is more in line with LLM, the author simply mentioned \"The LLM is asked to summarize the semantics of the relation into a sentence\".\n3. It has been mentioned many times in the abstract and introduction that using LLM can solve the long tail problem, but there is no experimental support at all in the experimental part.\n4. Generally speaking, the use of LLM is more suitable for few-shot tasks. It is recommended that the author consider the few-shot KGC task verification.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive review! We address your concerns as follows.\n\n \n>  ${\\bf Q1}$: I don’t quite agree with the long-tail problem\nof entity distribution mentioned in the article that the use of entity description\ninformation can solve the problem. In papers such as KG-BERT, there is no\nmention of the description that PLM can solve the long-tail problem. Is this\nmotivation valid?   \nThere is a gap between the\nmotivation of the article and the final experiment. The content of the experimental part is not unified with the motivation. The most important thing is\nto deal with the long-tail problem. The experimental part should at least be\nsupported by data.  \n It has been mentioned many times in the abstract\nand introduction that using LLM can solve the long tail problem, but there is\nno experimental support at all in the experimental part.\n\n${\\bf Answer}$: \nThe benefits of using PLM to process entity description information for long-tail entities have been observed in [1], which\nmentions that \n(line 21, right column, page 1)\n“Textual descriptions provide rich information for numerous semantic-related tasks, which brings an opportunity to learn informative representations for long-tail entities.\" In addition, we have provided experiments and analysis in Section 4.4 to evaluate the performance on long-tail entities. Specifically, Figure 3 shows that PLM-based approaches (such as StaR [2] and MEM-KGC [3]) have better performance on long-tail entities when compared with triple-based approaches (such as TransE and RotatE), and our approach has a more substantial performance improvement than the aforementioned approaches.  \nNotably, our KICGPT method performs comparably or even better on long-tail entities (leftmost groups in Figure 3) compared to highly popular entities (rightmost groups in Figure 3) in the KGs.\n\nThese results demonstrate the motivation and effectiveness of our proposed method for solving the long-tail problem.\n\nMoreover, we would like to emphasize that addressing the challenges posed by the long-tail distribution is only one of our contributions. Our approach also offers the following contributions:\n\n1. As demonstrated in Table 1 and Figure 3, our KICGPT approach achieves overall state-of-the-art (SOTA) performance, and improves the performance not only on long-tail entities but also on almost all entities. This performance improvement is not limited to specific entity types or subsets, but rather across the entire Knowledge Graph. \n\n2. Our work is the first to combine Language Models (LLMs) with triple-based Knowledge Graph Completion (KGC) methods \nto address\nthe \nknowledge graph completion\ntask. \n\n> ${\\bf Q2}$: The introduction of some key steps is not very clear, such as Text Self-Alignment, which feels to have a certain impact on the experimental\nresults, but it is not introduced in detail.    \nFrom the experimental results of WN18RR, the Text Self-Alignment part is relatively important, but the author’s introduction to this part is too simple. How to obtain a fluent expression that is more in line with LLM, the author simply mentioned ”The LLM is asked to summarize the semantics of the relation into a sentence”.\n\n${\\bf Answer}$: Thank you for bringing this to our attention. As we are limited by page constraints, a full example is only shown in Appendix A.2, which illustrates how to perform text self-alignment by providing the entire prompt. In the revised version, we will provide more detailed descriptions of the text self-alignment process in the main text.\n\nIn the following, we give the workflow of the text self-alignment. First, for relation $r$, we get its analogy pool (Section 3.3.1).  Then we order the analogy pool (Section 3.3.2). The ordered demonstrations are organized with prompt instructions that let the LLM find the relationship (refer to prompt in Appendix A.2). Finally, we use instruction prompts to let the LLM output only one concise sentence (section 3.4), which describes the relationship between the head entity (i.e., using placeholder [H] or 'B') and the tail entity (i.e., using placeholder [T] or 'A').\nBy assigning real entities in triplets to placeholders, this sentence can be directly used as a description for the relation between entities.\n\n\n> ${\\bf Q3}$:  Generally speaking,\nthe use of LLM is more suitable for few-shot tasks. It is recommended that the\nauthor consider the few-shot KGC task verification.\n\n\n${\\bf Answer}$: The original KG has few triples for the long-tail entities. Thus, KGC for long-tail entities can be viewed as a few-shot task, in which each KG entity has only a few associated triples. Specifically, for each KG entity, a ''shot'' for that entity refers to its occurrence in a triple. Therefore, the number of degrees for each entity is the number of shots for that entity.\n\nIn Section 4.4, we group entities based on the logarithm of their degrees (i.e., number of shots). The experimental results demonstrate that the proposed model performs well on few-shot cases (i.e., the leftmost groups in Figure 3).\n\n\n> ${\\bf Q4}$:  The article lacks some methodological content, and more is to preprocess or carefully process the data, and the rationality of the corresponding processing method is difficult to weigh.\n\n\n${\\bf Answer}$: First, combining LLMs and KG structure is reasonable. The interaction between the knowledge in LLM parameters and the original KG leads to mutual alignment and enhancement. As can be seen from Table 1, KICGPT performs significantly better than ChatGPT and triple-based baselines. In this process, our in-context learning strategies also play an important role in bridging the KG structure with LLM reasoning.\n\nFor the detailed design \nusefulness of the components \n(retriever, processing demonstrations, prompt engineering)\ncan be seen from the ablation studies in Section 4.3.  Moreover, they are well-justified. For example, the significance of the initial order is also observed in the recent research on task reordering [4].\nBesides, simultaneously considering the diversity and correlation of demonstrations is important to ICL [5], which is similar to our construction of an analogy pool and a supplementary pool. We will add more discussions in the final revision.\n\nWe will correct typos and polish the writing in the final revision.\n\nSincerely thanks again for your patient and valuable review. If you have any other questions, please do not hesitate to contact us.\n\nReference: \n\n[1] Language models as knowledge embeddings, IJCAI 2022. \n\n[2] Structure-augmented text representation learning for efficient knowledge graph completion, WWW 2021.\n\n[3] Mem-KGC: Masked entity model for\nknowledge graph completion with pre-trained language\nmodel, IEEE Access, 2021.\n\n[4] Is ChatGPT Good at Search?\nInvestigating Large Language Models as Re-Ranking Agent, arxiv 2023.\n\n[5] Compositional Exemplars for In-context Learning, ICML 2023.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "1. The article proposes to use LLM for KGC tasks, which breaks the paradigm of traditional KGC methods.\n2. The author proposes to use Analogy Pool and Supplement Pool to reorder entities, which is innovative.\n3. Experiments were carried out on the general FB15K-237 and WN18RR to verify the effectiveness of the proposed method.", "reasons_to_reject": "1. The article lacks some methodological content, and more is to preprocess or carefully process the data, and the rationality of the corresponding processing method is difficult to weigh.\n2. There is a gap between the motivation of the article and the final experiment. The content of the experimental part is not unified with the motivation. The most important thing is to deal with the long-tail problem. The experimental part should at least be supported by data.\n3. The introduction of some key steps is not very clear, such as Text Self-Alignment, which feels to have a certain impact on the experimental results, but it is not introduced in detail.", "questions_to_authors": "1. I don't quite agree with the long-tail problem of entity distribution mentioned in the article that the use of entity description information can solve the problem. In papers such as KG-BERT, there is no mention of the description that PLM can solve the long-tail problem. Is this motivation valid?\n2. From the experimental results of WN18RR, the Text Self-Alignment part is relatively important, but the author's introduction to this part is too simple. How to obtain a fluent expression that is more in line with LLM, the author simply mentioned \"The LLM is asked to summarize the semantics of the relation into a sentence\".\n3. It has been mentioned many times in the abstract and introduction that using LLM can solve the long tail problem, but there is no experimental support at all in the experimental part.\n4. Generally speaking, the use of LLM is more suitable for few-shot tasks. It is recommended that the author consider the few-shot KGC task verification.", "paper_topic_and_main_contributions": "This paper proposes to integrate a LLM and a triple-based KGC retriever to KGC task, which can alleviate the long-tail problem without incurring additional training overhead. The proposed KICGPT model has a certain degree of innovation in part Knowledge Prompt, but the article lacks some methodological support or richer engineering experiments, and the experimental part has no way to verify the rationality of the motivation of this article, and the solution to the long tail problem is directly handed over to LLM. If so, it cannot be used as the motivation of this article in essence. Therefore, I think it is debatable whether this article needs to be accepted.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Missing_References": "None", "Typos_Grammar_Style_and_Presentation_Improvements": "1. There are some wrong words in the article, such as test-based in the abstract, shouldn't it be text-based?"}, "extra_text": {}}, {"review_id": "iQBJEHB6uq", "reviewer": "Reviewer_C1He", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your thoughtful comments. \n\nAs you said, we will revise the paper to avoid redundancy in the final revision.\n\nSincerely thanks for your encouraging rating and valuable review. If you have any other questions, please do not hesitate to contact us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "The paper points out there are novel approaches contribution which I think can benefit the community. Even though there are many existing KGC triple-based methods. However, the KICGPT \"combines a vast external knowledge base inside LLM with the structural information in KGs\", which is a unique solution to address the KGC tasks. Also, they specially designed a strategy for KGC tasks called Knowledge Prompt, which further improve the performance.", "reasons_to_reject": "I believe the paper is well-structured and has valuable contributions. However, there are a few instances where the concept of incomplete triple representation is repeated in both the introduction and problem setting sections.", "questions_to_authors": "", "paper_topic_and_main_contributions": "The paper points out the significance of Knowledge Graph Completion (KGC) in addressing incomplete triples within Knowledge Graphs (KGs). Specifically, this paper centers its focus on the link prediction task within KGC for incomplete KGs. It identifies limitations inherent in two existing KGC methods, prompting the need for an alternative approach.\nIn response to these limitations, the paper introduces KICGPT—a novel approach that leverages in-context learning (ICL) through a strategy termed Knowledge Prompt. Notably, KICGPT showcases promising results in predicting missing entities within KGs. The authors assert that their approach is noteworthy for being the first to synergize Large Language Models (LLMs) with triple-based KGC methods, thereby achieving state-of-the-art performance with minimal training overhead.", "excitement": 4, "reproducibility": 4, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "luSKuzC3V8", "reviewer": "Reviewer_A54b", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. We address your concerns as follows.\n\n> ${\\bf Q1}$: The prompt engineering method\nrelies heavily on the behavior of the LM, so it might not generalize well to other\nLLMs.\n\n\n${\\bf Answer}$: While we agree that the choice of LLM is important, our model is agnostic to the specific LLM used.\n\nThe LLM in our approach serves as a black box for the re-ranking task, regulated by specific prompt instructions (as detailed in Appendix A.2). Our model's variable-length and truncated prompt design (Section 3.3.3) allows it to accommodate existing LLMs with varying token length limits.\n\nTo further validate our model's adaptability, we plan to extend our experiments to include open-source LLMs like Llama-2. This will allow us to explore the use of visible parameters or embeddings in conjunction with Knowledge Graph structures.\n\n\n> ${\\bf Q2}$: The ablation study reveals that the improvement over the trivial prompt\nis modest, raising questions about its effectiveness relative to the cost.\n\n${\\bf Answer}$: To clarify, in the ablation experiments, the \"Trivial prompt\" variant only removes the prompt engineering component (Section 3.3.3) from KICGPT. As such, this variant still benefits from our special demonstration pools (Section 3.3.1) and ordering methods (Section 3.3.2), which are also supported by the initial order provided by the retriever. The performance gap observed in this variant is solely due to the absence of prompt engineering, which incurs a negligible cost (i.e., adding prompt engineering of Section 3.3.3). To prevent confusion, we will rename this variant to \"Without prompt engineering\" in the revised version.\n\nSincerely thanks again for your valuable review. If you have any other questions, please do not hesitate to contact us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- Using an LLM as a re-ranker for KGC is an interesting idea, and this paper provides a study of prompt selection strategies. \n- The proposed approach surpasses prior work in two benchmarks, demonstrating the effectiveness of this method.\n- This paper clearly articulates the key ideas and the motivations behind them. Both the design choices and experimental setup are sound and reasonable.\n", "reasons_to_reject": "- In the experiments, this approach was implemented using a proprietary language model (ChatGPT). The prompt engineering method relies heavily on the behavior of the LM, so it might not generalize well to other LLMs.\n- The ablation study reveals that the improvement over the trivial prompt is modest, raising questions about its effectiveness relative to the cost.\n", "questions_to_authors": "", "paper_topic_and_main_contributions": "This paper tackles a problem of knowledge graph completion (KGC) and proposes an approach (KICGPT) that integrates in-context learning and a triple-based KGC retriever. Concretely, this paper focuses on the link prediction task where a model is asked to predict either a head entity or a tail entity given an incomplete triple (i.e., (head, relation, ?) or (?, relation, tail)). First, a triple-based KGC retriever ranks candidate entities. Subsequently, an LLM is used as a re-ranker with curated prompts (e.g., different types of demonstrations and their order). The re-ranked entities are obtained by four rounds of interaction with the LLM. Additionally, the text self-alignment step  can be conducted if a raw text in KG is not LLM friendly (i.e., not natural sentences).\n\nThis approach is evaluated on the KGC benchmarks such as FB15k-235 and WN18RR and compared with both text-based and triple-based approaches. In terms of MRR and Hits@k, this approach (w and w/o text self-alignment) outperforms all prior approaches on both benchmarks. \n", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "5ssPETHlA6", "reviewer": "Reviewer_dRov", "summary": "", "questions": "Question A: Section 3.3.3 describes a multi-round prompting strategy for re-ranking the resulting entities. Were the answers from ChatGPT consistent, throughout the various prompting turns, across all test cases in the two datasets? How are cases during which the OpenAI model would return an un-expected output at one or more dialogue turns are handled?\nQuestion B: I understand from the manuscript that the results from ChatGPT are not fully-conditioned to the entities that are returned by the retriever. Did the output from ChatGPT include only permutations of the original set of candidate entities, across all test instances?\nQuestion C: While the baseline solutions appear to improve their performance as the group of entity degrees is increasing (i.e. in Figure 3), a similar behaviour is not observed for the KICGPT systems. This is more evident for the Hits@1 case. The manuscript could benefit from some additional clarification about why this behaviour is observed.", "limitations": "", "rating": 3, "confidence": 4, "soundness": 4, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for your constructive comments. The typo about the dataset labels in Table 2 will be corrected in the revision.  We address your concerns as follows.\n\n> ${\\bf Q1}$: Section 3.3.3 describes a multi-round prompting strategy for re-ranking the resulting entities. Were the answers from ChatGPT consistent,\nthroughout the various prompting turns, across all test cases in the two datasets? How are cases\nduring which the OpenAI model would return an unexpected output at one or more dialogue turns are handled?\n\n${\\bf Answer}$: As shown by the examples in Appendix A.2, the instructions usually do very well in limiting the LLM output to a given set of candidate answers and help keep the consistency of the LLM output in each round. \n\nHowever, there are still some rare cases that need to be handled. \n\n\n1. Outputting entities beyond the candidate set: In this case, we directly filter out the entities that are not in the candidate set.\n\n2. Losing some candidate entities: In this case, we append the lost entities according to their corresponding initial ordering (by retriever) at the end of LLM output. Note that this case typically occurs when the LLM is very confident about the top answers and it tends to omit the others.\n\n3. Damaged output: In this case, we cannot identify a valid ordering from the LLM output. To handle this, KICGPT directly uses the initial ordering of the retriever as the final decision (i.e., it degenerates to the retriever).\n\nThese rare cases are very rare. For example, in fb15k-237, such rare cases account for only 2.26\\% (926 queries out of 40932). For Wn18RR, we do not have rare cases because we explicitly let the LLM to score  and\nrank\nthe entities.\n\n> ${\\bf Q2}$: I understand from the manuscript that the results from ChatGPT are not fully-conditioned to the entities that are returned by the retriever. Did the output from ChatGPT include only permutations of the original set of candidate entities, across all test instances?\n\n${\\bf Answer}$: Please refer to the answer to the previous question.\n\n>${\\bf Q3}$: While the baseline solutions appear to improve their performance as the group of entity degrees is increasing (i.e. in Figure 3), a similar behaviour is not observed for the KICGPT systems. This is more evident for the Hits@1 case. The manuscript could benefit from some additional clarification about why this behaviour is observed.\n\n${\\bf Answer}$: Thanks for your insightful question. Unlike most baselines that show a performance increase with an increase in degree, the performance of KICGPT in Figure 3 follows a distinct pattern. It initially peaks on long-tail entities, then decreases with an increase in degree, and finally reaches another peak on popular entities. This can be explained as follows:\n\nFor the long-tail entities, very few entities are directly related to them, and retrievers can easily include all these entities in the candidate set. Moreover, we observe that the answers to long-tail queries are highly likely to be included in the union of the analogy pool and candidate set. Therefore, because the true answers are far more reasonable than the other entities, the LLM can use its powerful capacity to find them out easily. This corresponds to the first performance peak in Figure 3.\n\nFor the middle-range entities (which fall between popular and long-tail entities), we observed that relatively more adversarial or confusing candidates are mixed with the true answers in the initial ordering. Identifying the true answers becomes more difficult in this case. Besides, compared with the long-tail and popular entities, we observed relatively more cases where the true answer is not in the candidate set. This results in a decrease in performance, explaining why KICGPT does not follow the pattern seen in baseline solutions.\n\nFor the popular entities in KG, the initial ordering is more reasonable. It places the true answers at the top of the list and confusing candidates at the back, which reduces the difficulty of LLM reordering. This ultimately results in another performance peak in Figure 3.\n\nWe believe that this clarification will improve our manuscript and we will include a more detailed discussion in the revision.\n\n> ${\\bf Q4}$: I understand that any performance improvements for such cases (presented in Section 4.4) can be mostly attributed to ChatGPT.\n\n${\\bf Answer}$: The performance improvement on long-tail entities is partially due to the external knowledge in LLM. However, we believe that the KG structure also plays an important role in performance improvement. \n\nOne contribution the KG structure offers is a set of candidate entities to LLM for answering queries. This not only restricts the LLM from generating unexpected output beyond the KG, but also provides the LLM the most likely candidate answers to the query.\n\nAnother contribution is to provide a good initial ordering of candidate entities, \n which can alleviate the challenge of reordering. This is consistent with a recent study that employed LLM in reordering tasks [1].\n\n The third contribution is that, even in the absence of supplement demonstrations for long-tail entities, the analogy pool can still aid LLMs in comprehending the semantics of long-tail queries.\n\n\n\nThe following table shows the performance of a pure ChatGPT (without the proposed KICGPT components on long-tail entities) on groups with different degrees on the FB15k-237 dataset. Note that this pure ChatGPT also has demonstrations, which are randomly selected from the original KG and use the same prompt template as KICGPT. However, the pure ChatGPT is asked to provide the answer directly, without the aid of the ordered candidate answers. As can be seen, KICGPT significantly outperforms pure ChatGPT even on long-tail entities. Therefore, we believe that the performance improvement on long-tail entities is not solely dependent on the LLM. All the components in KICGPT (including the retriever (ordered candidates) and Knowledge Prompt (well-selected, ordered, and prompt-organized demonstrations)) also play critical roles in achieving state-of-the-art performance.\n\n|  Model | degree 0 | degree 1 | degree 2-3 | degree 4-7 |\n|---|---|---|---|---|\n| KICGPT | 0.5278 | 0.4706 | 0.4988 | 0.5009 |\n| ChatGPT | 0.3721 | 0.3357 | 0.3662 | 0.3479 |\n\nHit@1 for long-tail performance on FB15k-237\n\n[1]  Is ChatGPT Good at Search?\nInvestigating Large Language Models as Re-Ranking Agent, arXiv:2304.09542, 2023.\n\n> ${\\bf Q5}$: RotatE is used as the retriever in the proposed architecture. However, this choice is not sufficiently justified in the manuscript. I believe it would be interesting to see the performance of the end system, using at least one more from the KG embedding models.\n\n\n${\\bf Answer}$: Thank you for your valuable suggestion. In order to choose the most suitable retriever for our model, we experimented with several lightweight methods, including TransE, RotatE, Distmult, and ComplEX, on a randomly sampled tiny dataset consisting of 200 instances from the validation set. As can be seen from the table below, RotatE is very effective. \n\n| Metric | RotatE | TransE | DistMult | ComplEx |\n|---|---|---|---|---|\n | Hit@1 | 0.341 | 0.292 | 0.265 | 0.279 |\n | Hit@10 | 0.610 | 0.583 | 0.516 | 0.544 |\n\nKICGPT performance for different retrievers on the sampled tiny dataset\n\nTo further support our decision, we plan to include experiment results based on another KG embedding method in the revised version of our paper.\n\nSincerely thanks again for your patient and valuable review. If you have any other questions or advice, please do not hesitate to contact us.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "- The paper is nicely written; the ideas and the train-of-thought of the experiments are clearly described.\n- Interesting method for coupling triple-based approaches for triple completion with larger language models.\n- Promising idea for including and ordering demonstrations of different nature (i.e. coming from an analogy and a supplement pool). The ordering aspect can be also particularly useful for cases when the length of the input sequence should be truncated.", "reasons_to_reject": "- The work is partially motivated by the challenges faced by triple-based approaches when dealing with long-tail entities. However, the manuscript does not highlight any particular considerations for long-tail triples, and I understand that any performance improvements for such cases (presented in Section 4.4) can be mostly attributed to ChatGPT (i.e. assuming that long-tail entities would lack suitable demonstration examples in their corresponding supplement pool). \n- Some details about how the experiments are conducted are missing (please refer to Question A and B below).\n- RotatE is used as the retriever in the proposed architecture. However, this choice is not sufficiently justified in the manuscript. I believe it would be interesting to see the performance of the end system, using at least one more from the KG embedding models.", "questions_to_authors": "Question A: Section 3.3.3 describes a multi-round prompting strategy for re-ranking the resulting entities. Were the answers from ChatGPT consistent, throughout the various prompting turns, across all test cases in the two datasets? How are cases during which the OpenAI model would return an un-expected output at one or more dialogue turns are handled?\nQuestion B: I understand from the manuscript that the results from ChatGPT are not fully-conditioned to the entities that are returned by the retriever. Did the output from ChatGPT include only permutations of the original set of candidate entities, across all test instances?\nQuestion C: While the baseline solutions appear to improve their performance as the group of entity degrees is increasing (i.e. in Figure 3), a similar behaviour is not observed for the KICGPT systems. This is more evident for the Hits@1 case. The manuscript could benefit from some additional clarification about why this behaviour is observed.", "paper_topic_and_main_contributions": "The paper proposes a framework for coupling a large language model with a triple-based retriever for knowledge graph completion (i.e. triple completion). The authors introduce a strategy for including relevant triple-demonstrations in a multi-prompt interaction with ChatGPT along with a prompt-engineering methodology for verbalising relation texts into fluent textual description using larger, OpenAI models. They compare their approach against a variety of triple- and text-based baselines (incl. ChatGPT on a zero-shot and one-shot setup) on two different triple completion benchmarks: FB15k-237 and WN18RR.", "excitement": 3, "reproducibility": 3, "ethical_concerns": "No", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"Typos_Grammar_Style_and_Presentation_Improvements": "- I believe the dataset labels (i.e. FB15k-237 and WN18RR) in Table 2 should be swapped."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper delves into the task of link prediction within incomplete KGs using an innovative technique that utilizes in-context learning (ICL) through a mechanism called Knowledge Prompt. They show that their method, KICGPT, demonstrates strong predictive abilities for missing entities in KGs. The authors claim that their approach is the first in combining Large Language Models (LLMs) with triple-based KGC methods for alleviating the long-tail problem in KGC. 

The main concerns raised by the reviewers were:
1. Lack of clear motivation for the long tail problem - this was adequately addressed in the rebuttal as acknowledged by the reviewer
2. Use of a proprietary LLM (ChatGPT) - in response the authors have now repeated the experiments using Llama2 which is great

A few other minor issues were also adequately addressed by the authors. 

Overall, I think the paper has some interesting ideas and a thorough evaluation. The community would benefit from these ideas and build upon them.

I request the authors to fit all the additional results provided during the rebuttal, in the main body of appendix of the paper, as appropriate.
Decision: Accept-Findings