KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge
Authors: Pengcheng Jiang
Lang Cao
Cao Xiao
Parminder Bhatia
Jimeng Sun
Jiawei Han
Venue: NeurIPS 2024 poster (2024)
Keywords: Knowledge Graphs
Large Language Models
Link Prediction
Primary area: natural_language_processing

Abstract: Knowledge Graph Embedding (KGE) techniques are crucial in learning compact representations of entities and relations within a knowledge graph, facilitating efficient reasoning and knowledge discovery. While existing methods typically focus either on training KGE models solely based on graph structure or fine-tuning pre-trained language models with classification data in KG, KG-FIT leverages LLM-guided refinement to construct a semantically coherent hierarchical structure of entity clusters. By incorporating this hierarchical knowledge along with textual information during the fine-tuning process, KG-FIT effectively captures both global semantics from the LLM and local semantics from the KG. Extensive experiments on the benchmark datasets FB15K-237, YAGO3-10, and PrimeKG demonstrate the superiority of KG-FIT over state-of-the-art pre-trained language model-based methods, achieving improvements of 14.4\%, 13.5\%, and 11.9\% in the Hits@10 metric for the link prediction task, respectively. Furthermore, KG-FIT yields substantial performance gains of 12.6\%, 6.7\%, and 17.7\% compared to the structure-based base models upon which it is built. These results highlight the effectiveness of KG-FIT in incorporating open-world knowledge from LLMs to significantly enhance the expressiveness and informativeness of KG embeddings.

KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge
Pengcheng Jiang Lang Cao Cao Xiaoi Parminder Bhatia
Jimeng Sun Jiawei Han
University of Illinois Urbana- Crhaana-hhampaign {pj20, langcao2, jimeng, hanj) edu
1GE HealthCare
danicaxiao@gmail.c com
Abstract
Knowledge Graph Embedding (KGE) techniques are crucial in learning compact representations of entities and relations within a knowledge graph, facilitating efficient reasoning and knowledge discovery. While existing methods typically focus either on training KGE models solely based graph structure or fine-tuning pre-trained language models with classification data in KG, KG-FIT leverages LLM-guided refinement to construct a semantically coherent hierarchical struc- ture entity clusters. By incorporating this hierarchical knowledge along with textual information during the fine-tuming process, KG-FIT effectively captures both global semantics from the LLM and local semanties from the KG. Extensive experiments the benchmark datasets FB15K-237 YAGO3-10 10, and PrimeKG demonstrate the superiority of KG-FIT over state-of-the-an pre-trained language model-based methods, achieving improvements of 14.4%, 13.5%, and 11.9% in the Hits 10 metric for the link prediction task, respectively. Furthermore, KG-FIT yields substantial performance gains of 12.6%, 6.7%, and 17.7% compared the structure-based base models upon which it is built. These results highlight the effectiveness of KG-FIT in incorporating open-world knowledge from LLMs significantly enhance the expressiveness and informativeness of KG embeddings.
Ang. HhsBB10 FB16K-25 297
LLM
Previous: Fine-tune
Classificatio Data
ciuaeit by sched, Sestiss
MBanMaac humbe Coogler
Open-Source LLMs/PLMs
Knowledge Sharing
KGs
FLM-bated KGS baaed
KG-FIT
KG-FIT: Fine-tune KG with LLM
Mitsig Primaito
Entity Knowledge
1, L'niity Descristion ot ("ltccia" "iacchutee lotoral apcrie Polkeer mode fundios..."
KG
Any LLMs
Entlty Hermihhy
KGs
Figure I: "Fine- tune LLM with KG" "Fine-tune KG with LLM".
Introduction
Knowledge graph (KG) is a powerful tool for representing and storing structured knowledge, with ap- plications spanning wide range of domains, such question answering [1, 2, 3, 4], recommendation systems [5, 6, 7], drug discovery [8, 9, 10], and clinical prediction [11, 12, 13]. Constituted by entities and relations, KGs form graph structure where nodes denote entities and edges represent relations among them. To facilitate efficient reasoning and knowledge discovery, knowledge graph embedding
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(KGE) methods [14, 15, 16, 17, 18, 19, 20, 21] have emerged, aiming to derive low- low-dimensional vector representations of entities and relations while preserving the graph's structural integrity.
While current KGE methods have shown success, many are limited to the graph structure alone, neglecting the wealth of open-world knowledge surrounding entities not explicitly depicted in the KG, which is manually created in most cases. This oversight inhibits their capacity to grasp the complete semantics of entities and relations, consequently resulting in suboptimal performance across downstream tasks. For instance, a KG might contain entities such as "Albert Einstein" and "Theory of Relativity", along with relation connecting them. However, the KG may lack the rich context and background information about Einstein's 's life, his other scientific contributions, and the broader impact of his work. In contrast, pre-trained language models (PLMs) and LLMs, having been trained on extensive literature, can provide a more comprehensive understanding of Einstein and his legacy beyond the limited scope of the KG. While recent studies have explored fine-tuning PLMs with KG triples [22, 23, 24, 25, 26, 27, 28, 29], this approach is subject to several limitations. Firstly, the training and inference processes are computationally expensive due to the large number of parameters in PLMs, making it challenging to extend to more knowledgeable LLMs. Secondly, the fine-tuned PLMS heavily rely on the restricted knowledge captured by the KG embeddings, limiting their ability to fully leverage the extensive knowledge contained within the language models themselves. As a result, these approaches may not adequately capitalize on the potential of LLMs to enhance KG representations, as illustrated in Fig. 1. Lastly, small-scale PLMs (c.g.. BERT) contain outdated and limited knowledge compared to modern LLMs, requiring re- re-training to incorporate new information, which hinders their ability to keep pace with the rapidly evolving nature of today's 's language models.
To address the limitations of current approaches, we propose KG-FIT (Knowledge Graph FIne- Tuning), a novel framework that directly incorporates the rich knowledge from LLMs into KG embeddings without the need for fine-tuning the LMs themselves. The term "fine-tuning" is used because the initial entity embeddings are from pre-trained LLMs, initially capturing global semantics.
KG-FIT employs a two-stage approach: (1) generating entity descriptions from the LLM and per- forming ALM-guides hierarchy construction to build semantically coherent hierarchical structure of entities, and (2) fine-tuning the KG embeddings by integrating knowledge from both the hierarchical structure and textual embeddings, effectively merging the open- open-world knowledge captured by the LLM into the KG embeddings. This results in enriched representations that integrate both global knowledge from L.MS and local knowledge from KGs.
The main contributions of KG-FIT outlined as follows:
(1) We introduce a method for automatically constructing a semantically coherent entity hierarchy using agglomerative clustering and LLM-guided refinement.
(2) We propose a fine-tuning lne-ttunng approach that integrates knowledge from the hicrarchical structure and pre- pre-trained text embeddings of entities, enhancing KG embeddings by incorporating open- world knowledge captured by the LLM.
(3) Through extensive empirical study on benchmark datasets, we demonstrate significant im- provements in link prediction accuracy over state-of-the-arr baselines.
2 Related Work
Structure-Based Knowledge Graph Embedding. Knowledge Graph Embedding methods that rely solely on graph structure aim to learn low-dimensional vector representations of entities and relations while preserving the graph's 's structural properties. TransE [14] models relations as translations in the embedding space. DistMult [15] is a simpler model that uses a bilinear formulation for link prediction. ComplEx [16] extends TransE to the complex domain, enabling the modeling of asymmetric relations. ConvE [17] employs a convolutional neural network to model interactions between entities and relations. TuckER [18] utilizes a Tucker decomposition to learn embeddings for entities and relations jointly. RotatE [19] represents relations as rotations in a complex space, which can capture various relation patterns, and HAKE [21] models entities and relations in an implicit hierarchical and polar coordinate system. These structure- based methods have proven effective in various tasks [30] but do not leverage the rich entity information available outside the KG itself.
PLM-Based Knowledge Graph Embedding. Recent studies have explored integrating pre-trained language models (PLMs) with knowledge graph embeddings to leverage the semantic information

---

Step Entity Embedding Initialization
Step 3. LLM-Guided Hierarchy Refinement
Step 4. Global Knowledge Guided Local KG Fine-Tuning
Input: Entty
Entiyy Block Rea
Emily 4pp
Refined Harsrrity
Mirimiatiion
Madmitzriion
Retational Trunsition
Bi Bl desolibe
Bingrage
Description
Codktai
Vinw
input
procoss
Dmb Berrinat
cubtaed ia with kithoer cofleieo
procsss
Coffee
bosell
Errb Patrerall
Parkno
Output
Cochiek
Coffee based
F fricoeeeiity used to make
LUM Buggestioro
Tea Ra--aned
PARENT HERGE
Output:
Boverago Pi
Text Embedding
INCLUDE
Inguu
Step Seed Hierarchy Construction
SPLIT
Men-Aicohelic
Bevergge
Wacarchical Clastering
Sithountin
HERDE
Black
Output
suggtes
Coffee
Llmer
Input
Seed Hiararchy
Sub-tree barstion
Outpun Embeedanng
Figure 2: Overview of KG-FIT. Input and Output are highlighted at each step. Step Obtain text embeddings for all entities in the KG, achieved by merging word embeddings with description embeddings retrieved from LLMs. Step 2: Hierarchical clustering is applied iteratively to all entity embeddings over various distance thresholds, monitored by Silhouette scorer to identify optimal clusters, thus constructing seed hierarchy where each leaf node represents a cluster of semantically similar entities. Step 3: Leveraging LLM guidance, the seed hierarchy is iteratively refined bottom-up through a series of suggested actions, aiming for a more accurate organization of KG entities with LLM''s s knowledge. Step 4: Use the refined hicrarchy along with KG triples and the initial entity embeddings to fine-tune the embeddings under series of distance constraints.
captured by PLMs. KG-BERT [22], PKGC [28], TagReal [29], and KG-LLM [31] train PLMs/LLMs with a full set of classification data and prompts. However, these approaches are computationally expensive due to the need to iterate over all possible positive/negative triples. LMKE [26] and SimKGC [27] adopt contrastive learning frameworks to tackle issues like expensive negative sampling and enable efficient learning for text-based KGC. KG- KG-S2S [24] and KGT5 [25] employ sequence- oo-sequence models to generate missing entities or relations in the KG. StAR [23] and CSProm-KC KG [32] fuse embeddings from graph-based models and PLMs. However, they are limited to small-scale PL.Ms and do not leverage the hierarchical and clustering information reflecting the LLM'' knowledge of entities. Fully LLM prompting based methods [33, 34] are costly and not scalable. In contrast, our proposed KG-FIT approach can be applied to any LLM, incorporating its knowledge through semantically coherent hierarchical structure of entities. This enables efficient exploitation of the extensive knowledge within LLMs, while maintaining the efficiency of structure-based methods.
3 KG-FIT Framework
We present KG-FIT (as shown in Fig. 2), framework for fine-tuning KG embeddings leveraging external hierarchical structures and textual information based on open knowledge. This framework comprises two primary components: (1) LLM-Guided Hierarchy Construction: This phase establishes a semantically coherent hierarchical structure of entities, initially constructing a seed hierarchy and then refining it using LLM-guided techniques, and (2) Knowledge Graph Fine-Tuning: This stage enhances the KG embeddings by integrating the constructed hierarchical structure, textual embeddings, and multiple constraints. The two stages combined open- open-world knowledge, leading to more comprehensive and contextually rich representations. Below we present more tecnical details. table of notations is placed in Appendix L.
enrich KG embeddings with
3.1 LLM-Guided Hierarchy Construction
We initiate the process by constructing a hierarchical structure of entities through agglomerative clustering, subsequently refining it using an LLM to enhance semantic coherence and granularity.
3

---

Step 1: Entity Embedding Initialization is the first step of this process, where we are given a set of entities& & = (ei..... e within a KG, and will enrich their semantic representations by generating descriptions using an L.M. Specifically for each entity e;, we prompt the L.L with template (e.g., Briefly describe [entity] with the format "[entity] (detailed in Appendix E.1)) prompting it to describe the entity from the KG dataset, thereby yielding concise natural language description d. Subsequently, the entity embedding v E Rddim()) and description embedding v. f(d)) E Rddims) are obtained using an embedding model f and concatenated to form the enriched entity representation V₁:
a [description]",
Vi v:].
(1)
Step 2: Seed Hierarchy Construction follows after entity embedding initialization. Here we choose agglomerative hierarchical clustering [35] over flat clustering methods like K-meam [36] for establishing the initial hierarchy. This choice is based on the robust hierarchical information provided by agglomerative clustering, which serves as strong foundation for LLM refinement, Using this hierarchical structure reduces the need for numerous LLM iterations to discern relationships between flat clusters, thereby lowering computational costs and complexity. Agglomerative clustering balances computational efficiency with providing the LLM a meaningful starting point for refinement. The clustering process operates on enriched entity representations V = (Vi...., vi} E |R|| using cosine distance and average linkage. The optimal clustering threshold r² is determined by maximizing the silhouette score [37] S" across range of thresholds [Tmin mmax E [o, 1]:
Toptim arg ( S* .labels,,))
(2)
where labels, are the clustering results at threshold r. This ensures that the resulting clusters are compact and well-separated based on semantic similarity. The constructed hierarchy forms a fully binary tree where each leaf node represents an entity. We use top-down algorithm (detailed in Appendix F.1) to replace the first encountered entity with its cluster based on the optimal threshold Toptim This process eliminates other entity leaves within the same cluster, forming the seed hierarchy Hyyed, where each leaf node is a cluster of entities defined by labels,,pin
Step 3: LLM-Guided Hierarchy Refinement (LHR) is then applied to improve the quality of the knowledge representation. As the seed hierarchy Hseed is binary tree, which may not optimally represent real-world entity knowledge, we further refine it using the LLM. The LLM transforms the seed hierarchy into the LLM-guided refined hierarchy HIHR through actions described below:
i. Cluster Splitting:
For each leaf cluster Coriginal E Clear (Hwaa), the LLM recursively splits into two subclusters using the prompt PSPLIT (Fig. 8 in Appendix E.2):
Csplit = (PL.M(PPPLIITT (Corigina));
Coriginal >> Csplit (Ci, Ck):
Eie where C;= c= eei.e,...,,,fc.. E. Et |
|Csptil , k is the total number of subclusters after recursively splitting Coriginal in binary manner. This procedure iterates until LLM indicates no further splitting or each subcluster has minimal entities, resulting in an intermediate hierarchy H.pli-
ii Bottom-Up Refinement: In the bottom-up refinement phase, the LLM iteratively refines the intermediate hierarchy Hapli produced by the cluster splitting step. The refinement process starts from the leaf level of the hierarchy and progresses upwards, considering each parent-child triple (Px, Pt, Pr), where P. represents the grandparent cluster, and Pt and P, represent the left and right child clusters of P., respectively. Let {C,, C2,, C -..... C(p)) denote the children of Pt, and {CT, C2 C%,..., ) denote the children of P,.
For each parent-child triple, the LLM is prompted with PREFINE P.., P., P.)) (Fig. 9 in Appendix E.2), which provides the names and entities of the grandparent and child clusters. The LLM then suggests a refinement action to update the triple based on its understanding of the relationships between the clusters. The refinement options are:
1. NO UPDATE: The triple remains unchanged, i.e.,
2. PARENT MERGE: All the children of P; and P, are merged into the grandparent cluster P.. resulting in P" {C₁, C22, Cy....,Co ...
Ci.,) C(, CT, C5 Og...., Cir.)). The original child clusters P.
and P. are removed from the hierarchy.
3. LEAF MERGE: P.
{e1, 62,....
.p.......,ppp. epp. 0. = 0. where (21, =1 Pr....

---

4. INCLUDE: One of the child clusters is absorbed into the other, while the grandparent cluster remains unchanged. This can happen in two ways:
P. P {C₁, C2 Cz:.... ....., CiAI p. and P. 0, or
P. = P PP C2. Cj...., CrP, and P/ = 0.
The LLM determines the most appropriate refinement action based on the semantic similarity and hierarchical relationships between the clusters. The refinement process continues bottom-up, iteratively updating the triples until the root of the hierarchy is reached. The resulting refined hierarchy is denoted as HLHR. We place more details of the process in Appendix E.2 and F.2.
3.2 Global Knowledge-Guided Local Knowledge Graph Fine-Tuning
Step 4: KG-FIT fine-tunes the knowledge graph embeddings by incorporating the hierarchical structure. text embeddings, and three main constraints: the hierarchical clustering constraint, text embedding deviation constraint, and link prediction objective.
To integrate the initial text embeddings (v; (dlm(f)) into the model, the entity embedding e, E _n is initialized as linear combination of random embedding ei E R" and the sliced text embedding v = [vil: Bl vfl: 3ll ER.. The relation embeddings, on the other hand, are initialized randomly:
Initialization of Entity and Relation Embeddings:
C; pc, + (1 p)y', Tÿ NO, 1/)
(4)
where P is hyperparameter controlling the ratio between the random embedding and the sliced text embedding. ry E I!m is the embedding of relation j. and ea is a hyperparameter controlling the standard deviation of the normal distribution N. This initialization ensures that the entity embeddings start close to their semantic descriptions but can still adapt to the structural information in the KG during training. The random initialization of relation embeddings allows the model to flexibly capture the structural information and patterns specific to the KG.
Hierarchical Clustering Constraint: The hierarchical constraint integrates the structure and rela- tionships derived from the adaptive agglomerative clustering and LLM-guided refinement process. This optimization enhances the embeddings for hierarchical coherence and distinct semantic clarity. The revised constraint consists of three tailored components:
d(e;, c')
By(d(e, (ot)))j)))))))))))))) p d(e;, - h-1
Laier Σ CLEE
A1d(e, c)
A2
As
(5)
Cluster Cohesion
(C)
J=1
Inter-level Cluster Separation
Hierarchical Distance Maintenance
where: e; and e; represent the entity and its embedding. C' is the cluster that entity belongs to. S., (C) represents the set of neighbor clusters of C where m the number of nearest neighbors (determined by lowest common ancestor (LCA) [38] in the hierarchy). c and c' denote the cluster embeddings of and C", which is computed by averaging all the embedding of entities under them с En,ec EC e; E IR"). Рj and Pj+1 are the embeddings of the parent nodes along the path from the entity (at depth h) the root, indicating successive parent nodes in ascending order. Each parent node is computed by averaging the cluster embeddings under it (p TF' Σc,e E R").. d(.s.) ) is the distance function used to measure distances between embeddings. As higher levels of abstraction encompass a broader range of concepts, and thus a strict maintenance of hierarchical distance may be less critical these levels, we introduce 3, - where 30 is the initial weight for the closest parent, typically a larger value, á is the decay rate, a positive constant that dictates how rapidly the importance decreases. X1. Ag. and À are hyperparameters. In Eq 5, Inter-level Chuster Separation aims to maximize the distance between an entity and neighbor clusters, enhancing the differentiation and reducing potential overlap the embedding space. This separation ensures that entities are distinctly positioned relative to non- member clusters, promoting clearer semantic divisions. Hierarchical Distance Maintenance encourages the distance between an entity and its parent nodes to be proportional to their respective levels in the hierarchy, with larger distances for higher-level parent nodes. This reflects the increasing abstraction and decreasing specificity, aligning the embeddings with the hierarchical structure of the KG. Cluster Cohesion enhances intra-cluster similarity by minimizing the distance between an entity and its own cluster center, ensuring that entities within the same cluster are closely embedded, maintaining the integrity of clusters.
(i.e., c
5

---

Semantic Anchoring Constraint: To preserve the semantic integrity of the embeddings, we introduce the semantic anchoring constraint, which is formulated as:
Cane
Σ d(e,, v/) ogEE
(6)
where E is the set of all entities, C₁ is the fine-tuned embedding of entity e; v is the sliced text embedding of entity c,, and d(.,-) -) is distance function. This constraint is crucial for large clusters, where the diversity of entities may cause the fine-tuned embeddings to drift from their original semantic meanings. This is also important when dealing with sparse KGs, the constraint helps prevent overfitting to the limited structural information available. By acting as a regularization term, mitigates overfitting and enhances the robustness of the embeddings [39].
Score Function- Based Fine- Tonn-TTning::: KG-FIT is a general framework applicable to existing KGE models [14, 15, 16, 17, 18, 19, 20J. These models learn low-dimensional vector representations of entities and relations in a KG, aiming to capture the semantic and structural information within the KG itself. In our work, we perform link prediction to enhance the model's ability to accurately predict relationships between entities within the KG. Its loss is defined as:
Liik
Σ (BLYME))ED
E(rog((> (logo(7 (,
1 [NG]
Σ RYEN,
logo(y fr Sr(6),
where 'D is the set of all triples in the KG, is sigmoid function. f+(:.-) is the scoring function (detailed in Appendix G and K) defined by the chosen KGE model that measures the compatibility between the head entity embedding e; and the tail entity embedding e, given the relation r, NG is the set of negative tail entities sampled for the triple (ei, Ez)), eny is the embedding of the negative tail entity nj, and 7 is a margin hyperparameter. The link prediction-base the scoring function for the true triples (e,, eÿ) while maximizing the margin between the scores of true triples and negative triples (e,, r, n2). This encourages the model assign higher scores to positive (true) triples and lower scores to negative triples, thereby enriching the embeddings with the local semantics in KG.
fine-tuning minimizes
Training Objective: The objective function of KG-FIT integrates three constraints:
= Cnier + Came + Ca
where Ç1+ S2. and Ç3 are hyperparameters that assign weights the constraints.
Note: During tunintnnng, the time complexity per epoch is +((E)+((() -n) where | is the number of entities, [T] is the number of triples, and is the embedding dimension. In contrast, classic PLM-based methods [22, 28, 29, 23] have a time complexity of O( L TiP.M) per epoch during fine-tuning, where L is the average sequence length and IPLM the hidden dimension of the PLM. This is typically much higher than KG-FIT's fine-tuning time complexity, as (7) L >> (|ε) + [T])-
4 Experiment
4.1 Experimental Setup
We describe our experimental setup follows. Datasets. We consider datasets that encompass various domains and sizes, ensuring comprehen- sive evaluation of the proposed model. Specif- ically, we consider three datasets: (1) FB15K- 237 [40] (CC BY 4.0) is subset of Freebase [41], large collaborative knowledge base, fo- cusing on common knowledge; (2) YAGO3-10
Table 2: Datasets statistics. #Ent/#Rell: number of entities/relations. #TriinWWalldd#Test number of triples contained the trainingyvaiiddaiionnsstnnn set.
Dataset
#Ent.
#Rel.
#'rain
#Valid
#Test
FB15k-237 YAGO3-10
14,541
237
272.115
17,535
20,466
123.182
37
1.079.440
5,000
5,000
PrimeKGi
10,344
11
100,000
3,000
3,000
[42] is subset of YAGO [43] (CC BY 4.0), which is a large knowledge base derived from multiple
sources including Wikipedia, WordNet, and GeoNames; (3) PrimeKG [44] (CC0 1.0) a biomedical KG that integrates 20 biomedical resources, detailing 17,080 diseases through 4,050,249 relation- ships. Our study focuses on a subset of PrimeKG, extracting 106,000 triples from the whole set, with processing steps outlined in Appendix B. Table 2 shows the statistics of these datasets.

---

Table 1: Link Prediction Performance Comparison. Results averaged values (of runs for FB 15K- 237/PrimeKG and of three runs for YAGO3-10) of head/tail entity predictions. Top-3 results for each metric are highlighted. "*" indicates the results taken from method's original paper. KG-FIT consistently outperforms both PLM- based models and structure- based base models across all datasets and metrics, demonstrating its effectiveness in incorporating open-world knowledge from M.M for enhancing KG embeddings.
FB1SK-237
YAGO3-10
PrimeKG
PLM-based Embodding Methods
Model
PLM
MR
MRR
H@1
H655
1@:10
MR
MRR H&
H@S
H6:10|
| MR
MRR
H&1
H
H
10
KG-BERT [22]*
.245
.158
.420
SLAR [23]*
RoBERTa
117
.296
.205
.482
PKGC [28]
RoBERTa
184
.342
.236
.411
.525
1225
.501
426
596
660
219
485
.391
565
625
C-LMKE [26]*
BERT
.306
218
.484
KGTS [25]
.276
.210
.414
.368
-528
KG-S2S [24]
T5
.336
257
.98
SimKGC [27]
BERT
.336
244
.511
168
.527
.524
679
742
CSProm- KG [32]
.269
1145
624
157
LLM Emb. (zcro shot)
TE:3-S
2044
.023
.002
035
.66
22741
.009
.000
.06
021
5581
.000
.000
.000
000
TB-3-L
1818
.000
.014
048
085
18780
.015
0000
.099
032
4297
.001
.000
.000
000
Structure-based Embldding Methods
Model
Frame
7
MR
MRR
H81
HO65
1o
MR
MRR
H8s
H685
He010 |
MR
MRR
H68 1
H65
H1610
Base [14]
233
.287
.192
.389
.478
1250
-500
.398
626
6655
182
8848
.000
044
.124
Trans
KG-FI7
LHR
Seed
142
122
.362
.345
.242
.264
.457
.478
.547
.568
529
952
.544
.520
.429
.63
6,38
650
.705
.700
69
80
.334
.298
.000
.000
.315
.342
.516
.536
Base [15)
283
.260
.163
349
.43
5501
A511
.365
.53
.615
174
.577
.455
.699
.782
DisMult
KG-FIT
Soud
184
.316
198
.115
.512
963
.186
.133
.591
.73
107
.589
.155
.715
.799
LHR
.226
.433
.529
.441
.666
78
Bose [16]
347
.252
161
.344
.439
6681
A63
.344
560
612
202
.614
.523
.728
.789
ComplEx
XG-FIT
LHR
Sood
.344
.325
.247
.223
.588
.436
.551
.523
842
.544
.600
.222
646
6613
669
94
.651
6:38
767
Base [17)
341
.312
.224
.401
.508
1105
.529
.511
.19
.63
144
.516
.466
6445
.760
ConvE
KG-F17
LHR
Seed
177
.318
.318
.241
.257
.415
.411
.525
912
885
.541
.461
628
.647
.695
9%
72
.627
.648
.547
.767
.757
.824
Base [18]
363
.320
.230
.417
.505
1110
.529
.454
.633
.690
171
.543
.442
.663
.737
TucKER
KG-FI7
Seed
LHR
175
144
.330
.319
.255
.241
.433
.148
.543
.521
874
838
.515
.538
.458
.466
.654
.61
.703
.708
62
'77
.640
.611
.542
.550
.779
.770
.820
.805
Bass[19]
.310
.205
.399
.502
.573
118
.191
.593
PRotulF
KG-FIT
Seed
LHR
119
160
.371
.355
.277
.257
.483
461
.572
.558
910
829
.550
.525
436
A64
662
648
.710
.033
75
69
.649
.638
.574
.538
.779
.745
.833
.809
Base [19]
190
.333
.241
.428
.528
1620
.955
AU2
550
670
57
.539
.447
696
.727
RotaiE
KG-FI7
LHR
Seed
141
120
.369
.354
.274
.261
.488
.464
.555
.570
744
790
.563
.529
.440
.475
.688
643
.722
7088
46
34
.622
.645
.532
.517
.758
.740
.877
.055
Base (20)
184
.344
.247
.35
.538
1220
530
A311
634
.681
95
.598
.515
.708
.760
HAKE
KC-FTT
Seed
162
.362
.358
.275
268
.485
.170
.572
.563
851
810
.511
.155
.474
.662
.617
.718
.703
82
.638
.510
.747
.808
Metrics. Following previous works, we use Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hits@N (H@N) N) to evaluate link prediction. MR measures the average rank of true entities, lower the better. MRR averages the reciprocal ranks of true entities, providing a normalized measure less sensitive to outliers. Hits N measures the proportion of true entities in the top N predictions.
Baselines. To benchmark the performance of our proposed model, we compared it against the state- of-the-art PLM-based methods including KG-BERT [22], StAR [23], PKGC [28], C-LMKE [26], KGTS [25]. KG-S2S [24], SimK [27], and CSProm-KG [32], and structure- based methods
including TransE [14], DistMult [15], ComplEx [16], ConvE [17), TuckER [18], pRotatE [19],
RotatE [19], and HAKE [20].
For most PLM-based models, due to their high training cost, we use the results reported in their respective papers.. We reproduce PKGC [28] for all three datasets, SimKGC [27] and CSProm-KG [32] for PrimeKG. In addition, we evaluate the capabilities of LLM beaebedddinn----malla-rr
Experimental Strategy:
embeddings (TE-3-S/L:
for zero-shot link prediction by ranking the cosine similarity between (c: + r) and Cy. For structure- based KGE models, we assess and present their best performance using optimal settings (shown in Table 12). For KG- FITF we provide a detailed hyperparameter study in Table 13. We use OpenAl's GPT-40 as the LLM for entity description generation and for LLM-guided hierarchy refinement. Beclemmbedding---laayg embedding initialization, which preserves semantics with flexible embedding slicing [45]. We set Tmin = 0.15 and Tmax 0.85 for seed hicrarchy construction. The values of Toptim for FB 15K-2377
is used for entity
7

---

YAGO3-10 10, and PrimeKG are 0.52, 0.49, and 0.33, respectively. The statistics of the seed and LHR hierarchies are placed in Appendix E3. For LCA, we designate the root as the grandparent (i.e.. two levels above) source cluster node and set m 5. We set p = 0.5, 80 = 1.2, /b 0.4, and use cosine distance for fine-tuning. Filtered setting [14, 19] is applied for link prediction evaluation. Hyperparameter studies and computational cost are detailed in Appendix I and H, respectively.
4.2 Results
We conduct experiments to evaluate the performance on link prediction. The results in Table 1 are averaged values from multiple runs with random seeds: ten runs for FB15K-237 and PrimeKG, and three runs for YAGO3-10. These averages reflect the performance of head/tail entity predictions.
Main Results. Table 1 shows that our KG-FIT framework con- sistently outperforms state-of-theearr PLM-based and traditional structure-based models across all datasets and metrics. This highlights KG-FIT' effectiveness in leveraging LLMs to en- hance KG embeddings. Specifically, KG-FITHAKE surpasses CSProm- KG by 6.3% and HAKE by 6.1% on FB15K-25 in outperforms CS-PromKG by 7.0% and
Valid MRR Training Steps on FB119-237
0.30
Volidation MBR
0.25
0.15
0.10
Hits@ 10; KG-FIT TuckER by 4.6% on YAGO3- 10; KG-FIT exceeds PKGC by 11.0% and ComplEx by 5.8% on PrimeKG. Additionally, with LLM-guided hierarchy refinement (LHR), XG-FIT achieves per- formance gains of 12.6% %. 6.7%, and 17.8% compared to the base models, and 3.0%, 1.9%, and 2.2% compared to KG-FIT with seed hierarchy, on FB15K-237, YAGO3-10, and PrimeKG,
DistMult (Base)
ComplEx (Base)
0.0
KG-FIT-GkIMMU
aC-FIT-ComplE
Valid Hits@110 ve Training Steps on PrimekG
Vaiidation HaNgno
04-
TransE: (Base]
respectively. All these findings highlight the effectiveness of KG-FIT for significantly improving the quality of structure-basse KG embeddings.
0.3
KG.FIT- Tran-TTwE--1
Trans- 2045--255 (Bass)
62
KC-FIT FIT- TaatT--nF-0
0.1
Figure 3 further illustrates the robustness of KG-FIT, showing superior validation performance across training steps compared to the corresponding base models, indicating its ability to fix both overfitting and underfitting issues of some structure-based KG embedding models.
30x 120K 160k 180K 200k
Training Steps
Figure 3: KG-FIT can mitigate overfitting (upper) and underfitting (lower) of structure-based models.
Effect of Constraints. We conduct an ablation study to evaluate the proposed constraints in Eq. 5 and 6, with results summarized in Table This analysis underscores the importance of each constraint: (1) Hierarchical Distance Maintenance is crucial for both datasets. Its removal significantly degrades performance across all metrics, highlighting the necessity of preserving the hierarchical structure in the embedding space. (2) Semantic Anchoring proves more critical for the denser YAGO3-10 graph, where each cluster contains more entities, making harder to distinguish between them based solely on cluster cohesion. The sparser FB 15K-237 dataset is less impacted by the absence of this constraint. Similar to the semantics anchoring, the removal of (3) Inter-level Cluster Separation significantly affects the denser YAGO3-10 10 more than FB15K-237 Without this constraint, entities in YAGO3-10 may not be well-separated from other clusters, whereas FB15K-237 is less influenced. Interestingly, removing (4) Cluster Cohesion has larger impact the sparser FB15K-237 than on YAGO3- 10. This difference suggests that sparse graphs rely more on the prior information provided by entity clusters, while denser graphs can learn this information more effectively from their abundant data.
Effect of Knowledge Sources. We explore the impact the qual- ity of L.MM-rffined hierarchies and pre-trained text embeddings on final performance, as illustrated in Figures 4 and 5, The results indicate that hierarchies constructed and text embeddings retrieved from more advanced LLMs consistently lead to improved perfor- mance. This finding underscores KG-FIT's capacity to leverage and evolve with ongoing advancements in LLMs, effectively uti- lizing the increasingly comprehensive entity knowledge captured by these models.
Table 3: Model efficiency on PrimeKG. T/Ep and Inf denote train- ing time/epoch and inference time. KG-FIT outperforms all the PLM- based models.
Method KG-BERT PRGC TagReal
TVPp
RoBERTs
170m
2900im
RoBERTs
190m
50m
LUKE
190m
Sihn
SIAR
RoBERTs
125m
KG-S2S
T5
30m
110m
Efficiency Evaluation. We evaluate the efficiency performance in Table 3. While pure structure-base models are the fastest, our model significantly outperforms all PLM-based models in both
SimKGC
BERT
20m
0.5m
CSPcom KG
BERT
15m
0.2m
RG-FI7 (ours)
LLM
1.2m
0.1m
Structure bused
8

---

Table 4: Ablation study for the proposed constraints. SA, HDM. ICS, CC denote Semantic Anchoring. Hierarchical Distance Maintenance, Inter-level Cluster Separation. and Cluster Cohesion, respectively. We use TransE and HAKE as the base models for KG-FIT on FBI5K-237 and YAGO3- 10, respectively.
FBISK-237 (KG-FITmma))
YAGO3-10 (KG-FITHAXE)
HDM
SA
ICS
CC
H&@5
Ho#10
MRR
H@@
H&B5
.362
264
.478
.58
.688
.444
662
.718
/
V
345
(
154
(42(
.558qumo)
.467 (1
664 (54
.799
(1
_.3
.54((
(52 mnn
.643 (00)
.695.
X
/
.343 q4.99.00
.244
449
449,
.338
(.00)
.544 (4m0
.453 (aar)
.69 (Lar7)
X
1332
.239 .239 (LiM5)
.377 m03)
[_99
.655
656 10)
X
(192
(389
1530
.43
.634
.61 (1097)
TransE MRR
TransE H@P
TransE H@P10
pRotalE: MRR
PRotalE: H@P
@RotatE H@P
38
58-
38
50
35
26-
56-
35
26-
56-
34
34
30
50
90
50
None
None G4
None Seed 52.5 G4
Nore Seed G2. Nore 62.5
None-Sed 50,5 G4
Figure 4: KG-FIT on FB15K-20 237 with different hierarchy types. None indicates no hierarchical information input. Seed denotes the seed hierarchy. G3.5/G4 denotes the LHRR hierarchy constructed by GPT-3.5/4o. I.RR
hicrarchies outperform the seed hierarchy, with more advanced I.I.Ms constructing higher-quality hicrarchies.
TransE MRR
TransE H@P1
TransE -H100
pRotaiE MRR
pRoteiEE - H@1
PRotsEE-HH11
40
50
50
38-
34-
54-
94
24
54
32
52-
32
22
52
BT RBT adt2
ada2 led
BT RBT ada2 te3
BT RBT 2002
RBT ado2 le
BT RAT ada led
Figure 5: KG-FIT on FB15K 237 with different text embedding. BT, RBT, ada2, and are BERT, RoBERTa, text-embett eerteembedding--e sieedinng-d------ Ada-002, and text- -ite-emmeeegn--eee 3-large, respectively, Seed hierarchy is used for all settings. It is observed that pre-trained text embeddings from LLMs are substantially better than those from small PL.Ms.
training and inference speed, consistent with previous analysis. It achieves 12 times the training speed of CSProm-KG KGG, the fastest PLMM-based method. Moreover, KG-FIT can integrate knowledge from any LLMs, unlike previous methods that are limited to small-scale PLMs, underscoring its superiority.
Visualization. Figure 6 demonstrates the effectiveness of KG-FIT in capturing both global and local semantics. The embeddings generated by KG-FIT successfully preserve the global semantics at both intra- and inter-levels. Additionally, KG-FIT excels in representing local semantics compared to the original HAKE model.
Conclusion
In this paper, introduced KG-FIT, novel framework for enhancing knowledge graph (KG) embeddings by leveraging the wealth of open-world knowledge captured by large language models (LLMs). KG-FIT seamlessly integrates LLM--driveed entity knowledge into the KG embedding process through a two-stage approach: LLM-guided hierarchy construction and global knowledge- guided local KG fine-tuning. By constructing a semantically coherent hierarchical structure of entities and incorporating this bierarchical knowledge along with textual information during fine-tuning, KG-FIT effectively captures both global semantics from the LLM and local semantics from the KG. Extensive experiments on benchmark datasets demonstrate the superiority of KG-FIT over state-of-the-art methods, highlighting its effectiveness integrating open-world knowledge from LLMs to significantly enhance the expressiveness and informativeness of KG embeddings. A key advantage of KG-FIT its flexibility incorporate knowledge from any LLM, enabling it to evolve and improve with ongoing advancements in language models. This positions KG-FIT as powerful and afture-proof framework for knowledge-infuee learning on graphs. Moreover, the enriched KG embeddings produced by KG-FIT have the potential to boost performance on a wide array of downstream tasks, such as question answering, recommendation systems, and drug discovery, among others. Our code and data are available at https: : /gi thub. com/ catn/pat-]. j/KG---BKKKKR FIT.

---

Mansdro:
Cistriiia
Merwvine
Canttrr
Contitin
CAA
Exetional
Exerfond
CAA
Exerdoral
Inftal Text Embeddiing
HAKE Embbddding
KG-FITrue''s Entedding
260
211
HV Drugo Purte Amaiaga
Antecegulent Druga
Homochill Draga
Atter Disodder
Contral Vapcdlrr Dbonders
Caro Abrormatiio
Alyliciig Agers
Targeted Thenay TKa
Reip raay Inu vica
Chaoric Lang Disesca
Crest Deberitics
Figure 6: Visualization of Entity Embedding (left to right: initial text embedding, HAKE embedding, and KG-FITHAKK embedding). Upper (local): Embeddings (dim=2048) of <Maraviroc, drug effect, CAA (Coronary artery atherosclerosis)> and <Cladribine, drug effect, Exertional dyspnea>, two parent-child triples selected from PrimeKG, polar coordinate system. In the polar coordinate system, the normalized entity embedding é is split to 61 = el: ? and e₂ = / + where n the hidden dimension, which serves as values on the x-axis and yaxis, respectively, which is consistent with Zhang ct al. [20]'s visualization strategy. Lower (global): t-SNE plots of different embeddings of sampled entities, with colors indicating clusters (e.g.. Maraviroc belongs the HIV Drugs cluster). Triangles indicate the positions of Maraviroc, CAA, Cladribine, and Exertional dyspnea. Observations: While the initial text embeddings capture global semantics, they to delineate local parent-child relationships within the KG, as seen in the intermingled polar plots. In contrast, HAKE shows more distinct grouping by modulus on the polar plots, capturing hierarchical local semantics, but fails to adequately capture global semantics. Our KG-FIT, notably, incorporates prior information from LLMs and is fine-tuned the KG, maintains global semantics from pre-trained text emheddings while hetter capturing local KG semantics, demonstrating its superior representational power across local and global scales.
6 Limitations
Although KG-FIT outperforms state-of-the-art PLM-based models on the FB15K-237, YAGO3- 10, and PrimeKG datasets, it does not outperform pure PLM-based methods lexical dataset WN18RR, as shown Table 5. This limitation is discussed with details in Appendix C. As a future work, will explore the integration of contrastive learning into the KG-FIT framework to enhance its capability to capture semantic relationships more effectively.
Moreover, KG-FIT's performance is influenced by the quality of the constructed hierarchy, particularly the seed hierarchy. To address this, we propose an automatic selection of the optimal binary tree based the silhouette score. However, if the initial clustering is suboptimal, may result in lower-quality hierarchy that affects KG-FIT's performance. Additionally, the bottom- up refinement process in proposed LHR approach updates each parent-child triple with only single operation (within four), which prioritizes efficiency and simplicity over performance. In future work, we plan explore cost-efficient methods for refining the hierarchy that integrate multiple operations for each triple update, striking a better balance between efficiency and performance.
7
Acknowledgement
The research was supported in part by US DARPA INCAS Program No. HR001 1-20--0--1 and BRIES Program No. HR0011 1-24-3-0325, National Science Foundation IIS- 19-511,,,,,, the Molecule Maker Lab Institute: An AI Research Institutes program supported by NSF under Award No. 2019897, and the Institute for Geospatial Understanding through an Integrative Discovery Environ- ment (I-GUIDE) by NSF under Award No. 21 2118329. Any opinions, findings, and conclusions or recommendations expressed herein are those of the authors and do not necessarily represent the views, either expressed implied, of DARPA or the U.S. Government.
10

---

References
[1] Xiao Huang, Jingyuan Zhang, Dingcheng Li, and Ping Li. Knowledge graph embedding based question answering. In Proceedings of the twelfth ACM international conference on web search and data mining. pages 105-113, 2019.
[2] Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, and Jure Leskovec. Qa-gnn: Reasoning with language models and knowledge graphs for question answering. arXiv preprint arXiv:2104.06378 2021.
[3] Weiguo Zheng, Hong Cheng, Lei Zou, Jeffrey Xu Yu, and Kangfei Zhao. Natural language question/answerrng:: Let users talk with the knowledge graph. In Proceedings of the 2017 ACM on Conference Information and Knowledge Management, pages 217-226, 2017.
[4] Michihiro Yasunaga, Antoine Bossclut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy Liang, and Jure Leskovee. Deep bidirectional Advances in Neural Information Processing Systems, 1-37339-373222 2022.
graph pretraining.
[5] Xiao Wang, Houye Ji, Chuan Shi, Bai Wang, Yanfang Ye, Peng Cui, and Philip S Yu. Het- erogeneous graph attention network. In The world wide web conference, pages 2022-2032, 2019.
[6] Hongwei Wang. Miao Zhao, Xing Xie, Wenjie Li, and Minyi Guo. Knowledge graph convolu- tional networks for recommender systems. In The world wide web conference, pages 3307-3313, 2019.
[7] Janneth Chicaiza and Priscila Valdiviezo-Diaz. A comprehensive survey of knowledge graph- based recommender systems: Technologies, development, and contributions. Information. 12(6):232, 2021.
[8] Sameh K Mohamed, Vít Nováček, and Aayah Nounu. Discovering protein drug targets using knowledge graph embeddings. Bioinformatics, 36(2):603-611 08 2019.
[9] Pengcheng Jiang, Cao Xiao, Tianfan Fu, and Jimeng Sun. Bi-level contrastive learning for knowledge-enhanced molecule representations, 2024.
[10] Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67, 2023.
[11] Junyi Gao, Chaoqi Yang, Joerg Heintz, Scott Barrows, Elise Albers, Mary Stapel, Sara Warfield, Adam Cross, and Jimeng Sun. Medml: fusing medical knowledge and machine learning models for early pediatric covid-19 hospitalization and severity prediction. Iscience, 25(9), 2022.
[12] Ricardo MS Carvalho, Daniela Oliveira, and Catia Pesquita. Knowledge graph embeddings for icu readmission prediction. BMC Medical Informatics and Decision Making, 23(1): 2023.
[13] Pengeheng Jiang, Cao Xiao, Adam Richard Cross, and Jimeng Sun. Grapheare: Enhaneing healthcare predictions with personalized knowledge graphs. In The Twelfth International Conference on Learning Representations, 2024.
[14] Antoine Bordes, Nicolas Usunier, Alberto Garcia Durran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. In C.J. Burges, L. Bottou, M. Welling, Z. Ghahramani, and K.Q. Weinberger, editors, Advances Neural Information Processing Systems, volume 26. Curran Associates, Inc., 2013.
[15] Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575. 2014.
[16] Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference machine learning, pages 207 10800 PMLR, 2016.
[17] Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[18] Ivana Balaževié, Carl Allen, and Timothy Hospedales. Tucker: Tensor factorization for knowledge graph completion. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-JJCNLP), pages 5185-5194, 2019.

---

[19] Zhiqing Sun, Zhi-Hong Deng, Jian- Yun Nie, and Jian Tang. Rotate: Knowledge graph em- bedding by relational rotation in complex space, In International Conference on Learning Representations, 2018.
[20] Zhanqiu Zhang, Jianyu Cai, Yongdong Zhang, and Jie Wang. Learning hierarchy-aware knowledge graph embeddings for link prediction. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 3065-3072, 2020.
[21] Yushi Bai, Zhitao Ying, Hongyu Ren, and Jure Leskovee. Modeling heterogeneous hierarchies with relation speatifo hyperbolic cones. Advances in Neural Information Processing Systems, 14:12316-123211 2021.
[22] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint 199.02390 2019.
[23] Bo Wang, Tao Shen, Guodong Long, Tianyi Zhou, Ying Wang, and Yi Chang. Structure- augmented text representation learning for efficient knowledge graph completion. In Proceed- ings of the Web Conference 2021, pages 1737-1748, 2021.
[24] Chen Chen, Yufei Wang, Bing Li, and Kwok-Yar Lam. Knowledge is flat: A seq2seq generative framework for various knowledge graph completion. In Proceedings of the 29th International Conference on Computational Linguistics, pages 4005-4017, 2022.
[25] Apoory Saxena, Adrian Kochsiek, and Rainer Gemulla. Sequence-to- pennence-to-seqeenee knowledge graph completion and question answering. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume Long Papers), pages 2814-2828, 2022.
[26] Xintao Wang, Qianyu He, Jiaqing Liang, and Yanghua Xiao. Language models as knowledge embeddings. arXiv preprint TrX1:22206.1261,, 2022.
[27] Liang Wang, Wei Zhao, Zhuoyu Wei, and Jingming Liu. Simkge: Simple contrastive knowledge graph completion with pre-trained language models. In Proceedings of the 60th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 4281 12944 2022.
[28] Xin Lv, Yankai Lin, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Peng Li, and Jie Zhou. Do pre-trained models benefit knowledge graph completion? reliable evaluation and a reasonable approach. Association for Computational Linguistics, 2022.
[29] Pengcheng Jiang, Shivam Agarwal, Bowen Jin, Xuan Wang, Jimeng Sun, and Jiawei Han. Text augmented open knowledge graph completion via pre-trained language models. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11161-11180 2023.
[30] Shaoxiong Ji, Shirui Pan, Erik Cambria, Pekka Marttinen, and Philip S. Yu. A survey on knowledge graphs: Representation, acquisition, and applications. IEEE Transactions on Neural Networks and Learning Systems, 33(2):494-514, 51,, 2022.
[31] Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprini arXiv:2308. 13916, 2023.
[32] Chen Chen, Yufei Wang, Aixin Sun, Bing Li, and Kwok-Yan Yan Lam. Dipping pims sauce: Bridging structure and text for effective knowledge graph completion via conditional soft prompting. In Findings of the Association for Computational Linguistics: ACL 2023, pages 11489-11503, 2023.
[33] Yanbin Wei, Qiushi Huang. Yu Zhang. and James Kwok. KICGPT: Large language model with knowledge in context for knowledge graph completion. In Houda Bouamor, Juan Pino, and Kalika Bali, editors, Findings of the Association for Computational Linguistics: EMNLP 2023, pages 8667-8683, Singapore, December 2023. Association for Computational Linguistics.
[34] Baolong Bi, Shenghua Liu, Yiwci Wang, Lingrui Mei, and Xueqi Cheng. Lpnl: Scalable link prediction with large language models. arXiv preprint arXiv:2401.1322) 2024.
[35] Daniel Müllerr. Modern hierarchical, agglomerative clustering algorithms. arXiv preprint 1rXiv:1109.2331 2011.
[36] K Krishna and M Narasimha Murty. Genetic k-means algorithm. IEEE Transactions on Systems, Man, and Cybernetics, Part B (Cybernetics), 29(3):433 1999.
[37] Peter J. Rousseeuw. Silhouettes: graphical aid to the interpretation and validation of cluster analysis. Journal of Computational and Applied Mathematics, 20:53-65, 1987.
12

---

[38] Baruch Schieber and Uzi Vishkin. On finding lowest common ancestors: Simplification and parallelization. SIAM Journal on Computing. 17(6):1253-122) 1988.
[39] Jay Pujara, Eriq Augustine, and Lise Getoor. Sparsity and noise: Where knowledge graph embeddings fall short. In Martha Palmer, Rebecca Hwa, and Sebastian Riedel, editors, Pro- ceedings of the 2017 Conference Empirical Methody in Natural Language Processing, pages 1751-1756, Copenhagen, Denmark, September 2017. Association for Computational Linguistics.
[40] Kristina Toutanova and Danqi Chen. Observed versus latent features for knowledge base and text inference. In Proceedings of the 3rd workshop on continuous vector space models and their compositionality, pages 57-66, 2015.
[41] Kurt Bollacker, Colin Evans, Praveen Paritosh, Tim Sturge, and Jamie Taylor. Freebase: collaboratively created graph database for structuring human knowledge. In Proceedings of the 2008 ACM SIGMOD international conference on Management of data, pages 1247-1250, 2008,
[42] Fabian M. Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowledge. In Proceedings of the 16th International Conference on World Wide Web, WWW '07, page 697-706, New York, NY, USA, 2007. Association for Computing Machinery.
[43] Fabian M Suchanek, Gjergji Kasneci, and Gerhard Weikum. Yago: a core of semantic knowl- edge. In Proceedings of the 16th international conference on World Wide Web, pages 697-706, 2007.
[44] Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Nature Scientific Data, 2023.
[45] Aditya Kusupati, Gantavya Bhatt, Aniket Rege, Matthew Wallingford, Aditya Sinha, Vivek Ra- manujan, William foward-Snydee Kaifeng Chen, Sham Kakade, Prateek Jain, al. Matryoshka representation learning. Advances in Neural Information Processing Systems, 35:30233-30249, 2022.
[46] George A Miller. Wordnet: lexical database for english. Communications of the ACM, 38(11):39 41, 1995.
[47] Prannay Khosla, Piotr Teterwak, Chen Wang, Aaron Sarna, Yonglong Tian, Phillip Isola, Aaron Maschinot, Ce Liu, and Dilip Krishnan. Supervised contrastive learning. Advances in neural information processing systems, 33:18661-18677 2020.
[48] Kun Xu, Liwei Wang, Mo Yu, Yansong Feng, Yan Song. Zhiguo Wang, and Dong Yu. Cross- lingual knowledge graph alignment via graph matching neural network. In Proceedings of the 57th Anmual Meeting of the Association for Computational Linguistics, pages 3156-3166 3161, 2019.
[49] Bayu Distiawan Trisedya, Jianzhong Qi, and Rui Zhang. Entity alignment between knowl- edge graphs using attribute embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 297-304, 2019.
[50] Qingheng Zhang, Zequn Sun, Wei Hu, Muhao Chen, Lingbing Guo, and Yuzhong Qu Multi- view knowledge graph embedding for entity alignment. arXiv preprint arXiv: D0Xiv:/906.023330 2019.
[51] Hailin Wang, Ke Qin, Rufai Yusuf Zakari, Guoming Lu, and Jin Yin. Deep neural aebsork-based relation extraction: an overview. Neural Computing and Applications, pages 1-21, 2022.
[52] Zhengyan He, Shujie Liu, Mu Li, Ming Zhou, Longkai Zhang, and Houfeng Wang. Learning entity representation for entity disambiguation. In Proceedings of the 51st Annual Meeting of the Association for Computational Linguistics (Volume 2: Short Papers), pages 30-34, 2013.
13

---

Contents of Appendix
A Ethics and Broader Impacts
15
PrimeKG Dataset Processing Subset Construction
15
C Results on WN18RR Dataset
16
Supplemental Implementation Details
17
D.1 KG-FIT ................................
17
D.2 Baseline Implementation
18
D.2.1
PLM-based Methods
18
D.2.2
Stureture-based Methods
18
E Prompts
19
E.1 Prompt for Entity Description
19
E,2 Prompts for LLM-Guided Hierarchy Refinement
19
F
Details of KG-FIT Hierarchy Construction
22
F.1
Seed Hierarchy Construction
22
F2
LLM-Guided Hierarchy Refinement
23
F3 Statistics of Constructed Hicrarchies
24
F.4 Examples of LLM-Guided Hierarchy Refinement
25
G Score Functions
26
H Computational Cost
27
H.1 Hardware and Software Configuration
27
H.2 Cost of Close-Source LLM APIs
27
1 Hyperparameter Study
27
J Downstream Applications of KG-FIT
29
Interpreting Score Functions in KG-FIT
30
Notation Table
33
14

---

A Ethics and Broader Impacts
KG-FIT is framework for enhancing knowledge graph embeddings by incorporating open knowledge from large language models. As a foundational research effort, KG-FIT itself does not have direct societal impacts. However, the enriched knowledge graph embeddings produced by KG-FIT could potentially be used in various downstream applications, such as question answering, recommendation systems, and drug discovery.
The broader impacts of KG-FIT depend on how the enhanced knowledge graph embeddings are used. On the positive side, KG-FIT could lead to more accurate and comprehensive knowledge representations, enabling better performance in beneficial applications like medical research and personalized recommendations. However, with any powerful technology, there is also potential for misuse. The enriched knowledge could be used to build systems that spread disinformation or make unfair decisions that negatively impact specific groups.
To mitigate potential negative impacts, we encourage responsible deployment of KG-FIT and the enhanced knowledge graph embeddings it produces. This includes carefully monitoring downstream applications for fairness, robustness, and truthfulness. Additionally, when releasing KG-FTT-enhanced knowledge graph embeddings, we recommend providing usage guidelines and deploying safeguards to prevent misuse, such as gated access and safety filters, as appropriate.
As KG-FIT relies on large language models, it may inherit biases present in the language models' training data. Further research is needed to understand and mitigate such biases. We also encourage future work on building knowledge graph embeddings that are more inclusive and less biased.
Overall, while KG-FIT is promising framework for advancing knowledge graph embeddings, it is important to consider its limitations and potential broader impacts. Responsible development and deployment will be key to realizing its benefits while mitigating risks. We are committed to fostering an open dialogue on these issues as KG-FIT and related technologies progress.
B PrimeKG Dataset Processing & Subset Construction
We describe the process of constructing subset of the PrimeKG [44] (version: V2, license: CCO 1.0) dataset. The goal is to create a highly-focused subset while leveraging additional information from the entire knowledge graph to assess KG embedding models' abilities on predicting drug- trigade relationships.
The dataset construction process involves several detailed steps ensure the creation of balanced training, validation, and testing sets from PrimeKG.
Validation/Testing Set Creation: We begin by selecting triples from the original PrimeKG where the type of the head entity is "drug" and the type of the tail entity is "disease." This selection yields a subset containing 42,631 triples, 2,579 entities, and 3 relations (" ("contraindice cation, "indication,
"off-label use"). From this subset, we randomly select 3,000 triples each for the validation and testing sets, ensuring overlap between the two.
Training Set Creation: We first extract the unique entities present in the validation and testing sets. These entities are used to ensure comprehensive coverage in the training set.
For each triple in the validation/testing set, search for triples involving either its head tail entity across the entire PrimeKG dataset, for the training set construction:
Step 1 involves searching for l-hop triples within the specified relations (" ("contraindicatioon, "indication," "off-label use")
Step 2 randomly enriches the training set with triples involving other relations, with a limit of up to 10 triples per entity.
Step 3 involves removing redundant triples and any triples that are symmetric to those in the validation/testing set, enhancing the challenge posed by the dataset. If the training contains fewer than 100,000 triples, we return to Step and continue the process until the desired size is achieved.
hntpp::/daatverrr harvaarr harvard. edu hwtp/dattee...o htt.:aawww.hhaallpppssssssmtddd. 10. 19w1:///0.11999000000000 IXA7BB
15

---

Following our methodology, we construct subset of the PrimeKG dataset that emphasizes drug-disease relations while incorporating broader context from the entire dataset. This subset is then split into training, validation, and testing sets, ensuring proper files are gen- erated for training and evaluating knowledge graph embedding (KGE) models. The dataset contains 10,344 entities with 11 types of relations: disease disease_phenntype_ee phw.ootyee.psstiieee drug_effect, indication, contraindication, disease_disease, disease_protein, disease_ disease_phenotype phenotyppe_negaiivee
exposure, exposure_disease, disease, drug-protein, off-labe use, drug_drug, The testing and validation sets include three specific relations ("contraindication, "indication," "off-label use").
C Results on WN18RR Dataset
Table 5: Link Prediction Results WNISRR. Results averaged values of ten independent runs of head/tail entity predictions. Top-6 results for each metric are highlighted in bold. "e" indicates the results taken from method's original paper.
WNISRR
PLM-based Embedding Methods
Model
PLM
MR
MRR
H@) 1
H@5
H@ 10
KG-BERT [22]*
BERT
97
.216
.041
.524
StAR [23]*
RoBERTa
51
.401
.243
.709
PKGC [28]
RoBERTs
160
.464
.441
.522
.540
C-LMKE [26]*
BERT
.523
.789
KGT5 |25|
T5
5088
.487
.544
KG-S2S [24]*
TS
.544
.531
.61
SimKGC [27]*
BERT
.666
.587
.800
CSProm- KG [32]*
BERT
.575
.522
.678
TE-3-S
1330
.141
.075
OpenAI-Embl (zero-shot)
TE-3-L
1797
.127
.074
.178
.235
Structure-based Embedding Methods
Model
Frame
MR
MRR
H@
H@5
10
Base[19]
2924
.460
.416
.506
.552
pRotatE
Sced
165
.666
.498
.584
.666
KG-FIT
LHR
78
.590
.511
.615
.722
Buse [19]
3365
.476
.429
.523
.572
RotatE:
Seed
144
.554
.501
.571
.712
KG-FIT
LHRR
75
.589
.519
.600
.719
Base [20]
3680
.490
.452
.535
.579
HAKE
Seed LHR
267
.541
.484
.590
.660
KG-FIT
172
.553
.488
.595
.695
Combination
PKGC w/ KG KG-FITTRratt' 's recull (X'
20)
70
.625
.540
.655
.791
KG-FITyemmt w/ SimKGC's Gruph-Bused Re-Ranking
73
.624
.532
.652
.745
In this section, we present our results and findings on the WNI8RR dataset. WN18RR [17] is derived from WordNet [46], comprehensive lexical knowledge graph for the English language. This subset addresses the test leakage problems identified in WN18. The statisties WN18RR are shown in Table 6. Taptim we found for the seed hierarchy construction on WN18RR is 0.44.
Our experiments reveal that although KG-FIT did not surpass the performance of C-LMKE and SimKGC, its integration with PKGC [28], employing KG-FIT as a recall model followed by re- ranking with PKGC, yielded comparable results these leading models.
Several factors contribute to these observations:
(1) WN18RR, being a lexical dataset, benefits significantly from pre-trained language model (PLM) based methods, which assimilate extensive lexical knowledge during fine-tuning. KG-FIT, relying on static knowledge graph embeddings, lacks this capability. However, as shown in Table 5, its combination with PKGC leverages the strengths both models, resulting in markedly improved performance.
16

---

Table 6: Statisties of WNI8RR. #Ent./4Rel: number of entities/rclations, #Train/ #Train/#Validd#estt number of triples contained the set.
Dataset
#Ent.
#Rel.
#Trrin
#Valid
#Test
WNISRR
40,943
11
86,835
3,034
3,134
(2) WN18RR has a very close semantic similarity between the head and tail entities. The dataset is full of lexical relations such as "hypernym", "member__meonym _mmoonmm","".."" "verb_grovr etc. C-LMKE and SimKGC, both utilizing contrastive learning [47] with PLMs, effectively exploit this semantic proximity during training. This ability to discern subtle semantic nuances contributes to their superior performance over other PLM-based approaches.
(3) SimKGC implements a "Graph-base basap Re-ranking" strategy that narrows the candidate pool to the k-hop neighbors of the source entity from the training set. This approach intuitively boosts performance by focusing on more relevant candidates. For fair comparison, our experiments did not adopt this specific setting across all datasets. Table 5, we showcase how this strategy could boost the performance for KG-FIT as well.
As future work, will explore the integration of contrastive learning into the KG-FIT framework to enhance its capability to capture semantic relationships more effectively.
D Supplemental Implementation Details
In this section, we provide more implementation details of both KG-FIT and baseline models, to improve the reproducibility of our work.
D.1 KG-FIT
Pre-computation. In our KG-FIT framework, there is a pre-computation step to avoid overhead during the fine-tuning phase. The data we need to pre-compute includes:
Cluster embeddings c: These are computed by averaging the all initial entity embeddings (v:) within each cluster.
2. Neighbor cluster IDs (Sm, (C) in Eq. 5): These are computed using the lowest common ancestor (LCA) approach, where we set the ancestor as the grandparent (i.e., two levels above the node) and search for at most m = 5 neighbor clusters.
3. Parent node IDs: These represent the node IDs along the path from cluster (leaf node) to the root of the hierarchy.
The pre-computed data is then used efficiently locate the embeddings of clusters (c, e') and parent nodes (p) for the hierarchical clustering constraint during fine-tuning. With this pre-computation, we significantly speed up the training process, the necessary information is readily available and does not need to be calculated on-the-fly.
Distance Function. We employ cosine distance as the distance metric for computing the hierarchical clustering constraint (Eq. 5) and the semantic anchoring constraint (Eq. 6). Cosine distance effectively captures the semantic similarity between embeddings, making it well-suited for maintaining the semantic meaning of the entity embeddings during fine-tuning. Cosine distance is also invariant to the magnitude of the embeddings, allowing the fine-tuned embeddings to adapt their magnitude based on the link prediction objective while preserving their semantic orientation with respect to the frozen reference embeddings. Moreover, cosine distance is independent of the link prediction objective, focusing on preserving the semantic properties of the embeddings.
Alternative distance metrics, such as Euclidean distance or LI norm, were considered but deemed much slower KG-FIT framework, aligning with the descriptions in OpenAl's document on the embedding models?.
Constraint Computation Options. During fine-tuning, each training step involves batch composed of one positive triple and b negative samples, where b is the negative sampling size. We provide two options for computing the constraints along with the link prediction score:
askeeatttl-aa-d--gaaaaaa questions
17

---

1. Full Constraint Computation: This option computes the distances for both the positive triple and the negative triples. For the positive triple, distances are computed for both the source and target entities. For the negative triples, distances are computed only for the source entities. The advantage of this option is that it allows the model converge in fewer steps and generally achieves better performance after fine-tuning. However, the drawback is that the computation for each batch is nearly doubled, as positive and negative triples are separately fed into the model for score computation.
Partial Constraint Computation: This option computes the distances only for the source entities in the negative batch. It is faster than the full constraint computation option and can achieve comparable results based on our observations.
D.2 Baseline Implementation
D.2.1 PLM-based Methods
PKGC. For PKGC on FB15K-237, we use the templates the authors constructed and released 3 for the base pre-trained langauge model. For YAGO3-10 and PrimeKG, we manually created templates for relations. For example, we converted the relation "isLeaderOf" to "[X] leader of [Y]." for YAGO3-I0 and converted the relation "drug effectt "drug [X] has effect [Y]". We choose TuckER as PKGC's backbone KGE recall model with hyperparameter X 100 for its overall great performance across all datasets. This means that we select top 50 results from TuckER and feed the shuffled entities into PKGC for re-ranking. We set batch size as 256 and run on NVIDIA A6000 GPU for both training and testing.
FB15K-237-N, and use RoBERTa-Lann
"drug_effect"
SimKGC. For SimKGC on PrimeKG, for fairness, we do not use the "graph-based re-ranking" strategy introduced in their paper [27), which adds biases the scores of known entities (in the training set) within h-hop of the source entity. The released code* was used for experiments. We set batch size 256 and run on NVIDIA A6000 GPU for both training and testing.
CSProm-KCE KG. For CSProm-KG on PrimeKG, we use ConvE as the backbone graph model, which is the best Hits @N performed model. The other settings paper. We use the code released by the authors to conduct the experiments.
the same as what reported in the
D.2.2 Sturcture-based Methods
TuckER and ConvE. For TuckER, we use its release code to run the experiments. For ConvE, we use its Pyrch implementation in PKGC's codebase", provided in the same framework as TuckER. It is worth noting that we do not use their proposed "label smoothing" setting for fair comparison.
TransE, DistMult, ComplEx, pRotatE, RotatE, For those KGE models, we reuse the code base released by RotatE [19]. As it provides unified framework and environment to run all those models, enabling fair comparison. Our code httee//odel__commoee py"") also adapts this framework as the foundation.
HAKE. HAKE's code" was built upon RotatE's repository described above. In work, we integrate the implementation of HAKE into our framework, which is also based on RotatE's.
The hyperparameters used are presented in Appendix I.
18

---

# when hint is available
<<(
Please provide brief description of the entity '(entity)' in the following format:
{entity) is [description].
For example:
apple is round fruit with red, green, yellow skin and crisp, juicy flesh.
HINT: (hint)
Now, describe {entity): ..
# when there is hint
...
Please provide a brief description of the entity '(entity)' in the following
format:
(entity) is [description].
For example:
apple is round fruit with red, green, yellow skin and crisp, juicy flesh.
Now, describe {entity):
Figure 7: Prompt for Entity Description.
E
Prompts
E.1 Prompt for Entity Description
Figure showcases the prompts used for entity description. In the prompt, instruct the LLM to provide a brief and concrete description of the input entity. We include an example, such as "apple round fruit with red, green, yellow skin and crisp, juicy flesh" to illustrate that the description should be concise yet cover multiple aspects. Conciseness is crucial to minimize noise. For datasets like FB FB15K-237 and WNISRR, where descriptions already available [23], description as a hint and ask the LLM to output the description in a unified format.
the original
E.2 Prompts for LLM-Guided Hierarchy Refinement
Fig. and show the main prompts we used for LLM-Guided Hierarchy Refinement (LHR).
Prompt for Cluster Splitting (PSPLIT): This prompt is designed to guide the I.I.M in identifying and splitting given cluster into meaningful subclusters based the characteristics of the entities. The prompt works as follows:
1. Input Entities:
The entities from the specified cluster are provided as input.
The LLM tasked with analyzing the entities determine if they can be grouped into distinct subclusters based on common attributes like characteristics, themes, or genres.
Analysis and Grouping:
htcpe://ggthabb.omm//t/t//////
19

---

Given entities from the cluster "[cluster_name?
Analyze the entities and determine if they be grouped into distinct and meaningful sub-clusters based on their characteristics, themes, or genres. If sub-clusters can be formed, provide clear and concise name for each sub-cluster that represents the common attribute of its entities. Each sub-cluster should be given new name that uniformly describes its entities. There needs to be differentiation in the names of different clusters.
The number of sub-clusters should be two. If the entities are already well-grouped and don't require further sub- clustering, simply provide the original cluster.
Provide the output in the following JSON format: "'json
{{
"Sub-clusted Name": ["Entity 1", "Entity 2", ...],
"Sub-cluster Name": ["Entity 3", "Entity 4", ...]
}
Example:
Cluster: Movies
Entities: The Godfather, The Shawshank Redemption, The Dark Knight, Forrest Gump, Inception, The Matrix
Output:
[(
"Drama": ["The Godfather", "The Shawshank Redemption", "Forrest Gump"),
"Action": ["The Dark Knight", "Inception", "The Matrix"]
}}
{cluster_name _amee Entities: (entities)
Cluster:
Output:
Figure 8: Prompt _LM_SPLIT S CLUSTER.
subclusters are formed, the L.L provides clear and concise names for each subcluster that represent the common attributes of their entities. Each subcluster is given unique name that uniformly describes its entities, ensuring differentiation between clusters.
Sub-cluster Naming:
Control of Sub-cluster Count: The prompt instructs the LLM to control the number of subclusters between and 5. If the entities are already well-grouped, no further sub-clustering is needed, and the original cluster returned.
The output format ensures structured and consistent results, facilitating easy integration into the hierarchy refinement process.
Prompt for Bottom-Up Refinement (PReFINE): This prompt guides the LLM refining the parent- child triples within the hierarchy. The refinement process involves several key steps:
1. Input Clusters: Two clusters, and along with their entities, are provided as input.
2. Analysis Clusters: The LLM analyzes the two clusters and their entities to determine the most appropriate update mode. The alignment of update modes to the actions described in the methodology section as follows:
Update Mode (Create New Cluster C): Aligns with No UPDATE. These two clusters cannot be merged, and no cluster belongs to any other.
Update Mode (Merge Cluster and B): Aligns with PARENT MERGE & LEAF MERGE. These two nodes can be merged. The name of the new merged node should be similar to both nodes.
20

---

Given the cluster "(cluster_name _aamte 1)': [[entities _]]]; and the cluster B [cluster_name__ 2)": [fentities_2{]]
Analyze two clusters their entities and determine update mode: Update Mode 1 Create New Cluster C: these two clusters cannot be merged, and no cluster belongs to any other.
Update Mode 2 Merge Cluster A B: these two clusters can merged. The name of two clusters should be similar and entities from two clusters should be similar.
Cluster Covers Cluster B: cluster B belongs to cluster A. cluster is subcluster of cluster A. The name of cluster should uniformly describe the entities from cluster A and the name cluster B. Update Mode 4 Cluster Covers Cluster A: cluster A belongs to cluster B. cluster A is subcluster of cluster В. The name of cluster should uniformly describe the entities from cluster and the name of cluster A.
Update Mode
You need to select update mode based two clusters. If you select mode 1, you should also suggest a name of new cluster. The new cluster name should uniformly describe two clusters.
If you select mode 2, you should suggest a name merged cluster. The new name should be similar to cluster A and B.
Example:
Cluster A 'Thermal Insulators': [cork, fiberglass, foam]; Cluster 'Electrical Conductors': [copper, aluminum, gold]; Select Mode 1.
Cluster 'Sedans': [Toyota Camry, Honda Accord, Fusion]; Cluster 'SUVs': [Honda CR-V, Toyota RAV4, Ford Escape]; Select Mode 2.
Cluster A 'Feline Species': [lions, tigers, cheetahs]; Cluster 'House Cats': [Siamese, Persian, Maine Coon]; Select Mode 3.
Cluster 'Leafy Vegetables': [lettuce, spinach, kale]; Cluster B 'Root Vegetables'; [carrots, potatoes, beets]; Select Mode 4.
Provide the output the following JSON format: -*isoo {(
supdate_modder or or 4,
"name": "merged cluster name or new cluster name"
}}
Output:
Figure 9: Prompt of
Update Mode 3 & (Cluster A Covers Cluster B): Aligns with INCLUDE where P' PUR.L' L, R' = G. Cluster B belongs to cluster A. Cluster B a subcluster cluster A. The name of cluster A should uniformly describe the entities from both clusters.
Output Format: The output includes the selected update mode and the suggested name for the new merged cluster, ensuring clarity and consistency in the hicrarchy refinement process.
21

---

These prompts, illustrated in Fig. 8 and Fig. 2, are essential for transforming the seed hierarchy into more refined and accurate LLM-guided hierarchy, enabling better hierarchical representation be learned by KG-FIT.
F Details of KG-FIT Hierarchy Construction
F1 Seed Hierarchy Construction
Algorithm Seed Hierarchy Construction
Require: Enriched entity representations V, range of thresholds [Tmlus Tmax
Ensure: Seed hierarchy Hseed
1:
2: I/ Step 1: Agglomerative Hierarchical Clustering
3: for [Tmin: Tmax do
4: labels, f AgglomeenaiveCCusseinggg
5: S, + SilhouctttSSoore V, labels,)
6: end for
7: Toptim arg (Sx
8:
9: I/ Step 2: Constructing the Initial Hierarchy
10: Hinit + ConnttrucBinnayy Tabellllssssss sppim))
11:
12: // Step 3: Top-Down Entity Replacement
13: visited clusters t !
14: ReplaceEntities WithClusters Humi, labels Togen iisited_clusteen
15:
16: // Step 4: Refinement Steps
17: Hseed RefineeHiiii)
18:
return Hased
20:
function RERRAACEEETTIISSWIICCUUSSSSSEEE visited_clusters) for node E 7. do
22:
if node a leaf then
24:
entity + GetEntity(node)
cluster <- GerCluster(entity, labels)
26:
if cluster E visited clusters then
27:
Remove node from H
28:
else
29:
Replace node with cluster iisited_clusted <- visited_cluste U [cluster}
30:
31:
end else
32:
33:
ReplaceEnttiessiilCltttss//ooooe labels, visited_ )lastdr_ssster) end if
34:
35:
end for
36:
end function
This section details the seed hierarchy construction mentioned in Section 3.1. Algorithm shows the pseudo code for this process.
The process begins with the agglomerative hierarchical clustering the enriched entity representa- tions V using range of thresholds [Tmin; Tmax]. For each threshold 7, the clustering labels (labels,) are obtained, and the silhouette score (Sz) is calculated. The optimal threshold Toptim is determined by selecting the threshold that maximizes the silhouette score.
22

---

Next, the initial hierarchy Hini is constructed based on the clustering labels obtained using the optimal threshold (labels,.gin) ) The ConstructBinaryTTee function builds a binary tree structure where each leaf node represents an entity.
The top-down entity replacement algorithm is then applied the initial hierarchy Hini- It traverses the hierarchy in a top-down manner, starting from the root node. For each node encountered during the traversal:
If the node is leaf, the entity associated with the node is retrieved using the GetEntity function.
The cluster to which the entity belongs is obtained using the GetCluster function and the labels,..tia
If the cluster has already been visited (i.e., it exists in the visited cloustrrs_esssseeeeeeee set), the leaf node is removed from the hierarchy.
If the cluster has not been visited, the leaf node is replaced with the cluster, and the cluster is added to the visited_clusters clusters set.
If the node is not leaf, the algorithm recursively applies the same process to its child nodes.
This top-down approach ensures that entities are included in their respective clusters as early as possible during the traversal of the hierarchy. By keeping track of the visited clusters, the algorithm avoids duplicating clusters in the hierarchy, resulting in a more compact and coherent representation.
Finally, the refinement steps (Refine function) are applied to the modified hierarchy to obtain the final seed hierarchy Hseed The refinement steps include handling empty dictionaries, single-entry dictionaries, and updating cluster assignments.
The resulting seed hierarchy Haeed represents a hierarchical organization of the entities, where each node is either a cluster containing a list of entities or a sub-hierarchy representing a more fine- grained grouping of entities. This seed hierarchy serves as a starting point for further refinement and incorporation of external knowledge in the subsequent steps of the KG-FIT framework.
F.2 LLM-Guided Hierarchy Refinement
LLM-Guided Hierarchy Refinement (LHR) is used to further refine the seed hicrarchy, which can better reflect relationships among entities. The process of LLM-Guided Hicrarchy Refinement can be divided into two steps: L.MM-Gidee Cluster Splitting and LLM-Guided Bottom-Up Hierarchy Refinement.
As described in Algorithm 2. the LLM-Guided Cluster Splitting algorithm is designed to iteratively split clusters in hierarchical structure with the guidance of large language model (LLM). The algorithm begins with an initial hierarchy seed Hwed and outputs a split hierarchy Hapli- Initially, the current cluster ID is set O. The recursive procedure RECURS SISSSIINNSSPPPSSSS __s___ST___SS CLUSTER is then defined to manage the splitting process. If the root of the current cluster is a list and its length is less than a predefined minimum number of entities in a leaf node WHIN_ENTITIESS IN LEAF), the function returns, indicating no further splitting is necessary. Otherwise, the root cluster's entities are passed to the LLM, which names the cluster and splits it into smaller clusters. If the split results in only one cluster, the function returns. For each resulting subcluster, the entities are assigned a new cluster ID, and the splitting process is recursively applied. If the root is not a list, the function iterates over each subcluster and applies the splitting procedure. Finally, the algorithm initiates the recursive splitting on the initial hierarchy seed and returns the modified hierarchy as Hapli--
The I. M-GGidded Bottom-Up Hierarchy Refinement algorithm (Algorithm 3) refines a previously split hierarchy Hsplit to produce more coherent and meaningful hierarchy HLHR. This process also guided by an LLM. The algorithm defines a recursive procedure RECURS LOR_OO_EFIIII BEEEEEEEEERHHHEEEEE which starts by checking the root of the current cluster is a list. If is. the LLM is used to name the cluster, and the updated hierarchy is returned. Otherwise, the procedure recursively refines the left and right child nodes. The children of these nodes are then evaluated, and the LLM suggests an update mode based on the names and children of the right and left nodes. Depending on the suggested update mode, the algorithm may add the right node under the left children, the left node under the right children, or merge the left and right clusters. If no update is suggested, the procedure continues. The updated hierarchy returned after applying the necessary refinements. The algorithm initiates the refinement process on the split hierarchy Hsplic and outputs the refined hierarchy HL
23

---

Algorithm 2 LLM-Guided Cluster Splitting
1: Input: Hsed
2: Output: Haplis
3: current cluster id 6 0
4: procedure RECURSION S_RIINNSSP_T_TE OPPLITCCUUSTERTTON
5: if' root is a list then
6: if length root CMMI_ENNTIIN LN__I______ LEAF then return
7:
8: end
9:
cluster_entities root
10:
cluster_name + LLM_LLMMAA HLM__AME__ CLUSTER((Custerr_ene
11:
splitted_clusters +- LLLMSPLITTCCLILI www___PLT_CCUSTTRtcusseee_ss_
cluster_entities)
length splitted_clusters
then
13:
return
14:
end
15:
for each (name, entities) in splitted clusters do
16:
rootfcurrent_ caat__clutte_rrree - entities current cluster id <- current cluster id 1 RECURSION SPPIINSSPTTT__ cuUSTEERrroocuurret__cuuteee
17:
18:
end for
20:
else
21:
for each (key, subcluster) in root do
22:
DeCURSSON_SPPIT_CCUSSEEKICA end for 24: end if
end procedure
26: RECURSION SPLIT_ DOCCURSOONSPLIT_COOO
SPLIT_CLUSTER((1,0
Hsplit < Hreed
28:
return Hapli
F.3 Statistics of Constructed Hierarchies
Table 7: Statistics of the hierarchies constructed by KG-FIT different datasets.
FB15K-231
YAGO3-10
PrimeKG
WN1SRR
Seed
LHR
Seed
LHR
Seed
LHR
Seed
LHR
# Cluster
5,226
5,073
31,832
30,465
4,048
3,459
16,1
14,230
Node
10,452
8,987
63,664
52,751
8,096
5,918
32,228
26,186
Max
115
115
1,839
1,839
72
40
58
Entity in a Cluster
Min
1
1
1
Avg
2.78
2.81
3.87
4.04
2.56
2.99
2.54
2.88
Max
46
40
81
26
57
43
Cluster Depth
Min
3
3
4
3
5
Avg
25.52
18.64
38.86
28.72
20.35
12.39
25.91
22.5
Max
2
74
2
135
34
2
37
# Branch of Node
Min
1
Avg
2.3
2
2.37
2.41
2
2.19
Table 7 presents the statistics of the hierarchies constructed by KG-FIT on four different datasets: PB15K-237, YAGO3-10, PrimeKG, and WN18RR. The table compares the seed hierarchy (Seed) with the LLM-Guided Refined hierarchy (LHR) show the changes and improvements after applying the LLM-guided hierarchy refinement process.
The number of clusters decreases across all datasets after refinement. For instance, in the FB15K-237 dataset, the clusters reduce from 5226 to 5073. Similarly, the number of nodes also decreases; for 2B15K-2377 nodes go from 10452 to 8987. The number of entities within each cluster sees

---

Algorithm 3 LLM-Guided Bottom-Up Hierarchy Refinement
1: Input: Hyplit
2: Output: HLHR
3: procedure RECURSION REFINE HIERARCHY(root)
4: if' root list then
5:
sedated_hierareee < LLM_NAME__C [AME_[LUSTER(OO)) return Jhierrrchy
6:
7:
else
8:
left_ node + RECURSION CECCURSIIO_REEINEE REFINE, SEIFINE_HIERARCHY(Oe) _ode)
9:
right_nodee node +- RECURSION REFINE_ CINIENN_IIERRRCC(((ChhIILLA
10:
left_children + Children(left_nonn node)
11:
right_childre < Children(right_non node)
update_mont + LLM LLM_JUDATTEnnames and children right and left node)
13:
if update_mooee mode = update then
14:
contimue
15:
else if update soceeeemoeeeee-eeeeeeeeeeeeeeeee == merge left and right then
16:
Merge left and right clusters update hierarchy name and children
17:
18:
else update_mode == left include right then
Add right_node under cffi_ccilreen update hierarchy name and children else if update_nne mode right include left then
20:
21:
22:
Add left_node under geght_children update hierarchy name and children 24: end if
25:
return updated_ hierarchy end if
26:
end procedure
28:
HLRH < RECURSION REFINE HIEERNEEEERAAHHYAAA (Aspit)
29:
return HIHRR
slight increase in the average number, with the maximum and minimum values remaining fairly constant. For example, the average number of entities per cluster in FB15K-237 increases from 2.78 to 2.81. The depth of the hierarchies shows a noticeable reduction, with the maximum and average depths decreasing. In PB15K-237, -233, the maximum depth goes from 46 to 40, and the average depth drops from 25.52 to 18.64. The branching factor of nodes also increases slightly, indicating a more interconnected structure: FB15K-237, the average number of branches per node rises from to 2.3. Similar patterns are observed in other datasets.
Overall, Table 7 illustrates that the refinement process effectively reduces the number of clusters and nodes, slightly increases the number of entities per cluster, decreases the depth of the hierarchy, and increases the branching factor. These changes suggest that the refinement process results in more compact and interconnected hierarchy, better reflecting relationships among entities. The general effect of the refinement is the creation of more streamlined and coherent hierarchical structure.
F.4 Examples of LLM-Guided Hierarchy Refinement
Table 8: An example of LLM-Guided Cluster Splitting.
Sab-cluster Name
Sub-cluster Entittee
Universities Colleges
Princeton University, Yale University. Hurvard University, Brown University, Darmouth College, Harvar Colloge, Yale College School
Medicine,
Examples in Table 8 and Table 9 illustrate how the LLM- hierarchy refinement process updates and organizes clusters and sub-clusters to create a more coherent and meaningful hierarchical structure.
Table & provides examples sub-clusters within a cluster the process cluster splitting. Fach sub-cluster is given name and lists the associated entities. For instance, one sub-cluster named
25

---

Table 9: Examples of LLM-Guided Bottom-Up Up Hierarchy Refinement.
A Name
Cluster
Claster
Cluster B
Update
Washington D.C. Sports Tomms
Washingoon Wiznns....
Detroil Sports Teams
Detroii Red Wings... ...
Updase
New York Regional Emilies
New York Island Fmiiiees
New York Locations and Cities
New Yoek Locaioon.
Puroni Merge
Foutball Positions
running back. ballback...
Foutball Positions
light end, defensive end,
Leaf Merge A Includes B
Los Angeles Region
Los Angeles, Southem Califomia
West Los Angeles Suburbs
Inglewood. Torrance
The Bryan Brothers
Bob Bryan. Mike Bryan
Tennis Athletes
Willinms Sisters...
Includes
"Universities and Colleges" includes entities such as Princeton University and Harvard College. Additionally, sub-cluster named "Medical Schools" includes entities such as Yale School of Medicine. The cluster of "Universities and Colleges" will be divided into the sub-clusters of "Universities" and "Colleges" in the next step. This example shows that LLM can divide the original large cluster into multiple smaller, refined ones.
Table 9 demonstrates various cases of cluster updates in LLM-Guided Bottom-Up Hierarchy Refine- ment. Each case lists two clusters (Cluster A and Cluster B) along with their entities and the type of update applied. In some cases, no update needed, and both clusters remain unchanged. In others, Cluster B is merged into Cluster A, indicating a hierarchical relationship. There are instances where entities from Cluster B are integrated into Cluster A. Additionally, some updates involve Cluster A including all entities of Cluster B, making Cluster B subset of Cluster A, or vice versa. These examples all show that the LLM correctly fixes the errors in the original seed hierarchy and refines the hierarchical structure to better reflect world knowledge.
Overall, these examples demonstrate how clusters are refined better represent the relationships between entities, leading to a more organized and efficient structure. These examples highlight the detailed organization of entities into meaningful sub-clusters, reflecting their natural groupings and relationships. The refinement process not only improves the overall structure but also enhances the clarity and accessibility of the hierarchy.
G Score Functions
Table 10: Score functions defined by the KGE methods tested in this work.
Model
Score Function f- L)
Parameters
TransE
h + t/l1/2
h,r,t E R
DistMult
h diag(r)t
h,r,t E R
ComplEx
Re(h" diag(r)t)
h,r,t C
ConvE
f(vee(f(h + W)t
h E R
TuckER
W x1 h X2 Wr Xx t
W, E Rd
PRotatF
-2C sin -
h, E C*
RotatE
|| or t||2
h,r,t Ch [r.] =
hm; fm R tm R
HAKE
Ilbm
I'm
tm|l2
\l| sisin((b,, + rp (,))
hp. tp E [0, 2T)² A E IR
The score functions defined by the structure-based KG embedding methods [14, 15, 16, 17, 18, 19, 20, 21] we tested in this papaer are shown in Table 10, Here is a paragraph explaining each notation in the table, with all explanations inline:
The table presents the score functions fr (h, t) used by various knowledge graph embedding (KGE) methods, where h, r, and t represent the head entity, relation, and tail entity embeddings, respectively. TransE uses translation- based score function with either LI or L2 norm, denoted by 111/2- where the embeddings are in real space R DistMult employs a bilinear score function with a diagonal relation matrix diag(r), and the embeddings are also in R ComplEx extends DistMult by using complex-valued embeddings in C* and takes the real part of the score, denoted by Re(-). ConvE applies 2D convolution operation, where h is the 2D reshaping of h, represents the
26

---

convolution, w is set of filters, vec(-) is a vectorization operation, and W is a linear transformation matrix. TuckER uses a Tucker decomposition with a core tensor W and relation-specific weights w, E RRdr where x, denotes the tensor product along the n-th mode. pRotatf: and RotatE, employ rotation-based score functions in complex space, with a representing the Hadamard product and Ir,l = constraining the relation embeddings to have unit modulus. HAKE combines modulus part (hm. tm E R tm IR4,) and phase part (hp. Fp, Ep [0, 2 hyperparameter balancing the two parts.
) in its score function, with
H Computational Cost
H.1 Hardware and Software Configuration
Based on the dataset size, we hybridly use two machines:
For FB 15-37777, PrimeKG, and WNISRR, experiments are conducted on a machine equipped with Two AMD EPYC 7513 32-Core Processors, 528GB RAM, eight NVIDIA RTX A6000 GPUs, and CUDA 12.4 and the NVIDIA driver version 550.76.
For YAGO3-10 10, due to its large size, experiments conducted on machine equipped with two AMD EPYC 7513 32-Core Processors. 528GB RAM, and eight NVIDIA A100 80GB PCIe GPUs. The system uses CUDA 12.2 and the NVIDIA driver version 535.129.03.
With a single GPU, takes about 2.5, 4.5, 2.0, and 1.1 hours for a KG-FIT model to achieve good performance on FB15K-237, YAGO3-10, PrimeKG, and WN18RR, respectively.
H.2 Cost of Close-Source LLM APIS
The costs of GPT-4o for entity description generation are $3.0. $24.8, $2.1, and $8.6 for FB15K- 237, YAGO3-10, PrimeKG, and WNISRR, respectively, proportional to their numbers of entities.
The cost of text embedding models (text-embedding-3 was totally about $0.8 to process all the KG datasets.
for entity embedding initialization
The costs of GPT-4o for LLM-Guided Hierarchy Refinement on FB FB15K-237, YAGO3-10. 10, PrimeKG, and WNI8RR are $10.0, $74.5, $8.7, and $34.4, respectively. This cost is almost propor- tional to the nodes consisted in the seed hierarchy.
I Hyperparameter Study
This section presents a comprehensive hyperparameter study for both structure- base models and proposed KG-FIT framework across different datasets. Table 11 outlines the range of hyperparameter values explored during the study. Table 12 showcases the optimal hyperparameter configurations for the base models that yielded the best performance. Similarly, Table 13 presents the best-performing hyperparameter settings for KG-FIT.
The hyperparamcter study aims to provide insights into the sensitivity of the models to various hyperaarameters and to identify the optimal configurations that maximize their performance on each dataset. By conducting thorough exploration of the hyperparameter space, ensure fair comparison between the base models and KG-FIT, and demonstrate the robustness and effectiveness of our proposed framework across different settings.
Table 11: Summary of hyperparameters we explored for both base models and KG-FIT.
Hyper-parameter
Studied Values
Batch Size
[64, 128, 256, 1024)
Negative Sampling Size ([G) in Eq 7)
[64, 128, 256, 400, 512, 1024)
Hidden Dimension Size n
[512, 768, 1024, 2048)
Score Margin y
(6.0, 9.0, 10.0, 12.0. 24.0, 60.0, 200.0)
Learning Rate
[5e-5, 2e-4, 1e-4, Se-4, 1e-3, 2e-3, 4e-3, Se-3, 1e-2, 2e-2, 5e-2)
A1. Au,
in range [0.1, I.0]
Çi. Co.
in range [0.1, 10.0]
27

---

Table 12: Best hyperparameters grid-searched for base models on different datasets.
FB15K-237
TransE
DistMult
ComplEx
ConvE
TuckER
pRotatE
RotatE
HAKE
Batch Size
1024
1024
1024
512
512
1024
1024
1024
Learning Rate
5e-4
lc-2
le-2
5e-3
Se-3
Se-4
5e-4
le-3
Negative Sampling
256
256
256
256
256
256
Hidden Dimension
1024
2048
1024
512
1024
2048
2048
2048
y
9.0
200.0
200.0
9.0
9.0
9.0
YAGO3-10
TransF
DistMult
ComplEx
ConvF:
TuckFIR
pRotaiF
RotatF
HAKE
Batch Size
512
512
512
128
128
256
1024
1024
Learning Rate
2e-3
lc-3
le-4
5e-5
le-4
Se-4
2e-3
2e-3
Negative Sampling
256
256
256
512
400
256
Hidden Dimension
1024
1024
1024
512
1024
1024
1024
1024
24.0
24.0
24.0
24.0
24.0
24.0
PrimeKG
TransF
DistMult
ComplEx
ConvE
TuckFR
pRotatF
RotatF
HAKE
Batch
512
512
512
128
512
512
512
Learning Rate
5e-4
2e-2
2c-3
Se-4
5e-5
lc-4
lc-4
lc-4
Negative Sampling
512
512
512
1024
1024
512
Hidden Dimension
1024
1024
1024
512
1024
2048
2048
2048
24.0
200.0
200.0
24.0
24.0
6.0
Table 13: Hyperparameters we used for KG-FIT with different base models on different datasets.
FB15K-237
TranS
DistMull
ComplEs
ConcE
TuckER
pRotatE
RotalE
HAKE
Leaming Rate
le.3
le-3
le.3
le-4
le-4
5e-4
le.3
Se-4
Negative Sumpling
512
512
512
512
512
512
Hiddenn Dimension
1024
512
512
512
1024
2048
2048
2048
9.0
Au. Au, Àa Ga
(1.0, 0.4, 0.5)
(0.5, 0.5. 0.5)
(0.5, 0.5, 0.5)
(0.8, D.6, (.2)
(0.8, 0.6, 0.2)
(1.0, 0.4, 0.5)
0.4,004, 0.5)
(1.0. 04, 0.5)
(0.5, 3.5)
(0.5, 0.5, 2.0)
(0.5, 0.5, 2.0)
(0.5, 0.5, 2.0)
(0.5, 0.5, 2.0)
(0.5, 0.5, 4.0)
(0.5, 0.5. 3.5)
(0.5, 0.5, 3.5)
YAGO3-10
Trams
DistMult
ComplEx
ConvE
TockER
pRotat
RotatE
HAKE
Batch Size Leaninng
256
256
256
128
256
256
256
1e-3
1e-4
Negative Sampling
256
256
512
256
512
Hidden Dimension
1024
1024
1024
512
1024
1024
1024
1024
24.01
24.0
24.01
Xy Ag
(1.0.
0.5)
(1.0,
0.2)
(1.0,
0.5)
0.5)
(0.5. 0.5, 0.5. 2.00
(0.5, 0.5, 2.0)
(0.2. 0.2, 3.0)
(0.2, 0.5, 2.00
(0.2, 0.5. 2.0)
(1.0) 1.0, 9.09
(1.0. 1.0, 9.0)
(1.0) 1.0, 8.0)
PeimeKG
TransE
DistMltt
ComplEx
ConvE
TuckER
pRotndE
RotatE
HAKE
Hatch Size
256
512
256
12%
128
256
512
512
26-2
2e-9
5o-4
Sc-5
le-4
le-4
Negative Sampling
1024
512
256
512
Hidden Dimension
1024
1024
20488
512
1024
2048
2044
2048
24.0
200.0
200.0
24.0
10.0
10.0
0.2)
G.
(0.5,
(0.5, 0.5, 2.0)
(0.2. 0.2, 5.0)
(0.5, 0.5, 6.0)
(0.5.0.5, 6.0)
(0.5, 0.5,
(1.0.
1.0. 7.0)
(1.0)
28

---

J Downstream Applications of KG-FIT
Fundamental
Matching KDs
Speed Prosimal Relationship Search for Retrieval Augmented Generation
Link Prediction
Q: Cann drinkin wa Lack
"Onnding Block
help reduce
Alvanadds, help
CIAATTEi InG.
diozal
(Sorclor
diseases. Flavonolds ontiousann proportles witich offer protective baoofiiz againet such conditions"
Bewarags
Triple Class fication
2000
S+3000 Car
True
@-
Home Autoriatio
Devicss
supportive information
Figure 10: Applications of KG-FIT.
The enhanced knowledge graph embeddings produced by KG-FIT can enable improved performance on various downstream tasks. As shown in Fig. 10, potential areas of application include:
Tasks transformed to fundamental KG tasks. KG-FIT's strong performance on link predic- tion can directly benefit fundamental KG tasks like knowledge graph-based question answering (KGQA) [1, 2, 4]. For example, given the question "Can drinking black tea help reduce the risk of cardiovascular diseases?", a KGQA system powered by KG-FIT embeddings could effectively traversely perform triple classification task for triples such <black tea, potentially beneficial to, cardiovascular diseases>, <black tea, is a, flavonoids--icc drink>, <flavonoids, may treat, cardiovascular diseases>, and provide an accurate answer based on the classification results.
Entity Matching Across KGs. KG-FIT's ability to capture both global and local semantics facilitates accurate entity matching across different knowledge graphs [48, 49, 50]. Consider two KGs. A and B, containing information about company's products. KG A lists the entity "GlobalTech Inc" launched "Smart Vision 3000", which is a product in "Smart Home Devices", while KG B mentions "SV3000 Camera" is released by "Global Tech Industries" in the category of "Home Automation Solution". By generating semantically rich embeddings that encode textual, hierarchical and relational information, KG-FIT can help identify that Vision 3000" and "SV3000 Camera", "GlobalTech Inc." and "Global Tech Industries" likely refer to the same entities, despite differences in surface form and graph structure.
Retrieval Augmented Generation with Graphs. The hierarchical nature of KG-FIT's embeddings enables efficient search for relevant information to augment language model-based text generation. In retrieval augmented generation setup, language model's output can be enhanced by retrieving and conditioning on pertinent information. KG-FIT's embeddings allow for quick identification of relevant entities and relationships via proximity search the semantic space. Moreover, by traversing the KG-FIT hierarchy, the system can gather additional context about an entity of interest. For instance, if the generation task involves the cardiovascular benefits of black tea, searching the KG-FIT hierarchy may surface related information on flavonoids and antioxidant properties, providing valuable context to guide the language model producing an informed and factually grounded response.
Other Tasks. KG-FIT's embeddings can also be leveraged for tasks such as relation extraction [51] and entity disambiguation [52]. By providing high-quality embeddings that encode both local and global information, KG-FIT can improve the accuracy and efficiency of these tasks. For example, in relation extraction, KG-FIT's embeddings can help identify the most likely relation between two entities given their positions in the hierarchy and their semantic proximity. In entity disambiguation, KG-FIT's embeddings can be used disambiguate between multiple entities with the same name by considering their relationships and positions within the knowledge graph hierarchy.
In summary, KG-FIT's robust embeddings, capturing both local and global semantics in a hierarchical structure, can significantly enhance variety of downstream applications, from fundamental KG tasks to entity matching and retrieval augmented generation. By providing semantically rich and efficiently searchable representations of KG entities and relationships, FG-FIT enables knowledge-intfe dinnuude-infuued AI systems that can better understand and reason over complex domains.
29

---

K Interpreting Score Functions in KG-FIT
In this section, we analyze how the entity embeddings in KG-FIT are interpreted in the (transitional) score functions defined by different base models.
Let h, t Rdim denote the head entity, relation, and tail entity embeddings, respectively, In KG-FIT, the entity embeddings h and are initialized as follows:
h hal, t = (t,; tx)
(9)
where hn tn E Rdin/2 represent the entity name embeddings and ha, ta Rddimm represent the entity description embeddings, obtained from the pre-trained text embeddings.
The inclusion of both h, and ha (similarly for t, and t.)) enhances the expressiveness of the entity representation. Starting from these embeddings, KG-FIT effectively captures more comprehensive understanding of each entity, leading to improved link prediction performance.
TransE: The TransE score function is defined
fh, t)
h r tl1p
(10)
where || Ilp denotes the Lp norm.
Expanding the score function using the KG-FIT embeddings:
(h, t)
[ ba] [tnn tu] [h, + Fn t, ha + ra tullp
(11)
where r = ral is the relation embedding learned during fine-tuning.
Interpretation:
hn +r, t This term represents the distance in the embedding space between the head and tail entities' name embeddings, adjusted by the relation embedding.
ha + ra tu: This term represents the distance in the embedding space between the head and tail entities' description embeddings, adjusted by the relation embedding.
The TransE score function considers the global semantic information by computing the translation distance between the head and tail entity embeddings, taking into account both the entity name and description embeddings.
DistMult: In the DistMult model, the score function is defined as the tri-linear dot product between the head entity embedding h, the relation embedding and the tail entity embedding t:
(+ (h,
(12)
In KG-FIT, the entity embeddings are initialized by concatenating the entity name embedding and the entity description embedding:
h hals = [tm, ta]
(13)
Now, let's expand the DistMult score function by substituting the KG-FIT entity embeddings:
Srh t) = (Ihn ha], r. [tn; ta]) (hn, Iw: tn) + (ha, ta)
=
dim/2
dim
(has Pais tn.i) +
(ha, Tax ta,s)
(14)
i=dim/2+1 2
where r = ra] is the relation embedding learned during fine-tuning.
Interpretation:
(hn, In; t,): This term captures the multiplicative interaction between the head entity's name embedding, the relation embedding, and the tail entity's 's name embedding.
30

---

(ba, Fd. Fd.ta): ty):
This term captures the multiplicative interaction between the head entity's 's description embedding, the relation embedding, and the tail entity" 's description embedding.
The DistMult score function captures the global semantic information by computing the tri-linear dot product between the head entity, relation, and tail entity embeddings. The dot product considers the interactions between the entity name embeddings and the entity description embeddings separately, allowing the model to capture the global semantic relatedness and attributional similarities.
ComplEx: In the ComplEx model, the score function is defined as:
[r t) Re((h, t))
= Re(r), Re(t)) + (Im(h), Re(r), Im(t))
(Re(h), Im(r), Im(()) (Im(h), Im(r), Re(t))
(15)
where Re(-) and Im(-) denote the real part and imaginary part of a complex number, and È represents the complex conjugate of t.
In KG-FIT, the entity embeddings are initialized as follows:
h ihds t tx + it,
(16)
where h tn E Rddim/2 represent the entity name embeddings (real part) and ha, ta E d represent the entity description embeddings (imaginary part).
The relation embedding r is also a complex-valus vector:
r r= Fy + ir;
(17)
where ry, E Rddim22 are learned embeddings that capture the intricate semantics of the relation in the complex space.
Thus, the score function using the KG-FIT embeddings becomes:
fr t) Re((h, t)) (hn, Fr, bn) + (ha, rr, ta) + (bn, ris ta) (ha, hn o r, o tn + haor, o r, o ta + hn o r o tu - ha o tn
(18)
Interpretation:
hn or, ot, and b or, oty These terms represent the fundamental interactions between the head and tail entity name embeddings modulated by the real part of the relation. They capture symmetric relationships where the semantic integrity of the relation maintained irrespective of the direction.
hn o r, o ta and -h r, o tn: These tross-terms incorporate the imaginary part of the relation embedding, introducing unique capability model antisymmetric relations. The inclusion of the imaginary components allows the score function to account for relations where the direction or the orientation between entities significantly alters the meaning or the context of the relation.
RotatE: In the RotatE model, each relation is represented rotation in the complex plane. The score function defined as:
fr (h, t) - -hor t|| (Re(h) Re(r) Im(h) Im(r) Re(t)) + + (Re(h) Im(r) + Im(h) o Re(r) - Im(t))
(19)
In RotatE with KG-FIT embeddings, we have:
h + ilds t tn + it
(20)
and the relation embedding is:
r= cos + sin (0,)
(21)
where 0, is the learned rotation angle for the relation.
31

---

Expanding the RotatE score function using KG-FIT embeddings, we have:
fr, t) = - h r t|| (h, cos( ha sin(O,) t..) + (h,, sin(0,) + ha cos(O,) t.)l [h,, cos(O,) tn) - (ha cos(O,) ) ta) + (hn o sin( (ha sin(0,))) (22)
Interpretation:
hn co,( t, and ha o cos(0,) ty: These terms represent the rotated head entity name and description embeddings respectively, which are then compared with the corresponding tail entity name and description embeddings. The cosine of the relation angle 0, scales the embeddings, effectively capturing the strength of the relationship between the entities.
hn (0,) and - had sin(),): These terms introduce phase shift in the entity embeddings based on the relation angle. The sine of the relation angle 0, allows for modeling more com- plex interactions between the head and tail entities, considering both the name and description information.
PRotatE; In the pRotatE model, the modulus of the entity embeddings is constrained such that [N] [4:] C, and the distance function is defined as:
(h -2 sin On 0,
0c Il
(23)
2
where On. 0,, and 0. represent the phases the head entity, relation, and tail entity embeddings, respectively. For KG-FIT embeddings, the entity embeddings complex and represented as:
h [h, ha),
=ttn ta)
(24)
The phases be calculated as: 0₂ arg( ar,( ha)), 0. arg((t.:: (t.: ta));
0, = arg([ri;r2)
(25)
where rı, 12 E IR dimm are the learned relation embeddings.
Thus, the pRotatE score function using KG-FIT embeddings becomes:
f+h -C sin arg((hn; ha)) + arg([r1;r2)) arg tni tal)
(26)
2
Interpretation:
pRotatE with KG-FIT, the entity phases 0j, and 0r are computed using both the name and description embeddings, while the relation phase 0, is learned through the relation embeddings rı and r2-
The model aims to minimize the phase difference ||sin (Oxte-=0) ][, for valid triples, considering both the entity name and description information. This allows pRotatE to capture the complex interactions between entities and relations in the knowledge graph.
HAKE: In the HAKE (Hierarchy-Awall Knowledge Graph Embedding) model, entities are embedded into polar coordinate space to capture hierarchical structures. The score function is a combination of radial and angular distances:
f, (h, t) aal hmod mood tmood B 3isin hppaae + Tphase
tpphsse
(27)
In KG-FIT, hmod = hys tmod = tu, hphase = arg(h,) and tphase = arg(t,). The learned relation embedding is r rg].
Thus, the HAKE score function using KG-FIT embeddings becomes:
= - ra - tall2- 8 sin arrg(h,) arg(r,) arg(ta) ||
(artm))ar((ta)))
(28)
Interpretation:
In HAKE with KG-FIT, the entity description embeddings are used to determine the modulus, while the entity name embeddings are used determine the phase. This approach seamlessly utilizes the information from both types of embeddings from pre-trained language models.
32

---

L Notation Table
Table 14 provides a comprehensive list of the notations used throughout this paper, along with their corresponding descriptions. This table serves as quick reference to help readers better understand the concepts presented our work.
Table 14: Notations and Descriptions in KG-FIT
Notation
Description
E
Set of entities in the knowledge graph
e, de
The i-th entity in the knowledge graph Text description entity e,
f
Text embedding model
Text embedding of entily e; Text embedding of description d;
Concatenated embedding of entity and description d:
Clustering threshold
Timi Timax! Toptim
Minimum and maximum clustering thresholds Optimal clustering threshold
S
Silhouette score
labels, Hysed Coniginal Csplit
Clustering results distance threshold
Seed hierarchy constructed by agglomerative clustering Original cluster in the seed hierarchy New clusters after splitting Cold Set of leaf clusters in the seed hierarchy
Claa Ci
The i-th new cluster after splitting
k Haphi Th (Pa, P.)
Number of new clusters after splitting
Intermediate hierarchy after cluster splitting
Set of parent-child triples in the intermediate hierarchy Parent-child triple (sub-tree) in the hierarchy -LMddddded refined hierarchy
HLHR c!
Sliced text embedding of entity Randomly initialized embedding of entity e; Hyperparameter controlling the density of randomized embedding KG-FIT FIT embedding of entity e;
P
e,
rj
Embedding relation j Dimension of entity embedding e; Dimension of relation embedding ry Hyperparameter controlling the standard deviation Cluster that an entity belongs to Neighbor cluster Cluster embedding of C' Cluster embedding of C" Set of nearest neighbor clusters of C Parent nodes
al mlm
C
clc
(C)
P
P Pf+1
Embeddings of successive parent nodes along the path from an entity to the root Distance function for measuring distances between embeddings Weight for the distance to the j-th parent node Initial weight for the closest parent node Decay rate for hierarchical distance maintenance Hyperparameters for the hierarchical clustering constraint Set of all triples in the knowledge graph Sigmoid function
d(
30
\1 A2, As
D
Margin hyperparameter for link prediction Scoring function defined by structure- based models negative tail entities sampled Negative tail entity
(
Ng
triple (c,, r, r,)s)
eny
Embedding the negative tail entity ny Hyperparameters for weighting the training objectives
S1 Co. C3
L
Average sequence length in PL.M-bsedd methods
1PPMM
Hidden dimension of the PLM
methods
33

---

NeurIPS Paper Checklist
Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
is fine include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: Appendix A.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low images are taken in low lighting. Or speech-to-text system might be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach address problems of privacy and faimess.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question:
For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [NA]
34

---

Justification:
the paper does not include theoretical results
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibiliyy
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: Please refer to Section 4 and Appendix I reproduce our results. We also release all the code and data.
Guidelines:
The answer NA means that the paper does include experiments.
If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
35

---

Answer: [Yes]
Justification:
We release our code and data in the uploaded zipped supplementary materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.. cc/
Sublic/guidess sondiidee/oodSuumiisee wwwsssssslllooooooooooooooo plyycyy) for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https:
/ //nips. wttpccccpppccccceeecec.eccccccct guides/dd/eee /CodeSubmiieeeee
/ttpa///ggiieee/eeeuuuubiiiiiiii ssiinnsssnneesese onPolicy) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: Section 4 and Appendix Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [Yes]
Justification: Section 4.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
36

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
In Appendix H.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https: / http///eurrrp.eeeee cctppblic//eeeeeeeceecccee htw.eettehhhiieennnncnnm
Answer: [Yes]
Justification: We have reviewed the NeurIPS Code of Ethics. The research conducted in this paper conform with it.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make to preserve anonymity (e.g.. if there is special consid- cration due to laws or regulations their jurisdiction..
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: In Appendix A.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
37

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of MI..
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [Yes]
Justification: In Appendix A.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Lieenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: Section 4 and Appendix B.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license (c.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thcode. com com/dataseta has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
38

---

If this information is not available online, the authors are encouraged to reach out to the asset's 's creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification: In this paper, we provide subset of PrimeKG, which is described in Appendix B, and we do not release new assets. We shared an anonymized URL to our code and data.
Guidelines:
The answer NA means that the paper does release new assets.
Researchers should communicate the details the dataset/code/moodll as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
39


Reviews:
[{"review_id": "gkTeEesfHL", "reviewer": "Reviewer_hiNn", "summary": "The paper introduces a framework called KG-FIT for enhancing knowledge graph embeddings by integrating knowledge from large language models (LLMs). KG-FIT enriches entity descriptions using LLMs and then constructs a semantically coherent hierarchical structure of entities. It finally fine-tunes KG embeddings using this hierarchy and textual information. Experiments on benchmark datasets (FB15K-237, YAGO3-10, PrimeKG) demonstrate its effectiveness in link prediction.", "questions": "- The proposed method leverages entity names to prompt large language models (LLMs) to generate corresponding descriptions. How does it address the issue of multiple entities having the same name?\n\n- Please see other questions in \"Weaknesses\".", "limitations": "NA.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- Using LLMs to enhance KG embeddings can capture comprehensive features. The proposed method demonstrates improvements in link prediction when compared to selected baseline methods.\n\n- The presentation of the paper is good.", "weaknesses": "- There are several related studies on using LLMs to enhance text information in KGs [1,2], which, however, were not discussed in the paper. Additionally, constructing a hierarchy seems redundant given the existing graph structure of the KG. More discussion and explanation are needed.\n\n- The rationale behind the technical design is unclear. For instance, the proposed method concatenates structural and textual embeddings to construct the hierarchy, and then linearly combines these embeddings for fine-tuning. What are the underlying reasons for these choices?\n\n- The method may be computationally expensive due to the use of LLMs and hierarchical refinement. In my view, it is not profitable to use LLMs to achieve ~0.02 Hits@1 improvements in link prediction.\n\n- Several recent strong baselines for KG link prediction, such as NBFNet [3] and AdaProp [4], which both achieve over 0.32 Hits@1 on FB15K-237, are absent from the experiments. It remains uncertain whether the proposed method can still improve the two baselines.\n\n[1] Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, Enhong Chen: Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models. LREC/COLING 2024: 11956-11968\n\n[2] Dawei Li, Zhen Tan, Tianlong Chen, Huan Liu: Contextualization Distillation from Large Language Model for Knowledge Graph Completion. EACL (Findings) 2024: 458-477\n\n[3] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, Jian Tang: Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021: 29476-29490\n\n[4] Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, Bo Han: AdaProp: Learning Adaptive Propagation for Graph Neural Network based Knowledge Graph Reasoning. KDD 2023: 3446-3457", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.\n\n---\n\n> ### **[W1.1]** *\"There are several related studies on using LLMs to enhance text information in KGs [1,2] ...\"*\n\nWe acknowledge this oversight. In our latest draft, we have cited [1] and [2] in line 83:\n\n- \"Some methods [1,2] are proposed to improve the performance of the aforementioned methods by enhancing the text information of entities/relations in KGs.\"\n\nInterestingly, these methods are complementary to Step 1 of KG-FIT. We conducted additional experiments replacing our entity descriptions with those generated by MPIKGC [1] (E&S strategy) and Contextualization Distillation (CD) [2] (ED strategy). For CD, we averaged the embeddings of all descriptions generated for each entity. \n\nResults on FB15K-237 with RotatE and HAKE are shown in **Table A in our rebuttal PDF**, which demonstrates that KG-FIT can be further enhanced by incorporating these methods.\n\n\n> ### **[W1.2]** *\"Additionally, constructing a hierarchy seems redundant given the existing graph structure of the KG.\"*\n\nConstructing a hierarchy offers key benefits:\n\n1. **Different levels of abstraction:**\n   KGs represent direct relationships but not hierarchical relationships or varying abstraction levels. KG-FIT's hierarchy captures broader semantic relationships not directly encoded in the KG.\n   - *Example:* In a biomedical KG, \"Aspirin\" and \"Ibuprofen\" might be directly connected as \"pain relievers\". However, an LLM-constructed hierarchy could group them under \"NSAIDs\", then \"Analgesics\", and finally \"Pharmaceuticals\", providing a richer semantic context.\n2. **Incorporating external knowledge:**\n   The LLM-constructed hierarchy in KG-FIT incorporates external knowledge absent from the original KG, enriching entity representations.\n   - *Example:* \"Apple\" and \"Microsoft\" grouped under \"Tech Giants\", \"Consumer Electronics\", and \"Fortune 500 Companies\", incorporating broader market knowledge not explicit in the original KG.\n3. **Handling sparse connections and improved generalization:**\n   KGs often have sparse entity connections. Hierarchical structure bridges gaps between semantically related but unconnected entities and enhances generalization to unseen entities or relationships.\n   - *Example:* In a medical KG, \"Heart Disease\" and \"Diabetes\" might not be directly connected, but both could be grouped under \"Chronic Diseases\". An LLM-constructed hierarchy could further classify them under \"Cardiovascular Diseases\" and \"Metabolic Disorders\", respectively. This organization bridges gaps and generalizes treatment or risk factors shared among similar diseases, providing a richer context for inference.\n\nThese hierarchies provide valuable semantic context beyond the explicit KG structure, enhancing overall embedding representational power.\n\nWe have also analyzed the effect of hierarchy in **Table 3 and Figure 3 in the paper**.\n\n\n\n> ### **[W2]** *\"The proposed method concatenates structural and textual embeddings to construct the hierarchy, ...\"*\n\nThis is a misunderstanding. Our method involves two distinct steps:\n\n1. **Hierarchy construction**: We use **only textual information** - *no structural embeddings*. The enriched entity embedding $v_i$ is a **concatenation of entity name embedding $v^e_i$ and description embedding $v^d_i$** (Equation 1).\n\n2. **Fine-tuning**: Entity embedding $e_i$ is **initialized as a linear combination** of a random embedding $e'_i$ and sliced text embedding $v'_i$ (Equation 5). This allows the integration of *LLM-derived semantics while adapting to KG structure*.\n\nStructural information from the KG is used *only in the link prediction objective (Equation 8)*.\n\n\n\n> ### **[W3]** *\"Expensive use of LLMs and hierarchical refinement.\"*\n\nWe appreciate this concern, but we respectfully disagree for two main reasons:\n\n1. **Significant improvements**: In many cases the improvements are substantial. For example, KG-FIT-HAKE w/ LHR on PrimeKG achieves a ~0.07 improvement in Hits@1 over KG-FIT-HAKE w/ seed hierarchy.\n2. **One-time, reasonable cost**: The use of LLMs is limited to the hierarchy construction, which is a one-time preprocessing step. As detailed in Appendix H.2, even for large-scale KGs like YAGO3-10, the cost remains reasonable when considering the deployment of KG-FIT in real-world applications.\n\n\n\n> ### **[W4]** *\"Several recent strong baselines for KG link prediction, such as NBFNet [3] and AdaProp [4] ...\"*\n\nWe appreciate the suggestion to compare with these strong GNN-based baselines. However, KG-FIT is designed for embedding-based methods, offering better scalability, interpretability, and lower computational requirements. Adapting KG-FIT to NBFNet or AdaProp would require substantial changes and might lose these benefits.\n\nKG-FIT constructs a hierarchical structure of entity clusters using **static embeddings,** while NBFNet and AdaProp use **dynamic entity representations** through message passing. Integrating these fundamentally different architectures would essentially amount to developing a new hybrid model, beyond our current scope. \n\nNonetheless, we believe KG-FIT's underlying philosophy could inspire improvements in GNN-based methods in future work.\n\n---\n\n> ### **[Q1]** *\"Issue of multiple entities having the same name\"*\n\nThank you for this question.\n\nFirst, as shown in Fig. 7 in Appendix E.1, the prompt asks the LLM to generate descriptions with a \"hint\" (an entity description from the original KG dataset). This helps differentiate entities with the same name during this step. In our paper, only YAGO3-10 does not have such descriptions, but it does not have this issue.\n\nSecond, we can also mitigate this by using strategies like MPIKGC/CD (mentioned in W1.2). Feeding the LLM triples of entities from the training set helps provide context and differentiate entities within the KG.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "cnZg4RpkkJ", "reviewer": "Reviewer_G3dr", "summary": "Knowledge graphs (KGs) are essential for representing structured knowledge in various domains. They consist of entities and relations, forming a graph structure for efficient reasoning and knowledge discovery. Current knowledge graph embedding (KGE) methods create low-dimensional representations of these entities and relations but often overlook extensive open-world knowledge, limiting their performance. Pre-trained language models (PLMs) and large language models (LLMs) offer a broader understanding but are computationally expensive to fine-tune with KGs. To address these issues, the authors propose KG-FIT, a framework that incorporates rich knowledge from LLMs into KG embeddings without fine-tuning the LLMs. KG-FIT generates entity descriptions from LLMs, constructs a hierarchical structure of entities, and fine-tunes KG embeddings by integrating this hierarchy with textual embeddings. This approach enhances KG representations, combining global knowledge from LLMs with local KG knowledge, significantly improving link prediction accuracy on benchmark datasets.", "questions": "1. Could the authors reconsider the motivation behind their approach? \n\n2. Why haven’t the authors considered incorporating published LLM-based baselines?\n\n3. Why do the authors continue to use KG embedding methods for efficient reasoning and knowledge discovery when LLMs possess very strong reasoning abilities for knowledge discovery?", "limitations": "Please refer to my weakness", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. Extensive experiments. The authors compare experiment performance with 8 baselines on three datasets and apply on 8 KG embedding backbones.\n\n\n2. Extensive experimental details description. For example, the hardware environment for running the experiment, data processing, prompts for interacting with large models, and codes to reproduce their results.\n\n3. Clear figures and presentations.", "weaknesses": "1.  The motivation needs to be reconsidered. The authors mention that using KG to fine-tune LLMs is computationally expensive. Many current research efforts do not fine-tune LLMs with KG. Instead, they use retrieval-based methods to explicitly provide the knowledge.\n\n2. LLM-based baselines should be considered. The authors extensively use LLMs in their methods. They also should incorporate the LLM-based methods. For example, those LLMs retrieval-based methods. Only comparing with small LM-based methods is not enough.\n\n3. The contribution is kind of limited. I do not know why the authors still use KG embedding methods for efficient reasoning and knowledge discovery since LLMs have very strong reasoning ability for knowledge discovery.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.\n\n---\n\n> ### **[W1/Q1]** (Reconsidering motivation)\n\nIf the retrieval process here refers to the \"retriever\" mentioned in KICGPT [1], it's important to note that:\n\n- The retriever in [1] is actually a KG embedding (KGE) model (RotatE in their case) used to generate top-ranked candidate object entities $o$ given a query $(s, r, ?)$. The LLM is then used to re-rank these candidates.\n\n- In this case, the final results rely heavily on the candidate generation handled by the KGE methods. This means that re-ranking methods like [1, 2, 3] can be used as an add-on to KG-FIT, where KG-FIT provides more accurate candidates, improving the final results. We demonstrate this by implementing KICGPT with KG-FIT and show the results in **Table B of our rebuttal PDF**.\n\nHowever, if the retrieval process does not involve KGE methods:\n\n1. Link prediction is a crucial task for evaluating knowledge discovery ability. To the best of our knowledge, there are currently no methods that use LLM with retrieval for this task.\n2. Knowledge discovery on an existing KG typically requires the model to be fine-tuned on the KG. This is because (1) we need a comprehensive view of all possible object entities $o$ given a query $(s, r, ?)$, and (2) the model must learn underlying patterns from the existing knowledge.\n3. Traditional retrieval methods without KGE cannot be directly applied for link prediction because they do not provide a systematic way to rank all possible object entities for a given query. They typically retrieve a small subset of relevant entities based on text similarity, which may miss many valid candidates. In contrast, KGE methods like KG-FIT learn to embed the entire KG structure, enabling them to score and rank all possible object entities for a given query.\n\nWe will clarify these points in the revised paper to highlight the unique strengths of KG-FIT and its potential to complement retrieval-based methods.\n\n> ### **[W2/Q2]** (Incorporating LLM-based baselines)\n\nThank you for this suggestion. We have added two recent LLM-based baselines: KICGPT [1] and KG-LLM [4].\n\nThe results are shown in **Table B of our rebuttal PDF** and will be added to Table 2 in the revised paper. Note that we only implemented KG-LLM on FB15K-237 due to its high computational cost.\n\n> ### **[W3/Q3]** (Why KG embedding methods)\n\nWe appreciate this question as it allows us to clarify the unique advantages of our approach. Let's compare different methods:\n\n1. Fine-tuning-based methods:\n   - Pros:\n     - Can adapt LLMs to specific KG domains\n     - Leverage the vast knowledge and reasoning capabilities of LLMs\n   - Cons:\n     - Computationally expensive, especially for large LLMs\n     - May overfit to small KGs due to the large number of parameters\n     - Difficult to update as the KG evolves, requiring retraining\n2. Re-ranking (Retrieval)-based methods:\n   - Pros:\n     - Leverage LLM knowledge without expensive fine-tuning\n     - Computationally efficient for inference\n   - Cons:\n     - Rely on pre-existing KG embeddings for candidate generation\n     - May miss global patterns and relationships in the KG\n     - Limited by the quality of the retrieval process and the retrieved context\n3. KG-FIT and KG embedding methods:\n   - Pros:\n     - Capture the global structure and patterns of the entire KG\n     - Computationally efficient\n     - Easily updatable as new knowledge is added to the KG\n     - Provide interpretable entity and relation representations\n   - Cons:\n     - Require a well-designed model architecture and training process to effectively incorporate LLM knowledge\n\nTo illustrate the importance of KGE methods for knowledge discovery, let's consider the following example.\n\n> **[Example]** Suppose we have a movie knowledge graph where entities represent actors, movies, directors, and genres, and relations represent facts like \"acted_in\", \"directed_by\", and \"belongs_to_genre\". Given a query $(s, r, ?)$ where $s$ is a specific actor, $r$ is \"likely_to_collaborate_with\", and the goal is to predict another actor $o$ that $s$ is likely to work with in the future, a KGE model like KG-FIT can learn from the global KG structure to infer patterns such as \"actors who have worked with the same directors or in similar genres are more likely to collaborate\". This allows KG-FIT to make informed predictions about potential actor collaborations, even if those specific actors have never worked together before. In contrast, a non-fine-tuned LLM-based method might have difficulty making such inferences if the actors' previous collaborations are not explicitly mentioned in the training set/retrieved context, as it lacks the global understanding of the interconnected nature of the film industry that KGE methods can capture.\n\nIt is also important to note that **KG embeddings enhanced by KG-FIT are complementary to LLM/PLM fine-tuning and re-ranking methods**. For example, PKGC [2] and TagReal [3] apply both fine-tuning and re-ranking, but they heavily rely on the \"retrieved\" results from their backbone KG embeddings. KG-FIT can  improve the quality of these backbone embeddings, leading to better overall performance.\n\nMoreover, KG-FIT strikes a balance between leveraging the strengths of LLMs and maintaining the desirable properties of KG embeddings. By incorporating LLM knowledge into the embedding space, KG-FIT can capture more semantic information and complex relationships while still being computationally efficient and interpretable.\n\n---\n\n**References**\n\n[1] (EMNLP 2023) KICGPT: Large language model with knowledge in context for knowledge graph completion.\n\n[2] (ACL 2022) Do pre-trained models benefit knowledge graph completion?\n\n[3] (ACL 2023) Text-augmented open knowledge graph completion via pre-trained language models.\n\n[4] (arXiv) Exploring large language models for knowledge graph completion.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "3CRB1OzGv4", "reviewer": "Reviewer_dKZ2", "summary": "This paper addresses the limitations of existing KGE models that focus either on graph structure or fine-tuning pre-trained language models. It introduces KG-FIT, which leverages LLM-guided refinement to incorporate hierarchical and textual knowledge, effectively capturing both global and local semantics. Experiments on benchmark datasets demonstrate KG-FIT's superiority, achieving significant performance improvements over state-of-the-art methods.", "questions": "Please refer to the weaknesses.", "limitations": "N/A", "rating": 4, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "1. The proposed method can automatically construct a semantically coherent entity hierarchy using agglomerative clustering and LLM-guided refinement, which is an interesting topic.\n2. The authors provide detailed illustrations for extensive empirical study on benchmark datasets and demonstrate significant improvements in link prediction accuracy.", "weaknesses": "1. The paper is not organized clearly, which is not friendly for understanding. For example, there is a lack the sensitivity study for the hyperparameters in the loss function. \n2. The comparable methods are old and lack the new ones in the last 2 years such as [1][2][3][4]. The performance is not comparable with the previous work.\n[1] Compounding Geometric Operations for Knowledge Graph Completion \n[2] Geometry Interaction Knowledge Graph Embeddings\n[3] KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion\n[4] Dual Quaternion Knowledge Graph Embeddings\n3. The paper lacks the analysis of time complexity as well as space complexity, which is necessary to study the efficiency of the model. \n4. There are some typos and It is commanded that the writing needs to be improved. \n(1) On page 5, line 163 “determined by lowest common ancestor” should be “determined by the lowest common ancestor” \n(2) On page 6, line 195 “is sigmoid function” should be “is the sigmoid function”", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your thoughtful review. We appreciate your recognition of our work's strengths and we address your concerns as follows.\n\n---\n\n> ### **[W1]** *\"there is a lack of the sensitivity study for the hyperparameters in the loss function\"*\n\nWe appreciate the reviewer's concern about the lack of a sensitivity study. To address this, we conducted a sensitivity analysis, presented in **Figure A in our rebuttal PDF**. We have also presented our hyperparameter study in **Appendix I in our paper**.\n\nFigure A demonstrates KG-FIT's robustness to hyperparameter variations:\n\n1. Left plot (λ1, λ2, λ3): Performance metrics remain stable across different combinations, indicating that KG-FIT is not overly sensitive to these hierarchical clustering constraint parameters.\n\n2. Right plot (ζ1, ζ2, ζ3): While there's some variation, performance remains consistently high over a range of ratios for these loss function component weights.\n\nThese results show that KG-FIT maintains good performance across various hyperparameter settings, addressing the reviewer's concern and suggesting that the model can be readily applied to new datasets without requiring extensive tuning.\n\n> ### **[W2]** *\"The comparable methods are old and lack the new ones in the last 2 years\"*\n\nThank you for suggesting these recent baselines. We have conducted additional experiments to compare KG-FIT with CompoundE [1], GIE [2], and DualE [4]. The results are presented in **Table C in our rebuttal PDF**.\n\nThese results demonstrate that KG-FIT not only compares favorably with but substantially improves upon recent state-of-the-art KGE methods. This underscores KG-FIT's effectiveness in leveraging LLM knowledge to enhance various KGE architectures.\n\nRegarding KRACL [3], as it employs a GNN-based approach, integrating it with KG-FIT presents unique challenges. We have added the exploration of KG-FIT's integration with GNN-based methods to our future work list.\n\nWe appreciate the reviewer's suggestion to include these recent baselines, as it has allowed us to further demonstrate KG-FIT's capabilities and versatility.\n\n\n\n> ### **[W3]** *\"The paper lacks the analysis of time complexity as well as space complexity\"*\n\nWe appreciate the reviewer's attention to this important aspect of our model. Here's a clarification:\n\n**For time complexity:**\nWe have analyzed KG-FIT's time complexity both theoretically (Lines 206-210) and empirically (Table 4, Lines 296-303). Our results demonstrate that KG-FIT is 12 times faster in training than the best PLM-based method, while maintaining inference speed comparable to backbone KGE methods.\n\n**For space complexity:**\nWhile we didn't explicitly state it in the paper, the space complexity of KG-FIT in terms of trainable parameters is the same as the backbone KGE models:\n\n$O(|E| * n + |R| * m)$\n\nWhere $|E|$ is the number of entities, $|R|$ is the number of relations, $n$ is the entity embedding dimension, and $m$ is the relation embedding dimension.\n\nThis is because the main trainable components of KG-FIT are:\n\n1. Entity embeddings: $O(|E| * n)$\n2. Relation embeddings: $O(|R| * m)$\n\nThe additional components introduced by KG-FIT (entity text embeddings and cluster embeddings) are not trainable parameters, but fixed inputs used during the forward pass. They do consume memory during runtime but do not increase the model's parameter count.\n\nThis space complexity is significantly lower than PLM-based methods, which often require gigabytes of memory for model parameters alone. For example, on the FB15K-237 dataset, KG-FIT's trainable parameters would only occupy approximately 60MB of memory (assuming 32-bit floating-point numbers and $n = m = 1024$).\n\n> ### **[W4]** (Typos in the paper)\n\nWe appreciate the reviewer's attention to detail. We will correct these typos:\n\n1. Page 5, line 163: \"determined by the lowest common ancestor\"\n2. Page 6, line 195: \"is the sigmoid function\"\n\nWe will thoroughly proofread the entire manuscript to improve clarity and precision. Thank you for helping us enhance the quality of our paper.\n\n\n\n\n\n---\n\n**References**\n\n[1] (ACL 2023) Compounding Geometric Operations for Knowledge Graph Completion.\n\n[2] (AAAI 2022) Geometry Interaction Knowledge Graph Embeddings.\n\n[3] (WWW 2023) KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion.\n\n[4] (AAAI 2021) Dual Quaternion Knowledge Graph Embeddings.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "Obeic2HHB4", "reviewer": "Reviewer_9oop", "summary": "This paper introduces KG-FIT, a general framework that enhances the expressiveness of existing Knowledge Graph Embedding (KGE) models by integrating LLMs. KG-FIT contains four key steps: First, it utilizes an LLM to generate descriptions for a set of given entities, forming an enriched entity representation by concatenating the entity's embedding with its description. Second, it constructs a semantically coherent seed hierarchical structure. Third, it leverages the real-world entity knowledge captured by the LLM to refine this hierarchical structure. Finally, it fine-tunes the knowledge graph embeddings by integrating the hierarchical structure with textual embeddings. Extensive experiments validate the effectiveness of KG-FIT.", "questions": "1. It is unclear the explicit description of $P$, $L$, $R$ in line 134.\n2. It remains unclear whether the efficiency evaluation of training time contains the duration required for seed hierarchy construction and LLM-guided hierarchy refinement stages.", "limitations": "The authors have discussed the limitations of their work in the paper.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The motivation of the proposed KG-FIT is clear and the paper is well-structured. \n2. Extensive experimental results demonstrate that KG-FIT can improve most KGE baseline models.\n3. Codes are provided for reproducibility.", "weaknesses": "1. The performance of KG-FIT heavily relies on the LLM used for generating entity descriptions and guiding the refinement of the seed hierarchy refinement. If the LLM lacks comprehensive real-world entity knowledge about the given entities or domains, the resulting embeddings may not suboptimal. Furthermore, this dependency could make KG-FIT less effective in the situations where LLM have limited coverage.\n2. If the LLM has limited coverage of a specific domain, the LLM guided hierarchy refinement process may yield incorrect results, potentially distorting rather than enhancing the structure of the well-formed seed hierarchy.\n3. The paper does not justify the selection of agglomerative hierarchical clustering and the use of silhouette score. Additional ablation studies will enhance this work.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's thoughtful comments and their recognition of our work's strengths. We address their concerns as follows:\n\n---\n\n> ### **[W1/W2]** (Dependency on LLM knowledge and potential issues with limited domain coverage)\n\nWe appreciate the reviewer's concern about KG-FIT's reliance on LLM knowledge. While we view this as a strength that allows our method to leverage vast and continually improving knowledge, we acknowledge the potential limitations in highly specialized domains. \n\nTo address this, we propose several mitigating strategies:\n\n1. Incorporating external knowledge: We can enhance LLM performance in specialized domains by using Retrieval-Augmented Generation (RAG) to incorporate domain-specific external knowledge bases during the description generation process.\n\n2. Leveraging KG context: For domains where external knowledge is limited, we can use the context within the Knowledge Graph itself to generate more informative entity descriptions. This approach ensures that even without extensive domain knowledge, the LLM can still provide useful descriptions based on the relationships and attributes present in the KG.\n\n3. Fallback to seed hierarchy: In cases where the LLM truly lacks domain-specific knowledge, our results show that the seed hierarchy alone still significantly improves KG embeddings. As demonstrated in Table 2 of our paper, KG-FIT with just the seed hierarchy (before LLM refinement) consistently outperforms base models across all datasets.\n\n4. Domain-specific LLMs: When available, using domain-specific LLMs can provide more accurate and relevant knowledge for specialized fields.\n\nThese strategies ensure that KG-FIT remains effective and adaptable across a wide range of domains, from general knowledge to highly specialized fields. Our experiments on diverse datasets (FB15K-237, YAGO3-10, PrimeKG) demonstrate KG-FIT's robustness and broad applicability, even when dealing with domain-specific knowledge.\n\nIn future work, we plan to explore methods for automatically selecting the most appropriate strategy based on the domain and available resources, further enhancing KG-FIT's versatility and performance.\n\n> ### **[W3]** (Justification for agglomerative clustering and silhouette score)\n\nWe appreciate the reviewer's suggestion for additional justification. We chose agglomerative clustering for its natural ability to create a hierarchical structure without pre-specifying the number of clusters, which is crucial for our approach. The silhouette score was selected as it balances both cluster cohesion and separation, providing a robust measure of clustering quality.\n\nTo address the reviewer's concern, we will conduct additional ablation studies comparing different clustering methods:\n\n1. Top-down agglomerative clustering: This variant will start with all entities in one cluster and progressively split them, potentially offering a different perspective on the hierarchy.\n2. K-means: We will implement a recursive K-means process, where we first cluster all entities, then recursively apply K-means to each resulting cluster until a stopping criterion is met (e.g., cluster size or maximum depth). This will create a top-down hierarchical structure.\n3. DBSCAN: We will use a similar recursive approach as with K-means, but DBSCAN's ability to detect noise points will allow us to create a hierarchy that potentially captures outliers at higher levels.\n\nFor evaluation metrics, alongside the silhouette score, we will also compare the Calinski-Harabasz index and Davies-Bouldin index.\n\nWe will include these results in the appendix of our revised paper, providing a comprehensive comparison of different clustering methods and their impact on KG-FIT's performance.\n\n---\n\n> ### **[Q1]** (Clarification on P, L, R in line 134)\n\nWe apologize for the lack of clarity. P, L, and R refer to Parent, Left child, and Right child, respectively. We will add this explanation to the paper for better understanding.\n\n> ### **[Q2]** (Clarification on efficiency evaluation)\n\nThe reported training time in Table 4 indeed focuses on the fine-tuning stage for a fair comparison with baselines. The hierarchy construction (Steps 1-3) is a one-time preprocessing step. For transparency, we will add the following breakdown to Appendix H:\n\n- Entity description & text embedding generation (with 15 threads): ~10 minutes for FB15K-237, ~1 hour for YAGO3-10, ~8 minutes for PrimeKG\n- Seed hierarchy construction: ~2 minutes for FB15K-237, ~8 minutes for YAGO3-10, ~1.5 minutes for PrimeKG\n- LLM-guided refinement (with 15 threads): ~12 minutes for FB15K-237,  ~1 hour for YAGO3-10, ~10 minutes for PrimeKG\n\nThese preprocessing times are relatively small compared to the overall training process, especially considering they're one-time operations that can be reused for multiple experiments or model variations. Moreover, the LLM-guided refinement step shows good scalability with parallel processing, which can further reduce preprocessing time for larger datasets.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
Dear Reviewers,

We sincerely thank you for your thoughtful and constructive feedback on our submission "Knowledge Graph Fine-Tuning Upon Open-World Knowledge from Large Language Models". We deeply appreciate the time and effort you've invested in reviewing our work.

> **[Strengths of Our Work]**

We are grateful that multiple reviewers recognized several strengths of our work:
1. The effectiveness of KG-FIT in leveraging knowledge from LLMs to significantly enhance KG embeddings for the link prediction task (All Reviewers).
2. The extensive experimental evaluation, including comparisons with numerous baselines across multiple datasets and KG embedding backbones (Reviewers *G3dr*, *dKZ2*, and *9oop*).
3. The clear presentation, including well-structured paper organization and clear figures (Reviewers *hiNn*, *G3dr*, and *9oop*).
4. The provision of code for reproducibility and detailed experimental setup descriptions (Reviewers *G3dr* and *9oop*).

> **[Our Responses to Weaknesses]**

We acknowledge the concerns raised and have addressed them in our **individual rebuttals**. We have also attached a **Rebuttal PDF** which includes several tables and a figure showing new experimental results to address reviewers' concerns. Here's an overview of its content and key findings:

1. Table A: Performance of KG-FIT augmented by LLM-based KG textual information enhancement methods (MPIKGC and CD) on FB15K-237.
   * Key finding: KG-FIT can be further enhanced by incorporating improved textual information, showing consistent improvements across all metrics.

2. Table B: Comparison with additional LLM-based baselines (KG-LLM and KICGPT) on FB15K-237 and WN18RR.
   * Key finding: KG-FIT outperforms KG-LLM and significantly enhances the performance of KICGPT when used as its backbone retriever, demonstrating its effectiveness in combining with LLM-based methods.

3. Figure A: Sensitivity analysis demonstrating KG-FIT-HAKE's robustness to hyperparameter variations.
   * Key finding: KG-FIT maintains stable performance across various hyperparameter settings, indicating its robustness and potential for easy adaptation to new datasets.

4. Table C: Comparison with additional recent KG embedding baselines (CompoundE, GIE, and DualE) across FB15K-237, YAGO3-10, and PrimeKG.
   * Key finding: KG-FIT consistently enhances the performance of these state-of-the-art KG embedding methods across all datasets, achieving new state-of-the-art results.

We have provided detailed, point-by-point responses to each reviewer's specific concerns in our individual rebuttals. We encourage you to refer to these for in-depth discussions on particular aspects of our work. We are committed to incorporating your valuable feedback to further improve our paper.

Thank you again for your time and expertise in reviewing our submission.

Meta-review: This is a paper on KG embeddings, where entity descriptions from LLMs are used to improve the embedding training. The paper yielded 4, 5, 5, 6. One reviewer who gave a 4 rating did not engage in discussion, whereas another later increased their score to 5. I have looked at author response for both these and other reviewers. Authors have provided additional experiments to provide good evidence for issues raised (e.g., comparison of other LLM-based enhancement approaches, comparison with other LLM-based baselines, experiments over more recent models, other clarifications especially regarding why do the research at all and rather simply use LLMs)

The positive reviewer also has interesting points such as what if LLM doesnt know these entities. Authors give some high level response to this question, but do not perform any experiments on the robustness of the model in such a case. This is an important question, and some experiments may have benefited the paper. Unfortunately, none(?) of the previous papers that combine LLMs with structure have done this experiment either... so while interesting, should not become the sole reason for rejection.

Finally, one criticism which is not addressed adequately is that some other papers like NBFNet (and more recently this https://arxiv.org/pdf/2311.03780) combine GNNs with LLMs, and have excellent performance on one of the hard datasets: (FB15K-237, 43 MRR, 33 Hits@1). The authors' results are much weaker. This is because the authors combine static embedding based models and not GNN-architectures that have additional power because they compute dynamic embeddings for a node in a query dependent way.  Authors relegate this combination (of their approach with GNNs) to future work. 

Overall, this is a paper where there aren't that many things wrong. Its a decent new idea, and good extensive experiments on a subset of models, leading to improvements compared to those models... not always generating a state of the art performance. I personally do not believe that a paper must only produce state of the art performance to get accepted. The idea of creating a hierarchy of entities and using that to train KGs is somewhat interesting. 

Is the paper groundbreaking? No. But, it will be a decent addition to NeurIPS if there is space.
Decision: Accept (poster)