KG-FIT: Knowledge Graph Fine-Tuning Upon Open-World Knowledge

Authors: pengcheng  jiang
lang  cao
cao  xiao
parminder  bhatia
jimeng  sun
jiawei  han
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 21:59:19

Abstract: Knowledge Graph Embedding (KGE) techniques are crucial in learning compact representations of entities and relations within a knowledge graph, facilitating efficient reasoning and knowledge discovery. While existing methods typically focus either on training KGE models solely based on graph structure or fine-tuning pre-trained language models with classification data in KG, KG-FIT leverages LLM-guided refinement to construct a semantically coherent hierarchical structure of entity clusters. By incorporating this hierarchical knowledge along with textual information during the fine-tuning process, KG-FIT effectively captures both global semantics from the LLM and local semantics from the KG. Extensive experiments on the benchmark datasets FB15K-237, YAGO3-10, and PrimeKG demonstrate the superiority of KG-FIT over state-of-the-art pre-trained language model-based methods, achieving improvements of 14.4\%, 13.5\%, and 11.9\% in the Hits@10 metric for the link prediction task, respectively. Furthermore, KG-FIT yields substantial performance gains of 12.6\%, 6.7\%, and 17.7\% compared to the structure-based base models upon which it is built. These results highlight the effectiveness of KG-FIT in incorporating open-world knowledge from LLMs to significantly enhance the expressiveness and informativeness of KG embeddings.

Main paper: 
 introduction: 
 Knowledge graph (KG) is a powerful tool for representing and storing structured knowledge, with applications spanning a wide range of domains, such as question answering [1, 2, 3, 4] , recommendation systems [5, 6, 7] , drug discovery [8, 9, 10] , and clinical prediction [11, 12, 13] . Constituted by entities and relations, KGs form a graph structure where nodes denote entities and edges represent relations among them. To facilitate efficient reasoning and knowledge discovery, knowledge graph embedding (KGE) methods [14, 15, 16, 17, 18, 19, 20, 21] have emerged, aiming to derive low-dimensional vector representations of entities and relations while preserving the graph's structural integrity.While current KGE methods have shown success, many are limited to the graph structure alone, neglecting the wealth of open-world knowledge surrounding entities not explicitly depicted in the KG, which is manually created in most cases. This oversight inhibits their capacity to grasp the complete semantics of entities and relations, consequently resulting in suboptimal performance across downstream tasks. For instance, a KG might contain entities such as "Albert Einstein" and "Theory of Relativity", along with a relation connecting them. However, the KG may lack the rich context and background information about Einstein's life, his other scientific contributions, and the broader impact of his work. In contrast, pre-trained language models (PLMs) and LLMs, having been trained on extensive literature, can provide a more comprehensive understanding of Einstein and his legacy beyond the limited scope of the KG. While recent studies have explored fine-tuning PLMs with KG triples [22, 23, 24, 25, 26, 27, 28, 29] , this approach is subject to several limitations. Firstly, the training and inference processes are computationally expensive due to the large number of parameters in PLMs, making it challenging to extend to more knowledgeable LLMs. Secondly, the fine-tuned PLMs heavily rely on the restricted knowledge captured by the KG embeddings, limiting their ability to fully leverage the extensive knowledge contained within the language models themselves. As a result, these approaches may not adequately capitalize on the potential of LLMs to enhance KG representations, as illustrated in Fig. 1 . Lastly, small-scale PLMs (e.g., BERT) contain outdated and limited knowledge compared to modern LLMs, requiring re-training to incorporate new information, which hinders their ability to keep pace with the rapidly evolving nature of today's language models.To address the limitations of current approaches, we propose KG-FIT (Knowledge Graph FIne-Tuning), a novel framework that directly incorporates the rich knowledge from LLMs into KG embeddings without the need for fine-tuning the LMs themselves. The term "fine-tuning" is used because the initial entity embeddings are from pre-trained LLMs, initially capturing global semantics.KG-FIT employs a two-stage approach: (1) generating entity descriptions from the LLM and performing LLM-guided hierarchy construction to build a semantically coherent hierarchical structure of entities, and (2) fine-tuning the KG embeddings by integrating knowledge from both the hierarchical structure and textual embeddings, effectively merging the open-world knowledge captured by the LLM into the KG embeddings. This results in enriched representations that integrate both global knowledge from LLMs and local knowledge from KGs.The main contributions of KG-FIT are outlined as follows:(1) We introduce a method for automatically constructing a semantically coherent entity hierarchy using agglomerative clustering and LLM-guided refinement. (2) We propose a fine-tuning approach that integrates knowledge from the hierarchical structure and pre-trained text embeddings of entities, enhancing KG embeddings by incorporating open-world knowledge captured by the LLM. (3) Through an extensive empirical study on benchmark datasets, we demonstrate significant improvements in link prediction accuracy over state-of-the-art baselines. 
related work: 
 Structure-Based Knowledge Graph Embedding. Knowledge Graph Embedding methods that rely solely on graph structure aim to learn low-dimensional vector representations of entities and relations while preserving the graph's structural properties. TransE [14] models relations as translations in the embedding space. DistMult [15] is a simpler model that uses a bilinear formulation for link prediction.ComplEx [16] extends TransE to the complex domain, enabling the modeling of asymmetric relations. ConvE [17] employs a convolutional neural network to model interactions between entities and relations. TuckER [18] utilizes a Tucker decomposition to learn embeddings for entities and relations jointly. RotatE [19] represents relations as rotations in a complex space, which can capture various relation patterns, and HAKE [21] models entities and relations in an implicit hierarchical and polar coordinate system. These structure-based methods have proven effective in various tasks [30] but do not leverage the rich entity information available outside the KG itself.PLM-Based Knowledge Graph Embedding. Recent studies have explored integrating pre-trained language models (PLMs) with knowledge graph embeddings to leverage the semantic information Step 2: Hierarchical clustering is applied iteratively to all entity embeddings over various distance thresholds, monitored by a Silhouette scorer to identify optimal clusters, thus constructing a seed hierarchy where each leaf node represents a cluster of semantically similar entities.Step 3: Leveraging LLM guidance, the seed hierarchy is iteratively refined bottom-up through a series of suggested actions, aiming for a more accurate organization of KG entities with LLM's knowledge.Step 4: Use the refined hierarchy along with KG triples and the initial entity embeddings to fine-tune the embeddings under a series of distance constraints.captured by PLMs. KG-BERT [22] , PKGC [28] , TagReal [29] , and KG-LLM [31] train PLMs/LLMs with a full set of classification data and prompts. However, these approaches are computationally expensive due to the need to iterate over all possible positive/negative triples. LMKE [26] and SimKGC [27] adopt contrastive learning frameworks to tackle issues like expensive negative sampling and enable efficient learning for text-based KGC. KG-S2S [24] and KGT5 [25] employ sequenceto-sequence models to generate missing entities or relations in the KG. StAR [23] and CSProm-KG [32] fuse embeddings from graph-based models and PLMs. However, they are limited to small-scale PLMs and do not leverage the hierarchical and clustering information reflecting the LLM's knowledge of entities. Fully LLM prompting-based methods [33, 34] are costly and not scalable. In contrast, our proposed KG-FIT approach can be applied to any LLM, incorporating its knowledge through a semantically coherent hierarchical structure of entities. This enables efficient exploitation of the extensive knowledge within LLMs, while maintaining the efficiency of structure-based methods.All the children of P l and P r are merged into the grandparent cluster P * , resulting in P ′ * = {C l 1 , C l 2 , C l 3 , . . . , C l |P l | , C r 1 , C r 2 , CThe LLM is tasked with analyzing the entities to determine if they can be grouped into distinct subclusters based on common attributes like characteristics, themes, or genres.Given entities from the cluster '{cluster_name}'.Analyze the entities and determine if they can be grouped into distinct and meaningful sub-clusters based on their characteristics, themes, or genres. If sub-clusters can be formed, provide a clear and concise name for each sub-cluster that represents the common attribute of its entities. Each sub-cluster should be given a new name that uniformly describes its entities. There needs to be differentiation in the names of different clusters.The number of sub-clusters should be two.If the entities are already well-grouped and don't require further subclustering, simply provide the original cluster.Provide the output in the following JSON format: ```json {{ "Sub-cluster 1 Name": ["Entity 1", "Entity 2", ...], "Sub-cluster 2 Name": ["Entity 3", "Entity 4", ...] }} ``È xample: Cluster: Movies Entities: The Godfather, The Shawshank Redemption, The Dark Knight, Forrest Gump, Inception, The Matrix Output: {{ "Drama": ["The Godfather", "The Shawshank Redemption", "Forrest Gump"], "Action": ["The Dark Knight", "Inception", "The Matrix"] }} Cluster: {cluster_name} Entities: {entities} Output: 3. Sub-cluster Naming: If subclusters are formed, the LLM provides clear and concise names for each subcluster that represent the common attributes of their entities. Each subcluster is given a unique name that uniformly describes its entities, ensuring differentiation between clusters. 4. Control of Sub-cluster Count: The prompt instructs the LLM to control the number of subclusters between 1 and 5. If the entities are already well-grouped, no further sub-clustering is needed, and the original cluster is returned.The output format ensures structured and consistent results, facilitating easy integration into the hierarchy refinement process.Prompt for Bottom-Up Refinement (P REFINE ): This prompt guides the LLM in refining the parentchild triples within the hierarchy. The refinement process involves several key steps:1. Input Clusters: Two clusters, A and B, along with their entities, are provided as input.The LLM analyzes the two clusters and their entities to determine the most appropriate update mode. The alignment of update modes to the actions described in the methodology section is as follows:• Update Mode 1 (Create New Cluster C): Aligns with NO UPDATE. These two clusters cannot be merged, and no cluster belongs to any other.  
kg-fit framework: 
 We present KG-FIT (as shown in Fig. 2 ), a framework for fine-tuning KG embeddings leveraging external hierarchical structures and textual information based on open knowledge. This framework comprises two primary components: (1) LLM-Guided Hierarchy Construction: This phase establishes a semantically coherent hierarchical structure of entities, initially constructing a seed hierarchy and then refining it using LLM-guided techniques, and (2) Knowledge Graph Fine-Tuning: This stage enhances the KG embeddings by integrating the constructed hierarchical structure, textual embeddings, and multiple constraints. The two stages combined to enrich KG embeddings with open-world knowledge, leading to more comprehensive and contextually rich representations. Below we present more tecnical details. A table of notations is placed in Appendix L.We initiate the process by constructing a hierarchical structure of entities through agglomerative clustering, subsequently refining it using an LLM to enhance semantic coherence and granularity.Step 1: Entity Embedding Initialization is the first step of this process, where we are given a set of entities E = {e 1 , . . . , e |E| } within a KG, and will enrich their semantic representations by generating descriptions using an LLM. Specifically for each entity e i , we prompt the LLM with a template (e.g., Briefly describe [entity] with the format "[entity] is a [description]".(detailed in Appendix E.1)) prompting it to describe the entity from the KG dataset, thereby yielding a concise natural language description d i . Subsequently, the entity embedding v e i ∈ R dim(f ) and description embeddingv d i = f (d i ) ∈ R dim(f )are obtained using an embedding model f and concatenated to form the enriched entity representation v i :EQUATIONStep 2: Seed Hierarchy Construction follows after entity embedding initialization. Here we choose agglomerative hierarchical clustering [35] over flat clustering methods like K-means [36] for establishing the initial hierarchy. This choice is based on the robust hierarchical information provided by agglomerative clustering, which serves as a strong foundation for LLM refinement. Using this hierarchical structure reduces the need for numerous LLM iterations to discern relationships between flat clusters, thereby lowering computational costs and complexity. Agglomerative clustering balances computational efficiency with providing the LLM a meaningful starting point for refinement. The clustering process operates on enriched entity representations V = {v 1 , . . . , v l } ∈ R |E|×2dim(f ) using cosine distance and average linkage. The optimal clustering threshold τ * is determined by maximizing the silhouette score [37] S * across a range of thresholds [τ min , τ max ] ∈ [0, 1]:EQUATIONwhere labels τ are the clustering results at threshold τ . This ensures that the resulting clusters are compact and well-separated based on semantic similarity. The constructed hierarchy forms a fully binary tree where each leaf node represents an entity. We use a top-down algorithm (detailed in Appendix F.1) to replace the first encountered entity with its cluster based on the optimal threshold τ optim . This process eliminates other entity leaves within the same cluster, forming the seed hierarchy H seed , where each leaf node is a cluster of entities defined by labels τoptim .Step 3: LLM-Guided Hierarchy Refinement (LHR) is then applied to improve the quality of the knowledge representation. As the seed hierarchy H seed is a binary tree, which may not optimally represent real-world entity knowledge, we further refine it using the LLM. The LLM transforms the seed hierarchy into the LLM-guided refined hierarchy H LHR through actions described below:i. Cluster Splitting: For each leaf cluster C original ∈ C leaf (H seed ), the LLM recursively splits it into two subclusters using the prompt P SPLIT (Fig. 8 in Appendix E.2):EQUATIONwhereC i = {e i 1 , e i 2 , . . . , e i |Ci| }, k i=1 |C i | = |C original | = |C split |, kis the total number of subclusters after recursively splitting C original in a binary manner. This procedure iterates until LLM indicates no further splitting or each subcluster has minimal entities, resulting in an intermediate hierarchy H split .ii. Bottom-Up Refinement: In the bottom-up refinement phase, the LLM iteratively refines the intermediate hierarchy H split produced by the cluster splitting step. The refinement process starts from the leaf level of the hierarchy and progresses upwards, considering each parent-child triple (P * , P l , P r ), where P * represents the grandparent cluster, and P l and P r represent the left and right child clusters of P * , respectively. Let {C l 1 , C l 2 , C l 3 , . . . , C l |P l | } denote the children of P l , and {C r 1 , C r 2 , C r 3 , . . . , C r |Pr| } denote the children of P r . For each parent-child triple, the LLM is prompted with P REFINE (P * , P l , P r ) (Fig. 9 in Appendix E.2), which provides the names and entities of the grandparent and child clusters. The LLM then suggests a refinement action to update the triple based on its understanding of the relationships between the clusters. The refinement options are:1. NO UPDATE: The triple remains unchanged, i.e., (P ′ * , P ′ l , P ′ r ) = (P * , P l , P r ).Step 4: KG-FIT fine-tunes the knowledge graph embeddings by incorporating the hierarchical structure, text embeddings, and three main constraints: the hierarchical clustering constraint, text embedding deviation constraint, and link prediction objective.To integrate the initial text embeddings (v i ∈ R dim(f ) ) into the model, the entity embedding e i ∈ R n is initialized as a linear combination of a random embedding e ′ i ∈ R n and the sliced text embeddingv ′ i = [v e i [: n 2 ]; v d i [: n 2 ]] ∈ R n .The relation embeddings, on the other hand, are initialized randomly:EQUATION)where ρ is a hyperparameter controlling the ratio between the random embedding and the sliced text embedding. r j ∈ R m is the embedding of relation j, and ψ is a hyperparameter controlling the standard deviation of the normal distribution N . This initialization ensures that the entity embeddings start close to their semantic descriptions but can still adapt to the structural information in the KG during training. The random initialization of relation embeddings allows the model to flexibly capture the structural information and patterns specific to the KG.The hierarchical constraint integrates the structure and relationships derived from the adaptive agglomerative clustering and LLM-guided refinement process. This optimization enhances the embeddings for hierarchical coherence and distinct semantic clarity.The revised constraint consists of three tailored components:L hier = ei∈E λ 1 d(e i , c) Cluster Cohesion -λ 2 C ′ ∈Sm(C) d(e i , c ′ ) |S m (C)| Inter-level Cluster Separation -λ 3 h-1 j=1 β j (d(e i , p j+1 ) -d(e i , p j )) h -1Hierarchical Distance Maintenance (5) where: e i and e i represent the entity and its embedding. C is the cluster that entity e i belongs to. S m (C) represents the set of neighbor clusters of C where m is the number of nearest neighbors (determined by lowest common ancestor (LCA) [38] in the hierarchy). c and c ′ denote the cluster embeddings of C and C ′ , which is computed by averaging all the embedding of entities under them (i.e., c = 1|C| ei∈C e i ∈ R n ). p j and p j+1 are the embeddings of the parent nodes along the path from the entity (at depth h) to the root, indicating successive parent nodes in ascending order. Each parent node is computed by averaging the cluster embeddings under it (p= 1 |P | Ci∈P c i ∈ R n ). d(•, •)is the distance function used to measure distances between embeddings. As higher levels of abstraction encompass a broader range of concepts, and thus a strict maintenance of hierarchical distance may be less critical at these levels, we introduce β j = β 0 • e -ϕj where β 0 is the initial weight for the closest parent, typically a larger value, ϕ is the decay rate, a positive constant that dictates how rapidly the importance decreases. λ 1 , λ 2 , and λ 3 are hyperparameters. In Eq 5, Inter-level Cluster Separation aims to maximize the distance between an entity and neighbor clusters, enhancing the differentiation and reducing potential overlap in the embedding space. This separation ensures that entities are distinctly positioned relative to non-member clusters, promoting clearer semantic divisions. Hierarchical Distance Maintenance encourages the distance between an entity and its parent nodes to be proportional to their respective levels in the hierarchy, with larger distances for higher-level parent nodes. This reflects the increasing abstraction and decreasing specificity, aligning the embeddings with the hierarchical structure of the KG. Cluster Cohesion enhances intra-cluster similarity by minimizing the distance between an entity and its own cluster center, ensuring that entities within the same cluster are closely embedded, maintaining the integrity of clusters.Semantic Anchoring Constraint: To preserve the semantic integrity of the embeddings, we introduce the semantic anchoring constraint, which is formulated as:EQUATIONwhere E is the set of all entities, e i is the fine-tuned embedding of entity e i , v ′ i is the sliced text embedding of entity e i , and d(•, •) is a distance function. This constraint is crucial for large clusters, where the diversity of entities may cause the fine-tuned embeddings to drift from their original semantic meanings. This is also important when dealing with sparse KGs, as the constraint helps prevent overfitting to the limited structural information available. By acting as a regularization term, it mitigates overfitting and enhances the robustness of the embeddings [39] .Score Function-Based Fine-Tuning: KG-FIT is a general framework applicable to existing KGE models [14, 15, 16, 17, 18, 19, 20] . These models learn low-dimensional vector representations of entities and relations in a KG, aiming to capture the semantic and structural information within the KG itself. In our work, we perform link prediction to enhance the model's ability to accurately predict relationships between entities within the KG. Its loss is defined as:EQUATIONwhere D is the set of all triples in the KG, σ is sigmoid function. f r (•, •) is the scoring function (detailed in Appendix G and K) defined by the chosen KGE model that measures the compatibility between the head entity embedding e i and the tail entity embedding e j given the relation r, N j is the set of negative tail entities sampled for the triple (e i , r, e j ), e nj is the embedding of the negative tail entity n j , and γ is a margin hyperparameter. The link prediction-based fine-tuning minimizes the scoring function for the true triples (e i , r, e j ) while maximizing the margin between the scores of true triples and negative triples (e i , r, n j ). This encourages the model to assign higher scores to positive (true) triples and lower scores to negative triples, thereby enriching the embeddings with the local semantics in KG.Training Objective: The objective function of KG-FIT integrates three constraints:EQUATION)where Metrics. Following previous works, we use Mean Rank (MR), Mean Reciprocal Rank (MRR), and Hits@N (H@N) to evaluate link prediction. MR measures the average rank of true entities, lower the better. MRR averages the reciprocal ranks of true entities, providing a normalized measure less sensitive to outliers. Hits@N measures the proportion of true entities in the top N predictions.Baselines. To benchmark the performance of our proposed model, we compared it against the stateof-the-art PLM-based methods including KG-BERT [22] , StAR [23] , PKGC [28] , C-LMKE [26] , KGT5 [25] , KG-S2S [24] , SimKGC [27] , and CSProm-KG [32] , and structure-based methods including TransE [14] , DistMult [15] , ComplEx [16] , ConvE [17] , TuckER [18] , pRotatE [19] , RotatE [19] , and HAKE [20] .Experimental Strategy: For most PLM-based models, due to their high training cost, we use the results reported in their respective papers.. We reproduce PKGC [28] for all three datasets, SimKGC [27] and CSProm-KG [32] for PrimeKG. In addition, we evaluate the capabilities of LLM embeddings (TE-3-S/L: text-embedding-3-small/large) for zero-shot link prediction by ranking the cosine similarity between (e i + r) and e j . For structure-based KGE models, we assess and present their best performance using optimal settings (shown in Table 12 ). For KG-FIT, we provide a detailed hyperparameter study in Table 13 . We use OpenAI's GPT-4o as the LLM for entity description generation and for LLM-guided hierarchy refinement. text-embedding-3-large is used for entity embedding initialization, which preserves semantics with flexible embedding slicing [45] . We set τ min = 0.15 and τ max = 0.85 for seed hierarchy construction. The values of τ optim for FB15K-237, YAGO3-10, and PrimeKG are 0.52, 0.49, and 0.33, respectively. The statistics of the seed and LHR hierarchies are placed in Appendix F.3. For LCA, we designate the root as the grandparent (i.e., two levels above) source cluster node and set m = 5. We set ρ = 0.5, β 0 = 1.2, ϕ = 0.4, and use cosine distance for fine-tuning. Filtered setting [14, 19] is applied for link prediction evaluation.Hyperparameter studies and computational cost are detailed in Appendix I and H, respectively.The output includes the selected update mode and the suggested name for the new or merged cluster, ensuring clarity and consistency in the hierarchy refinement process.These prompts, illustrated in Fig. 8 and Fig. 9 , are essential for transforming the seed hierarchy into a more refined and accurate LLM-guided hierarchy, enabling better hierarchical representation to be learned by KG-FIT. end for 36: end function This section details the seed hierarchy construction mentioned in Section 3.1. Algorithm 1 shows the pseudo code for this process. 
include:: 
 One of the child clusters is absorbed into the other, while the grandparent cluster remains unchanged. This can happen in two ways:• P ′ * = P ′ l = {C l 1 , C l 2 , C l 3 , . . . , C l |P l | , P r }, and P ′ r = ∅, or • P ′ * = P ′ r = {P l , C r 1 , C r 2 , C r 3 , . . . , C r |Pr| }, and P ′ l = ∅.The LLM determines the most appropriate refinement action based on the semantic similarity and hierarchical relationships between the clusters. The refinement process continues bottom-up, iteratively updating the triples until the root of the hierarchy is reached. The resulting refined hierarchy is denoted as H LHR . We place more details of the process in Appendix E.2 and F.2.We conduct experiments to evaluate the performance on link prediction. The results in Table 1 are averaged values from multiple runs with random seeds: ten runs for FB15K-237 and PrimeKG, and three runs for YAGO3-10. These averages reflect the performance of head/tail entity predictions. Main Results. We conduct an ablation study to evaluate the proposed constraints in Eq. 5 and 6, with results summarized in Table 4 . This analysis underscores the importance of each constraint: (1) Hierarchical Distance Maintenance is crucial for both datasets. Its removal significantly degrades performance across all metrics, highlighting the necessity of preserving the hierarchical structure in the embedding space.(2) Semantic Anchoring proves more critical for the denser YAGO3-10 graph, where each cluster contains more entities, making it harder to distinguish between them based solely on cluster cohesion. The sparser FB15K-237 dataset is less impacted by the absence of this constraint. Similar to the semantics anchoring, the removal of (3) Inter-level Cluster Separation significantly affects the denser YAGO3-10 more than FB15K-237. Without this constraint, entities in YAGO3-10 may not be well-separated from other clusters, whereas FB15K-237 is less influenced. Interestingly, removing (4) Cluster Cohesion has a larger impact on the sparser FB15K-237 than on YAGO3-10. This difference suggests that sparse graphs rely more on the prior information provided by entity clusters, while denser graphs can learn this information more effectively from their abundant data. Effect of Knowledge Sources. We explore the impact of the quality of LLM-refined hierarchies and pre-trained text embeddings on final performance, as illustrated in Figures 4 and 5 . The results indicate that hierarchies constructed and text embeddings retrieved from more advanced LLMs consistently lead to improved performance. This finding underscores KG-FIT's capacity to leverage and evolve with ongoing advancements in LLMs, effectively utilizing the increasingly comprehensive entity knowledge captured by these models.Efficiency Evaluation. We evaluate the efficiency performance in Table 3 . While pure structure-based models are the fastest, our model significantly outperforms all PLM-based models in both None Seed G3.5 G4 30 training and inference speed, consistent with our previous analysis. It achieves 12 times the training speed of CSProm-KG, the fastest PLM-based method. Moreover, KG-FIT can integrate knowledge from any LLMs, unlike previous methods that are limited to small-scale PLMs, underscoring its superiority.Visualization. Figure 6 demonstrates the effectiveness of KG-FIT in capturing both global and local semantics. The embeddings generated by KG-FIT successfully preserve the global semantics at both intra-and inter-levels. Additionally, KG-FIT excels in representing local semantics compared to the original HAKE model. 
conclusion: 
 In this paper, we introduced KG-FIT, a novel framework for enhancing knowledge graph (KG) embeddings by leveraging the wealth of open-world knowledge captured by large language models (LLMs). KG-FIT seamlessly integrates LLM-derived entity knowledge into the KG embedding process through a two-stage approach: LLM-guided hierarchy construction and global knowledgeguided local KG fine-tuning. By constructing a semantically coherent hierarchical structure of entities and incorporating this hierarchical knowledge along with textual information during fine-tuning, KG-FIT effectively captures both global semantics from the LLM and local semantics from the KG. Extensive experiments on benchmark datasets demonstrate the superiority of KG-FIT over state-of-the-art methods, highlighting its effectiveness in integrating open-world knowledge from LLMs to significantly enhance the expressiveness and informativeness of KG embeddings. A key advantage of KG-FIT is its flexibility to incorporate knowledge from any LLM, enabling it to evolve and improve with ongoing advancements in language models. This positions KG-FIT as a powerful and future-proof framework for knowledge-infused learning on graphs. Moreover, the enriched KG embeddings produced by KG-FIT have the potential to boost performance on a wide array of downstream tasks, such as question answering, recommendation systems, and drug discovery, among others. Our code and data are available at https://github.com/pat-jj/KG-FIT. where n is the hidden dimension, which serves as values on the x-axis and y-axis, respectively, which is consistent with Zhang et al. [20] 's visualization strategy. Lower (global): t-SNE plots of different embeddings of sampled entities, with colors indicating clusters (e.g., Maraviroc belongs to the HIV Drugs cluster). Triangles indicate the positions of ▲ Maraviroc, ▲ CAA, ▲ Cladribine, and ▲ Exertional dyspnea. Observations: While the initial text embeddings capture global semantics, they fail to delineate local parent-child relationships within the KG, as seen in the intermingled polar plots. In contrast, HAKE shows more distinct grouping by modulus on the polar plots, capturing hierarchical local semantics, but fails to adequately capture global semantics. Our KG-FIT, notably, incorporates prior information from LLMs and is fine-tuned on the KG, maintains global semantics from pre-trained text embeddings while better capturing local KG semantics, demonstrating its superior representational power across local and global scales. 
limitations: 
 Although KG-FIT outperforms state-of-the-art PLM-based models on the FB15K-237, YAGO3-10, and PrimeKG datasets, it does not outperform pure PLM-based methods on a lexical dataset WN18RR, as shown in Table 5 . This limitation is discussed with details in Appendix C. As a future work, we will explore the integration of contrastive learning into the KG-FIT framework to enhance its capability to capture semantic relationships more effectively. Moreover, KG-FIT's performance is influenced by the quality of the constructed hierarchy, particularly the seed hierarchy. To address this, we propose an automatic selection of the optimal binary tree based on the silhouette score. However, if the initial clustering is suboptimal, it may result in a lower-quality hierarchy that affects KG-FIT's performance. Additionally, the bottom-up refinement process in our proposed LHR approach updates each parent-child triple with only a single operation (within four), which prioritizes efficiency and simplicity over performance. In future work, we plan to explore cost-efficient methods for refining the hierarchy that integrate multiple operations for each triple update, striking a better balance between efficiency and performance.  
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [Yes] Justification: In Appendix A.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: Section 4 and Appendix B.Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. 
 
 Appendix: KG-FIT is a framework for enhancing knowledge graph embeddings by incorporating open knowledge from large language models. As a foundational research effort, KG-FIT itself does not have direct societal impacts. However, the enriched knowledge graph embeddings produced by KG-FIT could potentially be used in various downstream applications, such as question answering, recommendation systems, and drug discovery.The broader impacts of KG-FIT depend on how the enhanced knowledge graph embeddings are used. On the positive side, KG-FIT could lead to more accurate and comprehensive knowledge representations, enabling better performance in beneficial applications like medical research and personalized recommendations. However, as with any powerful technology, there is also potential for misuse. The enriched knowledge could be used to build systems that spread disinformation or make unfair decisions that negatively impact specific groups.To mitigate potential negative impacts, we encourage responsible deployment of KG-FIT and the enhanced knowledge graph embeddings it produces. This includes carefully monitoring downstream applications for fairness, robustness, and truthfulness. Additionally, when releasing KG-FIT-enhanced knowledge graph embeddings, we recommend providing usage guidelines and deploying safeguards to prevent misuse, such as gated access and safety filters, as appropriate.As KG-FIT relies on large language models, it may inherit biases present in the language models' training data. Further research is needed to understand and mitigate such biases. We also encourage future work on building knowledge graph embeddings that are more inclusive and less biased.Overall, while KG-FIT is a promising framework for advancing knowledge graph embeddings, it is important to consider its limitations and potential broader impacts. Responsible development and deployment will be key to realizing its benefits while mitigating risks. We are committed to fostering an open dialogue on these issues as KG-FIT and related technologies progress.We describe the process of constructing a subset of the PrimeKG1 [44] (version: V2, license: CC0 1.0) dataset. The goal is to create a highly-focused subset while leveraging additional information from the entire knowledge graph to assess KG embedding models' abilities on predicting drug-disease relationships.The dataset construction process involves several detailed steps to ensure the creation of balanced training, validation, and testing sets from PrimeKG.Validation/Testing Set Creation: We begin by selecting triples from the original PrimeKG where the type of the head entity is "drug" and the type of the tail entity is "disease." This selection yields a subset containing 42,631 triples, 2,579 entities, and 3 relations ("contraindication," "indication," "off-label use"). From this subset, we randomly select 3,000 triples each for the validation and testing sets, ensuring no overlap between the two.Training Set Creation: We first extract the unique entities present in the validation and testing sets. These entities are used to ensure comprehensive coverage in the training set.For each triple in the validation/testing set, we search for triples involving either its head or tail entity across the entire PrimeKG dataset, for the training set construction:Step 1 involves searching for 1-hop triples within the specified relations ("contraindication," "indication," "off-label use").Step 2 randomly enriches the training set with triples involving other relations, with a limit of up to 10 triples per entity.Step 3 involves removing redundant triples and any triples that are symmetric to those in the validation/testing set, enhancing the challenge posed by the dataset. If the training set contains fewer than 100,000 triples, we return to Step 2 and continue the process until the desired size is achieved.Following our methodology, we construct a subset of the PrimeKG dataset that emphasizes drug-disease relations while incorporating broader context from the entire dataset. This subset is then split into training, validation, and testing sets, ensuring proper files are generated for training and evaluating knowledge graph embedding (KGE) models. The dataset contains 10,344 entities with 11 types of relations:disease_phenotype_positive, drug_effect, indication, contraindication, disease_disease, disease_protein, disease_phenotype_negative, exposure_disease, drug_protein, off-label use, drug_drug. The testing and validation sets include three specific relations ("contraindication," "indication," "off-label use"). In this section, we present our results and findings on the WN18RR dataset. WN18RR [17] is derived from WordNet [46] , a comprehensive lexical knowledge graph for the English language. This subset addresses the test leakage problems identified in WN18. The statistics of WN18RR are shown in Table 6 . τ optim we found for the seed hierarchy construction on WN18RR is 0.44.Our experiments reveal that although KG-FIT did not surpass the performance of C-LMKE and SimKGC, its integration with PKGC [28] , employing KG-FIT as a recall model followed by reranking with PKGC, yielded comparable results to these leading models.Several factors contribute to these observations:(1) WN18RR, being a lexical dataset, benefits significantly from pre-trained language model (PLM) based methods, which assimilate extensive lexical knowledge during fine-tuning. KG-FIT, relying on static knowledge graph embeddings, lacks this capability. However, as shown in Table 5 , its combination with PKGC leverages the strengths of both models, resulting in markedly improved performance. (2) WN18RR has a very close semantic similarity between the head and tail entities. The dataset is full of lexical relations such as "hypernym", "member_meronym", "verb_group", etc. C-LMKE and SimKGC, both utilizing contrastive learning [47] with PLMs, effectively exploit this semantic proximity during training. This ability to discern subtle semantic nuances contributes to their superior performance over other PLM-based approaches.(3) SimKGC implements a "Graph-based Re-ranking" strategy that narrows the candidate pool to the k-hop neighbors of the source entity from the training set. This approach intuitively boosts performance by focusing on more relevant candidates. For a fair comparison, our experiments did not adopt this specific setting across all datasets. In Table 5 , we showcase how this strategy could boost the performance for KG-FIT as well.As a future work, we will explore the integration of contrastive learning into the KG-FIT framework to enhance its capability to capture semantic relationships more effectively.In this section, we provide more implementation details of both KG-FIT and baseline models, to improve the reproducibility of our work.Pre-computation. In our KG-FIT framework, there is a pre-computation step to avoid overhead during the fine-tuning phase. The data we need to pre-compute includes:1. Cluster embeddings c: These are computed by averaging the all initial entity embeddings (v i ) within each cluster.2. Neighbor cluster IDs (S m (C) in Eq. 5): These are computed using the lowest common ancestor (LCA) approach, where we set the ancestor as the grandparent (i.e., two levels above the node) and search for at most m = 5 neighbor clusters.3. Parent node IDs: These represent the node IDs along the path from a cluster (leaf node) to the root of the hierarchy.The pre-computed data is then used to efficiently locate the embeddings of clusters (c, c ′ ) and parent nodes (p) for the hierarchical clustering constraint during fine-tuning. With this pre-computation, we significantly speed up the training process, as the necessary information is readily available and does not need to be calculated on-the-fly.Distance Function. We employ cosine distance as the distance metric for computing the hierarchical clustering constraint (Eq. 5) and the semantic anchoring constraint (Eq. 6). Cosine distance effectively captures the semantic similarity between embeddings, making it well-suited for maintaining the semantic meaning of the entity embeddings during fine-tuning. Cosine distance is also invariant to the magnitude of the embeddings, allowing the fine-tuned embeddings to adapt their magnitude based on the link prediction objective while preserving their semantic orientation with respect to the frozen reference embeddings. Moreover, cosine distance is independent of the link prediction objective, focusing on preserving the semantic properties of the embeddings.Alternative distance metrics, such as Euclidean distance or L1 norm, were considered but deemed much slower in KG-FIT framework, aligning with the descriptions in OpenAI's document on the embedding models2 .Constraint Computation Options. During fine-tuning, each training step involves a batch composed of one positive triple and b negative samples, where b is the negative sampling size. We provide two options for computing the constraints along with the link prediction score:1. Full Constraint Computation: This option computes the distances for both the positive triple and the negative triples. For the positive triple, distances are computed for both the source and target entities. For the negative triples, distances are computed only for the source entities. The advantage of this option is that it allows the model to converge in fewer steps and generally achieves better performance after fine-tuning. However, the drawback is that the computation for each batch is nearly doubled, as positive and negative triples are separately fed into the model for score computation. For PKGC on FB15K-237, we use the templates the authors constructed and released 3 for FB15K-237-N, and use RoBERTa-Large as the base pre-trained langauge model. For YAGO3-10 and PrimeKG, we manually created templates for relations. For example, we converted the relation "isLeaderOf" to "[X] is a leader of [Y]." for YAGO3-10 and converted the relation "drug_effect" "drug [X] has effect [Y]". We choose TuckER as PKGC's backbone KGE recall model with hyperparameter X = 100 for its overall great performance across all datasets. This means that we select top 50 results from TuckER and feed the shuffled entities into PKGC for re-ranking. We set batch size as 256 and run on 1 NVIDIA A6000 GPU for both training and testing.SimKGC. For SimKGC on PrimeKG, for fairness, we do not use the "graph-based re-ranking" strategy introduced in their paper [27] , which adds biases to the scores of known entities (in the training set) within k-hop of the source entity. The released code 4 was used for experiments. We set batch size as 256 and run on 1 NVIDIA A6000 GPU for both training and testing.CSProm-KG. For CSProm-KG on PrimeKG, we use ConvE as the backbone graph model, which is the best Hits@N performed model. The other settings we use are the same as what reported in the paper. We use the code 5 released by the authors to conduct the experiments.TuckER and ConvE. For TuckER, we use its release code 6 to run the experiments. For ConvE, we use its PyTorch implementation in PKGC's codebase 7 , provided in the same framework as TuckER.It is worth noting that we do not use their proposed "label smoothing" setting for fair comparison.TransE, DistMult, ComplEx, pRotatE, RotatE. For those KGE models, we reuse the code base 8 released by RotatE [19] . As it provides a unified framework and environment to run all those models, enabling a fair comparison. Our code ("code/model_common.py") also adapts this framework as the foundation.HAKE. HAKE's code 9 was built upon RotatE's repository described above. In our work, we integrate the implementation of HAKE into our framework, which is also based on RotatE's.The hyperparameters we used are presented in Appendix I. Figure 7 showcases the prompts we used for entity description. In the prompt, we instruct the LLM to provide a brief and concrete description of the input entity. We include an example, such as "apple is a round fruit with red, green, or yellow skin and crisp, juicy flesh" to illustrate that the description should be concise yet cover multiple aspects. Conciseness is crucial to minimize noise. For datasets like FB15K-237 and WN18RR, where descriptions are already available [23] , we use the original description as a hint and ask the LLM to output the description in a unified format.Fig. 8 and 9 show the main prompts we used for LLM-Guided Hierarchy Refinement (LHR).Prompt for Cluster Splitting (P SPLIT ): This prompt is designed to guide the LLM in identifying and splitting a given cluster into meaningful subclusters based on the characteristics of the entities. The prompt works as follows:1. Input Entities: The entities from the specified cluster are provided as input.The process begins with the agglomerative hierarchical clustering of the enriched entity representations V using a range of thresholds [τ min , τ max ]. For each threshold τ , the clustering labels (labels τ ) are obtained, and the silhouette score (S τ ) is calculated. The optimal threshold τ optim is determined by selecting the threshold that maximizes the silhouette score.Next, the initial hierarchy H init is constructed based on the clustering labels obtained using the optimal threshold (labels τoptim ). The ConstructBinaryTree function builds a binary tree structure where each leaf node represents an entity.The top-down entity replacement algorithm is then applied to the initial hierarchy H init . It traverses the hierarchy in a top-down manner, starting from the root node. For each node encountered during the traversal:• If the node is a leaf, the entity associated with the node is retrieved using the GetEntity function.• The cluster to which the entity belongs is obtained using the GetCluster function and the labels τoptim .• If the cluster has already been visited (i.e., it exists in the visited_clusters set), the leaf node is removed from the hierarchy.• If the cluster has not been visited, the leaf node is replaced with the cluster, and the cluster is added to the visited_clusters set.• If the node is not a leaf, the algorithm recursively applies the same process to its child nodes.This top-down approach ensures that entities are included in their respective clusters as early as possible during the traversal of the hierarchy. By keeping track of the visited clusters, the algorithm avoids duplicating clusters in the hierarchy, resulting in a more compact and coherent representation.Finally, the refinement steps (Refine function) are applied to the modified hierarchy to obtain the final seed hierarchy H seed . The refinement steps include handling empty dictionaries, single-entry dictionaries, and updating cluster assignments.The resulting seed hierarchy H seed represents a hierarchical organization of the entities, where each node is either a cluster containing a list of entities or a sub-hierarchy representing a more finegrained grouping of entities. This seed hierarchy serves as a starting point for further refinement and incorporation of external knowledge in the subsequent steps of the KG-FIT framework.LLM-Guided Hierarchy Refinement (LHR) is used to further refine the seed hierarchy, which can better reflect relationships among entities. The process of LLM-Guided Hierarchy Refinement can be divided into two steps: LLM-Guided Cluster Splitting and LLM-Guided Bottom-Up Hierarchy Refinement.As described in Algorithm 2, the LLM-Guided Cluster Splitting algorithm is designed to iteratively split clusters in a hierarchical structure with the guidance of a large language model (LLM). The algorithm begins with an initial hierarchy seed H seed and outputs a split hierarchy H split . Initially, the current cluster ID is set to 0. The recursive procedure RECURSION_SPLIT_CLUSTER is then defined to manage the splitting process. If the root of the current cluster is a list and its length is less than a predefined minimum number of entities in a leaf node (MIN_ENTITIES_IN_LEAF), the function returns, indicating no further splitting is necessary. Otherwise, the root cluster's entities are passed to the LLM, which names the cluster and splits it into smaller clusters. If the split results in only one cluster, the function returns. For each resulting subcluster, the entities are assigned a new cluster ID, and the splitting process is recursively applied. If the root is not a list, the function iterates over each subcluster and applies the splitting procedure. Finally, the algorithm initiates the recursive splitting on the initial hierarchy seed and returns the modified hierarchy as H split .The LLM-Guided Bottom-Up Hierarchy Refinement algorithm (Algorithm 3) refines a previously split hierarchy H split to produce a more coherent and meaningful hierarchy H LHR . This process is also guided by an LLM. The algorithm defines a recursive procedure RECURSION_REFINE_HIERARCHY, which starts by checking if the root of the current cluster is a list. If it is, the LLM is used to name the cluster, and the updated hierarchy is returned. Otherwise, the procedure recursively refines the left and right child nodes. The children of these nodes are then evaluated, and the LLM suggests an update mode based on the names and children of the right and left nodes. Depending on the suggested update mode, the algorithm may add the right node under the left children, the left node under the right children, or merge the left and right clusters. If no update is suggested, the procedure continues. The updated hierarchy is returned after applying the necessary refinements. The algorithm initiates the refinement process on the split hierarchy H split and outputs the refined hierarchy H LHR .Algorithm 2 LLM-Guided Cluster Splitting end if 27: end procedure 28: H LRH ← RECURSION_REFINE_HIERARCHY(H split ) 29: return H LHR slight increase in the average number, with the maximum and minimum values remaining fairly constant. For example, the average number of entities per cluster in FB15K-237 increases from 2.78 to 2.81. The depth of the hierarchies shows a noticeable reduction, with the maximum and average depths decreasing. In FB15K-237, the maximum depth goes from 46 to 40, and the average depth drops from 25.52 to 18.64. The branching factor of nodes also increases slightly, indicating a more interconnected structure; in FB15K-237, the average number of branches per node rises from 2 to 2.3. Similar patterns are observed in other datasets.Overall, Table 7 illustrates that the refinement process effectively reduces the number of clusters and nodes, slightly increases the number of entities per cluster, decreases the depth of the hierarchy, and increases the branching factor. These changes suggest that the refinement process results in a more compact and interconnected hierarchy, better reflecting relationships among entities. The general effect of the refinement is the creation of a more streamlined and coherent hierarchical structure. Examples in Table 8 and Table 9 illustrate how the LLM-guided hierarchy refinement process updates and organizes clusters and sub-clusters to create a more coherent and meaningful hierarchical structure.Table 8 provides examples of sub-clusters within a cluster in the process of cluster splitting. Each sub-cluster is given a name and lists the associated entities. For instance, one sub-cluster named Additionally, a sub-cluster named "Medical Schools" includes entities such as Yale School of Medicine. The cluster of "Universities and Colleges" will be divided into the sub-clusters of "Universities" and "Colleges" in the next step. This example shows that LLM can divide the original large cluster into multiple smaller, refined ones. -∥h + r -t∥ 1/2 h, r, t ∈ R k DistMult h ⊤ diag(r)t h, r, t ∈ R k ComplEx Re(h ⊤ diag(r)t) h, r, t ∈ C k ConvE f (vec(f (h * ω))W)t h, r, t ∈ R k TuckER W ×1 h ×2 wr ×3 t wr ∈ R dr pRotatE -2C sin θ h +θr -θ t 2 1 h, r, t ∈ C k RotatE -∥h • r -t∥2 h, r, t ∈ C k , |ri| = 1 HAKE -∥hm • rm -tm∥2 -λ∥ sin((hp + rp -tp)/2)∥1 hm, tm ∈ R k , tm ∈ R k + , hp, rp, tp ∈ [0, 2π) k , λ ∈ RThe score functions defined by the structure-based KG embedding methods [14, 15, 16, 17, 18, 19, 20, 21] we tested in this papaer are shown in Table 10 . Here is a paragraph explaining each notation in the table, with all explanations inline:The table presents the score functions f r (h, t) used by various knowledge graph embedding (KGE) methods, where h, r, and t represent the head entity, relation, and tail entity embeddings, respectively. TransE uses a translation-based score function with either L1 or L2 norm, denoted by ∥ • ∥ 1/2 , where the embeddings are in real space R k . DistMult employs a bilinear score function with a diagonal relation matrix diag(r), and the embeddings are also in R k . ComplEx extends DistMult by using complex-valued embeddings in C k and takes the real part of the score, denoted by Re(•). ConvE applies a 2D convolution operation, where h is the 2D reshaping of h, * represents the convolution, ω is a set of filters, vec(•) is a vectorization operation, and W is a linear transformation matrix. TuckER uses a Tucker decomposition with a core tensor W and relation-specific weights w r ∈ R dr , where × n denotes the tensor product along the n-th mode. pRotatE and RotatE employ rotation-based score functions in complex space, with • representing the Hadamard product and |r i | = 1 constraining the relation embeddings to have unit modulus. HAKE combines a modulus part (h m , t m ∈ R k , t m ∈ R k + ) and a phase part (h p , r p , t p ∈ [0, 2π) k ) in its score function, with a hyperparameter λ ∈ R balancing the two parts. With a single GPU, it takes about 2.5, 4.5, 2.0, and 1.1 hours for a KG-FIT model to achieve good performance on FB15K-237, YAGO3-10, PrimeKG, and WN18RR, respectively.The costs of GPT-4o for entity description generation are $3.0, $24.8, $2.1, and $8.6 for FB15K-237, YAGO3-10, PrimeKG, and WN18RR, respectively, proportional to their numbers of entities.The cost of text embedding models (text-embedding-3-large) for entity embedding initialization was totally about $0.8 to process all the KG datasets.The costs of GPT-4o for LLM-Guided Hierarchy Refinement on FB15K-237, YAGO3-10, PrimeKG, and WN18RR are $10.0, $74.5, $8.7, and $34.4, respectively. This cost is almost proportional to the nodes consisted in the seed hierarchy.This section presents a comprehensive hyperparameter study for both structure-based base models and our proposed KG-FIT framework across different datasets. Table 11 outlines the range of hyperparameter values explored during the study. Table 12 showcases the optimal hyperparameter configurations for the base models that yielded the best performance. Similarly, Table 13 presents the best-performing hyperparameter settings for KG-FIT.The hyperparameter study aims to provide insights into the sensitivity of the models to various hyperparameters and to identify the optimal configurations that maximize their performance on each dataset. By conducting a thorough exploration of the hyperparameter space, we ensure a fair comparison between the base models and KG-FIT, and demonstrate the robustness and effectiveness of our proposed framework across different settings. SmartVision 3000 The enhanced knowledge graph embeddings produced by KG-FIT can enable improved performance on various downstream tasks. As shown in Fig. 10 , potential areas of application include:• Tasks transformed to fundamental KG tasks. KG-FIT's strong performance on link prediction can directly benefit fundamental KG tasks like knowledge graph-based question answering (KGQA) [1, 2, 4] . For example, given the question "Can drinking black tea help reduce the risk of cardiovascular diseases?", a KGQA system powered by KG-FIT embeddings could effectively traversely perform triple classification task for triples such as <black tea, potentially beneficial to, cardiovascular diseases>, <black tea, is a, flavonoids-rich drink>, <flavonoids, may treat, cardiovascular diseases>, and provide an accurate answer based on the classification results.• Entity Matching Across KGs. KG-FIT's ability to capture both global and local semantics facilitates accurate entity matching across different knowledge graphs [48, 49, 50] . Consider two KGs, A and B, containing information about a company's products. KG A lists the entity "GlobalTech Inc" launched "SmartVision 3000", which is a product in "Smart Home Devices", while KG B mentions "SV3000 Camera" is released by "Global Tech Industries" in the category of "Home Automation Solution". By generating semantically rich embeddings that encode textual, hierarchical and relational information, KG-FIT can help identify that "SmartVision 3000" and "SV3000 Camera", "GlobalTech Inc." and "Global Tech Industries" likely refer to the same entities, despite differences in surface form and graph structure. • Retrieval Augmented Generation with Graphs. The hierarchical nature of KG-FIT's embeddings enables efficient search for relevant information to augment language model-based text generation. In a retrieval augmented generation setup, a language model's output can be enhanced by retrieving and conditioning on pertinent information. KG-FIT's embeddings allow for quick identification of relevant entities and relationships via proximity search in the semantic space. Moreover, by traversing the KG-FIT hierarchy, the system can gather additional context about an entity of interest. For instance, if the generation task involves the cardiovascular benefits of black tea, searching the KG-FIT hierarchy may surface related information on flavonoids and antioxidant properties, providing valuable context to guide the language model in producing an informed and factually grounded response. • Other Tasks. KG-FIT's embeddings can also be leveraged for tasks such as relation extraction [51] and entity disambiguation [52] . By providing high-quality embeddings that encode both local and global information, KG-FIT can improve the accuracy and efficiency of these tasks. For example, in relation extraction, KG-FIT's embeddings can help identify the most likely relation between two entities given their positions in the hierarchy and their semantic proximity. In entity disambiguation, KG-FIT's embeddings can be used to disambiguate between multiple entities with the same name by considering their relationships and positions within the knowledge graph hierarchy.In summary, KG-FIT's robust embeddings, capturing both local and global semantics in a hierarchical structure, can significantly enhance a variety of downstream applications, from fundamental KG tasks to entity matching and retrieval augmented generation. By providing semantically rich and efficiently searchable representations of KG entities and relationships, KG-FIT enables knowledge-infused AI systems that can better understand and reason over complex domains.• ⟨h d , r d , t d ⟩: This term captures the multiplicative interaction between the head entity's description embedding, the relation embedding, and the tail entity's description embedding.The DistMult score function captures the global semantic information by computing the tri-linear dot product between the head entity, relation, and tail entity embeddings. The dot product considers the interactions between the entity name embeddings and the entity description embeddings separately, allowing the model to capture the global semantic relatedness and attributional similarities.ComplEx: In the ComplEx model, the score function is defined as:EQUATIONwhere Re(•) and Im(•) denote the real part and imaginary part of a complex number, and t represents the complex conjugate of t.In KG-FIT, the entity embeddings are initialized as follows:EQUATIONwhere h n , t n ∈ R dim/2 represent the entity name embeddings (real part) and h d , t d ∈ R dim/2 represent the entity description embeddings (imaginary part).The relation embedding r is also a complex-valued vector:EQUATIONwhere r r , r i ∈ R dim/2 are learned embeddings that capture the intricate semantics of the relation in the complex space.Thus, the score function using the KG-FIT embeddings becomes:EQUATIONInterpretation:• h n • r r • t n and h d • r r • t d : These terms represent the fundamental interactions between the head and tail entity name embeddings modulated by the real part of the relation. They capture symmetric relationships where the semantic integrity of the relation is maintained irrespective of the direction.• h n • r i • t d and -h d • r i • t n : These cross-terms incorporate the imaginary part of the relation embedding, introducing a unique capability to model antisymmetric relations. The inclusion of the imaginary components allows the score function to account for relations where the direction or the orientation between entities significantly alters the meaning or the context of the relation.In the RotatE model, each relation is represented as a rotation in the complex plane. The score function is defined as:EQUATIONIn RotatE with KG-FIT embeddings, we have:EQUATIONand the relation embedding is:EQUATIONwhere θ r is the learned rotation angle for the relation.Expanding the RotatE score function using KG-FIT embeddings, we have:EQUATIONInterpretation:• h n • cos(θ r ) -t n and h d • cos(θ r ) -t d : These terms represent the rotated head entity name and description embeddings respectively, which are then compared with the corresponding tail entity name and description embeddings. The cosine of the relation angle θ r scales the embeddings, effectively capturing the strength of the relationship between the entities. • h n • sin(θ r ) and -h d • sin(θ r ): These terms introduce a phase shift in the entity embeddings based on the relation angle. The sine of the relation angle θ r allows for modeling more complex interactions between the head and tail entities, considering both the name and description information.pRotatE: In the pRotatE model, the modulus of the entity embeddings is constrained such that |h i | = |t i | = C, and the distance function is defined as:EQUATION)where θ h , θ r , and θ t represent the phases of the head entity, relation, and tail entity embeddings, respectively. For KG-FIT embeddings, the entity embeddings are complex and represented as:EQUATIONThe phases can be calculated as:EQUATIONwhere r 1 , r 2 ∈ R dim/2 are the learned relation embeddings.Thus, the pRotatE score function using KG-FIT embeddings becomes: Interpretation:In pRotatE with KG-FIT, the entity phases θ h and θ t are computed using both the name and description embeddings, while the relation phase θ r is learned through the relation embeddings r 1 and r 2 .The model aims to minimize the phase difference sin θ h +θr-θt 2 1 for valid triples, considering both the entity name and description information. This allows pRotatE to capture the complex interactions between entities and relations in the knowledge graph.In the HAKE (Hierarchy-Aware Knowledge Graph Embedding) model, entities are embedded into polar coordinate space to capture hierarchical structures. The score function is a combination of radial and angular distances:EQUATIONIn KG-FIT, h mod = h d , t mod = t d , h phase = arg(h n ), and t phase = arg(t n ). The learned relation embedding is r = [r n ; r d ].Thus, the HAKE score function using KG-FIT embeddings becomes:EQUATION)Interpretation:In HAKE with KG-FIT, the entity description embeddings are used to determine the modulus, while the entity name embeddings are used to determine the phase. This approach seamlessly utilizes the information from both types of embeddings from pre-trained language models.Table 14 provides a comprehensive list of the notations used throughout this paper, along with their corresponding descriptions. This table serves as a quick reference to help readers better understand the concepts presented in our work. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).38th Conference on Neural Information Processing Systems (NeurIPS 2024).https://dataverse.harvard.edu/dataset.xhtml?persistentId=doi:10.7910/DVN/IXA7BMhttps://platform.openai.com/docs/guides/embeddings/frequently-asked-questionshttps://github.com/THU-KEG/PKGChttps://github.com/intfloat/SimKGChttps://github.com/chenchens190009/CSProm-KGhttps://github.com/ibalazevic/TuckERhttps://github.com/THU-KEG/PKGC/blob/main/TuckER/model.pyhttps://github.com/DeepGraphLearning/KnowledgeGraphEmbeddinghttps://github.com/MIRALab-USTC/KGE-HAKE 


Decision: This is a paper on KG embeddings, where entity descriptions from LLMs are used to improve the embedding training. The paper yielded 4, 5, 5, 6. One reviewer who gave a 4 rating did not engage in discussion, whereas another later increased their score to 5. I have looked at author response for both these and other reviewers. Authors have provided additional experiments to provide good evidence for issues raised (e.g., comparison of other LLM-based enhancement approaches, comparison with other LLM-based baselines, experiments over more recent models, other clarifications especially regarding why do the research at all and rather simply use LLMs)

The positive reviewer also has interesting points such as what if LLM doesnt know these entities. Authors give some high level response to this question, but do not perform any experiments on the robustness of the model in such a case. This is an important question, and some experiments may have benefited the paper. Unfortunately, none(?) of the previous papers that combine LLMs with structure have done this experiment either... so while interesting, should not become the sole reason for rejection.

Finally, one criticism which is not addressed adequately is that some other papers like NBFNet (and more recently this https://arxiv.org/pdf/2311.03780) combine GNNs with LLMs, and have excellent performance on one of the hard datasets: (FB15K-237, 43 MRR, 33 Hits@1). The authors' results are much weaker. This is because the authors combine static embedding based models and not GNN-architectures that have additional power because they compute dynamic embeddings for a node in a query dependent way.  Authors relegate this combination (of their approach with GNNs) to future work. 

Overall, this is a paper where there aren't that many things wrong. Its a decent new idea, and good extensive experiments on a subset of models, leading to improvements compared to those models... not always generating a state of the art performance. I personally do not believe that a paper must only produce state of the art performance to get accepted. The idea of creating a hierarchy of entities and using that to train KGs is somewhat interesting. 

Is the paper groundbreaking? No. But, it will be a decent addition to NeurIPS if there is space.

Reviews: review: paper_summary: paper_summary: The paper introduces a framework called KG-FIT for enhancing knowledge graph embeddings by integrating knowledge from large language models (LLMs). KG-FIT enriches entity descriptions using LLMs and then constructs a semantically coherent hierarchical structure of entities. It finally fine-tunes KG embeddings using this hierarchy and textual information. Experiments on benchmark datasets (FB15K-237, YAGO3-10, PrimeKG) demonstrate its effectiveness in link prediction.
strength_weakness: strength_weakness: - There are several related studies on using LLMs to enhance text information in KGs [1,2], which, however, were not discussed in the paper. Additionally, constructing a hierarchy seems redundant given the existing graph structure of the KG. More discussion and explanation are needed.

- The rationale behind the technical design is unclear. For instance, the proposed method concatenates structural and textual embeddings to construct the hierarchy, and then linearly combines these embeddings for fine-tuning. What are the underlying reasons for these choices?

- The method may be computationally expensive due to the use of LLMs and hierarchical refinement. In my view, it is not profitable to use LLMs to achieve ~0.02 Hits@1 improvements in link prediction.

- Several recent strong baselines for KG link prediction, such as NBFNet [3] and AdaProp [4], which both achieve over 0.32 Hits@1 on FB15K-237, are absent from the experiments. It remains uncertain whether the proposed method can still improve the two baselines.

[1] Derong Xu, Ziheng Zhang, Zhenxi Lin, Xian Wu, Zhihong Zhu, Tong Xu, Xiangyu Zhao, Yefeng Zheng, Enhong Chen: Multi-perspective Improvement of Knowledge Graph Completion with Large Language Models. LREC/COLING 2024: 11956-11968

[2] Dawei Li, Zhen Tan, Tianlong Chen, Huan Liu: Contextualization Distillation from Large Language Model for Knowledge Graph Completion. EACL (Findings) 2024: 458-477

[3] Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal A. C. Xhonneux, Jian Tang: Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction. NeurIPS 2021: 29476-29490

[4] Yongqi Zhang, Zhanke Zhou, Quanming Yao, Xiaowen Chu, Bo Han: AdaProp: Learning Adaptive Propagation for Graph Neural Network based Knowledge Graph Reasoning. KDD 2023: 3446-3457
questions: questions: - The proposed method leverages entity names to prompt large language models (LLMs) to generate corresponding descriptions. How does it address the issue of multiple entities having the same name?

- Please see other questions in "Weaknesses".
limitations: limitations: NA.
paper_summary: paper_summary: Knowledge graphs (KGs) are essential for representing structured knowledge in various domains. They consist of entities and relations, forming a graph structure for efficient reasoning and knowledge discovery. Current knowledge graph embedding (KGE) methods create low-dimensional representations of these entities and relations but often overlook extensive open-world knowledge, limiting their performance. Pre-trained language models (PLMs) and large language models (LLMs) offer a broader understanding but are computationally expensive to fine-tune with KGs. To address these issues, the authors propose KG-FIT, a framework that incorporates rich knowledge from LLMs into KG embeddings without fine-tuning the LLMs. KG-FIT generates entity descriptions from LLMs, constructs a hierarchical structure of entities, and fine-tunes KG embeddings by integrating this hierarchy with textual embeddings. This approach enhances KG representations, combining global knowledge from LLMs with local KG knowledge, significantly improving link prediction accuracy on benchmark datasets.
strength_weakness: strength_weakness: 1.  The motivation needs to be reconsidered. The authors mention that using KG to fine-tune LLMs is computationally expensive. Many current research efforts do not fine-tune LLMs with KG. Instead, they use retrieval-based methods to explicitly provide the knowledge.

2. LLM-based baselines should be considered. The authors extensively use LLMs in their methods. They also should incorporate the LLM-based methods. For example, those LLMs retrieval-based methods. Only comparing with small LM-based methods is not enough.

3. The contribution is kind of limited. I do not know why the authors still use KG embedding methods for efficient reasoning and knowledge discovery since LLMs have very strong reasoning ability for knowledge discovery.
questions: questions: 1. Could the authors reconsider the motivation behind their approach? 

2. Why haven’t the authors considered incorporating published LLM-based baselines?

3. Why do the authors continue to use KG embedding methods for efficient reasoning and knowledge discovery when LLMs possess very strong reasoning abilities for knowledge discovery?
limitations: limitations: Please refer to my weakness
paper_summary: paper_summary: This paper addresses the limitations of existing KGE models that focus either on graph structure or fine-tuning pre-trained language models. It introduces KG-FIT, which leverages LLM-guided refinement to incorporate hierarchical and textual knowledge, effectively capturing both global and local semantics. Experiments on benchmark datasets demonstrate KG-FIT's superiority, achieving significant performance improvements over state-of-the-art methods.
strength_weakness: strength_weakness: 1. The paper is not organized clearly, which is not friendly for understanding. For example, there is a lack the sensitivity study for the hyperparameters in the loss function. 
2. The comparable methods are old and lack the new ones in the last 2 years such as [1][2][3][4]. The performance is not comparable with the previous work.
[1] Compounding Geometric Operations for Knowledge Graph Completion 
[2] Geometry Interaction Knowledge Graph Embeddings
[3] KRACL: Contrastive Learning with Graph Context Modeling for Sparse Knowledge Graph Completion
[4] Dual Quaternion Knowledge Graph Embeddings
3. The paper lacks the analysis of time complexity as well as space complexity, which is necessary to study the efficiency of the model. 
4. There are some typos and It is commanded that the writing needs to be improved. 
(1) On page 5, line 163 “determined by lowest common ancestor” should be “determined by the lowest common ancestor” 
(2) On page 6, line 195 “is sigmoid function” should be “is the sigmoid function”
questions: questions: Please refer to the weaknesses.
limitations: limitations: N/A
paper_summary: paper_summary: This paper introduces KG-FIT, a general framework that enhances the expressiveness of existing Knowledge Graph Embedding (KGE) models by integrating LLMs. KG-FIT contains four key steps: First, it utilizes an LLM to generate descriptions for a set of given entities, forming an enriched entity representation by concatenating the entity's embedding with its description. Second, it constructs a semantically coherent seed hierarchical structure. Third, it leverages the real-world entity knowledge captured by the LLM to refine this hierarchical structure. Finally, it fine-tunes the knowledge graph embeddings by integrating the hierarchical structure with textual embeddings. Extensive experiments validate the effectiveness of KG-FIT.
strength_weakness: strength_weakness: 1. The performance of KG-FIT heavily relies on the LLM used for generating entity descriptions and guiding the refinement of the seed hierarchy refinement. If the LLM lacks comprehensive real-world entity knowledge about the given entities or domains, the resulting embeddings may not suboptimal. Furthermore, this dependency could make KG-FIT less effective in the situations where LLM have limited coverage.
2. If the LLM has limited coverage of a specific domain, the LLM guided hierarchy refinement process may yield incorrect results, potentially distorting rather than enhancing the structure of the well-formed seed hierarchy.
3. The paper does not justify the selection of agglomerative hierarchical clustering and the use of silhouette score. Additional ablation studies will enhance this work.
questions: questions: 1. It is unclear the explicit description of $P$, $L$, $R$ in line 134.
2. It remains unclear whether the efficiency evaluation of training time contains the duration required for seed hierarchy construction and LLM-guided hierarchy refinement stages.
limitations: limitations: The authors have discussed the limitations of their work in the paper.

Comments: title: Thank You for Your Recognition
Thanks for the response
comment: Dear Reviewer dKZ2,

Thank you for your insightful comments and suggestions. In our author response, we have provided additional experimental results, analyses, and clarifications to address your concerns.

As the **discussion period nears its end (in 24 hours)**, we would be grateful if you could take a moment to review our response and let us know if there are any remaining concerns or if our clarifications have adequately addressed your points.

We are grateful for the time and expertise you have shared in reviewing our work.

Sincerely,

The Authors
Dear Reviewer hiNn,

We greatly appreciate your thoughtful review and the time you have taken to provide detailed feedback on our work. In our author response, we have addressed your valuable comments regarding related work, motivation, and the several technical aspects of KG-FIT.

As **the discussion period nears its end (in 24 hours)**, we would be grateful if you could take a moment to review our response and let us know if there are any remaining concerns or if our clarifications have adequately addressed your points.

Thank you once again for your efforts in evaluating our submission.

Sincerely,

The Authors
Thanks for the response. After carefully considering your feedback as well as the comments from other reviewers, I have decided to maintain my rating since this paper does not yet meet the rigorous standards expected for publication in NeurIPS. I'd like to see an improved version after a major revision.
Thank you for your thoughtful review and for recognizing the strengths of our work! We appreciate your positive evaluation and are glad our responses addressed your concerns. Your feedback has been invaluable in refining our approach. Please let us know whenever you have any further questions during this reviewer-author discussion period. We are happy to discuss and provide any additional information. 

Thank you again for your support, and we look forward to continuing our research in this exciting area.
Thanks for the response. I have read all the reviews and responses, and am satisfied with the response to my comments. I will maintain my positive review score.