CAB-KGC: Context-Aware BERT for Knowledge Graph Completion

Authors: Haji Gul, Ajaz Ahmad Bhat
Primary area: learning on graphs and other geometries & topologies
Keywords: Knowledge Graph Completion (KGC), Tail Prediction, Context-Aware BERT (CAB-KGC), Large Language Models (LLMs), BERT

Abstract:
Knowledge graph completion (KGC) seeks to predict missing entities (e.g., heads or tails) or relationships in knowledge graphs (KGs), which often contain incomplete data. Traditional embedding-based methods, such as TransE and ComplEx, have improved tail entity prediction but struggle to generalize to unseen entities during testing. Textual-based models mitigate this issue by leveraging additional semantic context; however, their reliance on negative triplet sampling introduces high computational overhead, semantic inconsistencies, and data imbalance. Recent BERT-based approaches, like KG-BERT, show promise but depend heavily on entity descriptions, which are often unavailable in KGs. Critically, existing methods  overlook valuable structural information in the KG related to the entities and relationships. To address these challenges, we propose Context-Aware BERT for Knowledge Graph Completion (CAB-KGC), a novel model that utilizes contextual information from linked entities and relations within the graph to predict tail entities. CAB-KGC eliminates the need for entity descriptions and negative triplet sampling, significantly reducing computational complexity while enhancing performance. Additionally, we introduce the Evaluation based on Distance from Average Solution (EDAS) criterion to the KG domain, enabling a more comprehensive evaluation across diverse metrics. Our experiments on standard datasets, including FB15k-237, WN18RR, CoDEx-S, and ConceptNet100K, demonstrate that CAB-KGC outperforms state-of-the-art methods on three datasets. Notably, CAB-KGC achieves improvements in Hit@1 of 6.88\%, 14.32\%, and 17.13\% on WN18RR, CoDEx-S, and ConceptNet100K, respectively. Furthermore, EDAS rankings establish CAB-KGC as the top-performing model, highlighting its effectiveness and robustness for KGC tasks.

Decision: reject
Reviewer scores: 3, 3, 6, 3, 3

Reviews and discussion:
<thread>
creation date:
1729314522112

thread_type:
official_review

<thread_object>
creator: reviewer_dgcv

<content>
summary:
This paper proposes a text-based KGC method named CAB-KGC. CAB-KGC finetunes a BERT model to complete the missing entity of a triple with the help of neighboring contexts, instead of entity descriptions. The proposed method gets rid of the high computational complexity imposed by negative sampling.

strengths:
1) The problem is clearly defined. 
2) The limitation of text-based methods, namely encoding negative samples, is clearly pointed out. 
3) The proposed method can alleviate the high computational overhead imposed by negative sampling.

weaknesses:
1) On page 2, line 2-7, The authors classify related works using BERT-based pre-trained language models, such as KG-BERT. This reviewer does not agree with such a claim to some extent. Given the limited number of parameters, BERT, especially BERT-base, cannot be considered as an LLM. In this case, this reviewer suggests the authors use "text-based" methods instead.
2) There are several non-concurrent state-of-the-art works that are not discussed or compared with the proposed method. 

a. For instance, the SOTA GNN-based method, NBF-Net [1], accepted to NeurIPS 2021, achieves Hits@1 performance of 0.321 and 0.497 on the FB15k237 and WN18RR datasets, respectively.

b. In addition, the authors did not completely report the performance of KICGPT [2] (Wei et al. 2024) referred in their paper. KICGPT is accepted to EMNLP2023 Findings, which achieves a Hits@1 of 0.327 and an MRR of 0.412 on the FB15k237 dataset, which outperforms the proposed method. 

Given the reasons above, it is inappropriate to claim that the proposed method is SOTA. 

3) This reviewer acknowledges that text-based methods underperform SOTA also have their merits. However, the performance of baseline methods should be completely referred and duly acknowledged. 

4) Moreover, there is another SOTA method DIFT [3], which the authors claimed that the paper is accepted to 2024, achieves a Hits@1 of 0.364 and a Hits@3 of 0.439 on the FB15k237 dataset. Nevertheless, The decision made by this reviewer is not based on the existence of [3] since it is a concurrent work of this submission. 

5) This paper does not report the Hits@10 performance of the proposed method and its baselines. Hits@10 is also a commonly used evaluation metric. It is acceptable if the proposed method cannot achieve desirable Hits@10 performance, as long as the issue is discussed and analysed. 

6) There are many duplicated references included in the reference list. Papers including but not limited to "Convolutional 2d knowledge graph embeddings" and KICGPT appear twice or even three times on pages 9-11. 

7) There are several minor Latex formatting issues. E.g. “Equations 1-??” on page 4. The correct symbol to represent membership is \\in, not \\epsilon (the misuses are shown in the 2nd paragraph, section 3, page 4).

[1] Neural Bellman-Ford Networks: A General Graph Neural Network Framework for Link Prediction (NeurIPS 2021) 

[2] KICGPT: Large Language Model with Knowledge in Context for Knowledge Graph Completion (EMNLP2023 Findings) 

[3] Finetuning Generative Large Language Models with Discrimination Instructions for Knowledge Graph Completion (on Arxiv, the authors of the paper claimed that it is accepted to ISWC2024)

questions:
This reviewer sincerely requests the authors to revise the manuscript carefully. 

For details, please see the weakness part above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729770175613

thread_type:
official_review

<thread_object>
creator: reviewer_k5ug

<content>
summary:
This paper introduces the CAB-KGC (Context-Aware BERT for Knowledge Graph Completion) model, which leverages contextual information from neighboring entities and relationships to predict tail entities, thus eliminating the reliance on entity descriptions. The model uses MLE to build loss, rather than contrastive training that requires negative sampling, which improves computational efficiency. 

Additionally, the paper evaluates model performance with an additional metric, the Evaluation based on Distance from Average Solution (EDAS), for more comprehensive assessment. Through experiments on the FB15k-237 and WN18RR datasets, CAB-KGC outperforms some baseline methods, showing improvements in metrics like Hit@1.

strengths:
1. The CAB-KGC model presents a novel approach by leveraging the contextual information of neighboring entities and relationships without relying on entity descriptions or negative triplet sampling, which is a common limitation in previous KGE and LLM-based methods. This removes the dependency on external textual information, making it applicable to a wider variety of KGs, especially those that lack entity descriptions. This design leads to more efficient training and improved evaluation performance.

2. The paper demonstrates thorough experimentation and validation of the proposed CAB-KGC model on standard benchmark datasets (FB15k-237 and WN18RR).

3. The introduction of the EDAS criterion also has the potential to influence future performance evaluation practices in the knowledge graph domain.

weaknesses:
1. Lack of novelty: The innovation in this work seems incremental, as it mainly builds on the SimKGC framework. The only major difference in the CAB-KGC model is that it does not require head entity descriptions and employs a classification loss (cross-entropy) instead of contrastive loss for training.
2. Presentation issues:
  
    2.1. Unclear figures: Figures are often unclear or poorly labeled, making it hard for readers to interpret their meaning. For instance, Figure 1 lacks detailed labeling and a proper explanation of how its components relate to the proposed methodology.

    2.2. Inconsistent mathematical notation: Symbols are used inconsistently. In the Introduction, the sets of entities and relations are referred to as $\mathcal{E}$ and $\mathcal{R}$, but in the Methodology section, they are denoted as $E$ and $R$. Additionally, the formulas presented lack rigor and are not sufficiently academic.

    2.3. Grammatical and typographical errors: The paper contains several issues with grammar and typos.

    2.4. Missing ablation studies: Ablation results and analyses are absent from the main text, and since reviewers are not required to consult the appendix, this omission is problematic. The authors should revise the structure of the paper.

3. Experimental design shortcomings:

    3.1. Small datasets: The experiments are conducted primarily on small datasets like WN18RR and FB15k-237. While these are common, evaluating the model on larger datasets, such as Wikidata5M, would better demonstrate the method’s generalizability.

    3.2. Training epoch limitations: The authors note that “The number of epochs was set to 30 for CAB-KGC and other models,” which may result in unfair comparisons since different models might require different numbers of training epochs.

    3.3. Lack of metric comparison: There is insufficient comparison between EDAS and traditional metrics like MRR, and the paper does not thoroughly explain the advantages of EDAS.

    3.4. No explanation of ablation results: The ablation results are not properly explained, making it difficult to assess their relevance or impact.

questions:
The authors can focus on addressing the third concern of the weaknesses (Experimental design shortcomings), providing additional results, clarifications, etc.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730414221446

thread_type:
official_review

<thread_object>
creator: reviewer_e1sp

<content>
summary:
This paper introduces CAB-KGC, which eliminates the need for entity descriptions and negative triplet sampling, reducing computation while improving performance. It leverages contextual information from neighboring entities and relationships to predict tail entities in knowledge graphs.

strengths:
1.The paper is clearly written and easy to follow.

2. The proposed CAB-KGC does not require negative sample training, enhancing training speed and resilience against negative sample selection, and eliminates reliance on entity descriptions, focusing solely on head and relationship contexts.

weaknesses:
1. The article only uses two datasets in its experiments and lacks large-scale datasets. The authors should consider supplementing the datasets.

2. The creation of Figure 1 is evidently too rough, including misaligned text and meaningless graphics in the small image on the left.

3. The model section of this paper is very brief, with the model being simply a BERT that takes in information from neighboring nodes. While this approach may be effective, I question whether the paper's innovation and interpretability are sufficient for acceptance at ICLR.

4 The paper repeatedly emphasizes the advantages of the proposed model on effiency; therefore, it would be beneficial to include comparative experiments on time complexity or runtime performance between the proposed model and the baseline.

questions:
Please refer to the "weaknesses" section.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730646114615

thread_type:
official_review

<thread_object>
creator: reviewer_y4mx

<content>
summary:
This paper proposes Context Aware BERT for Knowledge Graph Completion (CAB-KGC), which introduces contextual information to entities and relationships in KG, eliminates the need for entity descriptions and negative sampling, and reduces computational complexity while improving performance. In addition, this paper proposes an EDAS evaluation method to more comprehensively assess the performance of model.

strengths:
S1. The paper focuses on knowledge graph completion, which is an important issue. The language model-based method discussed in this paper exhibit a certain novelty compared to traditional structure-based approaches.

S2. The Evaluation based on Distance from Average Solution (EDAS) criteria used in the paper is a relatively novel evaluation metric that can better evaluate model performance in the presence of multiple metrics.

weaknesses:
W1. This work has limited technical contributions. This approach simply concatenates entities, relationships, and their contexts and inputs them into the language model, then calculates the probability of all available entities as tail entities. The idea lacks novelty.

W2. The description of methodology is not clear enough. The paper does not clearly specify the output of language model or how the language model output is used to compute the probability of the tail entity. Although Figure 2 suggests that the authors intend to use the embedding of CLS token for multi-class classification, this is not explicitly stated in the text.

W3. The paper suffers from a lack of consistency in its symbolic notation, which may lead to confusion for readers. For example, in the problem formulation paragraph of the methodology section, there are three different symbol expressions for knowledge graph G. 

W4. The paper contains numerous errors in details. For example, the Hadamard product symbol in the score function of RotatE is incorrect; there is an undefined symbol g in equation (11); the citations for ChatGPTzero-shot and ChatGPTone-shot are incorrect in Table 2.

questions:
How were the smaller datasets, such as NATION, LOCATION, COUNTRY, SPORT, and UML, constructed? Additionally, why were the ablation experiments conducted on these smaller datasets instead of on larger benchmarks like FB15k-237 and WN18RR?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730682453742

thread_type:
official_review

<thread_object>
creator: reviewer_aktk

<content>
summary:
This paper introduces a novel approach for Knowledge Graph Completion (KGC) named Context-Aware BERT for Knowledge Graph Completion (CAB-KGC). The goal of CAB-KGC is to predict missing entities or relationships in knowledge graphs by leveraging contextual information. Unlike traditional embedding-based methods that struggle with unseen entities and relationships, CAB-KGC utilizes the contextual data from neighboring nodes and relationships, integrating these insights with a BERT-based architecture to enhance prediction accuracy for tail entities. 
The authors also propose a new evaluation metric, Evaluation based on Distance from Average Solution (EDAS), to address potential inconsistencies in existing metrics like Mean Reciprocal Rank (MRR) and Hit@k. EDAS provides a more comprehensive assessment by considering deviations from average performance across several criteria. 
CAB-KGC is evaluated against state-of-the-art KGC methods on benchmark datasets FB15k-237 and WN18RR, showing improvements in standard metrics, particularly in Hit@1 and MRR. The experimental results demonstrate that CAB-KGC outperforms baseline methods across different datasets, indicating that incorporating both head and relationship contexts into BERT can improve KGC model accuracy. The paper also includes ablation studies to validate the contributions of each component within the CAB-KGC model, showing that combining head and relationship contexts yields the best results.
CAB-KGC addresses limitations in current KGC approaches by utilizing both structural and contextual information without relying on negative sampling or entity descriptions, and the EDAS metric offers an alternative way to rank model performance comprehensively.

strengths:
1.	Introduction of CAB-KGC for Knowledge Graph Completion: The paper presents a novel method, CAB-KGC, which leverages context-aware information from both head entities and relationships. This is an original approach for enhancing knowledge graph completion tasks, aiming to address limitations of existing embedding-based and LLM-based models. 2.
2.	Proposal of EDAS as a New Evaluation Metric: The introduction of the EDAS (Evaluation based on Distance from Average Solution) metric is a unique contribution. EDAS aims to offer a more comprehensive assessment by incorporating both positive and negative deviations, which could provide a more nuanced evaluation of model performance, especially on ranking tasks. 
3.	Comparative Experimental Results: The paper conducts extensive experiments comparing CAB-KGC with multiple state-of-the-art methods, particularly on FB15k-237 and WN18RR datasets. The results demonstrate that CAB-KGC achieves competitive performance, with notable improvements on the Hit@1 metric.

weaknesses:
1. Formatting and Layout Issues:

•	The paper has several formatting inconsistencies that detract from its professionalism. For instance, the template content for acknowledgments has not been removed from the end of the main text, which is distracting and unpolished.

•	In the appendix, there are large blank spaces between figures and tables, and the line thickness in tables is inconsistent, affecting visual uniformity.

•	The title for Figure 2 is positioned too close to the bottom margin, overlapping with the page number, which can cause readability issues. These layout errors suggest a need for a thorough review of formatting before submission.

2. Insufficient Detail in Methodology and Figures:

•	The paper’s core methodological explanation is sparse and lacks clarity, particularly in the section describing CAB-KGC. Statements such as "The CAB-KGC proposed method incorporates the importance of contextual information obtained from the head entity and the relationship and integrates with BERT" are vague and need further elaboration.

•	Figures intended to illustrate the model, especially Figure 1, have limited accompanying descriptions, which could hinder reader understanding. Additionally, some figures, particularly in the Appendix, are complex and lack explanatory text, making it challenging for readers to interpret them effectively.

3. Weak Justification for the New Evaluation Metric (EDAS):

•	While EDAS is introduced as a novel evaluation metric, the paper does not sufficiently motivate its necessity. The limitations of current evaluation standards (such as MRR and Hit@k) are only briefly mentioned, and it is unclear why EDAS would provide a significant advantage. A more detailed comparison of EDAS with traditional metrics, highlighting specific scenarios where EDAS offers clearer insights, would strengthen this contribution.

4. Limited Innovation in CAB-KGC:

•	CAB-KGC’s novelty is unclear. The model combines context-aware techniques with BERT embeddings, which, while valuable, may not be a groundbreaking innovation within the knowledge graph completion (KGC) domain. The approach might appear as an incremental improvement over existing models rather than a fundamentally new technique. It would be beneficial for the authors to clarify CAB-KGC’s unique contributions and differentiate it from similar methods.

5. Insufficient Experimental Validation:

•	The paper’s experiments are conducted on only two datasets, FB15k-237 and WN18RR, which limits the demonstration of the model’s generalizability. Adding results from a dataset with entity descriptions would provide a more comprehensive evaluation and demonstrate the model’s adaptability across diverse knowledge graphs.

•	The experimental setup also lacks an ablation study on a wider set of datasets or with a broader variation of components, such as testing CAB-KGC in contexts with different entity types and relationship structures.

6. Imbalance in Content Focus:

•	Although CAB-KGC is the main proposed method, much of the paper focuses on EDAS, which may dilute the impact of CAB-KGC. A better balance could be achieved by providing more detailed insights into CAB-KGC’s architecture, implementation, and performance analysis. Readers might expect the majority of the paper to focus on the core model rather than the evaluation metric.

7. Page Length and Presentation Issues:

•	The main content is slightly shorter than the ICLR recommended 9-page limit, which might suggest a lack of comprehensive analysis or additional experimental validation. Expanding sections on methodology and experiments could provide a fuller picture of the model’s contributions.

•	In terms of presentation, the writing style sometimes lacks precision, and the images could benefit from clearer, more thorough captions and descriptions. Improved organization and clarity would enhance readability and comprehension.

questions:
1. Formatting: There are formatting issues, such as the residual template text and inconsistent spacing. Will you address these for better readability?
2. EDAS Motivation: Can you elaborate on the limitations of traditional metrics that EDAS addresses, specifically in the context of KGC?
3. Methodology Clarity: The methodology has vague phrases like “incorporates contextual information.” Could you clarify this with more detail?
4. Broader Validation: Why were only two datasets used, and do you plan to test CAB-KGC’s generalizability on more datasets?
5. Figure Explanations: Could you add clearer captions or appendix notes to explain complex figures like Figure 1?
6. Ablation Analysis: Can you discuss the contribution of each component in CAB-KGC, specifically “head context” and “relationship context”?
7. Computational Cost: CAB-KGC is computationally intensive. Could you clarify the trade-offs between this cost and performance gains?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732611541645

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Revised Manuscript and A list of changes made

comment:
Dear Reviewers

We would like to sincerely thank you for a very thorough evaluation of our submission and your constructive feedback. Your thoughtful suggestions have significantly contributed to improving the clarity and quality of our paper. We have carefully addressed each of the points raised and made the necessary revisions to enhance the presentation and strengthen the content of our work.
We greatly appreciate the valuable time and effort you put in reviewing our manuscript, and we believe that the revisions made will help in making the paper more polished and comprehensive. Thank you again for your valuable input. We trust that the revisions adequately address the reviewer’s concerns and contribute to enhancing the overall quality and clarity of our work. We look forward to the reviewer's final evaluation based on these updates. 


__Below are the key revisions made to the manuscript.__ 

1. __Introduction:__ Included a new paragraph addressing the limitations of commonly used assessment metrics in KGC.
2. __Table 1:__ Updated to highlight the distinctive structural features of knowledge graphs that our model leverages.
3. __Methodology and Figure 1:__ Revised thoroughly to provide a detailed explanation of our model's working.
4. __Datasets:__ Expanded experiments to include two additional datasets, CoDEx-S and ConceptNet100K.
5. __Experiments Section:__ Added a new subsection discussing the advantages of the EDAS assessment framework.
6. __Metrics Reporting:__ Provided comprehensive results for common metrics across four datasets: FB15k-237, WN18RR, CoDEx-S, and ConceptNet100K.
7. __Ablation Studies:__ Introduced a new section analyzing the contributions of our model’s components.
8. __EDAS Rankings:__ Added a table presenting the EDAS-based rankings of all evaluated models.

_The code will be shared upon the acceptance of the paper to ensure proper access and usage_.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732612567715

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Clarifying the Novelty and Impact of CAB-KGC

comment:
We appreciate the reviewers' feedback and would like to address the concerns regarding the novelty and technical contributions of our work. While our model builds on prior research, its contributions are substantial and address critical limitations in the state-of-the-art methods. Our proposal is simple but meaningful and powerful. We want to explicity exploit multiple structural features of knowledge graphs for tail prediction that no previous works have leveraged. Below, we provide a detailed explanation of the innovations introduced in our work, supported by experimental evidence and technical insights.

___Novel Contributions of CAB-KGC___

1. **Head Context Extraction Beyond Descriptions**:  
   CAB-KGC utilizes **head context** ($H_c$), a combination of all neighboring *relationships* and *entities* connected to the head entity. The *entities* information helps the model identify how the head entity $h$ is positioned within its local neighborhood (e.g., is it a hub, part of a chain, etc.). The neighbouring *relationships* information offers semantic understanding of the types of relationships the head entity $h$ is generally involved in. This approach captures a richer structural and relational representation of the head entity that NNKGC and similar models fail to consider as these models consider neighboring *entities only*.

2. **Incorporation of Relationship Context**:  
   Unlike NNKGC, SimKGC and other existing methods, the CAB-KGC model uniquely incorporates **relationship context** ($R_c$), which includes all entities associated with the operational relationship $r$ in the knowledge graph. This provides the model with a global perspective on the nature of the relation $r$ itself and the patterns or clusters involving the relation $r$. $R_c$ therefore provides __generalization__ ability.  By considering global patterns of the relation, it acts as a regularizer, ensuring that the model aligns with broader relational constraints in the KG. Existing methods, including NNKGC, SimKGC, don't exploit this, thereby failing to fully leverage the structural richness of knowledge graphs.

3. **Independence from Entity Descriptions**:  
   Unlike SimKGC and other LLM-based methods, CAB-KGC does not rely on entity descriptions for its operation. This is a critical distinction because many real-world knowledge graphs lack descriptive information for entities. The ability to perform KGC without entity descriptions broadens the applicability of CAB-KGC, making it suitable for datasets where descriptive metadata is sparse or unavailable. For example, we showed that our model demonstrated a substantial improvement in Hits@1 compared to state-of-the-art methods *without using entity descriptions*.

4. **Classification Loss as Training Strategy**:  
   The adoption of cross-entropy loss is not merely a substitution for contrastive loss but is better aligned with the KGC task. _Contrastive Loss_ as used in simKGC requires extensive negative sampling, which increases computational cost and introduces noise, especially on large-scale knowledge graphs. _Cross-Entropy Loss_ eliminates the reliance on negative sampling, which is computationally efficient and better suited for large-scale and incomplete knowledge graphs.

5. **Significant Performance Gains**:  
   CAB-KGC achieves **state-of-the-art results on multiple datasets**, including FB15k-237, WN18RR, ConceptNet100K  and CoDEx-S with CAB-KGC as top performer in three of four datasets. These results underscore the effectiveness of the proposed model in reliably predicting tail entities across diverse and challenging datasets.

6. **Introduction of EDAS in KGC**:  
   Furthermore our work introduces EDAS as a useful evaluation framework for Knowledge Graph Completion (KGC), addressing key limitations of existing metrics. Unlike traditional methods (e.g., MRR, Hits@K), which are often inconsistent, dataset-biased, and assessed in isolation, EDAS provides a unified, aggregated, and dataset-agnostic metric. Ours is the __first application of EDAS in KGC__, demonstrating its ability to holistically balance precision and coverage while penalizing biases inherent to specific datasets. Our empirical results validate its effectiveness, highlighting CAB-KGC’s good performance across multiple datasets—in assigning it the top EDAS rank outcompeting Llama-based DIFT model and others. EDAS thus enhances both the fairness and simplicity of KGC model evaluations.

In summary, the novelty of CAB-KGC lies in its **context-aware design, independence from entity descriptions, and efficient training paradigm**. These contributions directly address limitations in existing methods, making CAB-KGC a significant step forward in KGC research. The experimental results further validate its impact, achieving state-of-the-art performance on challenging benchmarks.


We hope this detailed explanation clarifies the innovative aspects of our work and demonstrates its value to the field.
</content>
</thread_object>

</thread>


Metareview: