Title: KG-BiLM: Knowledge Graph Embedding via Bidirectional Language Models
Venue: NeurIPS 2025
Accepted: no
Keywords: Knowledge Graph Embedding

Language Model

Transformer

Attention Mechanism
Abstract: Recent advances in knowledge representation learning (KRL) highlight the urgent necessity to unify symbolic knowledge graphs (KGs) with language models (LMs) for richer semantic understanding. However, existing approaches typically prioritize either graph structure or textual semantics, leaving a gap: a unified framework that simultaneously captures global KG connectivity, nuanced linguistic context, and discriminative reasoning semantics. To bridge this gap, we introduce KG-BiLM, a bidirectional LM framework that fuses structural cues from KGs with the semantic expressiveness of generative transformers. KG-BiLM incorporates three key components: (i) Bidirectional Knowledge Attention, which removes the causal mask to enable full interaction among all tokens and entities; (ii) Knowledge-Masked Prediction, which encourages the model to leverage both local semantic contexts and global graph connectivity; and (iii) Contrastive Graph Semantic Aggregation, which preserves KG structure via contrastive alignment of sampled sub-graph representations. Extensive experiments on standard benchmarks demonstrate that KG-BiLM outperforms strong baselines in link prediction, especially on large-scale graphs with complex multi-hop relations—validating its effectiveness in unifying structural information and textual semantics.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the KG-BiLM framework, which realizes knowledge graph embedding through a bidirectional language model, aiming to unify the semantic understanding of symbolic knowledge graphs and language models. The core innovations include bidirectional knowledge attention, knowledge mask prediction, and contrastive graph semantic aggregation. Experiments show that KG-BiLM outperforms comparative methods in the link prediction task, verifying the effectiveness of fusing structural and semantic information.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths：\n1) This paper proposes a \"bidirectional reasoning + contrastive aggregation\" framework, providing new ideas for solving challenges such as long-tail entities and multi-hop relationships in knowledge graphs. Experiments verify the necessity of each component. \n2) The paper has a complete structure and sufficient experiments, demonstrating the generalization ability of the proposed method.  \n3) The code and dataset are open-sourced, enhancing the reproducibility of the research.\n\nWeaknesses:\n\n1） Modern knowledge graphs are often updated. Does the model need to be retrained when new entities or relationships are added?  \n2） The paper does not mention the capability of incremental learning. For example, how does the model handle new entities when there is no textual description?  \n3） The paper mentions using a 24-layer Transformer with 1024-dimensional embeddings. How does the training time and resource consumption of such an architecture perform on large-scale datasets?  \n4） It is recommended that the authors supplement detailed descriptions of training time and memory usage. For instance, how much GPU time is required to train on Wikidata5M, and whether it is more efficient than baseline models?\n\nQUESTIONS:\n1) Please provide the specific training time consumption on each dataset, GPU memory usage, and efficiency comparison with baseline models to evaluate the engineering practicality.  \n2) Does the model support incremental updates? When new entities have no text descriptions, how is representation accuracy maintained? It is recommended to add transfer learning experiments in dynamic scenarios.  \n3) Please supplement the comparison results with large language models such as GPT-4o and LLaMA-3 to ensure the timeliness of the baselines.\nIf the authors can address these questions, I will consider raising my score.\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 3, "REVIEWER_CONFIDENCE": 4, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-06-24"}

comments: SUMMARY:
- This paper proposes a KG-BiLM model to learn knowledge graph embedding from bidirectional language models. KG-BiLM employs  Bidirectional Knowledge Attention, Knowledge-Masked Prediction, and Contrastive Graph Semantic Aggregation to achieve text-graph co-learning for better performance.

STRENGTHS AND WEAKNESSES:
- Equation4-7 are the most basic transformer design, there is no need to take up so much space in the article, they can be removed and more experiments and analysis can be added.
- Though KG-BiLM can be transferable in the zero-shot setting, authors do not explore the transferability of KG-BiLM in the inductive KGC tasks. I think traditional inductive settings can better reflect KG-BiLMs zero-shot performance.
- In the visualization experiments, authors do not compare the learned representations from baseline methods. It seems that many baselines using contrastive learning for KGC can achieve similar entity clusterings. 
- The training / inference efficiency of KG-BiLM is not explored in this paper.
- See the question section for more question about this paper.

QUESTIONS:
- After reading this paper, I can not find which LM you used as intialization weight for KG-BiLM. Does it means KG-BiLM train a transformer model from scratch?
- For the CGSA module, maximizing the mutual information in subgraphs is a common idea for current graph and KG papers [1]. Can you explain the core novelty of CGSA compared with existing methods?
- Nowadays there are a lot of further improvements on transformer details, such as position embedding, attentions, do the authors try to validate the results from these new module designs?


[1] Subgraph Neighboring Relations Infomax for Inductive Link Prediction on Knowledge Graphs

LIMITATIONS:
Yes
TITLE: Official Review
CLARITY: 2
IMPACT: 3
ORIGINALITY: 3
DATE: 2025-06-28

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces KG-BILM, a unified knowledge representation learning framework that integrates knowledge graph structural information with the semantic capabilities of language models. It combines three key components: (1) Bidirectional Knowledge Graph Attention; (2) Knowledge-Masked Prediction; and (3) Contrastive Graph Semantic Aggregation. The framework aims to unify KG structures with textual semantics, therefore achieving higher information density, integrating linguistic cues, and enabling zero-shot encoding. Experimental results on standard benchmarks like WN18RR, FB15k-237, and Wikidata5M demonstrate KG-BILM's superior performance in link prediction, particularly for complex multi-hop relations, compared to traditional KGE methods.\n\nSTRENGTHS AND WEAKNESSES:\n### Strengths\n\n- **Quality**: Technically robust approach, with a well-designed architecture that integrates bidirectional attention, masked prediction, and contrastive learning. Experimental results show strong performance in link prediction for complex graphs.\n- **Originality**: KG-BILM introduces a novel hybrid approach that bridges the gap between symbolic learning and generative LMs. The use of bidirectional attention to capture inter-triple dependencies and contrastive learning for structural-semantic alignment is innovative and distinct from traditional KGE or encoder-based KRL methods.\n- **Significance**: The framework’s ability to unify structural and textual semantics enable its practability in fusing knowledge from different modalities. Its zero-shot capabilities enhance its transferability across domains.\n- **Clarity (partial)**: The introduction and motivation are clearly articulated, effectively highlighting the limitations of existing KGE and KRL methods, the description of KG-BILM’s components is precise.\n\n\n\n### Weakness\n\n- **Quality**: No discussion of computational efficiency or scalability, which is critical for large-scale KGs like Wikidata5M\n- **Significance**: While the paper outlines promising applications (e.g., healthcare, education), it does not provide concrete examples or case studies demonstrating these benefits, reducing the perceived impact. \n- **Originality**: While innovative, the paper could better differentiate KG-BILM from recent works combining symbolic and neural models [1] [2] [3].\n\n\n\n[1] Taming Language Models for Text-attributed Graph Learning with Decoupled Aggregation, ACL, 2025\n\n[2] Learning on large-scale text-attributed graphs via variational inference, ICLR, 2023\n\n[3] Differentiable Neuro-Symbolic Reasoning on Large-Scale Knowledge Graphs, NeurIPS, 2023\n\nQUESTIONS:\nplz see above weakness part\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 5, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 2, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new bidirectional language model framework, KG-BiLM, for knowledge graph embedding. The authors emphasized the importance of global KG connectivity and presented three components to fuse the KG structural cues to LM: 1. A new mask strategy that removes the masks over entities in the text but connected by KG, 2. The mask over prediction will be mixed by entities and text to a ratio, forcing the model to learn both structural and semantic clues. 3. A contrastive learning diagram: the positive views are from the same entity description or subgraph but with different random dropout masks. With the proposed components, KG-BiLM performs well on 4 datasets and exhibits zero-shot learning ability on Wikidata5M.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n-- The writing is clear. The contribution and components are well explained.\n-- The results are good, and the ablation study supports the effectiveness of three components.\n-- The new mask strategy by graph structure is inspiring.\n\nWeaknesses:\n-- Illustrations of figures are confusing and not clear.\n-- Deeper analysis is missing for a) Which component is important for zero-shot reasoning b) the individual contribution from text description and graph structure\n-- Missing related work introduction for GNN-based KGE models, which are used as baselines.\n\nQUESTIONS:\n-- NBFNet serves as an important baseline as it outperforms KG-BiLM on one dataset.  However, its illustration in related work seems missing. \n\n-- Captions for figures. Figure 1 is not easy to understand. It’s hard to locate which component the descriptions is talking about. And in figure 1, the author claims that translation-based model rates 1-N triples almost identically, but many works(TransH, TransR…) based on TransE solved it. This is confusing. For figure 2, the caption does not have any description.\n\n-- In B.1, the author claimed that (ii) most existing geometry-based models must be retrained whenever a new graph is introduced, hampering scalability. However, recent work ULTRA https://arxiv.org/pdf/2310.04562 shows that GNN-based models can be trained from scratch and show 0-shot inference ability on new datasets. Also, from the experiment it seems that KG-BiLM also trains on each new dataset separately. The zero-shot reasoning ability in 4.5 is tested on Wikidata5M again. Do you have a uniform training checkpoint for all dataset and enable cross-dataset zero-shot inference? \n\n-- Following point 3, the code provided in the link lacks a detailed readme file. It’s very hard to judge if KG-BiLM is capable of point 3.\n\n-- The author mentions the model’s zero-shot capability in lines 177-181. Is KMP the only component that contributes to zero-shot capability?\n\n-- In general, the structural and linguistic patterns are entangled whenever it’s mentioned in the text. Where does the model improvement come from? Utilizing structural or linguistic patterns? What’s the case when the model is predicting on datasets without entity description?\n\n-- In line 167, the author describes Position-Shifted Loss Assignment as a key innovation and describes 3 benefits in line 174. However, no following experiments supports its claimed benefits (a single position shifted KMP scheme.)\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-02"}