## 3 KGs Learning
### Transformer-Based Learning
#### 3.4.2 Fusion of Topology and Logical Rules  
The Fuse Topology contexts and Logical rules in Language Model (FTL-LM) [135] enhances knowledge graph (KG) learning by combining graph topology information (structural relationships between nodes) with logical rules. This model imposes constraints on relationships within the graph through logical rules while learning the structural interactions of nodes and edges via topology information. For instance, it improves the reliability of relation prediction using rules like \"If X is related to Y, and Y is related to Z, then X is also related to Z\". This approach excels in domains where logical consistency is critical, such as healthcare and the law.  
The Medical Bidirectional Encoder Representations from Transformers (MCBERT) [136] system learns the relationships among diseases, drugs, and symptoms using domain-specific KGs. By modeling complex relationships in medical data using KGs, MCBERT facilitates efficient analysis and prediction. It is particularly useful for tasks such as drug-disease interactions, prescription suitability evaluations, and patient profiling, supporting medical professionals in decision-making.  
The Adaptive Hierarchical Transformer with Memory (AHTM) [137] combines a hierarchical Transformer architecture with model compression techniques to effectively generate KG embeddings for large-scale medical datasets. This method accounts for the high-dimensional nature of medical data and learns key patterns and relationships while reducing memory usage and computational costs. For example, it processes extensive medical datasets, including patient records, pharmaceutical databases, and genomic information to produce embeddings that maintain predictive accuracy while enhancing computational efficiency.  
Ref. [138] explored the potential of integrating large language models (LLMs) with graph data to expand and utilize KGs. This survey proposes methods for combining LLMs' language understanding and generation capabilities with graph-based learning to effectively learn new entities and relationships. By merging the structural information of graph data with the rich textual representations of LLMs, this approach achieves strong performance in various tasks, such as KG-based question answering, summarization, and relation prediction.  
\begin{table}
\begin{tabular}{c c c c c} \hline \hline
**Method** & **Core Technique** & **Pros** & **Cons** & **Typical Use Cases** \\ \hline RNN (e.g., LSTM/GRU) & Sequential processing of tokens over time & Efficient, Simple & Limited, Forgetful & Small-scale, Streaming \\ \hline Transformer (e.g., BERT) & Self-attention mechanism (non-sequential) & Powerful, Accurate & Expensive, Memory intensive & Large-scale, Complex \\ \hline \hline \end{tabular}
\end{table}
Table 1: Comparison of RNN-based methods vs. transformer-based methods.  
The Unsupervised Semantics and Syntax into heterogeneous Graphs (USS-Graph) [139] adopts unsupervised learning techniques to efficiently learn from graph data. This model simultaneously captures the structural features and syntactic semantics of a graph and models node and edge interactions with precision. By automatically learning significant patterns and relationships in the graph without explicit labels, it demonstrates high scalability, even with large datasets. This approach is applicable to graph-based text classification, relation extraction, and other similar tasks.