- GCN. [15] Graph Convolutional Network (GCN) is a classical model that works by performing a linear approximation to spectral graph convolutions.
- GraphSAGE. [7] GraphSAGE is a GNN model that focuses on inductive node classification, but can also be applied for transductive settings.
- GAT. [16] Graph Attention Network (GAT) introduces the attention mechanism to capture the importance of neighboring nodes when aggregating information from the graph.
- RevGAT. [33] RevGAT combines reversible connectivity with a deep network architecture to form a deep and efficient GNN.
- NodeFormer. [46] NodeFormer is a scalable graph Transformer for large-scale graphs, which achieves all-pair message passing with linear complexity. In the table, we denote NodeFormer by NFormer.
- GIN. [17] Graph Isomorphism Network (GIN) overcomes the drawbacks of previous MPNN-based methods, which struggles to differentiate various graph structures based on the generated graph embeddings.
- JKNet. [18] Jumping knowledge Network (JKNet) adaptively varies the neighborhood ranges for individual node, enabling enhanced structure-aware node representation.
- MoNet. [47] Mixture Model Network (MoNet) captures and represents the structural properties of graphs by incorporating multiple localized perspectives of each node's neighborhood.
- APPNP. [48] Approximate Personalized Propagation of Neural Predictions (APPNP) is specifically developed for semi-supervised learning tasks on graph-structured data. It utilizes personalized propagation to iteratively enhance node predictions by incorporating comprehensive information from both local and global contexts.
- DistilBERT. [14] DistilBERT is another distilled version of BERT. The student model DistillBert shares a similar overall architecture with the teacher model BERT. DistillBert differs from BERT by removing the token-type embeddings and pooled components, and also reducing the number of layers.
- ELECTRA. [11] ELECTRA is a pre-training model for NLP tasks that introduces Discriminative Pre-training. ELECTRA enhances the efficiency and effectiveness by generating more efficient representations through a discriminative approach.
- BERT. [10] Bidirectional encoder representations from Transformers (BERT) utilizes a transformer architecture that employs self-attention mechanisms to capture word relationships within sentences. This enables the model to effectively consider both the preceding and succeeding contexts of a word, facilitating bidirectional language comprehension. BERT undergoes unsupervised pre-training on an extensive text corpus, where it predicts masked words within sentences and acquires the ability to encode contextual information.
- RoBERTa. [12] Robustly Optimized BERT (RoBERTa) is a variant of the BERT. RoBERTa incorporates additional modifications during pre-training to optimize its performance. In the pretraining phase, it trains on a vast corpus of unlabeled text data by employing masked language modeling (MLM) while excluding the next sentence prediction (NSP) task. RoBERTa significantly expands the amount of training data used, enabling the acquisition of more comprehensive and robust language representations.
- DeBERTa. [13] Decoding-enhanced BERT with Disentangled Attention (DeBERTa) brings forth two significant enhancements: disentangled attention and heterogeneous layer integration. By dividing the attention heads into distinct groups, disentangled attention empowers the model to better capture dependencies among words, enabling more targeted and specialized attention. This enhancement greatly improves the model's ability to capture long-range dependencies in a more effective manner.