SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 2: Different downstream task results (\%) with the corresponding std ( $\pm$ ) on two datasets. The best performances are in bold and the second runners are shaded in gray, where * denotes a significant improvement according to the Wilcoxon signed-rank significance test.
Task	Node Classification	Link Prediction
Dataset	PubMed	MIMIC-III	PubMed	MIMIC-III
Metric	Macro-F1	Micro-F1	Macro-F1	Micro-F1	AUC	MRR	AUC	MRR
M2V	15.35	20.27	19.69	29.24	74.53	89.58	75.05	88.32
(±1.27)	(±3.01)	(±0.62)	(±1.57)	(±3.79)	(±2.05)	(±0.41)	(±0.23)
HIN2Vec	11.57	18.92	19.12	28.05	74.21	90.56	73.46	88.10
(±1.23)	(±2.78)	(±1.32)	(±1.44)	(±5.49)	(±1.06)	(±0.41)	(±0.14)
ConvE	16.06	19.16	24.44	32.89	76.48	92.27	69.56	84.88
(±3.69)	(±4.00)	(±1.28)	(±0.86)	(±4.31)	(±0.57)	(±0.36)	(±0.25)
ComplEx	13.93	18.27	9.82	21.39	79.81	91.79	63.86	81.40
(±2.59)	(±4.12)	(±0.56)	(±3.12)	(±0.97)	(±0.48)	(±0.42)	(±0.40)
SimKGC	21.97	30.83	51.62	58.50	79.62	91.43	67.73	84.86
(±3.51)	(±3.10)	(±1.81)	(±1.52)	(±2.72)	(±0.48)	(±1.69)	(±0.54)
RGCN	12.50	18.50	7.19	14.55	72.08	88.20	57.31	73.91
(±2.36)	(±1.41)	(±0.77)	(±3.25)	(±1.13)	(±0.47)	(±0.71)	(±0.57)
HAN	15.29	16.95	6.98	14.73	70.57	87.89	-	-
(±2.87)	(±2.71)	(±0.58)	(±1.69)	(±1.58)	(±0.62)	-	-
HGT	11.98	20.12	8.03	17.79	77.24	89.63	64.01	81.54
(±2.23)	(±3.89)	(±0.87)	(±0.83)	(±3.50)	(±0.84)	(±0.36)	(±0.56)
HeCo	10.32	18.01	10.78	15.26	65.04	83.29	53.13	71.81
(±1.12)	(±0.87)	(±0.41)	(±1.52)	(±1.26)	(±0.72)	(±0.47)	(±0.35)
SHGP	10.80	19.28	11.34	17.44	68.22	85.34	54.49	72.58
(±3.03)	(±0.91)	(±1.29)	(±1.49)	(±2.71)	(±0.48)	(±0.33)	(±0.24)
LM	40.10	44.71	54.51	61.27	60.20	84.23	51.21	74.22
(XRoBERTa)	(±4.62)	(±3.68)	(±1.50)	(±1.22)	(±2.78)	(±1.71)	(±0.17)	(±0.26)
LM	59.43	61.53	70.26	72.67	51.71	80.54	50.66	72.36
(GPT-2)	(±4.73)	(±3.43)	(±1.43)	(±0.90)	(±3.67)	(±2.49)	(±0.74)	(±0.86)
LM	58.29	60.57	66.25	70.14	60.97	83.00	51.44	75.09
(DRoBERTa)	(±2.44)	(±2.11)	(±1.60)	(±1.52)	(±2.98)	(±0.40)	(±0.14)	(±0.29)
LM	13.83	22.70	14.32	24.59	72.35	88.86	58.62	78.78
+RGCN	(±0.73)	(±3.25)	(±0.87)	(±1.17)	(±4.34)	(±1.46)	(±0.50)	(±0.10)
LM	12.81	21.79	10.49	20.57	82.97	89.98	65.01	82.28
+HGT	(±1.22)	(±3.54)	(±0.41)	(±0.97)	(±3.91)	(±0.88)	(±0.20)	(±0.30)
WalkLM	60.42*	62.33*	75.16*	77.89*	85.65*	94.16*	82.15*	92.78*
(±2.62)	(±3.13)	(±0.93)	(±0.70)	(±3.28)	(±0.37)	(±0.67)	(±0.68)
training tasks jointly based on a target set of relations to learn node embeddings and meta-paths on HINs.
(2) Relation learning-based methods: ConvE [6] proposes to use 2D convolution over embeddings and multiple layers of non-linear features to model KGs. ComplEx [49] handles a large number of binary relations using complex-valued embeddings on KGs. SimKGC [52] proposes to elicit the implicitly stored knowledge from BERT and designs a text-based contrastive learning mechanism for knowledge graph completion.
(3) Supervised heterogeneous graph neural networks (HGNNs): RGCN [45] proposes to apply GCN to model HINs or KGs. HAN [53] proposes to learn the importance between a node and its meta-path based neighbors on HINs. HGT [20] proposes to use each edge's type to parameterize the transformer-based self-attention architecture on HINs. For the above supervised HGNNs, we use link prediction loss introduced in GraphSAGE [14] to achieve unsupervised learning (i.e., without any node labels), following existing studies on HINs [11, 61].
(4) Unsupervised HGNNs: HeCo [54] proposes a co-contrastive learning mechanism for HGNNs. SHGP [63] designs a self-supervised pre-training method for HGNNs.
Settings. We mainly compare ten algorithms under the setting of unsupervised graph learning. The full code for this work is available<sup>7</sup>. All the models are optimized through the Adam optimizer and
<sup>7</sup>https://github.com/Melinda315/WalkLM
Table 3: Node classification results (\%) in the few-shot setting with Macro-F1 (abbr. Ma-F1) and Micro-F1 (abbr. Mi-F1) metrics.
Dataset Setting Metric	PubMed	MIMIC-III
1 shot	3 shot	5 shot	1 shot	3 shot	5 shot
Ma-F1	Mi-F1	Ma-F1	Mi-F1	Ma-F1	Mi-F1	Ma-F1	Mi-F1	Ma-F1	Mi-F1	Ma-F1	Mi-F1
ComplEx	9.31	12.51	10.32	13.26	10.12	15.94	2.82	5.29	2.00	3.03	3.87	9.26
M2V	9.86	13.42	10.27	12.56	12.97	14.98	5.83	8.72	3.91	5.11	3.40	4.49
ConvE	13.23	13.45	8.84	10.93	11.25	13.53	5.85	6.75	5.61	7.24	6.31	7.69
RGCN	9.34	11.02	8.57	10.58	10.84	13.43	4.97	5.82	5.43	6.28	5.22	5.73
HIN2Vec	8.46	10.54	9.04	12.79	10.96	17.39	5.72	10.33	4.90	5.72	3.57	4.91
SHGP	8.94	12.79	9.12	11.73	10.53	15.14	4.12	6.35	5.36	6.58	4.47	5.34
WalkLM	28.09*	30.94*	32.11*	35.35*	35.41*	37.68*	23.33*	27.96*	34.19*	40.49*	41.12*	46.83*
the learning rate is searched in [1e-4, 1e-2]. The hyper-parameters of baselines are chosen carefully based on either grid search or their official source codes. For all the methods, we use a five-fold cross-validation for a more reliable evaluation of the model's performance. All the experiments are performed with two NVIDIA GTX 3090 Ti GPUs. For link prediction, we deploy the Large Margin Nearest Neighbor (LMNN) technique based on the embeddings generated by our Wa1kLM. Then we construct feature vectors for edges.
Main Results. For node classification, we train a separate one-layer MLP classifier based on the learned embeddings on  $80\%$  of the labeled nodes and predict the remaining  $20\%$ . All the methods are trained in an unsupervised manner without classification labels. We evaluate Wa1kLM with Macro-F1 (across all labels) and Micro-F1 (across all nodes).
As shown in Table 2, our proposed WalkLM has superior performance, indicating the importance of leveraging both semantic and structural information in attributed graphs. WalkLM achieves  $138.59\%$  performance gains on PubMed over the second-best performance on average while achieving  $39.37\%$  average performance gains on MIMIC-III. Specifically, SimKGC achieves second-best performance by effectively employing text-based contrastive learning, which leverages BERT to capture a rich set of semantic information. Compared with SimKGC, WalkLM can effectively combine the complex semantic and graph structure information of attributed graphs, so as to accurately model the complex attributes of nodes. Although the HGNNs can naturally model attributes, their unsupervised training mechanisms likely do not align well with the downstream prediction task of node classification.
Results in the Few-shot Setting. Since one key challenge of node classification lies in the generalizability and adaptability of models [63], we design a few-shot setting to evaluate models in extending knowledge to unseen scenarios and adapting to new tasks with limited training data.
As shown in Table 3, our framework can stay strong with a small size of training data, where we win a  $275.45\%$  performance gain over the second-best performance on average. Through the novel textualization process that converts general attributed graphs into text-like sequence data, our proposed Wa1kLM can leverage the capabilities of modern language models for graph representation learning. With the extensive pre-training of LMs on broad text corpora, Wa1kLM can easily understand meaningful node attributes given a new graph, while the random walk strategy further allows it to capture graph structures. Consequently, Wa1kLM maintains superior performance even in the few-shot setting. Similar to the main results in Table 2, the ranking of baselines is fluctuating across datasets, where ConvE and M2V continue to exhibit promising performance. Note that ComplEx and HIN2Vec exhibit notable improvements in this setting, likely because HIN2Vec can also learn node representations based on random walks and ComplEx can capture fine interactions through complex-valued vectors, thus being more capable of capturing comprehensive node information before supervision.
5.3 Link Prediction
For link prediction, we train all models with the randomly selected  $80\%$  links and evaluate towards the  $20\%$  held-out links. We use the Hadamard function to construct feature vectors for node pairs and train a two-layer MLP classifier on the  $80\%$  training links. We evaluate Wa1kLM with AUC (area under the ROC curve) and MRR (mean reciprocal rank). Note that, HAN cannot predict links on MIMIC-III for its restriction to embed only one type of node at a time, and thus it cannot predict links between different types of nodes on MIMIC-III [53].
(a)  $N$  on node classification
(b)  $N$  on link prediction
(c)  $\alpha$  on node classification
Figure 3: Analysis of the number of sampled walks  $N$  and the termination probability  $\alpha$ .
(d)  $\alpha$  on link prediction
As shown in Table 2, our fine-tuned Wa1kLM demonstrates outstanding performance in uncovering latent associations among nodes in attributed graphs. In general, Wa1kLM outperforms all ten baselines with an average of  $5.97\%$  performance gain over the second-best performance, showing that our proposed framework can learn accurate edge representation for link prediction. As a RW-based method, M2V can effectively employ meta-path-guided random walks to capture topological information and trace meta-path to understand relations between nodes. As relation-learning methods, ConvE and ComplEx design different deep neural models to evaluate triplets. ConvE can achieve good performance by using convolutions over embeddings to mine relations between entities. ComplEx can capture relations via complex-valued embeddings, so as to better represent the complex inherent relations among entities. Compare to M2V, ConvE, and ComplEX, Wa1kLM can effectively capture the complex relations by providing characteristic graph traits and reconstructing network proximity of nodes that inherit from RWs. On the other hand, the unsupervised HGNNs, especially HeCo and SHGP, perform rather poorly, again because their training mechanisms are not aligned with the link prediction task. Such observation is consistent with the results in the recent work [72], showing the heterogeneous approaches that only preserve certain-type entities fail to capture accurate representations for all kinds of nodes.
To better understand our proposed techniques, we closely study our framework by selecting different LMs and varying the graph-aware LM fine-tuning mechanism.
Compared with the graph-based baselines, the LM-based models (e.g., LM (XRoBERTa $^8$ ), LM (GPT-2 $^9$ ), and LM (DRoBERTa $^{10}$ )) are able to learn accurate and rich node attributes. However, it is difficult for them to mine the relations between nodes in the attributed graph, where all of them perform worse than the RW-based and relation learning-based methods in the link prediction task. Considering the overall performance of the above three LMs on two different tasks and the goal of learning graph embedding, we choose LM (DRoBERTa) as our starting point for fine-tuning.
Furthermore, we show that LM can further effectively integrate with existing heterogeneous graph algorithms, such as  $\mathrm{LM} + \mathrm{RGCN}$  and  $\mathrm{LM} + \mathrm{HGT}$ , resulting in a notable performance enhancement over their individual methods. Compared with  $\mathrm{LM} + \mathrm{RGCN}$  and  $\mathrm{LM} + \mathrm{HGT}$ , our proposed graph-aware LM fine-tuning can achieve the largest improvement gains based on the chosen LM (DRoBERTa) in both node classification and link prediction tasks, showing the effectiveness of capturing topological information together with semantics in modeling attributed graphs. The detailed analysis of the ablation studies is shown in Appendix A.3.
5.5 Hyper-parameter Studies
In this subsection, we investigate the model sensitivity on the number of sampled walks  $N$  and the termination probability  $\alpha$ , which are the major hyper-parameters in WalkLM. For the space limitation, we show results on PubMed in Figure 3 and the results on MIMIC-III in Appendix A.4. Overall, WalkLM is not sensitive to the two hyper-parameters, where its performance increases slowly with  $N$  and  $\alpha$ . Note that, too small  $N$  or large  $\alpha$  can cause the textualization data  $|C|$  to lose
<sup>8</sup>https://github.com/facebookresearch/fairseq/tree/main/examples/xlmr
$^{9}$ https://github.com/openai/gpt-2
$^{10}$ https://github.com/huggingface/transformers/tree/main/examples/research_projects/distillation
(a) M2V on PubMed
(b) ConvE on PubMed
(e) M2V on MIMIC-III
(f) ConvE on MIMIC-III
(c) HGT on PubMed
(g) HGT on MIMIC-III
Figure 4: Visualization of different types of node embeddings on PubMed and MIMIC-III.
(d) WalkLM on PubMed
(h) WalkLM on MIMIC-III
sufficient information, while too large  $N$  or small  $\alpha$  lead to extensive  $|C|$  and increase computational costs for fine-tuning. Setting  $N$  around  $3 \times 10^{5}$  and  $\alpha$  around 0.05 seems appropriate to generate sufficient textual sequences, which can achieve a good balance of performance and efficiency. We additionally investigate the model's sensitivity to the quantity of language model masking samples, aiming to elucidate the parameter's influence on the performance of downstream tasks. The detailed experimental results are shown in Appendix A.4.
5.6 Visualization
For an intuitive comparison, we visualize the embedding space of different types of nodes which are learned by M2V, ConvE, HGT, and our Wa1kLM, respectively. Specifically, we select gene/disease nodes on PubMed and patient/visit/disease nodes on MIMIC-III. The embeddings are further transformed into the 2-dimensional Euclidean space via the t-SNE algorithm [50]. The nodes and links are both colored according to their types.
As shown in Figure 4, M2V, ConvE, and HGT have blurred boundaries and even overlaps between different types of nodes, which are hard to distinguish. Our WalkLM shows the clearest boundaries between different types of nodes and the best within-type compactness, which indicate it can automatically organize heterogeneous nodes in a uniform space. Moreover, by connecting different types of nodes according to the relations in the data, we find that WalkLM can provide more discriminate distributions for different types of relations than others. The visualizations clearly demonstrate the advantages of WalkLM in capturing both attribute semantics and topological structures on graphs.
In this paper, we propose a novel uniform language model fine-tuning framework for attributed graph embedding. The proposed WalkLM consists of two key modules, which encapsulate both attribute semantics and graph structures and obtain unsupervised generic graph representations. We conduct extensive experiments to demonstrate the superior effectiveness of WalkLM against state-of-the-art baselines. For future work, it is intriguing to further design more sophisticated techniques and empirical evaluations toward the leverage of LMs and generalize our work to modern LLMs.
This work was supported in part by the National Natural Science Foundation of China (No. 6230071268); Fujian Provincial Youth Education and Scientific Research Project under Grant JAT220811. Carl Yang was not supported by any fund from China.
[64] Zhilin Yang, Zihang Dai, Yiming Yang, Jaime Carbonell, Russ R Salakhutdinov, and Quoc V Le. XLnet: Generalized autoregressive pretraining for language understanding. Advances in neural information processing systems, 32, 2019.
[65] Liang Yao, Chengsheng Mao, and Yuan Luo. Kg-bert: Bert for knowledge graph completion. arXiv preprint arXiv:1909.03193, 2019.
[66] Liang Yao, Jiazhen Peng, Chengsheng Mao, and Yuan Luo. Exploring large language models for knowledge graph completion. arXiv preprint arXiv:2308.13916, 2023.
[67] Michihiro Yasunaga, Antoine Bosselut, Hongyu Ren, Xikun Zhang, Christopher D Manning, Percy S Liang, and Jure Leskovec. Deep bidirectional language-knowledge graph pretraining. Advances in Neural Information Processing Systems, 35:37309-37323, 2022.
[68] Da Yu, Saurabh Naik, Arturs Backurs, Sivakanth Gopi, Huseyin A Inan, Gautam Kamath, Janardhan Kulkarni, Yin Tat Lee, Andre Manoel, Lukas Wutschitz, et al. Differentially private fine-tuning of language models. In International Conference on Learning Representations, 2022.
[69] Donghan Yu, Chenguang Zhu, Yiming Yang, and Michael Zeng. Jaket: Joint pre-training of knowledge graph and language understanding. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 11630-11638, 2022.
[70] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems, 31, 2018.
[71] Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018.
[72] Rui Zhang, Arthur Zimek, and Peter Schneider-Kamp. A simple meta-path-free framework for heterogeneous network embedding. In Proceedings of the 31st ACM International Conference on Information & Knowledge Management, pages 2600-2609, 2022.
[73] Wen Zhang, Yushan Zhu, Mingyang Chen, Yuxia Geng, Yufeng Huang, Yajing Xu, Wenting Song, and Huajun Chen. Structure pretraining and prompt tuning for knowledge graph transfer. In Proceedings of the ACM Web Conference 2023, pages 2581-2590, 2023.
[74] Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang, Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applications. AI open, 1:57-81, 2020.
[75] Qi Zhu, Carl Yang, Yidan Xu, Haonan Wang, Chao Zhang, and Jiawei Han. Transfer learning of graph neural networks with ego-graph information maximization. Advances in Neural Information Processing Systems, 34:1766-1779, 2021.
[76] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Graph contrastive learning with adaptive augmentation. In Proceedings of the Web Conference 2021, pages 2069-2080, 2021.
[77] Zhaorui Zhu, Hongyi Yu, Caiyao Shen, Jianping Du, Zhixiang Shen, and Zhenyu Wang. Causal language model aided sequential decoding with natural redundancy. IEEE Transactions on Communications, 2023.
A.1 Experiments based on KG datasets
Datasets. We conduct extensive experiments on two new real-world KG datasets, i.e., Freebase  $^{11}$  and FB15K-237  $^{12}$ . Freebase contains a graph of books, films, sports, and locations. The nodes and edges are extracted according to [61]. A large portion of books are labeled into eight genres of literature. Each labeled book has only one label. FB15K-237 is a standard dataset in the knowledge graph community, which contains 310,116 triples with 14,541 entities and 237 relation types. Since we did not manually label the nodes, we only predicted whether a triple is correct or not on this dataset. We matched the entities with Wikidata entities and obtained metadata from Wikidata, and constructed a rough attribute graph dataset by using the names and descriptions of the nodes as textualized features of the nodes, and directly applying their original relationship text as the edge textualized attributes.
Node Classification. As shown in Table 4, our proposed WalkLM has superior performance, indicating the importance of leveraging both semantic and structural information in attributed graphs. WalkLM achieves  $40.24\%$  performance gains on Freebase over the second-best performance on average. Specifically, as a text-based Knowledge graph completion method, SimKGC can effectively employ text-based contrastive learning to capture a rich set of semantic information. Compared with SimKGC, WalkLM can effectively combine the complex semantic and graph structure information of attributed graphs, so as to accurately model the complex attributes of nodes.
Link Prediction. We evaluate WalkLM with AUC and MRR. As shown in Table 4, our fine-tuned WalkLM demonstrates outstanding performance in uncovering latent associations among nodes in attributed graphs. In general, WalkLM outperforms all ten baselines with an average of  $2.05\%$  performance gain over the second-best performance, showing that our proposed framework can learn accurate edge representation for link prediction. ComplEx and ConvE consistently demonstrate promising performance by effectively capturing generic node representations. However, as a text-based Knowledge graph completion method, SimKGC can sometimes outperform others in terms of the MRR metric, where SimKGC can enhance semantic similarity between nodes through contrastive learning based on bi-encoder architecture and three types of negatives. Compared with ConvE, ComplEX, and SimKGC, WalkLM can effectively capture the complex relations by providing text-based semantic traits of characteristic graph and reconstructing network proximity of nodes that inherit from RWs.
A.2 Graph-level Classification
Compared with node or edge classification, aggregating node embeddings for graph-level classification needs more context information. Furthermore, graph-level classification presents its own set of challenges, which require holistic capturing of graph structures and often do not rely much on attributes. Therefore, it is difficult to find a universal representation learning approach that solves all different levels of graph mining tasks. Technically, adapting our method to graph-level classification necessitates some subtle decisions to make (such as whether to include graph ID as a virtual node). We've conducted a preliminary analysis on aggregating our learned node embeddings for graph-level tasks. Specifically, we adopt a widely-used MUTAG  $^{13}$  dataset and use mean accuracy as the metric [48, 71]. The results on the popular MUTAG dataset are listed in Table 5. Although the findings are encouraging and show the potential of WalkLM, further studies are still needed to establish a clear advantage of our approach over SOTA graph classification baselines.
A.3 Detailed Ablation Studies
From Table 6, we have the following observations: (1) Compared with the graph-based baselines, the LM-based models (e.g., LM (XRoBERTa), LM (GPT-2), and LM (DroBERTa)) are able to learn accurate and rich node attributes, leading to superior performance in node classification. For the PubMed dataset distributed on 8 classes, LM (XRoBERTa), LM (GPT-2), and LM (DroBERTa) achieve  $63.78\%$ ,  $135.04\%$ , and  $130.89\%$  performance gains over the second-best performance on
11http://www.freebase.com/
$^{12}$ https://paperswithcode.com/dataset/fb15k-237
<sup>13</sup>https://ls11-www.cs.tu-dortmund.de/staff/morris/graphkerneldatasets
Table 4: Different downstream task results  $(\%)$  with the corresponding std  $(\pm)$  on two KG datasets. The best performances are in bold and the second runners are shaded in gray, where \* denotes a significant improvement according to the Wilcoxon signed-rank significance test.
Task	Node Classification	Link Prediction
Dataset	Freebase	Freebase	FB15K-237
Metric	Macro-F1	Micro-F1	AUC	MRR	AUC	MRR
M2V	25.74±1.12	50.25±2.57	80.68±1.81	88.97±0.93	90.35±0.50	96.78±0.19
HIN2Vec	15.56±1.07	43.67±2.12	80.04±3.01	90.90 ±0.87	79.68±0.83	92.85±0.40
ConvE	25.13±1.83	49.31±3.45	88.14±1.03	93.57±0.42	92.88±0.42	97.57±0.15
ComplEx	20.25±1.62	49.43±3.57	84.01±1.43	91.46±0.56	95.03±0.35	97.88±0.22
SimKGC	35.88±0.87	56.12±0.45	87.33±1.51	94.21±0.34	93.80±0.31	97.62±0.30
RGCN	15.37±1.54	45.86±1.03	82.75±0.89	91.52±0.64	85.88±0.35	89.84±0.19
HAN	14.25±1.77	39.30±2.18	80.73±1.37	91.61±0.34	82.06±0.53	89.31±0.89
HGT	19.97±1.34	47.99±2.56	81.94±1.84	89.65±0.43	87.41±0.69	94.62±0.34
HeCo	23.95±1.45	48.62±1.13	79.32±0.86	87.40±0.32	78.82±0.37	90.41±0.23
SHGP	13.83±1.27	39.07±1.39	78.37±1.77	85.52±0.69	78.56±0.33	89.84±0.21
XRoBERTa	48.10±2.01	67.95±0.97	73.94±1.62	88.17±0.91	75.62±0.72	91.10±0.71
GPT-2	49.24±2.12	68.28±1.37	60.45±2.43	83.29±1.87	68.87±1.21	85.23±1.73
DRoBERTa	51.76±1.24	69.51±0.73	79.22±1.85	91.21±1.17	84.15±0.63	93.39±0.39
LM+RGCN	28.38±0.63	53.37±2.27	83.63±1.81	96.38±0.67	87.72±0.50	94.47±0.46
LM+HGT	20.79±0.67	48.73±3.13	83.09±1.23	89.79±0.35	88.18±0.61	94.85±0.27
WalkLM	55.01±2.67*	71.36±1.53*	92.11±2.24*	96.54±0.56*	95.65±0.18*	98.45±0.33*
Table 5: Accuracy results (%) of graph-level classification on MUTAG.
Dataset	MUTAG
Model	HIN2Vec	ConvE	ComplEx	LM (DRoBERTa)	WalkLM w/o. graph-ID	WalkLM
Accuracy	78.72	77.64	78.69	79.23	79.77	81.39*