SlowGuess/NeurIPS_2023_OCR huggingface dataset:
average, respectively. For the MIMIC-III dataset on the total 19 classes, LM (XRoBERTa), LM (GPT-2), and LM (DRoBERTa) achieve  $5.17\%$ ,  $30.17\%$ , and  $20.12\%$  average performance gains compared to the second-best performance, respectively.
(2) The choice of LMs can affect the performance of fine-tuning. Due to different pre-training corpora, LM (XRoBERTa) performs worse than LM (DroBERTa) in most cases. Moreover, LM (GPT-2) achieves an average of  $3.30\%$  improvement over LM (DroBERTa) in node classification, while LM (DroBERTa) achieves an average of  $7.08\%$  improvement over LM (GPT-2) in link prediction. Considering the overall performance of the above three LMs on two different tasks and the goal of learning graph embedding, we choose LM (DroBERTa) as our starting point for fine-tuning.
(3) Furthermore, LM can further effectively integrate with existing heterogeneous graph algorithms, resulting in a notable performance enhancement over their individual methods. Specifically, compared with RGCN, LM + RGCN achieves an average of  $50.38\%$  improvement in node classification, and achieves up to  $6.59\%$  improvements in link prediction. Compared with HGT, LM + HGT achieves up to  $30.64\%$  improvements in node classification and  $7.42\%$  improvements in link prediction.
(4) Compared with LM + RGCN and LM + HGT, our proposed graph-aware LM fine-tuning can achieve the largest improvement gains based on the chosen LM (DROBERTa) in both node classification and link prediction tasks, showing the effectiveness of capturing topological information together with semantics in modeling attributed graphs. Specifically, our Wa1kLM outperforms the chosen LM (DROBERTa) by up to  $13.45\%$  in node classification. In Particular, our Wa1kLM achieves up to  $59.70\%$  improvements in link prediction, which demonstrates our Wa1kLM can better learn accurate edge representation for link prediction by the graph-aware LM fine-tuning.
A.4 Detailed Hyper-parameter Studies
We show the results of the model sensitivity on the number of sampled walks  $N$  and the termination probability  $\alpha$  on MIMIC-III in Figure 5. Overall, Wa1kLM is not sensitive to the two hyper-parameters, where its performance increases slowly with  $N$  and  $\alpha$ . Note that, setting  $N$  around  $3\times 10^{5}$  and  $\alpha$  around 0.05 seems appropriate to generate sufficient textual sequences and limit computational costs
Table 6: The detailed ablation results  $(\%)$  with the corresponding std  $(\pm)$  on two datasets. The best performances are in bold and the second runners are shaded in gray, where  $^*$  denotes a significant improvement according to the Wilcoxon signed-rank significance test.
Task	Node Classification	Link Prediction
Dataset	PubMed	MIMIC-III	PubMed	MIMIC-III
Metric	Macro-F1	Micro-F1	Macro-F1	Micro-F1	AUC	MRR	AUC	MRR
M2V	15.35(±1.27)	20.27(±3.01)	19.69(±0.62)	29.24(±1.57)	74.53(±3.79)	89.58(±2.05)	75.05(±0.41)	88.32(±0.23)
HIN2Vec	11.57(±1.23)	18.92(±2.78)	19.12(±1.32)	28.05(±1.44)	74.21(±5.49)	90.56(±1.06)	73.46(±0.41)	88.10(±0.14)
ConvE	16.06(±3.69)	19.16(±4.00)	24.44(±1.28)	32.89(±0.86)	76.48(±4.31)	92.27(±0.57)	69.56(±0.36)	84.88(±0.25)
ComplEx	13.93(±2.59)	18.27(±4.12)	9.82(±0.56)	21.39(±3.12)	79.81(±0.97)	91.79(±0.48)	63.86(±0.42)	81.40(±0.40)
SimKGC	21.97(±3.51)	30.83(±3.10)	51.62(±1.81)	58.50(±1.52)	79.62(±2.72)	91.43(±0.48)	67.73(±1.69)	84.86(±0.54)
RGCN	12.50(±2.36)	18.50(±1.41)	7.19(±0.77)	14.55(±3.25)	72.08(±1.13)	88.20(±0.47)	57.31(±0.71)	73.91(±0.57)
HAN	15.29(±2.87)	16.95(±2.71)	6.98(±0.58)	14.73(±1.69)	70.57(±1.58)	87.89(±0.62)	-	-
HGT	11.98(±2.23)	20.12(±3.89)	8.03(±0.87)	17.79(±0.83)	77.24(±3.50)	89.63(±0.84)	64.01(±0.36)	81.54(±0.56)
HeCo	10.32(±1.12)	18.01(±0.87)	10.78(±0.41)	15.26(±1.52)	65.04(±1.26)	83.29(±0.72)	53.13(±0.47)	71.81(±0.35)
SHGP	10.80(±3.03)	19.28(±0.91)	11.34(±1.29)	17.44(±1.49)	68.22(±2.71)	85.34(±0.48)	54.49(±0.33)	72.58(±0.24)
LM	40.10(±4.62)	44.71(±3.68)	54.51(±1.50)	61.27(±1.22)	60.20(±2.78)	84.23(±1.71)	51.21(±0.17)	74.22(±0.26)
(XRoBERTa)	59.43(±4.73)	61.53(±3.43)	70.26(±1.43)	72.67(±0.90)	51.71(±3.67)	80.54(±2.49)	50.66(±0.74)	72.36(±0.86)
LM	58.29(±2.44)	60.57(±2.11)	66.25(±1.60)	70.14(±1.52)	60.97(±2.98)	83.00(±0.40)	51.44(±0.14)	75.09(±0.29)
(DRoBERTa)	13.83(±0.73)	22.70(±3.25)	14.32(±0.87)	24.59(±1.17)	72.35(±4.34)	88.86(±1.46)	58.62(±0.50)	78.78(±0.10)
+RGCN	12.81(±1.22)	21.79(±3.54)	10.49(±0.41)	20.57(±0.97)	82.97(±3.91)	89.98(±0.88)	65.01(±0.20)	82.28(±0.30)
WalkLM	60.42*(±2.62)	62.33*(±3.13)	75.16*(±0.93)	77.89*(±0.70)	85.65*(±3.28)	94.16*(±0.37)	82.15*(±0.67)	92.78*(±0.68)
Table 7: Different downstream task results (%) with ratio of masked samples  $\mathbf{m}$  on PubMed.
Task	Node Classification	Link Prediction
Metric	Macro-F1	Micro-F1	AUC	MRR
m = 0.05	52.97	56.33	83.16	93.47
m = 0.15	60.42*	62.33*	85.65*	94.16*
m = 0.25	53.80	56.09	82.92	93.75
m = 0.35	52.22	55.61	82.38	92.72
for fine-tuning, which can achieve a good balance of performance and efficiency. Furthermore, for the ratio of masked samples  $\mathbf{m}$ , the specific results are listed in Table 7. Overall, Wa1kLM is sensitive to  $\mathbf{m}$ , where the optimal value across different tasks is 0.15, which is consistent with the empirical selection in our paper and the previous work [7, 31, 40, 60]
(c)  $\alpha$  on node classification
Figure 5: Analysis of the number of sampled walks  $N$  and the termination probability  $\alpha$ .