SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathrm {t c l + t d k}} = - \left(\log \frac {\exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {k} ^ {c}\right) / \tau\right)}{\sum_ {c ^ {\prime} = 1 , c ^ {\prime} \neq c} ^ {N} \exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {k} ^ {c ^ {\prime}}\right) / \tau\right)} + \log \frac {\exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {z} ^ {n}\right) / \tau\right)}{\sum_ {n ^ {\prime} = 1 , n ^ {\prime} \neq n} ^ {N} \exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {z} ^ {n ^ {\prime}}\right) / \tau\right)}\right). \tag {9}
$$
A.3 Code License
The code of CS-TAG uses the MIT license. Please refer to the GitHub repository for license details.
B Datasets
B.1 Dataset Format
For each dataset in CS-TAG, we provide three different files. We store the graph-type data available to dgl in the .pt format in Pytorch. For the node classification dataset, two types of features are stored, the adjacency matrix and the node labels. For the Ele-Computers and Ele-Photo datasets, we also store the year features of the nodes (the year of the comments posted by the user). For the link prediction dataset, it contains only the adjacency matrix information. We use a .txt file to store the text attributes of each dataset. The .csv file stores node-id, node-label, category, and text to provide a clearer picture of the dataset for subsequent researchers.
B.2 Dataset Construction
The construction of the text-attributed graph datasets includes the following three steps. First, preprocessing the text attributes in the original dataset, including removing missing values, removing non-English statements, removing abnormal symbols, length truncation, etc. Second, building the graph. The linking relationship between nodes has been provided in the original data of the dataset constructed in CS-TAG, such as the information of "also view", and "also buy" of the products in the Amazon dataset (indicating the two product ids that are jointly purchased or viewed), and the citation relationship between papers in DBLP, etc. Note that when obtaining the final graph data, self-edges and isolated nodes need to be removed. Third, refining the constructed graph. For the node classification dataset, the nodes in the graph need corresponding numerical node labels. We convert the categories of nodes in the original data to numerical node labels in the graph. For some datasets that are divided in a specific form (e.g., by the year of publication of a paper), we also need to store additional information about the nodes.
B.3 Dataset Details
CS-TAG includes 8 datasets, whose details are described as follows. The statistics of the datasets is shown in Table 5.
ogbn-arxiv-TA dataset is derived from ogbn-arxiv. The corresponding task is to predict the categories of the papers, which is formulated as a 40-class classification problem. The text attributes of each paper node are extracted from its title and abstract in ogbn-arxiv. Note that ogbn-arxiv solely provides node embeddings acquired through shallow text encoders like Skip-Gram. Different from them, we use different PLMs to model the node attributes to get the initial node features and delve into the performance of employing multiple PLM features in downstream GNNs.
Books-Children/History datasets are extracted from the Amazon-Books dataset. Books-Children consists of items with the second-level label "Children", while Books-History consists of items with the second-level label "History". The nodes in the dataset are books, and the edges mean two books are frequently co-purchased or co-viewed. The label of each dataset is the three-level label of the book. We choose the title and description of the book itself as the text attributes of the node. The task is to classify books into 24 and 12 categories, respectively.
Ele-Computers/Photo datasets are extracted from the Amazon-Electronics dataset. Ele-Computers consists of items with the second-level label "Computers", while Ele-Photo consists of items with the second-level label "Photo". The two datasets are extracted from the updated 2018 Amazon Computer and Amazon Photo datasets [27]. The nodes in the dataset are electronics related products, and the edge between two products means that they are frequently co-purchased or co-viewed. The label of each dataset is the three-level label of the electronics products. We adopt user reviews on the item as its text attribute. Since the item has multiple reviews, we mainly adopt the review with the highest number of votes. For some items lacking highly votes reviews, we randomly adopt a user review as the text attribute. The task on the two datasets is to classify electronics products into 10 and 12 categories, respectively.
Sports-Fitness dataset is extracted from the Amazon-Sports dataset. It consists of items with the second-level label "Fitness". The nodes in the dataset are the fitness-related items, and the edge between two items means that they are frequently co-purchased or co-viewed. The label of the dataset is the three-level label of the items. The task on this dataset is to classify items into 13 categories.
Table 5: Statistics of text-attributed graph dataset used in CS-TAG.
Dataset	Nodes	Edges	Class	Split Scheme	Split Ratio	Task Type	Metric	Max length
ogbn-arxiv-TA	169,343	1,166,243	40	Time	54/18/28	Node Class.	Acc, F1	512
Books-Children	76,875	1,554,578	24	Random	60/20/20	Node Class.	Acc, F1	256
Books-History	41,551	358,574	12	Random	60/20/20	Node Class.	Acc	256
Ele-Computers	87,229	721,081	10	Time	72/17/11	Node Class.	Acc, F1	256
Ele-Photo	48,362	500,928	12	Time	60/20/20	Node Class.	Acc, F1	512
Sports-Fitness	173,055	1,773,500	13	Random	20/10/70	Node Class.	F1	64
CitationV8	1,106,759	6,120,897	-	Time	99/1/1	Link Prediction	MRR	256
GoodReads	676,084	8,582,306	-	Random	90/2/8	Link Prediction	Hits@K	24
CitationV8 is a directed graph dataset, representing the citation relationship among a subset of papers extracted from DBLP [38]. It is constructed following the form of ogbl-citation2 [34]. The corresponding task is to predict the missing citations given some existing citations. Specifically, for each source paper, two of its references are randomly dropped, and the prediction model tries to rank the missing two references higher than 2,000 negative reference candidates. The negative references are randomly-sampled from all the previously published papers that are not referenced by the source paper. We adopt the title and abstract of each paper as its node text attributes.
GoodReads dataset is extracted from the world's largest book review site Goodreads. Its nodes are books and edges are identified by similarity relationships between books provided in the website. The corresponding task is to predict the similar relationships between books. We expect the model to rank the true correlations over the false ones. Specifically, we rank each true correlation among a set of 5,000 randomly-sampled negative correlation.
B.4 Datasets License
The datasets follow the MIT license. Please refer to the GOOD GitHub repository for license details.
C Experiment Settings
C.1 Experimental Settings of GNNs
We conduct experiments on 9 GNN models described in A.1 on 6 node classification datasets. We use the aforementioned 5 PLMs with different parameter scales to model the node attributes and form the initial node features of the graph data. Each experiment is repeated three times and the evaluation metrics are accuracy and F1-score. The parameters shared by all GNN models include epochs, model layers, hidden units, learning rate, and dropout ratio, and their values are set to 1000, {2,3}, {64,128,256}, {1e-04 ~ 1e-02}, 0.2, respectively. Besides these hyperparameters, for GAT we freeze the number of heads to 3 and set the ratio of attention-drop out to 0 by default. For GraphSAGE model, we use mean pool to aggregate the neighbor information, and for JKNet, we use cat to aggregate the features. For APPNP, we set the teleport probability to 0.1 by default and the number of propagation steps is set to 2. For MoNet, we set the pseudo coordinate dimensions in GMMConv to 2 and 3, and we set the number of kernels in GMMConv layer to 2. Since it mostly does not converge at epoch=1000, we set its maximum epoch to 2000. For GIN, we set the number of mlp layers as 2. The eval patience of all models is set to 1. We use cross-entropy loss with the AdamW optimizer to train and optimize all the above models. GNNs are mainly derived from the implementation in the DGL library.
C.2 Experimental Settings of PLMs
We conduct experiments on 5 PLMs with different parameters described in A.1 on 6 node classification dataset. Considering the efficiency of the language models, we conduct experiments on each dataset only once. The parameters shared by all the PLMs in A.1 include epochs  $e$ , label smoothing factor  $lsf$ , learning rate  $lr$ , warm-up epochs  $w$ , batch size, and eval patience. The label smoothing factor is used to calculate the cross-entropy loss which is set in GNN as well. The  $w$  denotes the duration of the warm-up phase, and  $w = 1$  means the duration of the warm-up is in one epoch. For all datasets we set  $e$ ,  $lsf$ ,  $w$ ,  $lr$  to 4, 0.1, 1, {5e-06~5e-04} , respectively. Due to different model parameters and different dataset sizes, we list the eval patience, batch size of different models on each dataset in Table 6. For PLMs at the same scale, we use the same batch size. For large-scale
Table 6: Batch size and eval patience of the different scale PLMs.
Datasets	Small Scale	Base Scale	Large Scale
Batch Size	Eval Patience	Batch Size	Eval Patience	Batch Size	Eval Patience
ogbn-arxiv-TA	100	50000	60	50000	60	50000
Books-Children	240	15000	90	15000	90	15000
Books-History	240	8000	90	8000	90	8000
Ele-Computeres	300	20000	180	20000	180	20000
Ele-Photo	100	5000	60	5000	60	5000
Sports-Fitness	800	10000	400	10000	400	10000
models, since full parameter fine-tuning is costly, we only fine-tune the last four encoder layers and its' effect is sometimes better than full parameter tuning. The experimental setup of several topological pre-training methods has been mentioned in the previous section, and for the topological pre-trained language models, we follow the same tuning strategy as the PLM-based methods.
C.3 Reproducibility
For all experiments, we select the best checkpoints according to the validation sets, and report the results. All the datasets and codes to reproduce the results in this paper are available at https://github.com/sktsherlock/TAG-Benchmark.
D Additional Experiment Results
D.1 Experimental Results for Node Classification
GNN-based methods. Table 3-5 lists all the experimental results for the node classification task. We first analyze the performance of different GNN-based methods on the tasks of node classification. Over the evaluation metric accuracy, one can see that RevGAT, GraphSAGE, and GAT perform the best on these datasets, which are less affected by the node features learned by different PLMs. While for the other GNN models, their performance is affected by the PLMs to different extents. GIN, which is commonly used for graph-level tasks, does not achieve better performance on node-level tasks. MoNet's performance on Books-Children and History datasets is even lower than MLP in most cases.
By investigating the impact of node features encoded by different PLMs on the downstream models, one can see that node features encoded by ELECTRA typically produce a larger gap between GNN and MLP. The failure of the ELECTRA model to generate higher quality node features may be due to its discriminative pre-training way. In contrast, the RoBERTa-Base model seems to have a better semantic understanding, with a difference between GNN and MLP of only 1.91 and 4.44 on the Books-History and Books-Children datasets. It is worth noting that the node features encoded by DeBERTa also perform poorly. However, DeBERTa performs better on NLP-related tasks, which indicates that the node features encoded by a language model cannot be judged by its performance on downstream tasks alone. The ineffectiveness of the DeBERTa to obtain better node features may be mainly due to its reduced corpus during pre-training.
Similar experimental conclusions can be drawn on the F1-score results as shown in Table 4. Note that the F1 score of downstream models is more significantly affected by different PLMs. For example, on Books-Children, the score of JKNet ranges from 19.79 to 30.94 under different PLMs.
PLM-based methods and Topological Pre-training Between Table 3 to 5, PLMs denote PLM-based methods, i.e., fine-tuning directly on the dataset without considering the topology. While TLM, TDK, TCL, and TMDC denote fine-tuning on models pre-trained for these four topological pre-training tasks. Observing the experimental results on these datasets, the PLM-basd method performs the worst on all models with all datasets. This indicates that it is not sensible to ignore topology and use only text attributes for representation learning on TAGs. In contrast, TMDC, a multi-task form of topology pre-training, performs the best, achieving the best on both metrics in the majority of experiments. This indicates that combining different topological structure pre-training methods can teach language models topological structure knowledge from different perspectives.
Table 9: Node classification accuracy and f1 score on Ele-Computers. We bold the best results for each row.
Methods	Ele-Computers (Accuracy)
Small Scale	Base Scale	Large Scale
BERT-Tiny	ELECTRA	DistilBERT	ELECTRA	BERT	RoBERTa	DeBERTa	ELECTRA	BERT	RoBERTa	DeBERTTa
LMs	PLM	58.32	59.88	60.51	60.80	60.40	61.10	61.68	59.63	60.70	61.22	61.96
GNNs	GCN	87.43	84.13	88.37	86.10	87.86	88.50	82.30	84.79	87.38	88.56	85.75
GAT	88.57	87.86	89.07	88.14	88.63	88.80	83.42	88.07	88.64	88.88	87.40
SAGE	87.90	86.43	88.67	86.97	88.30	88.87	82.81	87.16	88.35	88.66	86.85
RevGAT	88.66	87.92	89.32	88.22	88.72	88.91	84.12	88.45	88.86	89.03	87.56
NFormer	86.81	85.43	87.96	86.05	86.95	87.12	81.56	86.26	87.23	87.65	84.46
GIN	78.07	83.18	83.18	73.36	83.02	83.98	71.53	71.64	81.20	84.09	75.13
JKNet	85.05	85.31	85.31	79.37	85.08	83.91	55.15	73.67	83.32	83.66	70.39
APPNP	83.62	84.96	84.96	78.15	84.51	84.97	62.09	76.51	83.02	83.83	74.90
MoNet	78.30	77.76	77.76	62.08	77.67	74.53	47.68	60.24	68.18	70.63	59.67
MLP	46.02	53.76	53.76	36.21	50.85	55.11	42.45	36.25	47.59	55.03	46.74
Co-Training	GCN	83.93	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
SAGE	86.04	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
TPT	TMLM	58.32	60.20	61.42	61.53	63.43	61.30	61.74	61.99	62.41	63.12	63.43
TDK	58.18	60.42	63.38	63.13	62.95	63.80	64.97	60.96	62.49	62.92	62.95
TCL	58.80	61.04	66.50	67.02	64.93	67.94	70.08	61.50	63.34	64.55	64.93
TMDC	58.86	61.17	66.94	68.24	69.45	67.50	69.71	65.02	65.74	65.79	66.08
Methods	Ele-Computers (F1 score)
Small Scale	Base Scale	Large Scale
BERT-Tiny	ELECTRA	DistilBERT	ELECTRA	BERT	RoBERTa	DeBERTa	ELECTRA	BERT	RoBERTa	DeBERTTa
LMs	PLM	44.55	47.65	51.47	48.77	52.53	53.39	52.10	48.46	51.57	55.28	55.04
GNNs	GCN	80.88	75.73	82.24	79.11	81.03	82.13	72.97	78.10	80.93	82.85	76.56
GAT	83.49	83.52	84.14	83.35	83.99	83.14	78.30	83.33	83.55	83.43	83.06
SAGE	82.85	81.38	84.21	82.18	83.67	84.44	76.42	81.81	82.93	83.13	80.98
RevGAT	83.65	83.76	84.55	83.56	84.43	84.69	79.02	83.46	83.69	87.79	83.51
NFormer	80.57	75.12	80.26	77.56	80.01	81.16	71.03	76.56	79.65	81.51	75.55
GIN	71.56	57.99	76.46	67.39	77.00	78.09	61.03	64.63	75.43	77.57	62.25
JKNet	77.01	67.90	73.99	67.83	75.02	70.26	41.49	63.58	72.35	69.78	58.30
APPNP	75.49	62.04	78.02	69.04	76.99	73.46	42.05	68.22	77.79	72.96	59.96
MoNet	48.56	35.22	52.69	40.03	49.86	53.60	43.65	39.88	45.65	54.12	48.55
MLP	32.20	16.23	40.06	21.00	37.41	41.22	26.21	19.88	34.25	42.34	32.73
Co-Training	GCN	80.79	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
SAGE	83.67	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM	OOM
TPT	TMLM	44.91	48.01	51.51	52.01	52.51	53.00	53.86	55.47	55.71	55.81	57.56
TDK	44.50	48.48	52.97	55.80	55.15	56.53	59.39	50.44	55.53	55.94	54.05
TCL	44.46	48.81	54.66	55.44	59.25	60.58	57.69	50.22	54.48	57.86	55.67
TMDC	45.05	49.19	54.91	55.93	59.12	61.35	59.39	58.10	58.44	60.52	60.06
D.2 Experimental Results for Link Prediction
We further show the experiment results for link prediction in Table 10. We mainly follow the code used for link prediction in OGB. We choose GCN and GraphSAGE as the backbone model in the link prediction experiments.
Table 10: Link Prediction results on CitationV8 and GoodReads.
Methods	CitationV8	GoodReads
Hits10	Hits50	Hits100	MRR	Hits10	Hits50	Hits100	MRR
PLM	BERT-Tiny	33.56 ± 1.56	48.15 ± 2.02	66.56 ± 0.56	41.23 ± 1.39	36.86 ± 2.04	52.45 ± 1.69	76.23 ± 1.11	42.15 ± 0.86
BERT-Base	38.86 ± 2.53	57.53 ± 1.96	72.44 ± 0.98	44.56 ± 1.23	43.96 ± 2.26	60.87 ± 1.43	79.22 ± 0.46	44.43 ± 1.15
GNN	T-GCN	50.89 ± 3.56	74.26 ± 2.16	90.23 ± 0.89	60.79 ± 0.28	61.47 ± 4.65	84.14 ± 2.15	90.43 ± 0.60	69.44 ± 0.56
T-SAGE	45.12 ± 3.26	66.23 ± 1.56	89.36 ± 0.99	54.64 ± 1.03	64.52 ± 3.18	82.65 ± 1.45	88.53 ± 0.61	74.36 ± 0.84
B-GCN	50.39 ± 4.56	75.12 ± 2.56	90.16 ± 0.46	60.04 ± 0.86	55.42 ± 5.83	85.04 ± 2.34	91.49 ± 1.25	65.12 ± 0.48
B-SAGE	44.12 ± 4.12	71.26 ± 1.67	89.12 ± 0.75	53.96 ± 1.24	54.05 ± 3.03	82.87 ± 0.89	89.61 ± 0.33	65.68 ± 1.13
TCL	BERT-Tiny	41.26 ± 1.49	57.26 ± 1.59	72.62 ± 0.96	47.26 ± 1.23	45.47 ± 2.53	61.56 ± 1.56	82.43 ± 0.49	55.12 ± 0.84
BERT-Base	46.58 ± 1.69	65.77 ± 2.01	72.46 ± 0.56	52.78 ± 2.03	52.59 ± 2.27	65.97 ± 1.23	85.56 ± 0.34	61.21 ± 1.23
T-GCN	65.23 ± 2.36	81.23 ± 1.36	92.56 ± 0.56	65.69 ± 0.42	69.58 ± 2.29	88.89 ± 0.56	93.12 ± 0.57	83.16 ± 1.04
T-SAGE	61.89 ± 3.21	80.12 ± 1.46	90.23 ± 0.89	55.70 ± 0.15	70.28 ± 2.36	85.12 ± 0.47	90.38 ± 0.57	81.12 ± 1.23
B-GCN	68.26 ± 2.66	84.56 ± 0.56	93.68 ± 0.26	70.16 ± 0.38	73.87 ± 3.36	92.82 ± 0.16	95.85 ± 0.15	85.12 ± 1.56
B-SAGE	62.36 ± 3.46	80.56 ± 1.56	92.56 ± 0.68	60.18 ± 0.15	75.16 ± 2.26	90.74 ± 0.16	94.01 ± 0.11	82.15 ± 1.15
On the CitationV8 dataset, the parameters shared by the GCN and GraphSAGE include epochs, model layers, hidden units, learning rate, and batch size, and their values are set to 200, {2, 3}, {128, 256}, {1e-04 ~ 1e-03}, 65536. We validate the model performance with the Hits@K and MRR metrics. We bold the best results on each row. In Table 10, the PLM indicates that we use LMs directly for link prediction. In GNN, we use the features generated by BERT-Tiny and BERT-Base to feed into GCN and SAGE, respectively. In TCL, we are subdivided into two approaches. The first is to pre-train LMs with TCL tasks and then finetune LMs for downstream link prediction tasks. The second is to use LMs pre-trained by TCL to encode text attributes. T-GCN(SAGE) denotes the node features obtained by BERT-Tiny. B-GCN(SAGE) denotes the node features obtained by BERT-Base. When we use TCL to pre-train the language models, we find that BERT-Tiny and BERT-Base improve on MRR by 6.03 and 8.22, respectively. We also find that the features obtained using TCL-BERT-Base
can significantly improve the results on downstream GNNs. For example, B-GCN improves by  $10\%$  on average on all evaluation criteria.
On the GoodReads dataset, the parameters shared by the GCN and GraphSAGE include epochs, model layers, hidden units, learning rate, and batch size, and their values are set to 200, {2}, {128}, {1e-04~5e-03}, 524288. We validate the model performance with the Hits@K and MRR metrics. We bold the best results on each row. Observing from Table 10 we can find that after using TCL to pre-train the language models and obtaining the corresponding node features, the significant improvements are obtained in all experiments. In addition to the improvement in performance, the variance of the model is also reduced in most experiments. This indicates that the node features generated by the language model trained by a suitable pre-training task can not only enhance the performance of the downstream model but also improve the stability of the downstream model to some extent.
D.3 Efficiency and Scalability of Co-Training Paradigm
Table 11: Effectiveness and Scalability of Co-Training with TCL.
Datasets	BERT-Tiny	Co-Training	BERT-Base
Co-Training	TCL	TCL
Acc	Memory	Time	Acc	Memory	Time	Acc	Memory	Time	Acc	Memory	Time
Arxiv	73.57	76.27%	44.0	71.55	27.59%	7.0	-	OOM	-	74.87	70.73%	130
Children	59.70	97.28%	15.5	54.11	19.76%	2.0	-	OOM	-	60.73	80.99%	30
History	85.09	85.74%	5.7	86.06	14.69%	1.3	-	OOM	-	86.80	98.73%	18
Photo	86.64	97.83%	14.6	73.86	22.75%	3.1	-	OOM	-	82.85	70.65%	120
# Average	76.25	89.38%	19.95	71.40	21.20%	3.4	-	OOM	-	76.31	80.28%	74.50
In this subsection, we present the accuracy, GPU memory cost on a single 32GB V100, and the total training time (min) of the co-training paradigm and topological PLMs across five datasets. BERT-Tiny+SAGE and BERT-Base+SAGE are selected as the co-training approaches. As one of the best topological pre-training tasks, topological contrastive learning (TCL) is selected to enhance the PLMs. The experimental results are presented in the Table 11. For the small language model BERT-Tiny, the co-training paradigm costs more memories ( $\sim 4X$ ) compared to the TCL and is much slower ( $\sim 6X$ ) than TCL. If we use the larger language model like BERT-Base, co-training models will be out-of-memory in a single 32GB V100. Thus, the proposed topological pre-training paradigm is more efficient and practical than the co-training ones.
D.4 Comparison between Shallow Text Encoders and PLMs
The traditional GNN pipelines generally encode the textual attributes of each node using a shallow model such as the Skip-Gram. Therefore, we extend our node text encoding approach by incorporating shallow models like Skip-Gram [29] and GloVe [30]. Experimental results (accuracy) on node classification task over ogbn-arxiv-TA and Books-Children datasets are presented in the Table 12. One can clearly see that GNNs equipped with deeper text encoders consistently outperform those with shallow encoders, verifying the importance of node attribute understanding.
Table 12: Node classification performance on shallow text encoders and PLMs. We bold the best result for each row.
Text Attribute Encoder	OGbn-arxiv-TA	Books-Children
GCN	GAT	SAGE	RevGAT	NFormer	GCN	GAT	SAGE	RevGAT	NFormer
Shallow	Skip-Gram	71.97	72.22	72.02	73.42	71.03	56.23	55.84	56.49	57.13	56.03
GloVe	72.12	72.54	72.48	73.51	72.04	57.02	56.58	57.22	58.12	56.86
PLMs	DistilBERT	73.39	73.48	74.48	74.68	73.56	58.19	57.91	59.33	59.28	58.03
BERT-Base	73.30	73.40	74.14	74.59	72.80	58.11	57.70	58.74	58.67	57.42
RoBERTa-Base	73.56	73.38	74.52	74.82	73.12	58.62	57.83	58.97	59.01	57.26
D.5 Topological Pre-training under Semi-supervised and Few-shot Learning Scenarios
In order to provide the more comprehensive evaluation and deeper insights into our topological pre-training strategies, we evaluate them in different scenarios.
D.5.1 Semi-supervised Learning
To effectively evaluate the performance of various methods within semi-supervised settings, we've adjusted the training ratio from  $20\%$  to  $100\%$ .  $20\%$  implies that only  $20\%$  of training samples are used in the model training process. BERT-Base is selected as the foundational text encoder model. Detailed experimental results concerning the node classification task across four datasets have been presented in Table 13. Notably, as the training ratio decreases, the benefits of topological pre-training approaches become even more significant. Among these strategies, TMDC demonstrates superior performance across all datasets within the semi-supervised context.