SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {\mathrm {t m l m}} \left(\boldsymbol {T} _ {\Phi} ^ {(s)} \mid \boldsymbol {T} _ {- \Phi} ^ {(s)}\right) = \frac {1}{m} \sum_ {i = 1} ^ {m} \log p \left(t _ {\phi_ {i}} \mid \boldsymbol {T} _ {- \Phi} ^ {(s)}; \theta\right). \tag {4}
$$
in which  $\theta$  denotes the learnable parameters.
Topological Contrastive Learning (TCL). Inspired by contrastive learning [41, 42, 43, 44], we propose a novel topological contrastive learning (TCL) task to capture the first-order topological information in the node level. Given a center node  $c$  and one of its neighbors  $n$ , their corresponding node-level (sentence/document-level) embedding is derived from their CLS token in the  $T_{\Phi}^{(c)}$  and can be formally defined as  $h^c, h^n$ . The objective of TCL is to bring the center node  $h^c$  closer to its neighbors  $h^n$  while pushing itself farther away from other nodes. Denoting the cosine similarity function as  $\mathrm{sim}(h^c, h^n) = h^{c^\top}h^n / \| h^c\| \| h^n\|$ . The objective of TCL is:
$$
\mathcal {L} _ {\mathrm {t c l}} = - \log \frac {\exp \left(\sin \left(\boldsymbol {h} ^ {c} , \boldsymbol {h} ^ {n}\right) / \tau\right)}{\sum_ {n ^ {\prime} = 1 , n ^ {\prime} \neq n} ^ {N} \exp \left(\sin \left(\boldsymbol {h} ^ {c} , \boldsymbol {h} ^ {n ^ {\prime}}\right) / \tau\right)}, \tag {5}
$$
where  $\tau$  denotes the temperature parameter,  $N$  denotes the batch size.
Topological Deepwalk Learning (TDK). TMLM and TCL mainly capture low-order structural information, while the higher-order structural information still needs to be captured by designing
Table 2: Accuracy comparison among GNNs on the ogbn-arxiv-TA within different PLMs' node features. "Scale" means the different versions of PLMs (number of parameters). "Diff" denotes the performance gap between the best and worst performers. We mark the best performer in each row with blue bold font and mark the best performer in each column with black bold font.
Scale	PLMs	Arxiv
GCN	GAT	SAGE	RevGAT	NFormer	GIN	JKNet	APPNP	MoNet	MLP
Small	BERT-Tiny	72.03	72.25	72.35	72.52	71.91	68.42	69.50	71.63	45.13	57.22
ELECTRA	68.45	70.97	69.63	71.12	69.45	58.09	62.87	59.55	36.65	36.58
DistilBERT	73.39	73.48	74.48	74.68	73.56	72.30	71.44	74.01	50.51	68.11
Base	ELECTRA	70.81	71.67	70.82	71.96	70.43	64.88	63.41	65.62	38.91	48.56
BERT	73.30	73.40	74.14	74.59	72.80	71.94	70.08	73.90	46.90	67.35
RoBERTa	73.56	73.38	74.52	74.82	73.12	72.63	69.40	74.01	44.53	69.31
DeBERTa	68.15	66.56	67.58	68.26	67.11	62.05	44.16	52.37	29.67	47.07
Large	ELECTRA	70.44	71.01	70.72	72.56	70.04	64.47	58.34	64.52	37.26	47.72
BERT	73.25	73.37	74.15	74.68	73.12	71.88	68.70	73.53	43.31	66.85
RoBERTa	73.95	73.72	74.64	74.99	73.12	73.10	68.10	74.17	44.01	69.51
DeBERTa	72.57	71.50	73.22	73.59	71.88	71.25	54.41	69.28	33.53	66.28
Diff	5.80	7.16	7.06	7.54	6.45	15.01	27.28	21.80	20.84	32.93
suitable tasks. Considering that algorithms like Deepwalk [45] can capture higher-order structural information in the graph, we try to use the node representations learned by Deepwalk to augment the representations learned by LM. We first feed the whole graph structure into Deepwalk to get the corresponding representation  $\pmb{k}^c$  of each node  $c$ . The objective of TDK is to bring the center node  $\pmb{h}^c$  closer to its representation learned from Deepwalk  $\pmb{k}^c$ . The objective of TDK is:
$$
\mathcal {L} _ {\mathrm {t d k}} = - \log \frac {\exp \left(\operatorname {s i m} \left(\boldsymbol {h} ^ {c} , \boldsymbol {k} ^ {c}\right) / \tau\right)}{\sum_ {c ^ {\prime} = 1 , c ^ {\prime} \neq c} ^ {N} \exp \left(\operatorname {s i m} \left(\boldsymbol {h} ^ {c} , \boldsymbol {k} ^ {c ^ {\prime}}\right) / \tau\right)}, \tag {6}
$$
Baselines. (1) For GNN-based methods, we select 9 popular GNN models: GCN [15], GAT [16], GraphSAGE [7], RevGAT [33], NodeFormer [46], GIN [17], JKNet [18], MoNet [47] and APPNP [48]. (2) For PLM-based methods, we select 5 different PLMs with different scales: a) Small parameter scale models including BERT-Tiny [10], ELECTRA-Small [11], and DistilBERT [14]. b) Base parameter scale models including BERT-Base [10], ELECTRA-Base [11], RoBERTa-Base [12], and DeBERTa-Base [13]. c) Large parameter scale models including BERT-Large [10], ELECTRA-Large [11], RoBERTa-Large [12], and DeBERTa-Large [13]. (3) For Co-training methods, due to scalability constraints, we only explore the effectiveness of this pipeline on combinations of BERT-Tiny with GCN and GraphSAGE. (4) For the topological pre-training of LMs, we conduct experiments on various datasets and different PLM basic models. In addition, we iteratively train the proposed three pre-training tasks at batch level (named TMDC) in a multi-task learning framework. Please refer to Appendix A.2 for more details.
Implementation details. GNNs are mainly implemented based on the DGL toolkit [49]. PLMs are obtained from Huggingface [50] and trained under a unified framework. Considering the recent rise of parameter-efficient fine-tuning, we only fine-tune the last four encoder layers of large-scale language models. Implementation details and hyperparameter selections are provided in Appendix C.
**Evaluations metric.** We investigate the performance of different baselines through two tasks: node classification and link prediction. For the node classification task, we use Accuracy and F1-Score to evaluate the model performance. For the link prediction task, we use MRR, Hits@10, Hits@50, and Hits@100 as metrics. Due to space limitations, we present the results of some node classification experiments in the main paper, and the remaining node-level experiments with link prediction results are presented in Appendix D.1 and D.2. In addition to the aforementioned datasets, we also conduct experiments on other types of datasets, as described in Appendix D.6 (a large-scale ogbn-papers100M dataset) and Appendix D.7 (two social network datasets), where readers can find detailed information on these datasets and the corresponding experimental results.
Table 3: Node classification of the three learning paradigms on six datasets. Sports corresponds to the F1 score as an indicator of experimental results and Accuracy for the rest of the data. We bold the best results for each dataset.
Way	PLM-Based	GNN-Based	Co-Training Based
Tiny	Base	T-GCN	B-GCN	T-SAGE	B-SAGE	GCN(T)	SAGE(T)
Arxiv	70.83	72.96	72.03	73.30	72.35	74.14	69.22↓	73.57↑
Children	49.85	59.91	57.07	58.11	57.57	58.74	54.75↓	59.70↑
History	83.06	86.09	84.52	85.04	84.79	85.12	83.52↓	85.09↑
Photo	73.75	77.53	82.42	82.70	83.25	83.27	83.32↑	86.64↑
Computers	58.32	60.40	87.43	87.86	87.90	88.30	83.93↓	86.04↓
Sports	81.47	86.02	84.93	86.16	87.06	87.34	85.06↑	85.87↓
4.1 Impact of Static Modeling of Attributes on GNNs
In this subsection, we analyze the impact of different node attribute modeling methods for downstream GNNs. Table 2 represents the effect of node classification on ogbn-arxiv-TA for different GNNs given different PLMs' features. The results on other datasets can be found in Appendix D.1. Observing Table 2, we find that RevGAT performs the best among all initial node features, while GAT and SAGE exhibit the second-best performance. They are relatively less affected by the different node features,  $7.54\%$ ,  $7.16\%$ , and  $7.06\%$  respectively. JKNet, APPNP, MoNet, and MLP are more influenced by the initial node features, which all come above 20. On the other hand, the node features encoded by RoBERTa, BERT, and DistilBERT generally perform better on all types of baselines. DeBERTa, which performs better on many downstream NLP-related tasks, is less effective. This may be because DeBERTa sees less corpus during pre-training, resulting in their inability to understand the semantics well when modeling the text directly on downstream tasks. Furthermore, we compare traditional shallow text encoders (e.g., Skip-Gram) with PLMs in Appendix D.4 for a more comprehensive analysis of the impact of text modeling on downstream GNNs.
Recently, LLMs (Large-Language Models) are continuing to energize areas such as knowledge graphs [51] and recommender systems [52]. It is still an open question on how to successfully apply LLMs to text-attributed graph learning. We have conducted a preliminary exploration of how to use LLMs to advance the representation learning on TAGs. Please refer to Appendix D.8 for the detailed results and discussions.
4.2 Pitfalls of Co-Training Paradigm
In this subsection, we analyze the performance of the Co-training paradigm versus the PLM-based, GNN-based paradigm in terms of node classification tasks. Tiny and Base in the PLM column represent BERT-Tiny and BERT-Base, respectively. T-GCN, T-SAGE, and B-GCN, B-SAGE represent the node features of BERT-Tiny and BERT-Base fed to downstream GCN and GraphSAGE respectively. GCN(T) and SAGE(T) then denote co-training BERT-Tiny with GCN and SAGE, respectively. We compare GCN(T), and SAGE(T) with the corresponding T-GCN, T-SAGE respectively. As shown in Table 3, SAGE(T) improves in four of the datasets compared to the T-SAGE methods, with a maximum improvement of  $3.39\%$  on the Photo dataset. However, GCN(T) performs worse than T-GCN on most datasets and is even weaker than BERT-Tiny  $1.61\%$  on ogbn-arxiv-TA. The Co-training framework requires simultaneous training of PLMs and GNNs. The memory requirement and time cost of this paradigm are significantly increased. In order to facilitate the co-training of PLMs and GNNs, we reduce either the batch size or the number of neighbors. The limited scalability leads to a significant reduction in the number of neighbors for GNN aggregation, which may compromise the effectiveness of message passing. To analyze the impact of scalability on the Co-Training method, we analyze the effect of the number of neighbors sampled per GNN layer on Co-training in Fig 4. As can be seen from it, the effect of the model mainly tends to increase as the number of neighbors increases. The detailed discussions on efficiency and scalability can be found in Appendix D.3.
4.3 Comparing PLM-based Methods with GNN-based Methods
In this subsection, we compare the PLM-based methods and the GNN-based methods in different datasets. As shown in Table 4, the column GNNs indicates the best results on all GNNs for a given PLM node feature. On the Children and History dataset, the PLM-based methods works better than
Table 4: Node classification accuracy comparison among PLM-based, GNN-based, and topological pre-training based on four datasets. The best method for each PLM on each dataset is shown in bold.
Scale	Model	Arxiv	History
PLM	GNNs	TMLM	TDK	TCL	TMDC	PLM	GNNs	TMLM	TDK	TCL	TMDC
Small	BERT-Tiny	70.83	72.52	70.83	71.50	71.55	71.17	83.06	85.03	85.76	85.79	86.06	86.88
ELECTRA	71.26	71.12	72.65	72.83	73.06	73.71	84.18	83.11	84.54	84.42	84.57	85.18
DistilBERT	72.50	74.68	73.53	74.38	74.89	75.50	85.81	85.67	85.76	86.29	86.28	86.88
Base	ELECTRA	72.67	71.96	73.51	74.33	74.26	75.56	85.64	83.79	85.77	85.88	86.62	86.41
BERT	72.96	74.59	73.97	74.23	74.87	76.11	86.09	85.28	86.24	86.46	86.80	86.82
RoBERTa	73.10	74.82	74.25	74.57	75.37	75.97	85.85	85.69	86.19	86.32	86.95	86.96
DeBERTa	73.82	68.26	74.26	75.01	75.15	75.99	86.16	82.31	86.00	86.46	87.01	86.94
Large	ELECTRA	72.42	72.56	74.76	73.82	74.17	75.58	86.13	83.56	86.39	86.49	86.82	86.28
BERT	73.24	74.68	75.01	74.31	75.15	75.75	86.24	85.15	86.47	86.73	86.93	86.94
RoBERTa	73.83	74.99	75.18	74.58	75.48	75.73	86.41	85.23	86.72	86.75	87.11	87.22
DeBERTa	74.57	73.59	75.92	75.20	75.58	76.20	87.00	84.89	87.11	87.26	87.30	87.32
Scale	Model	Children	Photo
PLM	GNNs	TMLM	TDK	TCL	TMDC	PLM	GNNs	TMLM	TDK	TCL	TMDC
Small	BERT-Tiny	49.85	57.86	54.27	53.43	54.11	54.66	73.75	84.12	74.30	73.99	73.86	74.92
ELECTRA	57.03	56.42	57.35	56.92	56.88	58.55	76.58	83.12	76.09	76.89	77.74	77.83
DistilBERT	59.90	59.33	60.03	60.23	60.60	61.38	77.51	84.34	77.81	79.69	81.85	82.52
Base	ELECTRA	59.09	56.42	59.93	60.27	60.21	60.83	77.84	82.98	78.27	80.18	81.47	82.82
BERT	59.91	58.74	60.34	60.43	60.73	61.43	77.53	84.46	78.54	81.04	82.85	84.09
RoBERTa	59.80	59.01	60.19	60.71	61.47	61.83	78.11	84.59	78.33	81.26	82.47	83.04
DeBERTa	60.26	50.72	60.73	61.39	61.92	62.20	78.37	81.44	79.27	81.34	83.07	83.80
Large	ELECTRA	58.28	56.59	60.51	59.31	59.29	61.31	77.25	83.00	79.21	78.44	79.56	81.32
BERT	60.65	58.90	60.84	61.15	61.50	62.06	77.72	84.21	78.95	79.26	80.74	81.14
RoBERTa	60.93	59.26	62.11	61.95	62.06	63.24	79.60	85.12	80.32	80.82	81.47	82.55
DeBERTa	61.61	56.34	61.91	62.51	62.37	62.46	79.63	82.55	80.45	81.33	82.33	82.70
Figure 4: Node classification on four datasets are conducted to analyse the sensitivity of the two Co-Training models, GCN(T) and SAGE(T) to the number of neighbors sampled per GNN layer. "Fanout" denotes the number of neighboring nodes to which the center node is directly connected.
the GNN-based. This is probably because their text attributes are more fully informative and thus the text attributes largely reflect the linking relationships between the nodes. Therefore the PLM-based methods model text attributes more strongly would be more advantageous in this case. And on the Photo dataset, the GNN-based method outperforms the PLM-based method across the board. This may be because the text attributes of the Photo dataset are composed of information from user reviews of the product. Some lower-quality reviews introduce a certain amount of noise to the text attributes, which will reduce the effectiveness of the PLM-based methods. In order to analyze the importance of node text attribute selection, we further conduct relevant experiments in Appendix D.9.
4.4 Validity of Topological Pre-training
In this subsection, we compare the three different pre-training methods with the PLM-based and GNN-based methods. We observe that on almost all PLMs and all datasets, different degrees of improvement can be achieved with the proposed pre-training methods. For these three individual pre-training tasks, we find that TCL leads to greater improvement in most cases. The difference in performance between TMLM and TCL, which both capture low-order topological structure information, indicates that learning topological structure knowledge from a node-level perspective may work better than a token-level. TDK, on the other hand, performs second only to TCL in most cases, which reflects to some extent the fact that PLMs can benefit from knowledge of the complex topology. Further, we try to combine these three pre-training tasks to optimize the model together (name TMDC). We first
perform token-level TMLM tasks on PLMs. The enhanced PLMs are then jointly optimized using both TCL and TDK tasks. (Detailed implementation can be found in Appendix A.2) Observing from the Table 4, we find that TMDC further improves performance in most cases. This indicates that different pre-training strategies can teach the PLM different topological knowledge from different perspectives, and this leads us to explore more pre-training tasks on TAGs in the future. To further analyze the effectiveness of these topological pre-training methods, we also test the performance of such topological pre-training models in other scenarios (e.g., semi-supervised learning and few-shot learning). Please refer to Appendix D.5 for detailed experimental results and discussions.
5 Discussion on the Practical Values
Text-attributed graphs have emerged as a prominent graph format, which finds extensive applications in modeling real-world tasks, such as the mentioned recommender systems. Our research concentrates on achieving a comprehensive understanding of the textual attributes embedded within a single node and the topological structural connections between nodes. For example, a famous example in recommender systems is the association between "diaper" and "beer", commonly co-purchased by customers, thereby establishing links between these items in the item-item graph. To achieve the optimal item representation, a prerequisite is to capture the inherent characteristics of a given item by modeling its metadata, such as title and descriptions. Simultaneously, it is imperative to incorporate valuable and unique signals derived from the graph's topological connections into the representation learning process. Given that real-world graph topology is usually shaped by human behaviors, there exist unique human perceptions and knowledge in the topology beyond the pure semantics (e.g., "diaper" and "beer" are semantically different but are connected in the co-purchased graph). Consequently, it is imperative to delve into the effective and efficient fusion of intrinsic semantics within individual nodes and the topological connections among different nodes on the text-attributed graphs. Moreover, the scope of our research also includes domains like user behavior-enhanced sponsored search, including AdsGNN [24], HBGLR [53], and PASS [54].
We establish the first comprehensive benchmark CS-TAG specifically designed to explore representation learning on TAGs. We collect and provide eight available text-attributed graph datasets to facilitate the NLP and GNN communities to focus and investigate the data together. Our benchmark provides a more comprehensive evaluation of different learning paradigms, validating their effectiveness and limitations. We will also continue to mine and construct more research-worthy TAGs to advance the continued healthy development of the field.
This research was funded by the National Science Foundation of China (No. 62172443), Open Project of Xiangjiang Laboratory (22XJ03010, 22XJ03005), the Science and Technology Major Project of Changsha (No. kh2202004), Hunan Provincial Natural Science Foundation of China (No. 2022JJ30053), and the High Performance Computing Center of Central South University.
[54] Zhoujin Tian, Chaozhuo Li, Zhiqiang Zuo, Zengxuan Wen, Lichao Sun, Xinyue Hu, Wen Zhang, Haizhen Huang, Senzhang Wang, Weiwei Deng, et al. Pass: Personalized advertiser-aware sponsored search. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2023.
[55] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Proceedings of ICML, 2019.
[56] Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. In Proceedings of KDD, 2022.
[57] Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11198, 2020.
[58] Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. Fame for sale: Efficient detection of fake twitter followers. Decision Support Systems, 2015.
[59] Shangbin Feng, Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. Twibot-20: A comprehensive twitter bot detection benchmark. In Proceedings of CIKM, 2021.
[60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[61] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng-Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.08239, 2022.
[62] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXiv:2204.02311, 2022.
A Baselines and Model Implementation Details
A.1 Baselines
We provide detailed descriptions on the GNNs and PLMs baselines used in the main experiments as follows.
In our work, we categorize three types of models, namely BERT-Tiny (4.4M), ELECTRA-Small (13.5M), and DistilBERT (66.4M), as small scale PLMs. ELECTRA-Base (109M), BERT-Base (110M), RoBERTa-Base (125M), and DeBERTa-Base (139M) are classified as base scale PLMs. ELECTRA-Large (334M), BERT-Large (340M), RoBERTa-Large (355M), and DeBERTa-Large (405M) are categorized as large scale PLMs.
A.2 Topological Pre-training Implementation
We introduce the implementation details on 4 topological pre-training tasks, TLM, TCL, TDK and TMDC on text-attributed graphs.
TMLM. We adopt the Masked Language Model (MLM) training implementation in huggingface [] for the TMLM task. For a given TAG, we first pre-sample each node with five neighbors, and subsequently concatenate the text of the original node with the text of the sampled neighboring nodes. Five corresponding center-neighbor pairs are obtained for each node. We then disorganize the dataset to form a new topologically augmented dataset. The new dataset is passed into the MLM training code to pre-train the language model. The learning rate of the TMLM is set to 5e-05 on all datasets and language models. For detailed code and implementation, please refer to the project repository.
TCL. Traditional Contrastive Learning (CL) focuses on the central node itself (using the augmented nodes to construct the positive pairs) [42]. For our Topological Contrastive Learning (TCL) task, we consider the central node and its neighbors to form the positive pairs. In particular, when loading the data in batches, we sample one of its neighbor nodes for each central node through the adjacency matrix of the TAG. For the central node and its sampled neighbor, we compute the contrasting loss by mapping the textual representations into the contrastive space through a projection head with shared parameters by following [41]. Therefore the  $\mathcal{L}_{\mathrm{tcl}}$  in the main text can be rewritten as
$$
\mathcal {L} _ {\mathrm {t c l}} = - \log \frac {\exp \left(\sin \left(z ^ {c} , z ^ {n}\right) / \tau\right)}{\sum_ {n ^ {\prime} = 1 , n ^ {\prime} \neq n} ^ {N} \exp \left(\sin \left(z ^ {c} , z ^ {n ^ {\prime}}\right) / \tau\right)}, \tag {7}
$$
where  $z^c$  is the representation of the node  $c$  in the contrastive space. The main learning parameters include the learning rate  $lr$  and epoch  $e$ . For all datasets and all language models we set  $e = 5, lr = 5e - 05$ . The projection head is a two-layer MLP with a hidden layer set to 128. The  $\tau$  in the contrastive loss is set to 0.2. Detailed pre-training commands on each dataset can be found in the CS-TAG GitHub repository.
TDK. For the TDK task, we first run the Deepwalk [45] algorithm over the TAG to obtain its topological-level representation (only the topology information of TAG is fed into Deepwalk). Then we pull the representation of the center node close to the representation learned from Deepwalk. In particular, when loading the data in batches, we load the representation learned from Deepwalk at the same time. Subsequently, we from the textual representation of the center node with its corresponding topological structure representation to form the positive pairs in contrastive learning. We follow the same contrastive learning process with TCL presented above. Therefore the  $\mathcal{L}_{\mathrm{tdk}}$  in the main text can be rewritten as
$$
\mathcal {L} _ {\mathrm {t d k}} = - \log \frac {\exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {k} ^ {c}\right) / \tau\right)}{\sum_ {c ^ {\prime} = 1 , c ^ {\prime} \neq c} ^ {N} \exp \left(\sin \left(\boldsymbol {z} ^ {c} , \boldsymbol {k} ^ {c ^ {\prime}}\right) / \tau\right)}, \tag {8}
$$
For all datasets in CS-TAG, we use a uniform code to obtain the corresponding topological-level representation. The relevant parameters are consistent with the parameter settings in TCL. Detailed pre-training commands on each dataset can be found in the CS-TAG GitHub repository.
TMDC. TMDC attempts to pre-train the language models jointly by combining the three pre-training tasks proposed above. As the forms of the TMLM and TCL, TDK tasks differ a lot, we use an iterative training way to perform the multi-task learning. In particular, we first obtain a token-level topologically augmented language model  $PLM_{tmlm}$  by TMLM. Subsequently, we combine TCL and TDK to jointly optimize the  $PLM_{tmlm}$ . The loss function of the joint optimization is shown as follows