A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking
Authors: Hao Yan
Chaozhuo Li
Ruosong Long
Chao Yan
Jianan Zhao
Wenwen Zhuang
Jun Yin
Peiyan Zhang
Weihao Han
Hao Sun
Weiwei Deng
Qi Zhang
Lichao Sun
Xing Xie
Senzhang Wang
Venue: NeurIPS 2023 Datasets and Benchmarks Poster (2023)
Keywords: Graph Representation Learning
Pretrained Language Models
Graph Neural Networks
Text-attributed Graphs

Abstract: Text-attributed graphs (TAGs) are prevalent in various real-world scenarios, where each node is associated with a text description. The cornerstone of representation learning on TAGs lies in the seamless integration of textual semantics within individual nodes and the topological connections across nodes. Recent advancements in pre-trained language models (PLMs) and graph neural networks (GNNs) have facilitated effective learning on TAGs, garnering increased research interest. However, the absence of meaningful benchmark datasets and standardized evaluation procedures for TAGs has impeded progress in this field. In this paper, we propose CS-TAG, a comprehensive and diverse collection of challenging benchmark datasets for TAGs. The CS-TAG datasets are notably large in scale and encompass a wide range of domains, spanning from citation networks to purchase graphs. In addition to building the datasets,  we conduct extensive benchmark experiments over CS-TAG with various learning paradigms, including PLMs, GNNs, PLM-GNN co-training methods, and the proposed novel topological pre-training of language models. In a nutshell, we provide an overview of the CS-TAG datasets, standardized evaluation procedures, and present baseline experiments. The entire CS-TAG project is publicly accessible at \url{https://github.com/sktsherlock/TAG-Benchmark}.

A Comprehensive Study on Text-attributed Graphs: Benchmarking and Rethinking
Hao Yan'*, Chaozhuo Li Ruosong Long', Chao Yan', Jianan ZhaoS, Wenwen Zhuang'
Jun Yin', Peiyan Zhang Weihao Han", Hao Sun', Weiwei Deng', Qi Zhang?,
Lichao Sun*, Xing Xie², Senzhang Wang
Central South University, Microsoft Research Asia, University Birmingham
Peking University, 'Hong Kong University of Science and Technology Université de Montréal, 'Microsoft, *Lehigh University
Abstract
Text-attributed graphs (TAGs) are prevalent in various real-world scenarios, where each node is associated with a text description. The cornerstone of representation learning on TAGs lies in the seamless integration of textual semanties within indi- vidual nodes and the topological connections across nodes. Recent advancements in pre-trained language models (PLMs) and graph neural networks (GNNs) have facil- itated effective learning on TAGs, garnering increased research interest. However, the absence of meaningful benchmark datasets and standardized evaluation proce- dures for TAGs has impeded progress in this field. In this paper, we propose CS- TAG, a comprehensive and diverse collection of challenging benchmark datasets for TAGs. The CS-TAG datasets are notably large in scale and encompass wide range domains, spanning from citation networks to purchase graphs. In addition to building the datasets, conduct extensive benchmark experiments over CS-TAG with various learning paradigms, including PLMs, GNNs, PLM-GNN co-training methods, and the proposed novel topological pre-training of language models. In a nutshell, we provide an overview the CS-TAG datasets, standardized evalua- tion procedures, and present baseline experiments. The entire CSGTAG project is publicly accessible at hhttps: http//gitttt /ggittu....... com https//ssssscccceeeeete Benchmark
Introduction
Graphs are ubiquitous in modeling the relational and structural aspects of real-world objects across various domains, such as social networks, transportation system networks, and biological protein- protein networks [ 3 In many real-world graphs, nodes are often associated with text attributes, giving rise to the text-attributed graphs (TAGs) [ 6 TAGs are prevalent in various scenarios, such as social graphs where each user is accompanied by textual description and paper citation graphs where textual content is linked to each respective paper 7 8]. The exploration of
learning methodologies applied to TAGs has emerged as a prominent research area within multiple fields, including graph learning, information retrieval, and natural language processing (
The nucleus of learning TAGs lies in the effective integration of both the node attributes (textual semantics) and graph topology (structural connections) to facilitate the learning of node represen- tations. The textual information associated with each node offers a wealth of semantic content, enabling the characterization of individual node properties, which could be captured by the pre- trained language models (PLMs) [00
B [4] Meanwhile, the structural information encoded
"Equal contribution and co-first authors. CCorrespondigg author.
37th Conference on Neural Information Processing Systems (NeurIPS 2023) Track on Datasets and Benchmarks.

---

PLM Based
GNN Based
Co- Trainiinng Based
Topological sraning Basod
PLM
GNN
Arthinction
PLM
GNN
PLM
Topeiag
PL Bosod
PLM
GNN
Nested
TRM
TRM
TRM
T-PLM
GNNN Dissed
Unewn
Problem:
Discard Topology
Probiem: No end-to-end pipeline
Problem: Efficiency &Salability
Whether works?
Figure 1: The traditional text attributed graph representation learning pipeline.
within the graph topology presents the inherent proximity relationships between nodes. Graph neural networks (GNNs) have proven to be effective in capturing such structural relations based on the
message-passing mechanism [7 16 17 I8 19 20 21 22)
PL.Mbased and GNN-based methods are two prevalent types of learning paradigms on TAGs as illustrated in Figure D PL.M-hased methods generally input the textual content derived from the target node into a pre-trained language model. However, the knowledge of topology resulting from the high non-linearity of graph structure within TAGs is largely discarded by the PLM-based based methods [9) Conversely, GNN-based methods are capable of preserving the intricate graph topology information with greater fidelity. Nevertheless, an inherent limitation plaguing GNN-based methods lies in the disconnected modeling process of node attributes and graph topology. Specifically, most GNNs pre- pre-model node attributes as static representations, treating them fixed and unlearnable parameters during the message passing process. Consequently, the gradients stemming from the learning objective of GNNs cannot be effectively back- proppppopggtted into the attribute modeling. This discrepancy in the training procedure hinders the attainment of an optimal solution, as it fails to guarantee end-to-end training, thereby impeding the overall effectiveness of the approach.
To simultaneously enjoy the merits of GNNs and LMs, several recent endeavors shed light on the co-training paradigm shown in Figure LMs and GNNs are combined in cascaded 23 24 25 or nested (5) manner, establishing a unified end-to-end training paradigm to model the node attributes and graph topology jointly. Despite its theoretical appeal, the co-training method suffers from severe scalability issues as its memory complexity is proportional to the graph size as neighborhood texts also encoded [9]. Motivated by the recent advancements in pre-traine pre-training techniques, novel inquiry emerges: Can we pre-train the language models to understand the graph topology? If we effectively encode topological information into LMs through appropriate pre-training tasks, LMs could serve as the foundational model for learning on TAGs. Topological pre-trained LMs eliminate the explicit GNN aggregations, thereby circumventing the efficiency challenges encountered in the co- co-training paradigm. However, the design of suitable and effective pre-training tasks to encode valuable knowledge derived from intricate graph topology into LMs remains an open question.
In order to delve deeply into the intricate interplay between textual semantics and graph topology within TAGs, we embark on unprecedented exploration to investigate the optimal training paradigm for various TAGs. Existing text text-attributed graph datasets (c.g., Cora (8). WikiCS (26). Amazon- Photo (27] cannot meet requirements, as they solely offer node attribute embeddings, devoid of the original textual sentences. To overcome this limitation, we meticulously curate a novel and com- prehensive dataset, dubbed CS-TAG. comprising eight distinct TAGs sourced from diverse domains. This carefully crafted dataset serves as a solid foundation for future research endeavors, facilitating in-depth investigations in this burgeoning field. Moreover, extensive experiments conducted on the CS-TAG dataset to provide comprehensive and reliable benchmarks. All aforementioned learning paradigms are thoroughly evaluated and analyzed. Experimental results and detailed discussions further reveal the underlying correlations between graph topology and textual attributes, drawing deep insight into the inherent characteristics of the TAGs. Our contributions are summarized as follows:
1. To the best of our knowledge, CS-TAG is the first open dataset specifically designed for text- attributed graphs. TAGs from a variety of fields are collected, cleaned, and organized as the final structured dataset. We provide researchers with original links and data cleaning codes to facilitate their access and reprocessing of these datasets in accordance with their research interests and requirements. The entire CS-TAG project is publicly accessible as an open source repository on Github, accessible at mtjttps:://ithhut
com/ wwwm//ktsserrlle ccwp///teeeeerlocc/AA---Beeeeoee Bennhhar
2. In contrast to previous topology-drived graph learning models, our work underscores the vital significance of deep node attribute modeling- This novel perspective sheds light on the design of next-generation GNNs by emphasizing the incorporation of deep node attribute understanding.
2

---

3. We investigate the novel problem of topological pre- pre-training of language models, aiming at teaching LMs to understand topological structures. This innovative training paradigm exhibits remarkable performance on the CS-TAG dataset in terms of effectiveness and efficiency, which contributes to broadening the scope of language model pre-training.
4. Extensive experiments are conducted across eight diverse datasets, focusing on two downstream tasks: node classification and link prediction. Such experiments serve as a rigorous evaluation of various learning paradigms, providing precise and dependable benchmarks for future endeavors.
2 Related Work
In this section, we first briefly introduce three popular learning paradigms for TAGs. After that, the comparisons between the existing graph learning benchmarks and the proposed CS-TAG are also discussed. Refer Appendix [ for more detailed reviews of the related models. We have implemented most of the algorithms discussed in this section in the repository.
PLM-based methods. The PLMS refer universal language models that possess enhanced semantic understanding due to their pre-training on a vast corpus (28). The early works on modeling textual attributes were based on shallow networks, e.g., Skip-Gram 29 and Glo In recent years, the backbone networks dominated by the pre-training-fime fenneffinnetttinggggg paradigm are rapidly scaling up: from [LM GPT[32], to BERT [10. RoBERTa (2]. DeBERTa [3] The large-scale models, which get fully trained with massive data, demonstrate superior performances on general NLP tasks. One of the most critical usages of PLMs is text representation, where the underlying semantics of texts are captured by low-dimessional embeddings. On the TAGs, the PLMs use the local textual information of each node to learn a good representation for the downstream task [9].
GNN-based methods. As graph representation learning enjoys explosive growth in machine learning, numerous research works have been proposed for various tasks including node classification [15], link prediction [21]. and so on, Graph neural networks are recognized as powerful tools for modeling graph data. Such methods (e.g.. GCN (1S. GAT [16), GraphSAGE 77, GIN [17, RevGAT (33) learn effective gressage-passing mechanisms such that information between the nodes can get aggregated for expressive graph representations. GNNs generally adopt the "cascade architecture" suggested by GraphSAGE for textual graph representation: node features are encoded independently using text modeling tools (e.g. PLMs) and subsequently aggregated by GNNs produce the final representation.
Co-training methods. The aforementioned two types of paradigms primarily focus on modeling partial information, which limits their ability learn comprehensive features. Several recent endeav- ors propose to co-train GNNs and LMs to enjoy the merits from both sides. Specifically, LMs and GNNs are combined in the cascaded 24 or nested manner (5). The outputs generated by LMs serve as inputs for GNNs, and vice versa. The parameters of both LM and GNN are updated through the pack-propagation of gradients from downstream tasks. However, this co-training paradigm suffers from serious scalability problems, as all neighbors need to be encoded from scratch by the LMs, incurring significant additional computational costs (
Benchmarks for graph representation learning. Several established graph benchmarks have been developed and widely adopted (34 35 27 36). However, when it comes to learning TAGs, these benchmarks exhibit notable deficiencies. Firstly, a majority of these datasets suffer from the absence of raw textual information, limiting the investigation of attribute modeling's effectiveness. Secondly, these datasets often neglect to explore the impact of text attribute modeling on GNNs. Thirdly, these datasets are predominantly small in scale. Thus, there is compelling necessity construct comprehensive large-scale dataset for TAGs.
3 CS-TAG: A Comprehensive Dataset and Benchmark for TAGs
In this section, we commence by providing a concise summary of the constructed CS-TAG benchmark in Section 3.1 Subsequently, we present the details of the construction CS-TAG in Section 3.2 including data collection, cleaning, and labeling. Moreover, we elucidate the details of GNN- PlM-based, and Co-training learning paradigms Section 3.3 Finally, the proposed topological pre-training of LMs is presented in Section 3.4
3

---

3.1 Overview of CS-TAG
In order to address the limitations inherent in prior researches, we propose the establishment of the text-attributed graph benchmark, dubbed CS-TAG, which serves standardized evaluation framework for assessing the efficacy of representation learning techniques on TAGs. To ensure scalability, CS-TAG includes datasets of varying sizes and incorporates scalable baselines consisting of PLMs, GNNs, and co-training methods. This enables researchers to evaluate the performance of their models across broad range of dataset scales. To enhance usability, we provide a modular pipeline that simplifies the implementation of different models within the CS-TAG. Such a modular architecture enables researchers to easily integrate their novel methods and compare them with existing approaches. In addition, we are committed to maintaining a public leaderboard for TAGs, serving as repository for the latest advancements the field. This platform will continuously update text-attributed graph datasets that possess practical and research value, fostering ongoing progress and collaboration within the community. Overall, CS-TAG serves as scalable, unified, modular, and consistently updated evaluation framework for assessing the performance of representation learning methods on eext-attributed graphs.
3.2 Dataset Construction
In order to thoroughly investigate the perfor- mance of different learning paradigms on TAGs, conduct extensive survey of various text- attributed graph datasets that have been pre- viously utilized in the literature. Our obser- vations reveal that many commonly employed node-level datasets are essentially text-attributed graphs. For instance, well-known citation graphs such Cora, PubMed, Citeseer (8), and ogbn-arxiv [34] are all TAGs. These datasets derive node attributes from textual information, such as the title and abstract of papers. Addi- tionally, academic collaboration networks such as Coauthor CS/Physics [27] set node attributes derived from keywords defined in the papers.
10.0M
Size of different TAGs
Edges
Chilcren Sports Finees
1.0M
oghn-ande-TA
Computers
Amazon Computers
Hestry
Photo
WKCS
Coauther Physics
100.0K
Amazon Photo
Coouthor CS
Pubmed
10.0K
Cheseer
Previous
Ours
1.0K
Nodes
1.0K
10.0K
100.0K
1.0M
Figure 2: The differences between the TAGs datasets CS-TAG (used for node classification) and the previous datasets.
However, while these datasets are frequently employed by GNNs, they possess obvious inad- equacies when exploring representation learning on TAGs. Firstly, most of these datasets lack the availability of raw textual information, bringing challenges to investigating the effectiveness of attribute modeling on these datasets. Secondly, these datasets generally overlook the exploration of text attribute modeling's 's impact on GNNs. A majority of these datasets employ simplistic ong-of-wordh models or traditional text encoding techniques like GloVe or Skip-Grai Gram to represent text attributes, which are kind of outdated. Lastly, these datasets are predominantly small in scale, leading to a lack of differentiation between different learning models across numerous datasets.
To address these limitations, we have taken proactive steps to collect and construct some novel datasets of TAGs. [ the number of nodes/edges in the previous datasets and the proposed CS-TAG. One can clearly see that the TAGs within CS-TAG are comparatively larger than the counterparts. Here, we present the details of shopping graphs as an example. We extract datasets from the Amazon dataset (37), including Books-Children/Hfittory Sports-Fitness. Nodes represent different types items, while edges indicate items that are frequently purchased or browsed together. Node labels are assigned based on the product category. To explore the influence of attributes in the aext-attribute graphs, distinct text attributes have been provided for each of these datasets. For example, in the Bookss-hildrenHHittory dataset, node attributes are derived from the title and description of the respective books, such as "Description: Collection of Poetry; Title: The golden treasury of poetry". The Sports-Fitness dataset only incorporates node attributes from the title of the sports items, such as "Girls Ballet Tutu Neon Orange". In the Ele-Computers/PPee datasct, node attributes obtained from high-rated reviews and product summaries, for instance,
Aic-Computers/Phott
and

---

Table Statistics of text-attributed graph dataset used in CS-TAG.
Dutuset
Nodes
Edges
Class
Domain
Modeling
Scale
Tisks
Raw Text
WikiCs
11,701
216.123
10
Wikipedia
GlooV
Medium
Node classification
Corn
2,708
5,429
7
Academic
Bag of words
Small
Node classification
Citeseav
3,327
4,732
Academic
Bog
Small
Node classification
Pubmed
19.717
44,338
3
Academic
Bag of words
Medium
Node classification
Previous
ughn-arxiv
169,343
1666,243
40
Academic
Skop Gram
Large
Node classification
Coauthor
18,335
81,894
Academic
Bag words
Mediam
Node classification
Coauthor Physics
34,493
247,900 977962
5
Academic
Bag words
Medium
Node classification
Amuzon Photo
7,487
119.043
10
E commerce
Dag of words
Small
Node classification
Computers
15,381
248,778
R.cammerce
Rag wards
Medium
Node classification
ogha-arxiv-1B TA
169,343
L,166,233
40
Academie
PLMS
Large
Node classilication
Books-Children
76.875
1,554,578
24
commerce
PLMs
Large
Node classification
Books-History
41,551
358,574
B.commerce
PLMS
Node classification
Ours
Ele-Photo
Ele-Computers
48,362
87,229
721.081
500.928
10
12
E commerce
E-commerce
PLMs
PLMs
Large
Large
Node classification
Node classification
Sports-Fitness
173,055
1,773,500
13
cammerce
PLMs
Large
Node classification
Citation V8
1,106,759
6,120,997
Academic
PLMS
Large
Link Prodiction
GoodReads
676,084
8,582.324
E E-commerce
PLMs
Large
Link Prediction
"Great camera for the price! This camera takes crystal clear photos and is cheap too!". Further details on the dataset construction process can be found in Appendix
Additionally, we construct two other datasets, Citation and GoodReads, for the link prediction task. The Citation V8 dataset represents citation network extracted from DBLP [38]. Node attributes in Citation V8 are derived from the titles and abstracts of research papers. Each edge signifies a citation relationship between two papers. The GoodReads dataset, the other hand, originates from a prominent book review website This dataset captures the "similar item" linking relationship between books and provides valuable information about the attributes of each book, such as the title and description. Therefore, we leverage the GoodReads dataset to construct link prediction tasks, which involve predicting relationships between similar books. Detailed descriptions of all the aforementioned datasets can be found in Appendix B
This
3.3 Conventional Learning Paradigms on TAGS
Existing learning paradigms TAGs can be broadly classified into three distinct categories: 1) GNN- based methods: These methods primarily leverage GNNs as the foundational model for capturing the underlying graph topology structures through pesssag--aassing mechanisms. 2) PL.M-based methods: These approaches rely on prevalent pre-trained language models to capture the semantics from the textual node attributes, which excel in their ability to comprehend text semantics and exhibit strong transferability. 3) Co-training methods: This paradigm involves the joint learning of GNNs and LMs under unified framework (23) enjoy the merits from both sides. Next, we will give the formulaic definitions of these three paradigms.
GNN-based Paradigm. GNNs are employed to propagate information across the graph nodes, allowing for the extraction of meaningful representations via message passing, which are formally defined as follows:
h (e++) = UPDATE (n) AGGREGATEE"""
(((",,eN(W)))
(I)
where A represents the layers of GNNs, / denotes the set of neighbors, denotes the target node, means the learning parameters in GNNs. Please note that, the initial node feature vector ( using pre-learned by PLMMs other shallow text encoder (e.g., Skip-Gram). Such attribute modeling phase is performed independently of the subsequent training of GNNs. Gradients from the GNN training objectives are unable to be back-propagated into the PLMs to update their parameters. And this decoupling of PLMs and GNNs impedes the overall effectiveness.
PLM-based Paradigm. baM-- methods leverage the effectiveness of pre-training techniques to enhance the modeling of text within each node. The formulation of these methods is as follows:
h (R++) UPDATE (1(0)
where zb denotes the learnable parameters in PLMs. PLMs advance the modeling of node text attributes. However, incorporating crucial topological context into PLM-based paradigms remains a challenge, particularly when the available textual data is limited.
hhtpss/wwwwgoodrraas.ccmm
5

---

PLM
PLM
PLM
PLM
Center Node
Neighbor
Center Node
Naighbor Node
Topological Masked Language Modaling
Topoiogical Contrastive Learning
Topological Deepwalk Learning
Figure 3: Illustrations of different topological pre-training methods.
Co-training Paradigm. GNNs and LMs are jointly trained under unified training framework: fe(A,T) = G (A, 2 (T)) θ = {w, {b}
(3)
where f denotes the learning function and θ denotes the entire learnable parameters, which derived from both the GNN and PLM modules. The outputs generated by the LM serve as input to the GNN. Notably. the gradients obtained from the GNN can be proppprppggtted the LM, enabling the update of its parameters. However, the co-training method faces significant scalability challenges. This is primarily due the memory complexity associated with encoding neighborhood texts, resulting in memory requirement that scales linearly with the size of the graph [9]
3.4 Topological Pre-training of Language Models
The incorporation of explicit GNN aggregations in the co-training paradigm introduces inherent challenges in terms of training complexity and resource requirements. This is primarily due to the simultaneous modeling of texts from both the center node and its neighbors. Therefore, this brings us to question: is there a training paradigm to enjoy the merits of graph topology while avoiding the explicit GNN operations? Inspired by the recent advancements our motivation lies in teaching language models to understand the topological structures. Three topological pre-training tasks are proposed to impart graph structures into the LMs, enabling them to better comprehend and capture the underlying topology.
pre-training techniques [39] 40).
Topological Masked Language Model (TMLM), Inspired by the task of masked language modeling, we propose a novel topological masked language model (TMLMM to capture the first-order connections on the token level. Given center node e and of its neighbors ni, their corresponding text is
formally defined as (e) {t(() <(e) ,....,t, &
(c) and ( = {t(")) (22 ..... (())) respectively. We
randomly replace subset of tokens in the center node (Cc) and (n) with a special token [MASK].
The objective of TMLM is to predict the masked tokens. Let (+) {0) 92 ....,
(c) (0 (m)))
represents the indexes of the m masked tokens in the sentence (() and (*)) Let T denote the
set of masked tokens in T(s) and (m)), and ((*) denote the set of observed (unmasked) tokens. The objective of TMLM is:
m log (t (T))
Lamim (T
m
in which 0 denotes the learnable parameters.
Topological Contrastive Learning (TCL). Inspired by contrastive learning 41 42 propose novel topological contrastive learning (TCL) task to capture the first-order topological information the node level. Given center node and one of its neighbors n, their corresponding node-level (sentence/documee--eeeea)) embedding is derived from their CLS token in the T and can be formally defined as h. h". The objective of TCL is bring the center node he closer to its neighbors htd while pushing itself farther away from other nodes. Denoting the cosine similarity function as shm h= = h h² //"||[/"||]| [I. The objective of TCL is:
exp(sim(h" h²)
Liel - log
(5)
Dn L,n exp(sim (he, h/7)
where denotes the temperature parameter, N denotes the batch size.
Topological Deepwalk Learning (TDK). TMLM and TCL. mainly capture low-order structural information, while the higher-order structural information still needs to be captured by designing

---

Table 2: Accuracy comparison among GNNs on the ogbn-arxiv-TA within different PLMs' node features. "Scale" means the different versions of PLMs (number of parameters). "Diff" denotes the performance gap between the best and worst performers. We mark the best performer each row with blue bold font and mark the best performer in each column with black bold font.
Arxiv
Scale
PLMS
GCN
GAT
SAGE
RevGAT
NFormer
GIN
JKNNet
APPNP
MoNet
MLP
BERI-Tim
72.03
72.25
72.35
72.52
71.91
68.42
69,50
71.63
45.13
57.22
Small
ELECTRÁ
68.45
70.97
69.63
71.12
69.45
58.09
62.87
59.55
36.65
36.58
DistlBERT
73.39
73.48
74.48
74.68
73.56
72.30
71.44
74.01
50.51
68.11
ELECTRA
70.81
71.67
70.82
71.96
70.43
64.88
63.41
65.62
38.91
48.56
BERT
73,30
73:40
74.14
74.59
72.80
71,94
70.08
73,90
46.90
67.35
Base
RoBERTa
73.56
73.38
74.52
74.82
73.12
72.63
69.40
74.01
44.53
69.31
DeBERTa
68.15
66.56
67,58
68.26
67.11
62.05
44.16
52.37
29.67
47.07
HI LECTTRA
70.44
71.01
70.72
72.56
70.04
64,47
58.34
64.52
37.26
47.72
BERT
73.25
73.37
74.15
74,68
73.12
71.88
68.70
73.53
43.31
66.85
Large
RoBERTs
73,95
73.72
74.64
74.99
73.12
73.10
68.10
74.17
44.01
69.51
DeBHRTa
72.57
71.50
73.22
73.59
71.88
71.25
54.41
69.28
33.53
66.28
5.80
7.16
7.06
7.54
6.45
15.01
27.28
21.80
20.84
32.93
suitable tasks. Considering that algorithms like Deepwalk 45 can capture higher-order structural information in the graph, we try to use the node representations learned by Deepwalk to augment the representations learned by LM. We first feed the whole graph structure into Deepwalk to get the corresponding representation &c of each node c. The objective of TDK is to bring the center node h closer to its representation learned from Deepwalk k The objective of TDK is:
cxp(sim(h" k²)/T)
Liak = -log
(6)
ΣN
ce== exp(sim(h kxd )/T)
4 Experiments
Baselines. (1) For GNN-based methods, we select 9 popular GNN models: GCN 15], GAT [6]]
GraphSAGE 77 RevGAT (33, NodeFormer [46], GIN [17. JKNet [88. MoNet 47 and
APPNP 48 (2) For PLM-based methods, we select 5 different PLMs with different scales: a) Small
parameter scale models including BERT-Tiny [O), ELECTRA-Smal [I], and DistilBERT [4. b)
Base parameter scale models including BERT-Base [0). ELECTRA-Base II. RoBERTa-Base [2]
and DeBERTa-Base [3]. c) Large parameter scale models including BERT-Large 10. ELECTRA- Large M. RoBERTa-Large [12], and DcBERTa-Large [13]. (3) For Co-training methods, due to scalability constraints, we only explore the effectiveness of this pipeline on combinations of BERT-Tiny with GCN and GraphSAGE. (4) For the topological pre-training of LMs, we conduct experiments on various datasets and different PLM basic models. In addition, we iteratively train the proposed three pre-training tasks at batch level (named TMDC) in multi-task learning framework. Please refer to Appendix A.2 for more details.
Implementation details. GNNs are mainly implemented based on the DGL toolkit 49 PLMs are obtained from Huggingface (50) and trained under a unified framework. Considering the recent rise of parameter-efficient fine-tuning, we only fine-tune the last four encoder layers of large-scale language models. Implementation details and hyperparameter selections are provided in Appendix C
Evaluations metric. We investigate the performance of different baselines through two tasks: node classification and link prediction. For the node classification task, we use Accuracy and FI-Score to evaluate the model performance. For the link prediction task, we use MRR, Hits @ 10, Hits @ 50, and @its@ 100 as metrics. Due to space limitations, we present the results of some node classification experiments in the main paper, and the remaining node-level experiments with link prediction results are presented Appendix [D1 and D.2. addition to the aforementioned datasets, we also conduct experiments on other types of datasets, as described in Appendix D.6 (a large-scale ogbn-papers (0M dataset) and Appendix [D.7 (two social network datasets), where readers can find detailed information on these datasets and the corresponding experimental results.
7

---

Table 3: Node classification of the three learning paradigms on six datasets. Sports corresponds to the Fl score as an indicator of experimental results and Accuracy for the rest of the data. We bold the best results for each dataset.
Way
PL.--Based
GNN-Based
Co- Trairinn Based
Tiny
Base
T-GCN
B-GCN
T-SAGE
B-SAGE
GCN(T)
SAGE(T)
Arxiv
70.83
72.96
72.03
73.30
72.35
74.14
69.22
73.571
Children
49.85
59.91
57.07
58.11
57.57
58.74
54.75
59.701
History
83.06
86.09
84.52
85.04
84.79
85.12
83.52
85.091
Photo
73.75
77.53
82.42
82.70
83.25
83.27
83.321
86.64
Computers
58.32
60.40
87.43
87.86
87.90
88.30
83,93
86.04
Sports
81.47
86.02
84.93
86.16
87.06
87.34
85.061
85.87
4.1 Impact of Static Modeling of Attributes on GNNs
In this subsection, analyze the impact of different node attribute modeling methods for downstream GNNs. Table represents the effect node classification on ogbn-arxiv-TA for different GNNs given different PLMs' features. The results other datasets can be found in Appendix [.I] Observing [ we find that RevGAT performs the best among all initial node features, while GAT and SAGE exhibit the second-best performance. They are relatively less affected by the different node features, 7.54%. 7.16%, and 7.06% respectively. JKNet, APPNP, MoNet, and MLP are more influenced by the initial node features, which all come above 20. On the other hand, the node features encoded by RoBERTa, BERT, and DistilBERT generally perform better on all types of baselines. DeBERTa, which performs better on many downstream NLP-related tasks, is less effective, This may be because DeBERTa sees less corpus during pre-training, resulting in their inability to understand the semantics well when modeling the text directly on downstream tasks. Furthermore, we compare traditional shallow text encoders (e.g., Skip-Graar Gram) with PLMs in Appendix D.4 for a more comprehensive analysis of the impact of text modeling on downstream GNNs.
Recently, LLMs (Large-Languag Models) are continuing to energize areas such as knowledge graphs (51) and recommender systems (52). It is still open question on how to successfully apply I.MSs to text-attributed graph learning. We have conducted a preliminary exploration of how to use LLMs advance the representation learning on TAGs. Please refer Appendix D.8 for the detailed results and discussions.
4.2 Pitfalls of Co- To-Training Paradigm
In this subsection, we analyze the performance of the Co-training paradigm versus the PLM-based, GNN-based paradigm in terms of node classification tasks. Tiny and Base in the PLM column repre- sent BERT-Tiny and BERT-Base, respectively. T-GCN, T-SAGE, and B-GCN, B-SAGE represent the node features of BERT-Tiny and BERT-Base fed downstream GCN and GraphSAGE respectively. GCN(T) and SAGE(T) then denote co-training BERT-Tiny with GCN and SAGE, respectively. We compare GCN(T), and SAGE(T) with the corresponding T-GCN, T-SAGE respectively. As shown in Table 3 SAGE(T) improves in four of the datasets compared to the T-SAGE methods, with a maxi- mum improvement of 3.39% on the Photo dataset. However, GCN(T) performs worse than T-GCN on most datasets and is even weaker than BERT-Tiny 1.61% on ogbn-arxiv-TA, The Co-training framework requires simultaneous training of PLMS and GNNs. The memory requirement and time cost of this paradigm are significantly increased. In order to facilitate the co-training of PLMs and GNNs, we reduce either the batch size or the number of neighbors. The limited scalability leads to a significant reduction in the number of neighbors for GNN aggregation, which may compromise the effectiveness message passing. To analyze the impact of scalability on the Co-Training method, we analyze the effect of the number of neighbors sampled per GNN layer Co-training in Fig 4 As can be seen from it, the effect of the model mainly tends to increase as the number of neighbors increases. The detailed discussions efficiency and scalability can be found in Appendix D.3
4.3 Comparing Pased Methods with GNN-based Methods
In this subsection, we compare the PlM-based methods and the GNN-based methods in different datasets. As shown Table the column GNNs indicates the best results on all GNNs for a given PLM node feature. On the Children and History dataset, the PLM-based methods works better than
8

---

Table 4: Node classification accuracy comparison among PLM-based, GNN-based, and topological pre-training based on four datasets. The best method for each PLM on each dataset is shown in bold.
Scule
Mudel
Arxiv
History
PLM
GNNS
TMLM
TDK
TCL
TMDC
PLM
GNNs
TMLM
TDK
TCL
TMDC
BERI-Tiny
70.83
72.52
70.83
71.55
83.06
85.03
85.76
85.79
86
86.88
Small
ELECTRA
71.26
71.12
72.65
72.83
73.06
73.71
84.18
83.11
84.54
84.12
84.57
85.18
DistBERT
72.50
74,68
73.53
74.38
74.89
75.50
85.81
85,67
85,76
86.29
86.28
86.88
RRECTRA
72.67
71.96
73.51
77.33
74.26
75.56
83.79
85.77
88
86.62
86.41
BERT
72.96
74.59
73.97
74,23
74.57
76.11
86.09
85.28
86.24
86.16
86,80
86,82
Basc
RoBERTa
75.37
86
N6.96
DeBERTa
73.82
68.26
74.26
75.01
75.15
75.99
86.16
$2.31
86.00
86.46
87.01
86.94
ELECTRA
72.42
72.56
74.76
73.82
74.17
75.58
86.13
83.56
86.39
86.49
86.82
86 28
BERT
73.24
74.68
75.01
74.31
75. 15
75.75
86.23
85.15
86.47
86.73
86
86.94
Large
RoBERTs
73.83
74.99
75.
74.58
75.48
75.73
86.41
85.23
86.72
86.75
87.11
887.22
DeBERTa
74.57
73.59
75.92
75.20
75.58
76.20
87.00
84,89
87.11
87.26
87.30
87.32
Scale
Model
Children
Photo
GNNs
TMLM
TCL
TMDC
PLM
GNNss
TMLM
TDK
TCL
TMDC
BERI-Tiny
49.85
$7.86
54.27
53.43
54.11
54.66
73.75
84.12
74.30
73.59
73.86
74.92
Small
ELECTRA
57.03
57.35
77.74
77.83
DiselBERT
59.90
59.33
60.03
60.23
60.60
61.38
77.51
84.34
77.81
79.69
81.85 81.85
82.52
RA
60:21
60.83
82.98
82.82
BERT
59.91
58.74
60.34
60:43
60.73
61.43
77.53
84.46
78.54
81.04
82.85
84.09
RoBERTa
59.80
59.01
60.19
60.71
61.47
61,83
78
$4.59
78.33
81.26
82.47
83.04
DeBBRTa
60.26
60:73
62.20
78:37
83.07
83,80
ELECTRA
58.28
$6.59
60.51
59.31
$9.29
61.31
77.25
$3.00
79.21
78.44
79.56
81.32
BERT
60.65
58.90
60.84
61.15
61.50
62.06
77.72
84.31
78.95
79.26
80.74
81.14
Large
Rо
60.93
59.26
62.11
61,95
62.06
63.24
79.60
85.12
80.32
80.82
81
82,55
DeBERTa
61.61
56.34
61.91
62.51
62.37
62.46
79.63
82.55
80.45
81,33
82.33
82.70
Anev
History
Computers
Photo
85
85
70
Accuracy
65
Accuracy
62
Accuracy
80
Accurssy
80
75
60
Layorel
Layer=?
60
Layer=1
Layer=2
70-
Layerw
Layer=
Layer=
Layer=2
Layer=l
Layer=]
70
55
2ayer
78
2
65
2
2 Fanout
2
2
2
Fanout
Fanout
Fanout
Figure 4: Node classification on four datasets are conducted to analyse the sensitivity of the two Co-Traing models, GCN(T) and SAGE(T) to the number of neighbors sampled per GNN layer. "Fanout" denotes the number of neighhoring nodes to which the center node is directly connected.
the GNN-based. This probably because their text attributes are more fully informative and thus the text attributes largely reflect the linking relationships between the nodes. Therefore the PLM-based methods model text attributes more strongly would be more advantageous in this case. And on the Photo dataset, the GNN-based method outperforms the PLM-based method across the board. This may be because the text attributes of the Photo dataset are composed of information from user reviews of the product. Some lower-quality reviews introduce certain amount of noise to the text attributes, which will reduce the effectiveness of the PLM-based methods. In order to analyze the importance of node text attribute selection, further conduct relevant experiments in Appendix D.9
4.4 Validity of Topological Pre-training
In this subsection, we compare the three different pre- pre-training methods with the based and GNN- based methods. We observe that on almost all PLMs and all datasets, different degrees of improvement can be achieved with the proposed pre-training methods. For these three individual pre-training tasks, we find that TCL leads to greater improvement in most cases. The difference in performance between TMLM and TCL, which both capture low-order topological structure information, indicates that learning topological structure knowledge from node-level perspective may work better than a teven- TDK, on the other hand, performs second only to TCL. in most cases, which reflects to some extent the fact that PLMS can benefit from knowledge of the complex topology. Further, we try to combine these three pre-training tasks to optimize the model together (name TMDC). We first

---

perform token-level TMLM tasks on PLMs. The enhanced PLMs are then jointly optimized using both TCL and TDK tasks. (Detailed implementation can be found in Appendix AA.)) Observing from the Table 4 we find that TMDC further improves performance in most cases. This indicates that different pre-training strategies can teach the PL.M different topological knowledge from different perspectives, and this leads us to explore more pre-training tasks on TAGG in the future. To further analyze the effectiivness of these topological pre-training methods, we also test the performance of such topological pre-training models in other scenarios (e.g., semi-supervised learning and few-shot learning). Please refer to Appendix D.5 for detailed experimental results and discussions.
Discussion on the Practical Values
Text-attributed graphs have emerged prominent graph format, which finds extensive applications in modeling real-world tasks, such as the mentioned recommender systems. Our research concentrates on achieving comprehensive understanding of the textual attributes embedded within single node and the topological structural connections between nodes. For example, a famous example in recommender systems the association between "diaper" and "beer", commonly co-purchased by customers, thereby establishing links between these items in the item-item graph. To achieve the optimal item representation, prerequisite to capture the inherent characteristics of given item by modeling its metadata, such as title and descriptions. Simultancously. it is imperative to incorporate valuable and unique signals derived from the graph 's topological connections into the representation learning process. Given that real-world graph topology is usually shaped by human behaviors, there exist unique human perceptions and knowledge in the topology beyond the pure semantics (e.g., "diaper" and "beer" are semantically different but are connected in the co co-purchased graph). Consequently, it imperative to delve into the effective and efficient fusion of intrinsic semantics within individual nodes and the topological connections among different nodes on the text-attributed graphs. Moreover, the scope of our research also includes domains like user behavior-enhanced sponsored search, including AdsGNN (24), HBGLR 53 and PASS [54).
6 Conclusion
We establish the first comprehensive benchmark CS-TAG specifically designed to explore represen- tation learning TAGs. We collect and provide eight available text-attributed graph datasets to facilitate the NLP and GNN communities to focus and investigate the data together. Our benchmark provides a more comprehensive evaluation of different learning paradigms, validating their effective- ness and limitations. We will also continue to mine and construct more research-worthy TAGs to advance the continued healthy development of the field.
Acknowledgement
This research was funded by the National Science Foundation of China (No. 62172443), Open Project of Xiangjiang Laboratory (22XJ03010, 22XJ03005), the Science and Technology Major Project of Changsha (No. kh2202004), Hunan Provincial Natural Science Foundation of China (No. 20223130053), and the High Performance Computing Center of Central South University.
10

---

References
[1] Yanqiao Zhu, Yuanqi Du, Yinkai Wang, Yichen Xu, Jieyu Zhang, Qiang Liu, and Shu Wu. A survey deep graph generation: Methods and applications. arXiv preprint 2203::2203.67744 2022.
[2] Hao Miao, Jiaxing Shen, Jiannong Cao, Jiangnan Xia, and Senzhang Wang. Mba-stnet: Bayes-enhanced discriminative multi-task learning for flow prediction. IEEE Transactions on Knowledge and Data Engineering, 2022.
[3] Zijian Zhang, Xiangyu Zhao, Hao Miao, Chunxu Zhang, Hongwei Zhao, and Junbo Zhang Autostl: Automated spatio-temporal multi-task learning. arXiv preprint arXiv:2304.09 091444 2023.
[4] Peiyan Zhang, Yuchen Yan, Chaozhuo Li, Senzhang Wang, Xing Xic, Guojie Song, and Sunghun Kim. Continual learning on dynamic graphs via parameter isolation. In Proceedings of SIGIR, 2023.
[5] Junhan Yang, Zheng Liu, Shitao Xiao, Chaozhuo Li, Defu Lian, Sanjay Agrawal, Amit Singh, Guangzhong Sun, and Xing Xie. Graphformers: Gnn-nested transformers for representation learning on textual graph. Advances in Neural Information Processing Systems, 2021.
[6] Xiaoxin He, Xavier Bresson, Thomas Laurent, and Bryan Hooi. Explanations as features: Llm-based features for text-attributed graphs. arXiv preprint arXiv:2305. 19523, 2023.
[7] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Proceedings of NeurIPS, 2017.
[8] Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi- Rad. Collective classification in network data. Al magazine, 2008
(9] Jianan Zhao, Meng Qu, Chaozhuo Li, Hao Yan, Qian Liu, Rui Li, Xing Xie, and Jian Tang. Learning on large-scale text-attributed graphs via variational inference. arXiv preprint arXiv:2210. 14709, 2022.
[10] Jacob Devlin, Ming- Wei Chang, Kenton Lee, and Kristina Toutanova. BERT: pre-training of deep bidirectional transformers for language understanding. CoRR, 2018.
[11] Kevin Clark, Minh- text encoders as discriminators rather than generators. In Proceedings ICLR, 2020.
Minh-Thang Thang Luong, Quoc V Le, and Christopher D Manning. Electra: Pre-training
[12] Alexis Conneau, Kartikay Khandelwal, Naman Goyal, Vishrav Chaudhary. Guillaume Wen- zek, Francisco Guzmán, Edouard Grave, Myle Out, Luke Zettlemoyer, and Veselin Stoyanov. Unsupervised cross-lingual representation learning at scale. In Proceedings of ACL, 2020.
[13] Pengcheng He, Xiaodong Liu, Jianfeng Gao, and Weizhu Chen. Deberta: Decoding-enhaneee bert with disentangled attention. In Proceedings of ICLR, 2021.
[14] Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. Distilbert, distilled version of bert: smaller, faster, cheaper and lighter. ArXiv, 2019.
[15] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. Proceedings of ICLR, 2017.
[16] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. In Proceedings of ICLR, 2018.
[17] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In Proceedings of ICLR, 2019.
[18] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Proceedings of ICML, 2018.

---

[19] Jinlong Du, Senzhang Wang, Hao Miao, and Jiaqiang Zhang. Multi-channel pooling graph neural networks. In IJCAI, 2021.
[20] Zhongyu Huang, Yingheng Wang, Chaozhuo Li, and Huiguang He. Going deeper into permutation--ensiiive graph neural networks. In Proceedings of ICML, pages 9377-9409 2022.
[21] Rui Li, Jianan Zhao, Chaozhuo Li, Di He, Yiqi Wang, Yuming Liu, Hao Sun, Senzhang Wang. Weiwei Deng, Yanming Shen, et al. House: Knowledge graph embedding with householder parameterization. In Proceedings of ICML, 2022.
[22] Yi Zhao, Chaozhuo Li, Jiquan Peng. Xiaohan Fang, Feiran Huang, Senzhang Wang, Xing Xie, and Jibing Gong. Beyond the overlapping users: Cross-domain recommendation via adaptive anchor link learning. In Proceedings of the 46th International ACM SIGIR Conference on Research and Development in Information Retrieval, pages 1488-1497, 2023.
[23] Jason Zhu, Yanling Cui, Yuming Liu, Hao Sun, Xue Li, Markus Pelger, Tianqi Yang, Liangjie Zhang, Ruofei Zhang, and Huasha Zhao. Textgnn: Improving text encoder via graph neural network in sponsored search. In Proceedings of WebConf, 2021.
[24] Chaozhuo Li, Bochen Pang. Yuming Liu, Hao Sun, Zheng Liu, Xing Xie, Tianqi Yang, Yanling Cui, Liangjie Zhang, and Qi Zhang. Adsgnn: Behavior- augmented relevance modeling in sponsored search. In Proceedings of SIGIR, 2021.
[25] Shuxian Bi, Chaozhuo Li, Xiao Han, Zheng Liu, Xing Xie, Haizhen Huang, and Zengxuan Wen. Leveraging bidding graphs for advertiser-awan relevance modeling in sponsored search. In Proceedings of EMNLP, 2021.
[26] Péter Mernyei and Cätälina Cangea. Wiki-cs: A wikipedia-basee based benchmark for graph neural networks. arXiv preprint rxXv:2007.02900 2020.
[27] Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint 18110588 2018.
[28] Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in neural information processing systems, 2020.
[29] Tomas Mikolov, llya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations of words and phrases and their compositionality. arXiv preprint arXiv: 10.44566 2013.
[30] Jeffrey Pennington, Richard Socher, and Christopher D Manning. Glove: Global vectors for word representation. In Proceedings of EMNLP. 2014.
[31] Matthew E Peters, Mark Neumann, Mohit Iyyer, Matt Gardner, Christopher Clark, Ken- Lee, and Luke Zeutlemoyer. Deep contextualized word representations. arXiv preprint 102.5=180.. 2018.
[32] Alec Radford, Karthik Narasimhan, Tim Salimans, Ilya Sutskever, et al. Improving language understanding by generative pre-training. 2018
[33] Guohao Li, Matthias Müller, Bernard Ghanem, and Vladlen Koltun. Training graph neural networks with 1000 layers. In Proceedings of ICMI., 2021.
[34] Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 2020.
[35] Scott Freitas, Yuxiao Dong, Joshua Neil, and Duen Horng Chau. A large-scale database for graph representation learning. arXiv preprint S0xiv:0011.07600 2020.
[36] Qinkai Zheng, Xu Zou, Yuxiao Dong, Yukuo Cen, Da Yin, Jiarong Xu, Yang Yang, and Jie Tang. Graph robustness benchmark: Benchmarking the adversarial robustness of graph machine learning, arXiv preprint
2021.
12

---

[37] Jianmo Ni, Jiacheng Li, and Julian McAuley. Justifying recommendations using distantly- labeled reviews and fine-gratd mne-grained aspects. Proceedings of EMNNP-IJCNLL 2019.
[38] Jie Tang, Jing Zhang, Limin Yao, Juanzi Li, Li Zhang, and Zhong Su. Ametminer: extraction and mining of academic social networks. In Proceedings of KDD, 2008.
[39] Suchin Gururangan. Ana Marasovié, Swabha Swayamdipta, Kyle Lo. Iz Beltagy, Doug Downey. and Noah A Smith. Don't stop pretraining: Adapt language models to domains and tasks. arXiv preprint arXiv:2004. 10964, 2020.
[40] Mingda Chen, Jingfei Du, Ramakanth Pasunuru, Todor Mihaylov, Srini Iyer, Veselin Stoyanov, and Zornitsa Kozareva. Improving in-context few-shot learning via self-supervised training. arXiv preprint arXiv:2205.01703, 2022.
[41] Yuning You, Tianlong Chen, Yongduo Sui, Ting Chen, Zhangyang Wang, and Yang Shen. Graph contrastive learning with augmentations. In Proceedings of NeurIPS, 2020.
[42] Yanqiao Zhu, Yichen Xu, Feng Yu, Qiang Liu, Shu Wu, and Liang Wang. Deep graph contrastive representation learning. arXiv preprint arXiv:2006.0411 2020.
[43] Hao Yan, Senzhang Wang. Jun Yin, Chaozhuo Li, Junxing Zhu, and Jianxin Wang. Hierarchical graph contrastive learning. In Proceedings of ECML PKDD, 2023.
[44] Senzhang Wang. Hao Yan, Jinlong Du, Jun Yin, Junxing Zhu, Chaozhuo Li, and Jianxin Wang. Adversarial hard negative generation for complementary graph contrastive learning, In Proceedings of SDM, 2023.
[45] Bryan Perozzi, Rami AI-Rfou, and Steven Skiena. Deepwalk: Online learning of social representations. In Proceedings of KDD, 2014.
[46] Qitian Wu, Wentao Zhao, Zenan Li, David P Wipf, and Junchi Yan. Nodeformer: A scalable graph structure learning transformer for node classification. Proceedings of NeurlPS, 2022.
[47] Federico Monti, Davide Boscaini, Jonathan Masci, Emanuele Rodola, Jan Svoboda, and Michael M Bronstein. Geometric deep learning on graphs and manifolds using mixture model cnns. In Proceedings of CVPR, 2017.
[48] Johannes Gasteiger, Aleksandar Bojchevski, and Stephan Günnemano. Predict then propagate: Graph neural networks meet personalized pagerank. In Proceedings of ICLR, 2019.
[49] Minjie Wang, Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centric, highly-performant package for graph neural networks. arXiv preprint arXiv: 1909.0131, 2019.
[50] Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, Rémi Louf, Morgan Funtowicz, Joe Davison, Sam Shleifer, Patrick von Platen, Clara Ma, Yacine Jernite, Julien Plu, Canwen Xu, Teven Le Scao, Sylvain Gugger, Mariama Drame, Quentin Lhoest, and Alexander M. Rush. Transformers: State-of-the- art natural language processing. In Proceedings of EMNLP, 2020.
[51] Shirui Pan, Linhao Luo, Yufei Wang, Chen Chen, Jiapu Wang, and Xindong Wu. Unifying large language models and knowledge graphs: A roadmap. arXiv preprint arXiv:2306.08300 2023.
[52] Wenjie Wang, Xinyu Lin, Fuli Feng, Xiangnan He, and Tat-Seng Chua. Generative recommen- dation: Towards next-generation recommender paradigm. arXiv preprint arXiv:230..03516 2023.
[53] Bochen Pang, Chaozhuo Li, Yuming Liu, Jianxun Lian, Jianan Zhao, Hao Sun, Weiwei Deng, Xing Xie, and Qi Zhang. Improving relevance modeling via heterogeneous behavior graph learning in bing ads. In Proceedings of KDD, 2022.
13

---

[54] Zhoujin Tian, Chaozhuo Li, Zhiqiang Zuo, Zengxuan Wen, Lichao Sun, Xinyue Hu, Wen Zhang, Haizhen Huang, Senzhang Wang, Weiwei Deng, al. Pass: Personalized advertiser- aware sponsored search. In Proceedings of the 29th ACM SIGKDD Conference Knowledge Discovery and Data Mining, 2023.
[55] Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In Proceedings of ICML, 2019.
[56] Wentao Zhang, Ziqi Yin, Zeang Sheng, Yang Li, Wen Ouyang, Xiaosen Li, Yangyu Tao, Zhi Yang, and Bin Cui. Graph attention multi-layer perceptron. In Proceedings of KDD, 2022.
[57] Fabrizio Frasca, Emanuele Rossi, Davide Eynard, Ben Chamberlain, Michael Bronstein, and Federico Monti. Sign: Scalable inception graph neural networks. arXiv preprint arXiv:2004.11199 2020.
[58) Stefano Cresci, Roberto Di Pietro, Marinella Petrocchi, Angelo Spognardi, and Maurizio Tesconi. Fame for sale: Efficient detection of fake twitter followers. Decision Support Systems, 2015.
[59] Shangbin Feng. Herun Wan, Ningnan Wang, Jundong Li, and Minnan Luo. Twibot-20: A comprehensive twitter bot detection benchmark. In Proceedings of CIKM, 2021.
[60] Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. Exploring the limits of transfer learning with unified text-to-text transformer. The Journal of Machine Learning Research, 2020.
[61] Romal Thoppilan, Daniel De Freitas, Jamie Hall, Noam Shazeer, Apoorv Kulshreshtha, Heng- Tze Cheng, Alicia Jin, Taylor Bos, Leslie Baker, Yu Du, et al. Lamda: Language models for dialog applications. arXiv preprint arXiv:2201.088 2022.
[62] Aakanksha Chowdhery, Sharan Narang, Jacob Devlin, Maarten Bosma, Gaurav Mishra, Adam Roberts, Paul Barham, Hyung Won Chung, Charles Sutton, Sebastian Gehrmann, et al. Palm: Scaling language modeling with pathways. arXiv preprint arXhh:2204.023111 2022.
14

---

A Baselines and Model Implementation Details
A.1 Baselines
We provide detailed descriptions on the GNNs and PLMs baselines used in the main experiments as follows.
GCN. [15] Graph Convolutional Network (GCN) is a classical model that works by performing a linear approximation spectral graph convolutions.
GraphSAGE. 7 GraphSAGE is a GNN model that focuses on inductive node classification, but also be applied for transductive settings.
GAT. [16] Graph Attention Network (GAT) introduces the attention mechanism to capture the importance of neighboring nodes when aggregating information from the graph.
RevGAT. [33] RevGAT combines reversible connectivity with a deep network architecture to form a deep and efficient GNN.
NodeFormer. 466 NodeFormer is scalable graph Transformer for large-scale graphs, which achieves all-pair message passing with linear complexity. In the table, we denote NodeFormer by NFormer.
GIN. (17) Graph Isomorphism Network (GIN) overcomes the drawbacks previous MPNN-based methods, which struggles to differentiate various graph structures based on the generated graph embeddings.
JKNet. [18) Jumping knowledge Network (JKNet) adaptively varies the neighborhood ranges for individual node, enabling enhanced structure-an awarr node representation.
MoNet. 47 Mixture Model Network (MoNet) captures and represents the structural properties of graphs by incorporating multiple localized perspectives of each node's s neighborhood.
APPNP. 48 Approximate Personalized Propagation of Neural Predictions (APPNP) is specifically
developed for semi-supervised learning tasks on graph-structure data. utilizes personalized propagation to iteratively enhance node predictions by incorporating comprehensive information from both local and global contexts.
DiStiIBERT. [14 DistilBERT's is another distilled version of BERT. The student model BistillBert shares a similar overall architecture with the teacher model BERT. DistillBert differs from BERT by removing the token-type embeddings and pooler components, and also reducing the number of layers.
ELECTRA. [1] ELECTRA is pre-training model for NLP tasks that introduces Discriminative Pre-training. ELECTRA enhances the efficiency and effectiveness by generating more efficient representations through a discriminative approach.
BERT. 10] Bidirectional encoder representations from Transformers (BERT) utilizes a transformer architecture that employs self-attention mechanisms to capture word relationships within sentences. This enables the model to effectively consider both the preceding and succeeding contexts of a word, facilitating bidirectional language comprehension. BERT undergoes unsupervised pre-training on an extensive text corpus, where it predicts masked words within sentences and acquires the ability encode contextual information.
RoBERTa. [12) Robustly Optimized BERT (RoBERTa) is a variant of the BERT. RoBERTa incorporates additional modifications during pre-training to optimize its performance. In the pre- training phase, it trains on vast corpus of unlabeled text data by employing masked language modeling (MLM) while excluding the next sentence prediction (NSP) task. RoBERTa significantly expands the amount of training data used, enabling the acquisition of more comprehensive and robust language representations.
DeBERTa. [13 Decoding-enhancee BERT with Disentangled Attention (DeBERTa) brings forth two significant enhancements: disentangled attention and heterogencous layer integration. By dividing the attention heads into distinct groups, disentangled attention empowers the model to better capture dependencies among words, enabling more targeted and specialized attention. This enhancement greatly improves the model's ability to capture long-range dependencies in a more effective manner.
In our work, we categorize three types of models, namely BERT- (4.4M), ELECTRA-Small (13.5M), and DistilBERT (66.4M), as small scale PLMs. ELECTRA-Base (109M), BERT-Bas (110M), RoBERTa-Base (125M), and DeBERTa--ase (139M) are classified as base scale PLMs. ELECTRA-Large (334M), BERT-Large (340M), RoBERTa-LLage (355M), and DeBERTa-L (405M) are categorized as large scale PLMs.
15

---

A.2 Topological Pre-training Implementation
We introduce the implementation details topological pre-training tasks, TLM, TCL. TDK and TMDC on text-attributed graphs.
TMLM. We adopt the Masked Language Model (MLM) training implementation in huggingface [] for the TMLM task. For a given TAG, we first pre-sample each node with five neighbors, and subsequently concatenate the text of the original node with the text of the sampled neighboring nodes. Five corresponding center-neighbor pairs are obtained for each node, We then disorganize the dataset to form new topologically augmented dataset. The new dataset is passed into the MLM training code to pre-train the language model. The learning rate of the TMLM is set to 5e-05 on all datasets and language models. For detailed code and implementation, please refer to the project repository.
TCL. Traditional Contrastive Learning (CL) focuses on the central node itself (using the augmented nodes to construct the positive pairs) 42). For our Topological Contrastive Learning (TCL) task, we consider the central node and its neighbors to form the positive pairs. In particular, when loading the data in batches, we sample one of its neighbor nodes for each central node through the adjacency matrix of the TAG. For the central node and its sampled neighbor, we compute the contrasting loss by mapping the textual representations into the contrastive space through projection head with shared parameters by following [41]. Therefore the Liel in the main text can be rewritten as
Lid -log exp(sim(2*, 2²)/T)
nn exp(sim(2* "0r )T))
where zf is the representation of the node in the contrastive space. The main learning parameters include the learning rate Ir and epoch e. For all datasets and all language models we set e = 5, Ir = 5e 05. The projection head is two-layer MLP with hidden layer set to 128. The in the contrastive loss is set to 0.2. Detailed pre-training commands on each dataset can be found in the CS-TAG GitHub repository.
TDK. For the TDK task, we first run the Deepwalk 45 algorithm over the TAG to obtain its topological-level representation (only the topology information of TAG is fed into Deepwalk). Then we pull the representation of the center node close to the representation learned from Deepwalk. In particular, when loading the data in batches, same time. Subsequently, we from the textual representation of the center node with its corresponding topological structure representation to form the positive pairs in contrastive learning. We follow the same contrastive learning process with TCL presented above. Therefore the Leak in the main text can be rewritten as
load the representation learned from Deepwalk at the
Lidk -log exp(sim(2",, kT) Ee=Le exp(sim(2*) (z kc )/T)
(8)
For all datasets CS-TAG, we use a uniform code to obtain the corresponding topological-level representation. The relevant parameters consistent with the parameter settings in TCL. Detailed pre-training commands on each dataset can be found in the CS-TAG GitHub repository.
TMDC. TMDC attempts to pre-train the language models jointly by combining the three pre-training tasks proposed above. As the forms of the TMLM and TCL., TDK tasks differ a lot, we use an iterative training way to perform the multi-task learning. In particular, we first obtain token-level topologically augmented language model PLM by TMLM. Subsequently, we combine TCL and TDK to jointly optimize the P PLMmmm- The loss function of the joint optimization is shown as follows
exp(sim(z*, (²)//
+ log N (alsimmm",22)//11
Ludtiik = - -(log N
(9)
Σ Ee= exp(sim(2 ( )/T)
Σ
En exp(sim (20, 2xd )²²)
A.3 Code License
The code of CS-TAG uses the MIT license. Please refer to the GitHub repository for license details.
16

---

B Datasets
B.1 Dataset Format
For each dataset in CS-TAG, we provide three different files. We store the graph-type data available to dgl in the pt format in Pytorch. For the node classification dataset, two types of features are stored, the adjacency matrix and the node labels. For the Cleppoomssers and Ele-Photo datasets, we also store the year features of the nodes (the year of the comments posted by the user). For the link prediction dataset, it contains only the adjacency matrix information. We use .Ixt file to store the text attributes of each dataset. The .cvv file stores node-id, node-label, category, and text to provide a clearer picture of the dataset for subsequent researchers.
B.2 Dataset Construction
The construction of the text-attributed graph datasets includes the following three steps. First, pre- processing the text attributes in the original dataset, including removing missing values, removing non-English statements, removing abnormal symbols, length truncation, etc. Second, building the graph. The linking relationship between nodes has been provided in the original data the dataset constructed in CS-TAG, such as the information of "also view", and "also buy" of the products in the Amazon dataset (indicating the two product ids that are jointly purchased or viewed), and the citation relationship between papers in DBLP, etc. Note that when obtaining the final graph data, self-edges and isolated nodes need to be removed. Third, refining the constructed graph. For the node classification dataset, the nodes in the graph need corresponding numerical node labels. We convert the categories of nodes in the original data to numerical node labels in the graph. For some datasets that are divided in specific form (e.g.. by the year of publication of a paper), we also need to store additional information about the nodes.
B.3 Dataset Details
CS-TAG includes 8 datasets, whose details are described as follows. The statistics of the datasets is shown in Table
ogbn-arxiv-TA dataset derived from ogbn-arxiv. The corresponding task is to predict the categories of the papers, which is formulated 40-class classification problem. The text attributes of each paper node are extracted from its title and abstract in ogbn-arxiv. Note that ogbn-arxiv solely provides node embeddings acquired through shallow text encoders like Skip Gram. Different from them, use different PLMs to model the node attributes to get the initial node features and delve into the performance of employing multiple PLM features in downstream GNNs.
Books-Childdeen/fiittor datasets are extracted from the Amazon- Amazon-Books dataset. Books-Childee consists of items with the second-level label "Children", while Books- the second-level label "History". The nodes in the dataset are books, and the edges mean two books frequeetly co-purchased book. We choose the title and description of the book itself as the text attributes of the node. The task is to classify books into 24 and 12 categories, respectively.
consists of items with
co-viewed. The label of each dataset is the three-level label of the
Ele- Cohouterss/Pooo datasets are extracted from the -eleztrneeecsonics dataset. Ele- Ele-Compusers consists of items with the second-level label "Computers", while Ele-Photo consists of items with the second-level label "Photo". The two datasets are extracted from the updated 2018 Amazon Computer and Amazon Photo datasets (27]. The nodes in the dataset are electronics related products, and the edge between two products means that they are frequently co-purchased or co-viewed. The label of each dataset is the three-level label of the electronics products. We adopt user reviews on the item as its text attribute. Since the item has multiple reviews, we mainly adopt the review with the highest number of votes. For some items lacking highly votes reviews, randomly adopt user review as the text attribute. The task on the two datasets is to classify electronics products into 10 and 12 categories, respectively.
Sports-Fitmes Fitness dataset extracted from the Amazon-Sports dataset. It consist of items with the second-level label "Fitness". The nodes in the dataset are the fitness-related items, and the edge between two items means that they are frequently co-purchased or co-viewed. The label of the dataset is the three-level label of the items. The task on this dataset is to classify items into 13 categories.
17

---

Table Statistics of text-attributed graph dataset used in CS-TAG.
Dataset
Nodes
Edges
Split Scheme
Split
Tass Type
Metric
length
ogbn-arxis-TA
169,343
1,166,243
Time
54/18/28
Node Class.
Acc,
512
Books-Children
76,875
1,554,578
24
Random
60/20/20
Node Class.
Acc, F1
256
Books-History
41,551
358,574
Random 60/20/20
Node Class.
Acc
256
Ele-Computers
87,229
721,081
Time 72/17/11
Node Class.
Acc,
256
Ele-Photo
48,362
500,928
12
Time 60/20/20
Node Class.
Acc, Fl
512
Sports-Fitmess
173,055
1,773,500
Random 20/10/70
Node Class.
F1
64
Citation V8
1,106,759
6,120.897
Time
99/1/1
Link Prediction
MRR
256
GoodReads
676,084
8,582,306
Random
90/2/8
Link Prediction
Hits @ K
24
Citation is a directed graph dataset, representing the citation relationship among a subset of papers extracted from DBLP (38]. It is constructed following the form of ogbl-citation2 [34]. The corresponding task is to predict the missing citations given some existing citations. Specifically, for randomly dropped, and the prediction model tries to rank
each source paper, two of its references the missing two references higher than 2,000 negative reference candidates. The negative references sondomly-sampleds from all the previously published papers that not referenced by the source paper. We adopt the title and abstract of each paper as its node text attributes.
GeodReads dataset is extracted from the world's largest book review site Goodreads, Its nodes are books and edges are identified by similarity relationships between books provided in the website. The corresponding task is to predict the similar relationships between books. We expect the model to rank the true correlations over the false ones. Specifically, we rank each true correlation among a set of 5,000 manommly-saplle negative correlation.
B.4 Datasets License
The datasets follow the MIT license. Please refer to the GOOD GitHub repository for license details.
C Experiment Settings
C.1 Experimental Settings of GNNS
We conduct experiments on 9 GNN models described in A.1 on 6 node classification datasets. We use the aforementioned 5 PLMS with different parameter scales to model the node attributes and form the initial node features of the graph data. Each experiment is repeated three times and the evaluation metrics are accuracy and Fl-score. The parameters shared by all GNN models include epochs, model layers, hidden units, learning rate, and dropout ratio, and their values are set to 1000, (2.3), 144,128,256,, (1e-04 1e-04 ~1e-02), 0.2, respectively. Besides these hyperparameters, for GAT freeze the number of heads to and set the ratio of attention- GraphSAGE model, we use mean pool to aggregate the neighbor information, and for JKNet, we cat aggregate the features. For APPNP, we set the teleport probability to 0.1 by default and the number of propagation steps is set to 2. For MoNet, we set the pseudo coordinate dimensions in GMMConv to 2 and 3, and set the number of kernels in GMMCony layer to 2. Since it mostly does not converge at epoch=1000, we set its maximum epoch to 2000. For GIN, we set the number of mip layers as The eval patience of all models is set 1. We use cross-entropy loss with the AdamW W optimizer to train and optimize all the above models, GNNs are mainly derived from the implementation in the DGL library.
out to 0 by default. For
C.2 Experimental Settings of PLMs
We conduct experiments on 5 PLMs with different parameters described in A.I 1 on 6 node classification dataset. Considering the efficiency of the language models, we conduct experiments on each dataset only once. The parameters shared by all the PLMs in A.1 include epochs e, label smoothing factor fs learning rate Ir, warm-up epochs w, batch size, and eval patience. The label smoothing factor is used to calculate the cross-entropy loss which is set in GNN well. The w denotes the duration of the warm-up phase, and w 1 means the duration of the warm-up in one epoch. For all datasets we set lsf. w, Ir to 4, 0.1, 1, (5e-06 ~5e-04 ). respectively. Due to different model parameters and different dataset sizes, we list the eval patience, batch size of different models on each dataset in Table 6 For PLMs at the same scale, we use the same batch size. For large-scale
18

---

Table 6: Batch size and eval patience of the different scale PLMs.
Small Scale
Base Scale
Large Scale
Datasets
Batch Size
Eval Patience
Batch Size
Eval Patience
Batch Size
Eval Patience
ogbn-arxiv-TA
100
50000
60
50000
60
50000
Books- Books-Children
240
15000
90
15000
90
15000
books-History
240
8000
90
8000
90
8000
Ele-Computeres
300
20000
180
20000
180
20000
Ele-Photo
100
5000
60
5000
60
5000
Sports-Fitness
800
10000
400
10000
400
10000
models, since full parameter fine-tuning is costly, we only fine-tune the last four encoder layers and its' effect is sometimes better than full parameter tuning. The experimental setup of several topological pre-training methods has been mentioned in the previous section, and for the topological pre-trained language models, we follow the same tuning strategy as the PLM-based methods.
C.3 Reproducibility
For all experiments, we select the best checkpoints according to the validation sets, and report the results. All the datasets and codes reproduce the results in this paper are available at http://gghtbb.oommkttttcctt//CCt BeC.see
D Additional Experiment Results
D.1 Experimental Results for Node Classification
GNN-based methods. Table 3-5 lists the experimental results for the node classification task. We first analyze the performance of different GNN- methods on the tasks of node classification. Over the evaluation metric accuracy, one can see that RevGAT, GraphSAGE, and GAT perform the best on these datasets, which are less affected by the node features learned by different PLMs. While for the other GNN models, their performance is affected by the PLMs to different extents. GIN, which is commonly used for graph-level tasks, does not achieve better performance node-level tasks. MoNet's performance on Gooks-Children and History datasets is even lower than MLP in most cases.
By investigating the impact of node features encoded by different PLMs on the downstream models, one can see that node features encoded by ELECTRA typically produce larger gap between GNN and MLP. The failure the ELECTRA model to generate higher quality node features may be due to its discriminative pre-training way. In contrast, the RoBERTa Base model seems to have better semantic understanding, with a difference between GNN and MLP of only 1.91 and 4.44 on the Books-History and Books-Children datasets. It is worth noting that the node features encoded by DeBERTa also perform poorly. However, DeBERTa performs better on NLP-related tasks, which indicates that the node features encoded by language model cannot be judged by its performance downstream tasks alone. The ineffectiveness of the DeBERTa to obtain better node features may be mainly due to its reduced corpus during pre- tre-training.
Similar experimental conclusions can be drawn on the FI-score results as shown Table Note that the Fl score of downstream models is more significantly affected by different PLMs. For example, on Books- Books-Children Children, the score of JKNet ranges from 19.79 to 30.94 under different PLMs.
PLM-based methods and Topological Pre-training Between Table to 5, PLMs denote PLM-based methods, i.c., fine-tuning directly on the dataset without considering the topology. While TLM, TDK, TCL, and TMDC denote fine-tuning on models pre-trained for these four topological pre-training tasks. Observing the experimental results on these datasets, the PLM-basd method performs the worst on all models with all datasets. This indicates that it is not sensible to ignore topology and use only text attributes for representation learning on TAGs. In contrast, TMDC, a multi-task form of topology pre-training, performs the best, achieving the best both metrics in the majority of experiments. This indicates that combining different topological structure pre-training methods can teach language models topological structure knowledge from different perspectives.
19

---

Table 7: Node classification accuracy on ogbn-arxiv-IA Books-Children, Books-Histonn and Ele- Photo. We bold the best results for each row.
Mediank
Small
Bise Scule
BHRT-Twy
FCTRA
REECTRA
NFRT
RoNFIRTS
DeRFSC
FLECTRRA
BERT
DaRFFRRT
72.96
72:43
73.83
70.97
70.73
RovGAT
71.12
74.59
74.82
73.56
74.90
69.45
72.80
70.04
GNNS
GIN
APPNP
59.55
73.00
MoNa
OCN
DOM
Co Traiting
SAGE
DOM
0UM
DOM
00M
OUM
00M
OOM
73:97
54.76
75.18
72.83
73.82
TPT
23.0
28.17
Small
BERT Tay
BLECTRA
BERT
DeBERGG
RoBERIG
57.03
59.91
58. 11
57.11
57.70
47.63
57.35
SAGE
58.21
RevGAT
59.01
59.26
GNN
GIN
55.37
51.90
JKN
52:33
57:73
50.24
54.73
34.70
43.04
32.55
52.61
OOM
0OM
OOM
Co- Co-Trining
SAGE
DOM
DOM
00M
OOM
TMLM
56.92
6043
TCL
56.88
6073
61.50
TMDD
58.55
61.38
60.83
61.43
61.83
02.20
61.31
62.06
63.34
Bocks-Hiilny
Large KoNFFKi
SART-Twy
ECTRA
DASIRERT
BLBCTRA
HFRT
RoRFRRT
DeBFKKh
FCTEA
LMss
84.18
85.81
96.09
85.85
86.16
86.13
86.24
66.41
GCN
82.08
52.46
85.04
85. 15.15
62.31
82.54
$4.95
58.51
84.22
GAT
82.85
84
80.71
81.
$4.36
54.47
SAGR
82.1
83.00
RevGAT
85.23
80.95
GNNS
GIN
62.62
73.69
$3.50
85.19
$4.01
19.81
83.29
83.34
82.89
JKNet
80.5
85.53
26, 77
83.45
$2.13
77.36
APPIN
78.65
79.91
$4.00
82.68
MoNia
28
72:72
90.17
70.66
60.79
$3.00
68.09
83.78
74,60
68.85
$2.73
DOM
00M
DOM
COM
00M
OOM
Traizing
SAGE
DOM
00M
OOM
DOM
00M
00M
O0M
TMLM
85.76
84.54
$3.76
55.77
86.24
36.19
96,00
86.39
$6.47
56.72
87.11
TDK
85.79
84.02
$6.29
55.38
36.45
86.32
86.46
86.89
86.73
56.75
87.26
DCL
86.05
84.57
85.25
86.62
36.80
86.95
87.01
86.81
$6.90
82.11
87.50
TMDC
86.88
88.18
$6.88
36.41
86.82
86.96
66.94
86.25
86.94
87.22
87.32
Metbods
Photo
Small Scale
Baee Scale
Large Scale
BERT Tay
ELECTRA
DGRIBERT
BLECTRA
BEKT
RoBERT
ICBERIo
ELECTRA
BERT
AnBEKJo
DEBERIa
LMSs
PLM
73.75
76.55
77.51
37.54
77.53
75.11
78.37
77.25
77.72
79.60
79.63
GCN
82.42
78:86
82.91
79.99
82:70
82.99
80.07
79.20
$2.01
53.82
80.76
GAT
83.82
82.83
83.75
$2.52
85.74
83.
79.47
K3.00
$3.48
$3.97
82.55
SAGE
83.25
80.90
$3.50
$1.79
85.27
83.91
81.44
81.05
$2.77
$4.15
91.85
RevGAT
84.12
85.12
84.34
$2.98
$4.46
84.59
82.59
$4.21
85.12
81.12
NUoomer
79.95
80.25
$2.09
$0.02
$1.79
$2.45
79.06
75.96
$1.23
$2.96
90.62
GIN
76.00
64.89
77.23
68.56
76.55
77.76
60.01
66.96
75.32
70.37
7095
JKNet
79.65
75.29
$0.41
76.53
79.72
79.13
60.13
74.35
79.18
78.61
69.57
APENP
79.24
70.77
$3.45
73.56
80.68
81.92
67.52
72.89
79.21
$2.02
76366
MoNec
76.24
65.55
72.84
06.39
75.17
73.67
57.48
66.24
71.08
T3.57
61.7
MLP
55.43
47.69
64.57
$1.24
02.1
05 65
54.12
09 99
50,83
56.26
58.76
GCN
83.32
00M
0UM
OOM
DOM
DOM
O0M
00M
OOM
OOM
DOM
Co-Trinnin
SAGE
86.64
00M
00M
0OM
DOM
0OM
00M
00M
OOM
OOM
00M
TMLM
74.30
76/09
77.81
78.07
78.54
78
70.23
70.21
78.05
$0.33
80.45
TDK
73.90
76.89
70.69
$0.18
81.04
81 26
K1.34
44
79.26
$0.83
81.33
TCL
73.86
77.74
$1.85
$1.47
82385
82.47
K3.07
79.56
80.74
$1.47
82.35
TMDC
74.92
77383
82.52
52.52
$4.09
83.04
83.80
81.32
$1.14
$2.55
82.70
20

---

Table 8: Node classification fl score on ogbn-arxiv- Books-Children, Ele-Photo, and Sports- Fitness. We bold the best results for each row.
Mediank
ugln-arsi--A Bise Scule
Small
BRRT-Twy
FCTRA
DSsHRRERT
RLRCTRRA
NFRT
RoNFIRTS
DeRFSCh
FLECTRRA
BERT
57,50
54.57
55.86
55.70
RovGAT
55.01
43.06
45.56
30.55
51.16
GNNS
GIN
APENP
33.67
54.13
MoNa
2015
20.
34.28
15.35
OCN
Co Traiting
SAGE
DOM
0UM
DOM
00M
00M
00M
OOM
57.23
56.22
TPT
80.00
60.56
Small
Tay
BERT
DeBERG
ELECTRA
ROBERIG
38.09
47.55
40.70
50.67
SAGE
47-37
47.82
GNN
GIN
JKN
38.92
7.98
13.54
16.16
40.95
37.99
20.75
21.14
OUM
OOM
00M
OOM
Co- Co-Trinnng
SAGE
DOM
DOM
00M
00M
OOM
TMLM
44.79
40.26
47.06
411
TCL
4043
42.93
49.72
52,63
TMDD
44.21
49.31
30.9
50009
52.38
49.96
51.11
52.55
Eic-Plato
Small
Large KoNFFI
SART-Twy
ECTRA
DSSiRRRT
ELECTRR
HFRT
RoBFRRT
DeRFKKh
FCTRA
LMss
66.11
70.82
30.48
70.04
71:21
72,10
87.97
70.96
73.69
OCN
75.64
72.02
75.35
16.49
20.90
74.91
96.54
72.95
GAT
77.24
15:22
77.26
96.54
76.97
71.30
73.00
75.00
76.77
77.47
15.66
SAGR
72.97
73.96
75.36
76.11
73.25
12.31
RevGAT
76.55
77.23
73.55
76.12
77.51
72.15
68.12
74.45
GNNS
GIN
53.35
63.75
00.26
69
63.50
51.73
55.00 00
82.80
57.46
IKNat
89.90
66.72
09.90
65.5
69.5
67.02
67.51
09.77
68.11 11
$4.92
89.86
71
73.88
63.26
MoNia
$7.11
14.98
34.56
20.30
23.25
51.75
DOM
00M
DOM
DOM
00M
00M
OOM
Traizing
SAGE
DOM
00M
DOM
00M
00M
OOM
TMLM
63.19
66.62
71.50
30.93
71.34
71.76
72.27
72.73
73.23
73.96
24.61
TDK
65.15
68.08
72.92
73,60
76.98
75.08
73.84
73.79
71.71
74,85
75.85
DCL
64.94
69.34
75.00
96.93
76.5 55.59
76.62
26.92
71.95
75,05
75.48
76.95
TMDC
66.23
2002
76.62
76.97
78.18
77.45
78.69
24.47
75.27
77.08
76.
Metbods
Sports Fimess
Small Scale
Riee Scale
Large Scale
BERT Tay
ELOCTRA
DGRIBERT
BLECTRA
BEKT
RoBERT1
1CBERIo
ELECTRA
BERT
AnBEKJo
DEBERD
LMSs
PLM
81.47
76.55
85.31
53.94
86.02
83.04
84.28
76.85
53.06
$4.16
86.21
GCN
84.03
82:24
86.15
53.46
86. 16
85.83
77.96
83.54
53.70
56.07
83.71
GAT
85.45
85.45
$6.58
55.57
86
86. 26.26
74,23
86.85
55.50
55.77
84.56
SAGE
87.05
85.05
67.51
55.70
87.34
87.39
85.80
57.46
$7.58
85.60
RevGAT
87.55
85.95
$7.88
$3.59
$7.45
87.56
77.79
87.03
57.68
$7.96
96.62
NU
83.69
$1.15
$4.56
$2.20
$1.42
$1.65
74.12
81.
$4.49
$5.00
82.26
GIN
83.50
71.3 31
74.91
71.51
$1.95
72.47
67.00
71.00
58.77
$3.53
79.86
JKNet
80.70
73.56
75.69
71.70
80.31
74.37
15.38
66.20
50,56
77.24
48.73
APENP
83.62
71.5 16.464
83.54
73.10
$4.59
82.61
36.81
73.37
55.18
$4.31
77.37
MoNec
59.95
45.56
65.80
$1.23
6902
69.56
58,59
70.01
72.12
65.13
MLP
58.47
43.65
70.09
49.56
73.69
74.90
52,86
56.33
74,08
77.57
71.41
GCN
85.05
85.21
0UM
OOM
DOM
0OM
O0M
00M
OOM
OOM
DOM
Co-Trinnin
SAGE
85.87
96.21
00M
0OM
DOM
0OM
00M
00M
OOM
OOM
0OM
TMLM
81.56
80.15
86.23
$4.45
86 35
83.05
R5.16
70.89
R5.45
84.85
86.51
TDK
81.40
82.35
96.48
55.51
86.88
84.92
96.04
80.47
85.65
85.50
86
TCL
82.04
83.20
67.44
56.72
8800%
86.67
K7.29
K2.05
86.64
$5.80
87.57
TMDC
83.15
84.21
88.56
$7.69
89001
38.39
88.56
84.51
87.89
$6.43
88.78
21

---

Table Node classification accuracy and fl score on Ele- Ele-Computers. We bold the best results for
each row.
Metiods
Ble- Bie-Comguees (Accuracy)
Scale ELECTRA
Rane: Stale
Lane Stale
BERI-Tay
DAMRRRRT
BLSCTRRA TEA
BERT
RoBERTs
IXBERT
ELECTRA
BPRT
KoBFRI
DERERI
IMs
PLM
58.92
59.85
00.51
40.50
60:40
61:10
61.88
50.63
50:70
61.22
61.96
DCN
87.41
84:15
63.30
56.10
87.86
88 38.50
82.
64.29
67:38
58.56
85.75
GAT
88.57
87.85
80.07
48.14
88.63
88.80
63.42
88.07
68.64
13.83
87.40
SAGE
87.90
86.43
K3.67
56.07
88.3
88.87
K2.81
87.
88.35
13.66
86.85
RAT
88.66
87.92
89.32
S5.22
88.72
88.91
84.12
HS.45
88.86
$0.83
87.56
GNNS
GIN
Neoomer
75.07
66.81
85.45
85.18
53.18
$7.96
73.36
$6.05
86
8502
83. 16.96
82.12
$1.56
71.53
86.35
71.64
67.23
61.30
57.85
54.00
$4.40
75
IKNa
85.05
85.31
85.31
79.37
85.08
83.01
55.15
73.67
K3.33
K3.66
70.99
APPNP
83.62
84-95
$4.00 96
75.15
84.51
84.97
62.00
76.51
$3.00
K3.83
74.90
MuNa
75.30
77.76
77.76
62.08
77.67
74.53
47.68
66.24
58.18
70,63
59.67
MLP
46.02
53.76
53.76
36.21
50.85
55.11
42,45
36.25
47.59
55.03
46.74
Co Traiting
GCN
SAGE
86.04
63.93
DOM
DOM
00M
OUM
0OM
OOM
DOM
DOM
00M
00M
OUM
00M
00M
00M
00M
OOM
OOM
00M
DOM
DOM
TMLM
58.32
60.29
62.42
61.55
63:45
61.30
61.74
61.99
52.41
63.12
63:45
TDK
58.18
60-42
60.38
63.13
62.95
63.80
61.97
60.96
62.49
62.92
62.95
TCL
58.90
61.04
66..0
67.02
64.93
67 94
20.08
61.50 61.50
63.34
64.55
64.95
TMDC
58.86
61.17
66. 94
68.74
69.45
67. 50
69.71
65.00
55.74
65.79
66.08
Methods
EleCompuitees (FI Score)
Small Scale BLECTRA
Base Scale
Large Scale
BERT Tay
DGMIBORT
BLECTRA
BCRT
R5OURIIS
DEBERG
ELECTRA
BERT
RoBIKI6
DRILKD
LMs
PLM
44.55
47.65
51.47
48.77
52.55
53.39
52.10
48,45
51.57
55.28
55.04
GCN
80.85
75.75
K2.24
79.31
81.03
82.15
72.57
78.10
50.93
52.85
76.56
GAT
83.49
83.52
$4.14
$3.35
83.99
83.14
74.30
83.33
$3.55
53.43
83.05
SAGE
82,85
81.35
$4.21
$2.18
83.67
84.43
76.42
$1.81
$2.90
$3.13
80.95
RevGAT
83.05
83.76
$4.55
83.56
$4.45
$4.69
79.02
R3.46
$3.09
$7.79
83.51
NFumer
80.57
75.12
50.26
77.56
80.01
81.16
71.03
76.56
70.65
$1.51
75.55
GEN
71.55
57.99
76.46
67.39
77.00
75.09
61.03
64.63
75.43
17.57
62.25
JKNet
77.01
67.90
73.99
67.53
75.02
70.26
41.49
03.55
72.35
69.78
58,30
APENP
75.49
62.04
75.00
09.04
76.99
73.46
42.08
08.22
77.79
72.96
$9.95
MoNer
43.55
35.22
$2.09
40.03
49.35
53.60
43.85
39.85
15.05
54.12
48.55
MLP
32.20
16.25
40.06
21.00
37.41
41.22
26,21
19.85
$4.25
42.34
32.75
Co-Triinigg
GCN
SAGE
83,67
80.70
0OM
0OM
00M
00M
OOM
0OM
0OM
0OM
0OM
00M
O0M
00M
00M
00M
OOM
00M
OOM
00M
COM
DOM
TMLM
44.91
48.01
51.51
$2.01
52.51
$3.00
$3.86
55.47
55.71
55.81
57.56
TDK
44,50
48.45
$2.97
55,80
55.15
56.53
99.39
30.44
55.53
55.94
54.05
TPT
TCL
44.45
48.81
54.66
55.44
59.25
60.58
$7.69
50.22
54,48
$7.86
55.67
TMDC
45.05
49.19
54.91
55.93
59.12
61.35
59.39
58.10
58.44
50.52
60.05
D.2 Experimental Results for Link Prediction
We further show the experiment results for link prediction in Table TO We mainly follow the code used for link prediction in OGB. We choose GCN and GraphS as the backbone model in the link prediction experiments.
Table 10: Link Prediction results on Citation V8 and GoodReads.
ChationVa Hin
GoodReads
Methods
Hazlo
Hineo
MRR
Hix
BERT Tly
33.50 +
48.15 + 2.02
06.56 = 0.55
41.23 2 1.39
36.86 : 2.04
52.45 A: 1.09
26.23 2.11
22865
PLM
BEXT-Rose
3886 + 2.53
57.53
1.96
72.44
01440000 0.98
44.56 + 1.25
43.96 + 2.26
60.57 + 1.43
79.22 e 0.45
44.43 1.15
T-GUN
3.56
74.36 2.16
90.23 - 0.89
60.79 0.78
61.47 4665
84 2.15
0.66
0.56
TSAGE
3:222
66.23 à:
39.36 0.99
54.61
1.03
64.52 3.18
82.65 1.45
0.65
74.36 0.84
GNN
B-GUN
15.12222.5
2..1
60.04
55.42
85.04 2.34
91.49 1.25
65.12 0.48
B-SAGF
71.26
0.712:0075
1.34
$4.05
82.37
0..35
1.13
BLXU Tiny
41.26
57.26 4: 1.59
72.62 0:96
4 1:23
45.47 +
61.56
0.49
034
BEXT-Rose
46.58
65.77
72.46 0.56
52.78 + 2.03
52.59
2.27
65.07 1.23
0.334
61.21 1.23
TCGCN
6
81.23 +: 1.36
92.56 0.56
63.89
0.42
08 à: 2.29
88
0.56
93.12
83 1.05
TSSAGE
61.89 3:21
M0.12 + 1.46
90.23 = 0.89
05.70:00.15
70.2% + 2.36
85.
1147
90.35 38 0.57
$1.12 1:25
B-OCN
68.26 x 2.66
$4.56 è 0.56
93.68 e 0.26
70.16 0.38
73,87 à 3.36
92.82 à 0.16
95.85 0.15
85.12 1.56
B SAGE
62.36 2
80.56 4: 1.56
92.56 0.68
00.15
7 2.26
00.7
0.10
$2.15 1.15
On the V8 dataset, the parameters shared by the GCN and GraphSAGE include epochs, model layers, hidden units. learning rate, and batch size, and their values are set to 200, (2, 3), [128. (56),[le-04 1e-04 -1c-03], 65536. We validate the model performance with the Hits @ K and MRR metrics. We bold the best results on each row. In Table the PLM indicates that we use LMs directly for link prediction. In GNN, we use the features generated by BERT-Tiny and BERT-Base to feed into GCN and SAGE, respectively. In TCL, we are subdivided into two approaches. The first to pre-train LMs with TCL tasks and then finetune LMs for downstream link prediction tasks. The second is to use LMs pre-trained by TCL to encode text attributes. T-GCN(SAGE) denotes the node features obtained by BERT-Tiny. B- B-GCN(SAGE) denotes the node features obtained by BERT-Base. When we use TCL to pre-train the language models, we find that BERT-Tiny and BERT-Base improve on MRR by 6.03 and 8.22, respectively. We also find that the features obtained using TCL-BERT-Bass
22

---

can significantly improve the results on downstream GNNs. For example, B-GCN improves by 10% on average on all evaluation criteria.
On the GoodReads dataset, the parameters shared by the GCN and GraphSAGE include epochs, model layers, hidden units, learning rate, and batch size, and their values are set to 200, (2), (128).( (11--04 -5e-03), 524288. We validate the model performance with the Hits K and MRR metrics. We bold the best results on each row. Observing from Table 10 we can find that after using TCL to pre-train the language models and obtaining the corresponding node features, the significant improvements are obtained in all experiments. model is also reduced in most experiments. This indicates that the node features generated by the language model trained by suitable pre-trate pre-training task can not only enhance the performance of the downstream model but also improve the stability of the downstream model to some extent.
addition the improvement in performance, the variance of the
D.3 Efficiency and Scalability of Co- Co-Triining Paradigm
Table 11: Effectiveness and Scalability of Co-Training with TCL.
BERT-Tiay
BERT-Base
Dalasets
Co-Training
TCL
Co- Co-Training
TCL
Acc
Memory
Time
Acc
Memory
Time
Acc
Memory
Time
Acc
Memory
Time
Arxiv
73.57
76.27%
44.0
71.55
27.59%
7.0
OOM
74.87
70.73%
130
Children
39.70
97.28%
15.5
54.11
19.76%
2.0
OOM
60.73
80.99%
30
History
85.09
85.74%
5.7
86.06
14.69%
1,3
OOM
86.80
98.73%
18
Photo 86.64
97.83%
73.86
22.75%
3.1
82.85
70.65%
# Averrage
76.25
89.38%
19.95
71.40
21.20%
3.4
OOM
76.31
80,28%
74,50
In this subsection, we present the accuracy, GPU memory on a single 32GB V100, and the total training time (min) of the co-training paradigm and topological PLMs across five datasets. BERT-Tny+SAGE and BERT-Base+SAGE are selected as the co- co-training approaches. As one of the best topological pre- tre-iraining tasks, topological contrastive learning (TCL) is selected to enhance the PLMS. The experimental results are presented the Table For the small language model BERT-Tiny, the co-training paradigm costs more memories (~ 4X) compared to the TCL. and is much slower (~ 6X) than TCL. If we use the larger language model like BERT-Base, co-training models will be out-of-memory in single 32GB V V100. Thus, the proposed topological pre-training paradigm more efficient and practical than the co-training ones.
D.4 Comparison between Shallow Text Encoders and PLMs
The traditional GNN pipelines generally encode the textual attributes of each node using a shallow model such as the Skip-Gram. Therefore, we extend our node text encoding approach by incorporating shallow models like Skip-Gram [29] and GloVe [30]. Experimental results (accuracy) on node classification task over -gbn-arxiv-TA and Books-Children datasets are presented in the Table [2] One can clearly see that GNNs equipped with deeper text encoders consistently outperform those with shallow encoders, verifying the importance of node attribute understanding.
Table 12: Node classification performance on shallow text encoders and PLMs. We bold the best result for each row.
Attribute Encoder
ogba arxiv-TA TA SAGE
Books Children SAGR
GCN
GAT
RevGAT
NFoomeer
GCN
GAT
RevGAT
NToormer
Skip Gram
Gkip-Gram GloNe
71.97
72.22
72.02
73.42
71.03
56.23
55.84
56.49
$7.13
56.03
Shallow
72.12
72.54
72.48
72.04
57.02
56.58
57.22
58.12
DistiBBRT
73.39
73.48
74.48
74.68
73.56
58.19
57.91
59.33
59.28
58.03
PLMs
BERT-Bac Base
73.30
73.40
74.14
74.59
72,80
58.11
57.70
58.74
58.67
57.42
RoBERTa-Base
73.56
73.38
74.52
74.82
73.12
58.62
57.83
58.97
59.01
57.26
D.5 Topological Pre-training under Semi-supervised and Few-shot Learning Scenarios
In order to provide the more comprehensive evaluation and deeper insights into our topological pre-training strategies, we evaluate them in different scenarios.
23

---

D.5.1 Semi-supervised Learning
To effectively evaluate the performance of various methods within semi-supervised settings, we've adjusted the training ratio from 20% to 100%. 20% implies that only 20% of training samples are used in the model training process. BERT Base is selected the foundational text encoder model. Detailed experimental results concerning the node classification task across four datasets have been presented in Table 13 Notably, as the training ratio decreases, the benefits of topological pre-training approaches become even more significant. Among these strategies, TMDC demonstrates superior performance across all datasets within the semi-supervised context.
Table 13: Topological pre-training methods under semi semi-supervised scenarios with different training ratios
Methods
Arxiv
History
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
PLM
57.76
62.56
67.12
70.15
72.96
70.86
75.61
80.18
83.28
86.09
TMLM
62.15
65.51
69.01
71.76
73.97
74.39
77.70
81.21
84.03
86.24
TDK
63.86
66.78
70.01
71.95
74.23
75.46
78.93
82.17
84.18
86.46
TCL
64.98
67.89
71.22
73.56
74.87
77.12
80.53
83,46
85.49
86.80
TMDC
65.48
68.91
72.92
75.09
76.11
78.24
81.48
84.23
85.80
86.82
Methods
Children
Photo
20%
40%
60%
80%
100%
20%
40%
60%
80%
100%
PLM
44.21
49.12
53,80
56.97
59.91
56.61
63.28
69.45
73.86
77.53
TMLM
48.52
51.88
55.38
58.13
60.34
66.72
70.08
73.58
76.33
78.54
TDK
49.46
52.98
56.21
58.15
60.43
70.07
73.59
76.82
78.76
81.04
TCL.
51.21
54.67
57.59
59.55
60.73
73.20
76.66
79.58
81.54
82.85
TMDC
52.88
56.17
58.91
60.41
61.43
75.54
78.83
81.57
83.07
84.09
D.5.2 Few-shot Learning
Table 14: Topological Pre-training methods under few-shot scenarios
Arxiv
Children
History
Sports
Methods
3-shot
5-shot
3-shot
5-shot
3-shot
5-shot
3-shot
5-shot
PLM
37.76
41.56
26.55
30.15
32.52
37.78
42.56
46.22
TMLM
40.08
45.21
31.86
36.89
35,51
40.15
44.18
50.15
41.15
47.42
34.56
39.26
36.58
42.26
45.69
52.68
TCL
43.26
49.58
38.26
42.12
38.95
44.12
48.58
55.64
TMDC
45.68
51.52
40.05
44.62
40.86
46.69
50.26
58.95
We've undertaken few-shot learning experiments over four datasets. The term "K-shot" denotes that merely K samples correspond each category within the training set. Based on the results in the Table [4] topological pre-training consistently enhances the performance of LMs across diverse few-shot scenarios.
D.6 Experimental Results on Large-scale Text-attributed Graphs
Considering the existence of large text-attributed graphs, we further execute preliminary experiments on the ogbn-papers 100M dataset [34 with 111,059 956 nodes and (611,685,877 edges. Given the substantial scale of this dataset, which poses challenges for many existing GNNs, we have chosen set of scalable GNNs (SGC [55) GAMLP [56] and SIGN (57). Additionally, we have selected several DistilBERT. Different training paradigms are systematically evaluated under the node classification task. Experimental results (accuracy) are demonstrated in the Table [5] GAMLP achieves the best performance among all the GNN models due to its adaptive node-wiss feature combination. Furthermore, it is noteworthy that the topological
prominent LMs including BERT-Tiny, ELECTRA,

---

pre- pre-training (referred to TPT) methods obtain the SOTA performance over this large dataset, demonstrating the superiority of the proposed pre-training tasks.
Table 15: Node classification results on Sebhn-apersleeee 100M
LMs PLM
GNNS
TPT
Models
SGC
SIGN
GAMLP
MLP
TMLM
TDK
TCL
TMDC
BERT-Timy
62.11
62.51
64.26
65.12
49.32
63.55
63.98
64.78
65.31
ELEECTA-SSaaI
61.01
61.07
63.06
64.98
47.26
62.24
63.13
63.98
65.16
DistilBERT
64.18
63.58
65.81
67.73
51.26
65.36
65.89
66.76
68.12
Table 16: Statics information of the Cresci-2015 and TwiBot-20
Datasets
User
Tweet
Edge
Human
Bot
Cresci-2015
5,301
2,827,757 33,488,192
7,086, 33,716,171
1,950
3,351
TwiBot-20
229,580
5,237
6,589
D.7 Text-attributed Graphs in Social Networks
In the realm of text-attributed graphs (TAGs), two prominent and prevalent categories are academic and e-commerce graphs. For example, all text-attributed graphs within the OGB benchmark [34] belong to these two domains. To boost the impact of our benchmark, we have broadened the spectrum of 'AAG domains by introducing new dimension, social networks, into our benchmark. To this end, we have incorporated two text-attributed graphs sourced from the widely-used social platform Twitter, named Cresci-2015 (58] and TwiBot-20 (59). These two datasets are collected for social bot detection. Each node corresponds to a user within Twitter, intrinsically linked to the tweets they have published. The underlying graph topology shaped by the relationships (e.g., following relations) among users. The labels attributed to each node signify whether the respective user is classified as bot or not. is worth noting that, owing to privacy concerns intrinsic to social network data, we provide a summarized representation of the outcomes and insights. The original datasets are available upon request through email, pending the acceptance of this paper. The detailed statisties of the datasets are presented in Table 16
Table 17: Node classification results on Cresci-2015 and TwiBot-20 dataset (accuracy)
Datasets
Models
LMs
GNNs SAGE
Co-Training SAGE
TPT
PLM
GCN
GAT
GCN
GAT
TMLM
TDK
TCL
TMDC
Tiny-Tiny
91.0
93.3
93.6
94.1
93.6
93.9
00M
91.5
92.1
93.0
94.1
ELECTRA Small
92.0
93.0
93.4
93.9
OOM
0OM
0OM
92.3
92.9
93.8
94.6
Cresci-2015
DisiiBERT
95.5
95.3
95.5
96.0
0OM
00M
00M
95.9
96.3
97.1
97.9
RoBERTa-Bee Base
97.0
96.2
96.6
96.9
0OM
00M
00M
97.5
97.9
98.3
98.8
BERT Tay
68.2
76.8
79.2
81.1
0OM
00M
00M
69.9
72.6
75.6
76.9
ELECTRA-Smin
69.5
75.6
78.8
80.1
0OM
0OM
00M
71.6
73.0
76.4
77.1
TwiBot-20
DistiBERRT
76.5
80.9
84.6
85.4
0OM
00M
00M
71.6
78.1
79.3
81.2
RoBERTs
81.6
84.9
COM
0OM
81.2
82.3
83.4
The node classification results (accuracy) are demonstrated in the Table [7] "OOM" stands for "Out- Of Of-Memory" 32GB V100. On the Cresci-2015 dataset, observe a noteworthy trend wherein PLM-based methods exhibit comparable or even superior performance compared to GNN-based methods, particularly the parameters of PLMs increase. RoBERTa consistently surpasses all three GNN models. On the TwiBot-20 dataset, the graph topology seems to be more important. The Graph Attention Network (GAT) emerges as the top-performing model on these datasets. Additionally, the employment of topological pre-training strategies brings performance enhancements for both datasets, underscoring their efficacy in advancing model capabilities.
D.8 Evaluations of LLMS on TAGs
In our benchmark, we have mainly studied PLMMs based on encoder architecture like BERT [10] However, most of the recent rapidly developing models in the NLP field are LLMs based on the
25

---

Following the experiment workflow denoted in TAPE [6,, we leverage LLMs to generate high-quality node features for TAGs. From the perspective
Decoder architecture represented by GPT [32.
of LMs, we incorporate the recent and prominent large language models as the baselines, including T3 (1IB) (6)) LaMDA (137B) [6], GPT-3 (175B) [28] and PaI.M (540B) [62]. In brief, we use I.L.Ms' inference APIs to generate explanations about the original text. These explanations are incorporated into the original text for fine-tuning the respective LMs. Finally, we extract features from LMs and use them to train the downstream GNNs. DistilBERT is selected as the feature extractor LMs, and GCN, GAT, and SAGE are selected as the downstream GNNs. Experimental results are presented in the Table 18 PaLM has consistently attained the most impressive performance across all downstream GNNs and datasets.
Table 18: Node classification experiments on the three datasets. The row "LLMs" denotes using L.LMs to generate explanations about the raw text to fine-tune the L.M and generate node features for downstream GNNs. The row "Raw" denotes using the original LMs to generate node features.
Arxiv GAT
Children GAT
Photo GAT
Methods
GCN
SAGE
GCN
SAGE
GCN
SAGE
Raw
DistiBERT
73.39
73.48
74.48
58.19
57.91
59.33
82.91
83.75
83.50
T5
73.51
73.80
74.22
58.61
58.14
59.12
83.76
84.75
84.33
LaMDA
74.06
74.55
74.92
59.67
59.56
60.86
84.23
84.95
84.59
LLMs
GPT-3
74.41
74.81
75.34
60.12
60.01
61.12
84.56
85,23
84.78
PaLM
75.22
76.43
76.72
61.59
61.26
62.23
85.45
85.95
85.69
D.9 Study on the Selection of Node Attributes
our previous experiments, we have observed phenomenon that LMs usually perform much better on datasets that use product descriptions as text attributes than those that use product reviews. In order to be more explicit about the effect node attribute selection on different models, we have reconstructed two datasets with product description as text attribute for Ele-Photo and Ele-Computers, where LMs performed poorly before. In the Table 19] datasets labeled with "RW" incorporate user reviews node attributes, while those marked "DS" employ product descriptions attributes. Different training paradigms are systematically evaluated under the node classification task. BERT- Base is selected as the foundational text encoder model. Evidently, the performance of PLMs when utilizing descriptions as node attributes demonstrates substantial enhancement in contrast to reviews. This observation underscores the pivotal role that the selection of node attributes plays in achieving desirable TAG representation learning.
Table 19: Experimental results (accuracy) of node classification datasets with different text attributes.
LMs PLM
GNNs GCN
TPT
Datasets
SAGE
GAT
TMLM
TDK
TCL
TMDC
Photo-RW
77.53
83.27
82.70
83.74
78.54
81.04
82.85
84.09
Photo-DS
85.07
84.86
83.72
85.16
86.15
86.49
87.26
88.15
Computers-RYs
61.96
88.30
87.86
88.63
63.43
82.85
64.93
69.45
Computers-DS
86.41
88.90
88.26
89.13
87.56
87.26
88.96
89.53
E
Broader Impact
Representation learning on text-attributed graphs is fast- growing and promising research field, and covers a wide range of applications. We start this benchmark to call more researchers' attention to this common data type. The proposed benchmark CS-TAG can significantly facilitate the development of the textual-attributed graph learning. CS-TAG deeply and extensively explores the paradigm of combining pre-trained language models (PLMs) with graph neural networks (GNNs), and provides comprehensive evaluation over multiple large constructed datasets. Nevertheless, there still
26

---

lots of research gaps need to be bridged. First, the sulf-supervised and unsupervised learning in eextual-attributed graphs are not included CS-TAG, which play an important role in graph data mining research. Second, CS-TAG does not pay much attention to link prediction task that has many applications in real world, such as recommendation system and drug discovery. Third, the interpretability of rextual-attributer graph learning is not discussed here. Comparing with the vectorized features in other graphs, such as chemical molecules, the text attributes can be directly understood by human. Therefore, the textual-attributed graphs are more human-intelliiible and have promising potential for interpretability research.
In the future, we will keep track on the newly emerged problems in rextull-attribuce graphs and provide more solid experimental results and detailed analyses to improve CS-TAG consistently. is an ongoing effort and we strive to continuously include more datasets and evaluate different methods advance the field.
27


Reviews:
[{"review_id": "LWya1Yewgx", "reviewer": "Reviewer_pXJE", "summary": "", "questions": "", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "1.\tThis paper presents a comprehensive and diverse collection of challenging benchmark datasets for text-attributed graphs, with the potential to inspire further research in this direction.\n2.\tThe detailed information regarding data collection and their statistics is well-explained.\n3.\tThe authors also propose several warm-up methods to explore the topology structure in LLMs. They conduct a series of experiments based on these methods to benchmark their results on the proposed datasets.\n", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "v8J90azdO9", "reviewer": "Reviewer_ghFj", "summary": "", "questions": "", "limitations": "**[Addressed the limitations and potential negative societal impact of their work]**\nNA at this stage. \n\n**[Constructive suggestions for improvement]**\n1. Many details are in appendix, which is absolutely fine, as long as the authors indicate clearly where to find additional details via coreference. \nMany of these coreferences are missing. \n    1. Mention link prediction results are in Appendix X, for instance in the paragraph before the conclusion. \n    2. Mention results on other datasets (like Table 2) are in Appendix X, l. 276-277.\n    3. Mention how TMDC is done briefly and link the details to the Appendix X, l. 320. \n\n2. Code documentation. This is not clear from the codebase how to run the benchmark, the authors need to specify scripts to call and also add a dir structure, which gives an overview of the scripts. \n\n**[Potential points to include]**\nI would like to see the discussions on efficiency and scalability (as mentioned above). ", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "**[Significance of the contribution]**\nHigh. See above in [Contributions].\n\n**[Relevance to the broader research community]**\nHigh. \n\n**[Quality of the research]**\nIn general the quality of this research is high. \n\n**[Ethical and social implications]**\nNA at this stage. ", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "7Paq6WiB94", "reviewer": "Reviewer_ZLyP", "summary": "", "questions": "", "limitations": "The findings may lack some sufficient explaination. For example, the co-training framework performs worse and the authors think \"This is because jointly optimizing GNNs and PLMs can be more difficult than training the respective models separately.\" If so, are the joint optimization well conducted and farily compared?", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed dataset are comprehensive, scalable and with raw text.\n2. The problem co-training GNN and LMs are interesting and useful.\n3. The authors have some interesting findings and may help the research field.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}, {"review_id": "f0XAqpTxZs", "reviewer": "Reviewer_fbKv", "summary": "", "questions": "", "limitations": "Yes", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "1. The proposed benchmarks are more large-scale than traditional datasets, with millions of nodes and edges.\n2. The authors have tested various LLMs to better model the semantic information.", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": ""}]

Author rebuttal:

Meta-review: All reviewers are in favor of accepting, the paper has a clear champion among the reviewers, and the rebuttal seems quite comphrehensive.
Decision: Accept (Poster)