SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Hao Yan $^{1*}$ , Chaozhuo Li $^{2*}$ , Ruosong Long $^{3}$ , Chao Yan $^{4}$ , Jianan Zhao $^{6}$ , Wenwen Zhuang $^{1}$ , Jun Yin $^{1}$ , Peiyan Zhang $^{5}$ , Weihao Han $^{7}$ , Hao Sun $^{7}$ , Weiwei Deng $^{7}$ , Qi Zhang $^{7}$ , Lichao Sun $^{8}$ , Xing Xie $^{2}$ , Senzhang Wang $^{1\dagger}$
<sup>1</sup>Central South University, <sup>2</sup>Microsoft Research Asia, <sup>3</sup>University of Birmingham
$^{4}$ Peking University,  $^{5}$ Hong Kong University of Science and Technology
<sup>6</sup>Université de Montréal, <sup>7</sup>Microsoft, <sup>8</sup>Lehigh University
Text-attributed graphs (TAGs) are prevalent in various real-world scenarios, where each node is associated with a text description. The cornerstone of representation learning on TAGs lies in the seamless integration of textual semantics within individual nodes and the topological connections across nodes. Recent advancements in pre-trained language models (PLMs) and graph neural networks (GNNs) have facilitated effective learning on TAGs, garnering increased research interest. However, the absence of meaningful benchmark datasets and standardized evaluation procedures for TAGs has impeded progress in this field. In this paper, we propose CS-TAG, a comprehensive and diverse collection of challenging benchmark datasets for TAGs. The CS-TAG datasets are notably large in scale and encompass a wide range of domains, spanning from citation networks to purchase graphs. In addition to building the datasets, we conduct extensive benchmark experiments over CS-TAG with various learning paradigms, including PLMs, GNNs, PLM-GNN co-training methods, and the proposed novel topological pre-training of language models. In a nutshell, we provide an overview of the CS-TAG datasets, standardized evaluation procedures, and present baseline experiments. The entire CS-TAG project is publicly accessible at https://github.com/sktsherlock/TAG-Benchmark.
Graphs are ubiquitous in modeling the relational and structural aspects of real-world objects across various domains, such as social networks, transportation system networks, and biological protein-protein networks [1, 2, 3, 4]. In many real-world graphs, nodes are often associated with text attributes, giving rise to the text-attributed graphs (TAGs) [5, 6]. TAGs are prevalent in various scenarios, such as social graphs where each user is accompanied by a textual description and paper citation graphs where textual content is linked to each respective paper [7, 8]. The exploration of learning methodologies applied to TAGs has emerged as a prominent research area within multiple fields, including graph learning, information retrieval, and natural language processing [9].
The nucleus of learning on TAGs lies in the effective integration of both the node attributes (textual semantics) and graph topology (structural connections) to facilitate the learning of node representations. The textual information associated with each node offers a wealth of semantic content, enabling the characterization of individual node properties, which could be captured by the pretrained language models (PLMs) [10, 11, 12, 13, 14]. Meanwhile, the structural information encoded
*Equal contribution and co-first authors.
Figure 1: The traditional text attributed graph representation learning pipeline.
within the graph topology presents the inherent proximity relationships between nodes. Graph neural networks (GNNs) have proven to be effective in capturing such structural relations based on the message-passing mechanism [7, 15, 16, 17, 18, 19, 20, 21, 22].
PLM-based and GNN-based methods are two prevalent types of learning paradigms on TAGs as illustrated in Figure 1. PLM-based methods generally input the textual content derived from the target node into a pre-trained language model. However, the knowledge of topology resulting from the high non-linearity of graph structure within TAGs is largely discarded by the PLM-based methods [9]. Conversely, GNN-based methods are capable of preserving the intricate graph topology information with greater fidelity. Nevertheless, an inherent limitation plaguing GNN-based methods lies in the disconnected modeling process of node attributes and graph topology. Specifically, most GNNs pre-model node attributes as static representations, treating them as fixed and unlearnable parameters during the message passing process. Consequently, the gradients stemming from the learning objective of GNNs cannot be effectively back-propagated into the attribute modeling. This discrepancy in the training procedure hinders the attainment of an optimal solution, as it fails to guarantee end-to-end training, thereby impeding the overall effectiveness of the approach.
To simultaneously enjoy the merits of GNNs and LMs, several recent endeavors shed light on the co-training paradigm as shown in Figure 1. LMs and GNNs are combined in a cascaded [23, 24, 25] or nested [5] manner, establishing a unified end-to-end training paradigm to model the node attributes and graph topology jointly. Despite its theoretical appeal, the co-training method suffers from severe scalability issues as its memory complexity is proportional to the graph size as neighborhood texts are also encoded [9]. Motivated by the recent advancements in pre-training techniques, a novel inquiry emerges: Can we pre-train the language models to understand the graph topology? If we can effectively encode topological information into LMs through appropriate pre-training tasks, LMs could serve as the foundational model for learning on TAGs. Topological pre-trained LMs eliminate the explicit GNN aggregations, thereby circumventing the efficiency challenges encountered in the co-training paradigm. However, the design of suitable and effective pre-training tasks to encode valuable knowledge derived from intricate graph topology into LMs remains an open question.
In order to delve deeply into the intricate interplay between textual semantics and graph topology within TAGs, we embark on an unprecedented exploration to investigate the optimal training paradigm for various TAGs. Existing text-attributed graph datasets (e.g., Cora [8], WikiCS [26], Amazon Photo [27]) cannot meet our requirements, as they solely offer node attribute embeddings, devoid of the original textual sentences. To overcome this limitation, we meticulously curate a novel and comprehensive dataset, dubbed CS-TAG, comprising eight distinct TAGs sourced from diverse domains. This carefully crafted dataset serves as a solid foundation for future research endeavors, facilitating in-depth investigations in this burgeoning field. Moreover, extensive experiments are conducted on the CS-TAG dataset to provide comprehensive and reliable benchmarks. All aforementioned learning paradigms are thoroughly evaluated and analyzed. Experimental results and detailed discussions further reveal the underlying correlations between graph topology and textual attributes, drawing deep insight into the inherent characteristics of the TAGs. Our contributions are summarized as follows:
1. To the best of our knowledge, CS-TAG is the first open dataset specifically designed for text-attributed graphs. TAGs from a variety of fields are collected, cleaned, and organized as the final structured dataset. We provide researchers with original links and data cleaning codes to facilitate their access and reprocessing of these datasets in accordance with their research interests and requirements. The entire CS-TAG project is publicly accessible as an open source repository on Github, accessible at https://github.com/sktsherlock/TAG-Benchmark.
2. In contrast to previous topology-driven graph learning models, our work underscores the vital significance of deep node attribute modeling. This novel perspective sheds light on the design of next-generation GNNs by emphasizing the incorporation of deep node attribute understanding.
3. We investigate the novel problem of topological pre-training of language models, aiming at teaching LMs to understand topological structures. This innovative training paradigm exhibits remarkable performance on the CS-TAG dataset in terms of effectiveness and efficiency, which contributes to broadening the scope of language model pre-training.
4. Extensive experiments are conducted across eight diverse datasets, focusing on two downstream tasks: node classification and link prediction. Such experiments serve as a rigorous evaluation of various learning paradigms, providing precise and dependable benchmarks for future endeavors.
In this section, we first briefly introduce three popular learning paradigms for TAGs. After that, the comparisons between the existing graph learning benchmarks and the proposed CS-TAG are also discussed. Refer to Appendix A for more detailed reviews of the related models. We have implemented most of the algorithms discussed in this section in the repository.
PLM-based methods. The PLMs refer to universal language models that possess enhanced semantic understanding due to their pre-training on a vast corpus [28]. The early works on modeling textual attributes were based on shallow networks, e.g., Skip-Gram[29] and GloVe[30]. In recent years, the backbone networks dominated by the pre-training-fine-tuning paradigm are rapidly scaling up: from ELMo[31], GPT[32], to BERT [10], RoBERTa [12], DeBERTa [13]. The large-scale models, which get fully trained with massive data, demonstrate superior performances on general NLP tasks. One of the most critical usages of PLMs is text representation, where the underlying semantics of texts are captured by low-dimensional embeddings. On the TAGs, the PLMs use the local textual information of each node to learn a good representation for the downstream task [9].
GNN-based methods. As graph representation learning enjoys explosive growth in machine learning, numerous research works have been proposed for various tasks including node classification [15], link prediction [21], and so on. Graph neural networks are recognized as powerful tools for modeling graph data. Such methods (e.g., GCN [15], GAT [16], GraphSAGE [7], GIN [17], RevGAT [33]) learn effective message-passing mechanisms such that information between the nodes can get aggregated for expressive graph representations. GNNs generally adopt the "castle architecture" suggested by GraphSAGE for textual graph representation: node features are encoded independently using text modeling tools (e.g. PLMs) and subsequently aggregated by GNNs to produce the final representation.
Co-training methods. The aforementioned two types of paradigms primarily focus on modeling partial information, which limits their ability to learn comprehensive features. Several recent endeavors propose to co-train GNNs and LMs to enjoy the merits from both sides. Specifically, LMs and GNNs are combined in the cascaded [24] or nested manner [5]. The outputs generated by LMs serve as inputs for GNNs, and vice versa. The parameters of both LM and GNN are updated through the back-propagation of gradients from downstream tasks. However, this co-training paradigm suffers from serious scalability problems, as all neighbors need to be encoded from scratch by the LMs, incurring significant additional computational costs [9].
Benchmarks for graph representation learning. Several established graph benchmarks have been developed and widely adopted [34, 35, 27, 36]. However, when it comes to learning on TAGs, these benchmarks exhibit notable deficiencies. Firstly, a majority of these datasets suffer from the absence of raw textual information, limiting the investigation of attribute modeling's effectiveness. Secondly, these datasets often neglect to explore the impact of text attribute modeling on GNNs. Thirdly, these datasets are predominantly small in scale. Thus, there is a compelling necessity to construct a comprehensive large-scale dataset for TAGs.
3 CS-TAG: A Comprehensive Dataset and Benchmark for TAGs
In this section, we commence by providing a concise summary of the constructed CS-TAG benchmark in Section 3.1. Subsequently, we present the details of the construction of CS-TAG in Section 3.2, including data collection, cleaning, and labeling. Moreover, we elucidate the details of GNN-based, PLM-based, and Co-training learning paradigms in Section 3.3. Finally, the proposed topological pre-training of LMs is presented in Section 3.4.
3.1 Overview of CS-TAG
In order to address the limitations inherent in prior researches, we propose the establishment of the text-attributed graph benchmark, dubbed CS-TAG, which serves as a standardized evaluation framework for assessing the efficacy of representation learning techniques on TAGs. To ensure scalability, CS-TAG includes datasets of varying sizes and incorporates scalable baselines consisting of PLMs, GNNs, and co-training methods. This enables researchers to evaluate the performance of their models across a broad range of dataset scales. To enhance usability, we provide a modular pipeline that simplifies the implementation of different models within the CS-TAG. Such a modular architecture enables researchers to easily integrate their novel methods and compare them with existing approaches. In addition, we are committed to maintaining a public leaderboard for TAGs, serving as a repository for the latest advancements in the field. This platform will continuously update text-attributed graph datasets that possess practical and research value, fostering ongoing progress and collaboration within the community. Overall, CS-TAG serves as a scalable, unified, modular, and consistently updated evaluation framework for assessing the performance of representation learning methods on text-attributed graphs.
3.2 Dataset Construction
In order to thoroughly investigate the performance of different learning paradigms on TAGs, we conduct an extensive survey of various text-attributed graph datasets that have been previously utilized in the literature. Our observations reveal that many commonly employed node-level datasets are essentially text-attributed graphs. For instance, well-known citation graphs such as Cora, PubMed, CiteSeer [8], and ogbn-arxiv [34] are all TAGs. These datasets derive node attributes from textual information, such as the title and abstract of papers. Additionally, academic collaboration networks such as Coauthor CS/Physics [27] set node attributes derived from keywords defined in the papers.
However, while these datasets are frequently employed by GNNs, they possess obvious inadequacies when exploring representation learning
Figure 2: The differences between the TAGs datasets in CS-TAG (used for node classification) and the previous datasets.
on TAGs. Firstly, most of these datasets lack the availability of raw textual information, bringing challenges to investigating the effectiveness of attribute modeling on these datasets. Secondly, these datasets generally overlook the exploration of text attribute modeling's impact on GNNs. A majority of these datasets employ simplistic bag-of-words models or traditional text encoding techniques like GloVe or Skip-Gram to represent text attributes, which are kind of outdated. Lastly, these datasets are predominantly small in scale, leading to a lack of differentiation between different learning models across numerous datasets.
To address these limitations, we have taken proactive steps to collect and construct some novel datasets of TAGs. Figure 2 illustrates the number of nodes/edges in the previous datasets and the proposed CS-TAG. One can clearly see that the TAGs within CS-TAG are comparatively larger than the counterparts. Here, we present the details of shopping graphs as an example. We extract datasets from the Amazon dataset [37], including Books-Children/History, Ele-Computers/Photo, and Sports-Fitness. Nodes represent different types of items, while edges indicate items that are frequently purchased or browsed together. Node labels are assigned based on the product category. To explore the influence of attributes in the text-attribute graphs, distinct text attributes have been provided for each of these datasets. For example, in the Books-Children/History dataset, node attributes are derived from the title and description of the respective books, such as "Description: Collection of Poetry; Title: The golden treasury of poetry". The Sports-Fitness dataset only incorporates node attributes from the title of the sports items, such as "Girls ballet Tutu Neon Orange". In the Ele-Computers/Photo dataset, node attributes are obtained from high-rated reviews and product summaries, for instance,
Table 1: Statistics of text-attributed graph dataset used in CS-TAG.
Dataset	Nodes	Edges	Class	Domain	Modeling	Scale	Tasks	Raw Text
Previous	WikiCS	11,701	216,123	10	Wikipedia	GloVe	Medium	Node classification	✘
Cora	2,708	5,429	7	Academic	Bag of words	Small	Node classification	✘
Citeseer	3,327	4,732	6	Academic	Bag of words	Small	Node classification	✘
Pubmed	19,717	44,338	3	Academic	Bag of words	Medium	Node classification	✘
ogbn-arxiv	169,343	1,166,243	40	Academic	Skip-Gram	Large	Node classification	✔
Coauthor CS	18,333	81,894	15	Academic	Bag of words	Medium	Node classification	✘
Coauthor Physics	34,493	247,962	5	Academic	Bag of words	Medium	Node classification	✘
Amazon Photo	7,487	119,043	10	E-commerce	Bag of words	Small	Node classification	✘
Amazon Computers	13,381	245,778	8	E-commerce	Bag of words	Medium	Node classification	✘
Ours	ogbn-arxiv-TA	169,343	1,166,243	40	Academic	PLMs	Large	Node classification	✔
Books-Children	76,875	1,554,578	24	E-commerce	PLMs	Large	Node classification	✔
Books-History	41,551	358,574	12	E-commerce	PLMs	Large	Node classification	✔
Ele-Computers	87,229	721,081	10	E-commerce	PLMs	Large	Node classification	✔
Ele-Photo	48,362	500,928	12	E-commerce	PLMs	Large	Node classification	✔
Sports-Fitness	173,055	1,773,500	13	E-commerce	PLMs	Large	Node classification	✔
CitationV8	1,106,759	6,120,897	-	Academic	PLMs	Large	Link Prediction	✔
GoodReads	676,084	8,582,324	-	E-commerce	PLMs	Large	Link Prediction	✔
"Great camera for the price! This camera takes crystal clear photos and is cheap too!". Further details on the dataset construction process can be found in Appendix B.
Additionally, we construct two other datasets, CitationV8 and GoodReads, for the link prediction task. The CitationV8 dataset represents a citation network extracted from DBLP [38]. Node attributes in CitationV8 are derived from the titles and abstracts of research papers. Each edge signifies a citation relationship between two papers. The GoodReads dataset, on the other hand, originates from a prominent book review website. This dataset captures the "similar item" linking relationship between books and provides valuable information about the attributes of each book, such as the title and description. Therefore, we leverage the GoodReads dataset to construct link prediction tasks, which involve predicting relationships between similar books. Detailed descriptions of all the aforementioned datasets can be found in Appendix B.
3.3 Conventional Learning Paradigms on TAGs
Existing learning paradigms on TAGs can be broadly classified into three distinct categories: 1) GNN-based methods: These methods primarily leverage GNNs as the foundational model for capturing the underlying graph topology structures through message-passing mechanisms. 2) PLM-based methods: These approaches rely on prevalent pre-trained language models to capture the semantics from the textual node attributes, which excel in their ability to comprehend text semantics and exhibit strong transferability. 3) Co-training methods: This paradigm involves the joint learning of GNNs and LMs under a unified framework [23] to enjoy the merits from both sides. Next, we will give the formulaic definitions of these three paradigms.
GNN-based Paradigm. GNNs are employed to propagate information across the graph nodes, allowing for the extraction of meaningful representations via message passing, which are formally defined as follows:
$$
\boldsymbol {h} _ {u} ^ {(k + 1)} = \operatorname {U P D A T E} _ {\omega} ^ {(k)} \left(\boldsymbol {h} _ {u} ^ {(k)}, \operatorname {A G G R E G A T E} _ {\omega} ^ {(k)} \left(\left\{\boldsymbol {h} _ {v} ^ {(k)}, v \in \mathcal {N} (u) \right\}\right)\right) \tag {1}
$$
where  $k$  represents the layers of GNNs,  $\mathcal{N}$  denotes the set of neighbors,  $u$  denotes the target node,  $\omega$  means the learning parameters in GNNs. Please note that, the initial node feature vector  $h_u^{(0)}$  using pre-learned by PLMs or other shallow text encoder (e.g., Skip-Gram). Such attribute modeling phase is performed independently of the subsequent training of GNNs. Gradients from the GNN training objectives are unable to be back-propagated into the PLMs to update their parameters. And this decoupling of PLMs and GNNs impedes the overall effectiveness.
PLM-based Paradigm. PLM-based methods leverage the effectiveness of pre-training techniques to enhance the modeling of text within each node. The formulation of these methods is as follows:
$$
\boldsymbol {h} _ {u} ^ {(k + 1)} = \operatorname {U P D A T E} _ {\boldsymbol {\psi}} ^ {(k)} \left(\boldsymbol {h} _ {u} ^ {(k)}\right) \tag {2}
$$
where  $\psi$  denotes the learnable parameters in PLMs. PLMs advance the modeling of node text attributes. However, incorporating crucial topological context into PLM-based paradigms remains a challenge, particularly when the available textual data is limited.
<sup>3</sup>https://www.goodreads.com/
Figure 3: Illustrations of different topological pre-training methods.
Co-training Paradigm. GNNs and LMs are jointly trained under a unified training framework:
$$
f _ {\Theta} (A, T) = \operatorname {G N N} _ {\omega} (A, \operatorname {P L M} _ {\psi} (T)), \Theta = \{\omega , \psi \} \tag {3}
$$
where  $f$  denotes the learning function and  $\Theta$  denotes the entire learnable parameters, which are derived from both the GNN and PLM modules. The outputs generated by the LM serve as input to the GNN. Notably, the gradients obtained from the GNN can be back-propagated to the LM, enabling the update of its parameters. However, the co-training method faces significant scalability challenges. This is primarily due to the memory complexity associated with encoding neighborhood texts, resulting in a memory requirement that scales linearly with the size of the graph [9].
3.4 Topological Pre-training of Language Models
The incorporation of explicit GNN aggregations in the co-training paradigm introduces inherent challenges in terms of training complexity and resource requirements. This is primarily due to the simultaneous modeling of texts from both the center node and its neighbors. Therefore, this brings us to a question: is there a training paradigm to enjoy the merits of graph topology while avoiding the explicit GNN operations? Inspired by the recent advancements in pre-training techniques [39, 40], our motivation lies in teaching language models to understand the topological structures. Three topological pre-training tasks are proposed to impart graph structures into the LMs, enabling them to better comprehend and capture the underlying topology.
Topological Masked Language Model (TMLM). Inspired by the task of masked language modeling, we propose a novel topological masked language model (TMLM) to capture the first-order connections on the token level. Given a center node  $c$  and one of its neighbors  $n$ , their corresponding text is formally defined as  $T^{(c)} = \{t_1^{(c)}, t_2^{(c)}, \ldots, t_k^{(c)}\}$  and  $T^{(n)} = \{t_1^{(n)}, t_2^{(n)}, \ldots, t_u^{(n)}\}$ , respectively. We randomly replace a subset of tokens in the center node  $T^{(c)}$  and  $T^{(n)}$  with a special token [MASK]. The objective of TMLM is to predict the masked tokens. Let  $\Phi^{(s)} = \{\phi_1^{(c)}, \phi_2^{(c)}, \ldots, \phi_{m-1}^{(n)}, \phi_m^{(n)}\}$  represents the indexes of the  $m$  masked tokens in the sentence  $T^{(c)}$  and  $T^{(n)}$ . Let  $T_{\Phi}^{(s)}$  denote the set of masked tokens in  $T^{(c)}$  and  $T^{(n)}$ , and  $T_{-\Phi}^{(s)}$  denote the set of observed (unmasked) tokens. The objective of TMLM is: