SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 13: Topological pre-training methods under semi-supervised scenarios with different training ratios
Methods	Arxiv	History
20%	40%	60%	80%	100%	20%	40%	60%	80%	100%
PLM	57.76	62.56	67.12	70.15	72.96	70.86	75.61	80.18	83.28	86.09
TMLM	62.15	65.51	69.01	71.76	73.97	74.39	77.70	81.21	84.03	86.24
TDK	63.86	66.78	70.01	71.95	74.23	75.46	78.93	82.17	84.18	86.46
TCL	64.98	67.89	71.22	73.56	74.87	77.12	80.53	83.46	85.49	86.80
TMDC	65.48	68.91	72.92	75.09	76.11	78.24	81.48	84.23	85.80	86.82
Methods	Children	Photo
20%	40%	60%	80%	100%	20%	40%	60%	80%	100%
PLM	44.21	49.12	53.80	56.97	59.91	56.61	63.28	69.45	73.86	77.53
TMLM	48.52	51.88	55.38	58.13	60.34	66.72	70.08	73.58	76.33	78.54
TDK	49.46	52.98	56.21	58.15	60.43	70.07	73.59	76.82	78.76	81.04
TCL	51.21	54.67	57.59	59.55	60.73	73.20	76.66	79.58	81.54	82.85
TMDC	52.88	56.17	58.91	60.41	61.43	75.54	78.83	81.57	83.07	84.09
D.5.2 Few-shot Learning
Table 14: Topological Pre-training methods under few-shot scenarios
Methods	Arxiv	Children	History	Sports
3-shot	5-shot	3-shot	5-shot	3-shot	5-shot	3-shot	5-shot
PLM	37.76	41.56	26.55	30.15	32.52	37.78	42.56	46.22
TMLM	40.08	45.21	31.86	36.89	35.51	40.15	44.18	50.15
TDK	41.15	47.42	34.56	39.26	36.58	42.26	45.69	52.68
TCL	43.26	49.58	38.26	42.12	38.95	44.12	48.58	55.64
TMDC	45.68	51.52	40.05	44.62	40.86	46.69	50.26	58.95
We've undertaken few-shot learning experiments over four datasets. The term "K-shot" denotes that merely K samples correspond to each category within the training set. Based on the results in the Table 14, topological pre-training consistently enhances the performance of LMs across diverse few-shot scenarios.
D.6 Experimental Results on Large-scale Text-attributed Graphs
Considering the existence of large text-attributed graphs, we further execute preliminary experiments on the ogbn-papers100M dataset [34] with 111,059,956 nodes and 1,615,685,872 edges. Given the substantial scale of this dataset, which poses challenges for many existing GNNs, we have chosen a set of scalable GNNs (SGC [55], GAMLP [56] and SIGN [57]). Additionally, we have selected several prominent LMs including BERT-Tiny, ELECTRA, and DistilBERT. Different training paradigms are systematically evaluated under the node classification task. Experimental results (accuracy) are demonstrated in the Table 15. GAMLP achieves the best performance among all the GNN models due to its adaptive node-wise feature combination. Furthermore, it is noteworthy that the topological
pre-training (referred to as TPT) methods obtain the SOTA performance over this large dataset, demonstrating the superiority of the proposed pre-training tasks.
Table 15: Node classification results on ogbn-papers100M
Models	LMs PLM	GNNs	TPT
SGC	SIGN	GAMLP	MLP	TMLM	TDK	TCL	TMDC
BERT-Tiny ELECTRA-Small DistilBERT	62.11	62.51	64.26	65.12	49.32	63.55	63.98	64.78	65.31
61.01	61.07	63.06	64.98	47.26	62.24	63.13	63.98	65.16
64.18	63.58	65.81	67.73	51.26	65.36	65.89	66.76	68.12
Table 16: Statics information of the Cresci-2015 and TwiBot-20
Datasets	User	Tweet	Edge	Human	Bot
Cresci-2015	5,301	2,827,757	7,086,134	1,950	3,351
TwiBot-20	229,580	33,488,192	33,716,171	5,237	6,589
D.7 Text-attributed Graphs in Social Networks
In the realm of text-attributed graphs (TAGs), two prominent and prevalent categories are academic and e-commerce graphs. For example, all text-attributed graphs within the OGB benchmark [34] belong to these two domains. To boost the impact of our benchmark, we have broadened the spectrum of TAG domains by introducing a new dimension, social networks, into our benchmark. To this end, we have incorporated two text-attributed graphs sourced from the widely-used social platform Twitter, named Cresci-2015 [58] and TwiBot-20 [59]. These two datasets are collected for social bot detection. Each node corresponds to a user within Twitter, intrinsically linked to the tweets they have published. The underlying graph topology is shaped by the relationships (e.g., following relations) among users. The labels attributed to each node signify whether the respective user is classified as a bot or not. It is worth noting that, owing to privacy concerns intrinsic to social network data, we provide a summarized representation of the outcomes and insights. The original datasets are available upon request through email, pending the acceptance of this paper. The detailed statistics of the datasets are presented in Table 16.
Table 17: Node classification results on Cresci-2015 and TwiBot-20 dataset (accuracy)
Datasets	Models	LMs PLM	GCN	GNNs SAGE	GAT	Co-Training	TMLM	TDK	TCL	TMDC
GCN	SAGE	GAT
Cresci-2015	BERT-Tiny	91.0	93.3	93.6	94.1	93.6	93.9	OOM	91.5	92.1	93.0	94.3
ELECTRA-Small	92.0	93.0	93.4	93.9	OOM	OOM	OOM	92.3	92.9	93.8	94.6
DistilBERT	95.5	95.3	95.5	96.0	OOM	OOM	OOM	95.9	96.3	97.1	97.9
RoBERTa-Base	97.0	96.2	96.6	96.9	OOM	OOM	OOM	97.5	97.9	98.3	98.8
TwiBot-20	BERT-Tiny	68.2	76.8	79.2	81.1	OOM	OOM	OOM	69.9	72.6	75.6	76.9
ELECTRA-Small	69.5	75.6	78.8	80.1	OOM	OOM	OOM	71.6	73.0	76.4	77.1
DistilBERT	76.5	80.9	84.6	85.4	OOM	OOM	OOM	77.6	78.1	79.3	81.2
RoBERTa-Base	78.6	81.6	84.9	85.9	OOM	OOM	OOM	80.1	81.2	82.3	83.4
The node classification results (accuracy) are demonstrated in the Table 17. "OOM" stands for "Out-Of-Memory" on a 32GB V100. On the Cresci-2015 dataset, we observe a noteworthy trend wherein PLM-based methods exhibit comparable or even superior performance compared to GNN-based methods, particularly as the parameters of PLMs increase. RoBERTa consistently surpasses all three GNN models. On the TwiBot-20 dataset, the graph topology seems to be more important. The Graph Attention Network (GAT) emerges as the top-performing model on these datasets. Additionally, the employment of topological pre-training strategies brings performance enhancements for both datasets, underscoring their efficacy in advancing model capabilities.
D.8 Evaluations of LLMs on TAGs
In our benchmark, we have mainly studied PLMs based on encoder architecture like BERT [10]. However, most of the recent rapidly developing models in the NLP field are LLMs based on the
Decoder architecture represented by GPT [32]. Following the experiment workflow denoted in TAPE [6], we leverage LLMs to generate high-quality node features for TAGs. From the perspective of LMs, we incorporate the recent and prominent large language models as the baselines, including T5 (11B) [60], LaMDA (137B) [61], GPT-3 (175B) [28] and PaLM (540B) [62]. In brief, we use LLMs' inference APIs to generate explanations about the original text. These explanations are incorporated into the original text for fine-tuning the respective LMs. Finally, we extract features from LMs and use them to train the downstream GNNs. DistilBERT is selected as the feature extractor LMs, and GCN, GAT, and SAGE are selected as the downstream GNNs. Experimental results are presented in the Table 18. PaLM has consistently attained the most impressive performance across all downstream GNNs and datasets.
Table 18: Node classification experiments on the three datasets. The row "LLMs" denotes using LLMs to generate explanations about the raw text to fine-tune the LM and generate node features for downstream GNNs. The row "Raw" denotes using the original LMs to generate node features.
Methods	GCN	Arxiv GAT	SAGE	GCN	Children GAT	SAGE	GCN	Photo GAT	SAGE
Raw	DistilBERT	73.39	73.48	74.48	58.19	57.91	59.33	82.91	83.75	83.50
LLMs	T5	73.51	73.80	74.22	58.61	58.14	59.12	83.76	84.75	84.33
LaMDA	74.06	74.55	74.92	59.67	59.56	60.86	84.23	84.95	84.59
GPT-3	74.41	74.81	75.34	60.12	60.01	61.12	84.56	85.23	84.78
PaLM	75.22	76.43	76.72	61.59	61.26	62.23	85.45	85.95	85.69
D.9 Study on the Selection of Node Attributes
In our previous experiments, we have observed a phenomenon that LMs usually perform much better on datasets that use product descriptions as text attributes than those that use product reviews. In order to be more explicit about the effect of node attribute selection on different models, we have reconstructed two datasets with product description as text attribute for Ele-Photo and Ele-Computers, where LMs performed poorly before. In the Table 19, datasets labeled with "RW" incorporate user reviews as node attributes, while those marked "DS" employ product descriptions as attributes. Different training paradigms are systematically evaluated under the node classification task. BERT-Base is selected as the foundational text encoder model. Evidently, the performance of PLMs when utilizing descriptions as node attributes demonstrates a substantial enhancement in contrast to reviews. This observation underscores the pivotal role that the selection of node attributes plays in achieving desirable TAG representation learning.
Table 19: Experimental results (accuracy) of node classification on datasets with different text attributes.
Datasets	LMs	GNNs	TPT
PLM	SAGE	GCN	GAT	TMLM	TDK	TCL	TMDC
Photo-RW	77.53	83.27	82.70	83.74	78.54	81.04	82.85	84.09
Photo-DS	85.07	84.86	83.72	85.16	86.15	86.49	87.26	88.15
Computers-RW	61.96	88.30	87.86	88.63	63.43	82.85	64.93	69.45
Computers-DS	86.41	88.90	88.26	89.13	87.56	87.26	88.96	89.53
E Broader Impact
Representation learning on text-attributed graphs is a fast-growing and promising research field, and covers a wide range of applications. We start this benchmark to call more researchers' attention to this common data type. The proposed benchmark CS-TAG can significantly facilitate the development of the textual-attributed graph learning. CS-TAG deeply and extensively explores the paradigm of combining pre-trained language models (PLMs) with graph neural networks (GNNs), and provides a comprehensive evaluation over multiple large constructed datasets. Nevertheless, there are still
lots of research gaps need to be bridged. First, the self-supervised and unsupervised learning in textual-attributed graphs are not included in CS-TAG, which play an important role in graph data mining research. Second, CS-TAG does not pay much attention to link prediction task that has many applications in real world, such as recommendation system and drug discovery. Third, the interpretability of textual-attributed graph learning is not discussed here. Comparing with the vectorized features in other graphs, such as chemical molecules, the text attributes can be directly understood by human. Therefore, the textual-attributed graphs are more human-intelligible and have promising potential for interpretability research.
In the future, we will keep track on the newly emerged problems in textual-attributed graphs and provide more solid experimental results and detailed analyses to improve CS-TAG consistently. It is an ongoing effort and we strive to continuously include more datasets and evaluate different methods to advance the field.