SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhixun Li $^{1}$  Liang Wang $^{2,3}$  Xin Sun $^{4}$  Yifan Luo $^{5}$  Yanqiao Zhu $^{6}$  Dingshuo Chen $^{2,3}$  Yingtao Luo $^{7}$  Xiangxin Zhou $^{2,3}$  Qiang Liu $^{2,3*}$  Shu Wu $^{2,3}$  Liang Wang $^{2,3,4}$  Jeffrey Xu Yu $^{1*}$ $^{1}$ Department of Systems Engineering and Engineering Management The Chinese University of Hong Kong  
 $^{2}$ Center for Research on Intelligent Perception and Computing State Key Laboratory of Multimodal Artificial Intelligence Systems Institute of Automation, Chinese Academy of Sciences  
 $^{3}$ School of Artificial Intelligence, University of Chinese Academy of Sciences  
 $^{4}$ Department of Automation, University of Science and Technology of China  
 $^{5}$ School of Cyberspace Security, Beijing University of Posts and Telecommunications  
 $^{6}$ Department of Computer Science, University of California, Los Angeles  
 $^{7}$ Heinz College of Information Systems and Public Policy, Machine Learning Department School of Computer Science, Carnegie Mellon University  
Primary contact: zxli@se.cuhk.edu.hk
Graph Structure Learning (GSL) has recently garnered considerable attention due to its ability to optimize both the parameters of Graph Neural Networks (GNNs) and the computation graph structure simultaneously. Despite the proliferation of GSL methods developed in recent years, there is no standard experimental setting or fair comparison for performance evaluation, which creates a great obstacle to understanding the progress in this field. To fill this gap, we systematically analyze the performance of GSL in different scenarios and develop a comprehensive Graph Structure Learning Benchmark (GSLB) curated from 20 diverse graph datasets and 16 distinct GSL algorithms. Specifically, GSLB systematically investigates the characteristics of GSL in terms of three dimensions: effectiveness, robustness, and complexity. We comprehensively evaluate state-of-the-art GSL algorithms in node-and graph-level tasks, and analyze their performance in robust learning and model complexity. Further, to facilitate reproducible research, we have developed an easy-to-use library for training, evaluating, and visualizing different GSL methods. Empirical results of our extensive experiments demonstrate the ability of GSL and reveal its potential benefits on various downstream tasks, offering insights and opportunities for future research. The code of GSLB is available at: https://github.com/GSL-Benchmark/GSLB.
Graphs, structures made of vertices and edges, are ubiquitous in real-world applications. A wide variety of applications spanning social network [51, 9], molecular property prediction [40, 14], fake news detection [45, 1], and fraud detection [23, 27] have found graphs instrumental in modeling complex systems. In recent years, Graph Neural Networks (GNNs) have attracted increasing attention due to their powerful ability to learn node or graph representations. However, most of the GNNs heavily rely on the assumption that the initial structure of the graph is trustworthy enough to serve as ground-truth for training. Due to uncertainty and complexity in data collection, graph structures are inevitably redundant, biased, noisy, incomplete, or the original graph structures are even unavailable, which will bring great challenges for the deployment of GNNs in real-world applications.
*Corresponding authors: Qiang Liu (qiang.liu@nlpr.ia.ac.cn), Jeffrey Xu Yu (yu@se.cuhk.edu.hk)
Table 1: An overview of GSLB. Both algorithms and datasets are divided into three categories: homogeneous node-level, heterogeneous node-level, and graph-level. The evaluation is divided into three dimensions: effectiveness, robustness, and complexity.
Algorithms
Homogeneous GSL	LDS [11], GRCN [48], ProGNN [18], IDGL [4], CoGSL [26], SUBLIME [28], GEN [39], STABLE [21], NodeFormer [43], SLAPS [10], GSR [54], HES-GSL [42]
Heterogeneous GSL	GTN [49], HGSL [53]
Graph-level GSL	HGP-SL [52], VIB-GSL [36]
Datasets
Homogeneous datasets	Cora [47], CiteSeer [47], Pubmed [47], ogbn-arxiv [15], Polblogs, Cornell [34], Texas [34], Wisconsin [34], Actor [37]
Heterogeneous datasets	ACM [49], DBLP [49], Yelp [29]
Graph-level datasets	IMDB-B [3], IMDB-M [3], COLLAB [46], REDDIT-B [46], MUTAG [5], PROTEINS [2], Peptides-Func [8], Peptides-Struct [8]
Evaluations
Effectiveness	Homogeneous node classification (Topology Refinement/Topology Inference), Heterogeneous node classification, Graph-level tasks
Robustness	Supervision signal robustness, Structure robustness, Feature robustness
Complexity	Time complexity, Space complexity
To mitigate the aforementioned problems, Graph Structure Learning (GSL) [4, 55, 30, 10, 57, 50] has become an important theme in graph learning. GSL aims to make the computation structure of GNNs more suitable for downstream tasks and improve the quality of the learned representations. While it is widespread in different communities and the research enthusiasm for GSL is increasing, there is no standardized benchmark that could offer a fair and consistent comparison of different GSL algorithms. Moreover, due to the complexity and diversity of graph datasets, the experimental setups in existing work are not consistent, such as varying ratios of the training set and different train/validation/test splits. This poses a great obstacle to a holistic understanding of the current research status. Therefore, the development of a standardized and comprehensive benchmark for GSL is an urgent need within the community.
In this work, we propose Graph Structure Learning Benchmark (GSLB), which serves as the first comprehensive benchmark for GSL. Our benchmark encompasses 16 state-of-the-art GSL algorithms and 20 diverse graph datasets covering homogeneous node-level, heterogeneous node-level, and graph-level tasks. We systematically investigate the characteristics of GSL in terms of three dimensions: effectiveness, robustness, and complexity. Based on these three dimensions, we conduct an extensive comparative study of existing GSL algorithms in different scenarios. For effectiveness, GSLB provides a fair and comprehensive comparison of existing algorithms on homogeneous node-level, heterogeneous node-level, and graph-level tasks, where we consider both homophilic and heterophilic graph datasets for homogeneous node-level tasks, and cover both Topology Refinement (TR, i.e., refining graphs from data with the original topology) and Topology Inference (TI, i.e., inferring graphs from data without initial topology) settings. For robustness, GSLB evaluates GSL models under three types of noise: supervision signal noise, structure noise, and feature noise. We also compare GSL algorithms with the models specifically designed to improve these types of robustness. For complexity, GSLB conducts a detailed evaluation of representative GSL algorithms in terms of time complexity and space complexity.
Through extensive experiments, we observe that: (1) GSL generally brings performance improvement for node-level tasks, especially on heterophilic graphs; (2) on graph-level tasks, current GSL models bring limited improvement and their performance varies greatly across different datasets; (3) most GSL algorithms (especially unsupervised GSL algorithms) show impressive robustness; (4) GSL models require significant time and memory overhead, making them challenging to deploy on large-scale graphs. In summary, we make the following three contributions:
Figure 1: A general framework of Graph Structure Learning (GSL). GSL methods start with input features and an optional initial graph structure. Its corresponding computation graph is refined/inferred through a structure learning module. With the learned computation graph, Graph Neural Networks (GNNs) are used to generate graph representations.
- We propose GSLB, the first comprehensive benchmark for graph structure learning. We integrate 16 state-of-the-art GSL algorithms and 20 diverse graph datasets covering homogeneous node-level, heterogeneous node-level, and graph-level tasks. An overview of our benchmark is shown in Table 1.
- To explore the ability and limitations of GSL, we systematically evaluate existing algorithms from three dimensions: effectiveness, robustness, and complexity. Based on the results, we reveal the potential benefits and drawbacks of GSL to assist future research efforts.
- To facilitate future work and help researchers quickly use the latest models, we develop an easy-to-use open-source library. Besides, users can evaluate their own models or datasets with less effort. Our code is available at https://github.com/GSL-Benchmark/GSLB.
In this section, we will briefly review the advances and basic concepts of GSL. Given an undirected graph  $\mathcal{G} = (\mathbf{A},\mathbf{X})$ , where  $\mathbf{A}\in \mathbb{R}^{N\times N}$  is the adjacency matrix,  $a_{uv} = 1$  if edge  $(u,v)$  exists and  $a_{uv} = 0$  otherwise, and  $\mathbf{X}\in \mathbb{R}^{N\times F}$  is the node features matrix,  $N$  is the number of nodes,  $F$  is the dimension of node features. Given an optional graph  $\mathcal{G}$ , the goal of GSL is to jointly optimize computation graph  $\mathcal{G}^{\star} = (\mathbf{A}^{\star},\mathbf{X})$  and the parameters of graph encoder  $\Theta_f$  to obtain high-quality node representations  $\mathbf{Z}^{\star}\in \mathbb{R}^{N\times F^{\prime}}$  for downstream tasks, where  $\mathbf{A}^{\star}$  is the refined graph by graph learner.
In general, the objective of GSL can be summarized as the following formula:
$$
\mathcal {L} _ {\mathrm {G S L}} = \mathcal {L} _ {\text {T a s k}} (\mathbf {Z} ^ {\star}, \mathbf {Y}) + \lambda \mathcal {L} _ {\text {R e g}} (\mathbf {A} ^ {\star}, \mathbf {Z} ^ {\star}, \mathcal {G}) \tag {1}
$$
where the first term  $\mathcal{L}_{\mathrm{Task}}$  refers to a task-specific objective with respect to the learned representation  $\mathbf{Z}^{\star}$  and ground-truth  $\mathbf{Y}$ , the second term  $\mathcal{L}_{\mathrm{Reg}}$  imposes constraints on the learned graph structure and representations, and  $\lambda$  is a hyper-parameter that controls the trade-off between the two terms. The general framework of GSL is shown in Figure 1.
3 GSLB: Graph Structure Learning Benchmark
In this section, we introduce the overview of Graph Structure Learning Benchmark, with considerations of algorithms (Section 3.1), datasets (Section 3.2) and evaluations (Section 3.3).
3.1 Benchmark Algorithms
Table 1 shows the overall 16 algorithms integrated in GSLB. They are divided into three categories: homogeneous GSL, heterogeneous GSL, and graph-level GSL. We briefly introduce each category in the following, and more details are provided in Appendix A.2.
Homogeneous GSL. Most of the existing GSL algorithms are designed for homogeneous graphs. They assume there is only one type of nodes and edges in the graph. We select 7 TR-oriented
algorithms including GRCN [48], ProGNN [18], IDGL [4], GEN [39], CoGSL [26], STABLE [21], and GSR [54]. For TI-oriented algorithms, we select SUBLIME [28], NodeFormer [43], SLAPS [10], and HES-GSL [42]. It is worth noting that TR-oriented algorithms can only be applied if the original graph structure is available, but we can construct a preliminary graph based on node features (e.g., kNN graphs or  $\epsilon$ -graphs).
Heterogeneous GSL. We integrate two representative heterogeneous GSL algorithms: Graph Transformer Networks (GTN) [49] and Heterogeneous Graph Structure Learning (HGSL) [53], which can handle the heterogeneity and capture complex interactions in heterogeneous graphs.
Graph-level GSL. Graph-level GSL algorithms aim to refine each graph structure in datasets. We select two graph-level algorithms: Hierarchical Graph Pooling with Structure Learning (HGP-SL) [52] and Variational Information Bottleneck guided Graph Structure Learning (VIB-GSL) [36].
To comprehensively and effectively evaluate the characteristics of GSL in the field of graph learning, we have integrated a large number of datasets from various domains for different types of tasks. For node-level tasks, to evaluate the most mainstream task of GSL, node classification, we use four citation networks (i.e., Cora, Citeseer, Pubmed [47]), and ogbn-arxiv [15], three website networks from WebKB (i.e., Cornell, Texas, and Wisconsin [34]), and a cooccurrence network Actor with homophily ratio ranging from strong homophily to strong heterophily. Subsequently, to validate the effectiveness of GSL in heterogeneous node classification, we utilized three heterogeneous graph datasets (i.e., DBLP [49], ACM [49], and Yelp [29]). To investigate the robustness of GSL, we further incorporate the Polblogs dataset for evaluation. For graph-level tasks, we select six public graph classification benchmark dataset from TUDataset [31] for evaluation, including IMDB-B [3], IMDB-M [3], RDT-B [46], COLLAB [46], MUTAG [5] and PROTEINS [2]. Each dataset is a collection of graphs where each graph is associated with a level. Besides, exploring whether GSL can capture long-range information is an exciting topic. Therefore, we have utilized recently proposed long-range graph datasets: Peptides-func and Peptides-struct [8]. See more details and statistics about datasets in Appendix A.1.
3.3 Benchmark Evaluations
To comprehensively investigate the pros and cons of GSL, our benchmark evaluations encompass three dimensions: effectiveness, robustness, and complexity. For effectiveness, GSLB provides a fair and comprehensive comparison of existing algorithms from three perspectives: homogeneous node classification, heterogeneous node classification, and graph-level tasks. In the case of homogeneous node classification, we evaluated them on both homophilic and heterophilic graph datasets, conducting experiments in both TR and TI scenarios. For graph-level, we evaluate graph-level GSL algorithms on TUDataset and long-range graph datasets for exploring the capabilities on graph-level tasks. For most datasets, we use accuracy as our evaluation metric. For robustness, GSLB evaluates three types of robustness: supervision signal robustness, structure robustness, and feature robustness. We control the count of labels to explore the supervision signal robustness of GSL and find that GSL exhibits excellent performance in the scenarios with few labels. We inject random structure noise and graph topology attacks to investigate the structure robustness. We also study the feature robustness by randomly masking a certain proportion of node features. For complexity, we conduct a detailed evaluation of representative GSL algorithms in terms of time complexity and space complexity. It will help to facilitate the deployment of GSL in real-world applications.
In this section, we systematically investigate the effectiveness, robustness, and complexity of GSL algorithms by answering the following specific questions:
- For effectiveness, RQ1: How effective are the algorithms on node-level representation learning (Section 4.2)? RQ2: Can GSL mitigate homophily inductive bias of traditional message-passing based GNNs (Section 4.2)? RQ3: How does GSL perform on heterogeneous graph datasets (Section 4.3)? RQ4: How effective are the algorithms on graph-level representation learning (Section 4.4)? RQ5: Can GSL methods capture long-range information on the graph (Appendix B)?
- For robustness, RQ6: How robust are GSL algorithms when faced with a scarcity of labeled samples? RQ7: How robust are GSL algorithms in the face of structure attack or noise? RQ8: How is the feature robustness of GSL? (Section 4.5)
- For complexity, RQ9: How efficient are these algorithms in terms of time and space (Section 4.6)?
- Otherwise, RQ10: What does the learned graph structure look like (Appendix B.2)?
All algorithms in GSLB are implemented by PyTorch [33], and unless specifically indicated, the encoders for all algorithms are Graph Convolutional Networks. All experiments are conducted on a Linux server with GPU (NVIDIA GeForce 3090 and NVIDIA A100) and CPU (AMD EPYC 7763), using PyTorch 1.13.0, DGL 1.1.0 [38] and Python 3.9.16.
4.2 Performance on node-level representation learning
For node-level representation learning, we conduct experiments on homogeneous graph datasets under both TR and TI scenarios, and use classification accuracy as our evaluation metric. Table 2 shows the experimental results of various GSL algorithms under the standard setting of transductive node classification task in the TR scenario. We can observe that: 1) Most GSL algorithms generally show improvements in node classification task, particularly on datasets with high heterophily ratio. Due to the presence of heterophilic connections in heterophily graphs, where nodes are often connected to nodes with different labels, it violates the homophily assumption of message-passing neural networks. As a result, traditional GNNs like GCN and GAT exhibit poor performance. However, GSL can improve significantly on heterophily graph datasets by learning new graph structures based on downstream tasks and specific learning objectives, thus enhancing the homophily of the graph and promoting the performance on node-level representation learning. 2) SUBLIME achieves optimal or near-optimal results on most datasets. It learns graph structure through contrastive learning in an unsupervised manner. As mentioned in the recent literature [10], optimizing graph structures solely based on label information is insufficient. Leveraging a large and abundant amount of unlabeled information can enhance the performance of GSL. 3) The scalability of GSL still needs improvement, as only a few models can be trained on large-scale datasets (e.g., ogbn-arxiv). We will discuss the scalability of GSL algorithms in detail in a subsequent section (Section 4.6).
Table 3 shows the experimental results of the transductive node classification task in the TI scenario. Some GSL algorithms are designed for TR scenario (i.e., GRCN, IDGL, etc.), so we use kNN graphs as their original graph structure. As we can observe, on the homophily graph datasets, GSL outperforms baselines, such as MLP,  $\mathrm{GCN}_{knn}$  and  $\mathrm{GAT}_{knn}$ . However, on the heterophily graph datasets, most GSL algorithms often have difficulty achieving better results than baseline models. As mentioned in earlier literature, a network with randomness tends to get better performance utilizing kNN for direct information propagation [17]. Therefore, traditional message-passing neural networks with kNN graphs demonstrate powerful performance. In addition, as observed in the TR scenario, models that leverage self-supervision to extract abundant unlabeled information often achieve better performance.
4.3 Performance on heterogeneous graph node-level representation learning
In this section, we evaluate the performance of GSL algorithms on heterogeneous node classification task and use Macro-F1 and Micro-F1 as our evaluation metrics. Table 4 shows the experimental results on heterogeneous graph datasets. By observing the results, we can find that: 1) Because GTN and HGSL consider both heterogeneity and structure learning, they generally outperform other models on heterogeneous graph datasets. 2) GSL algorithms generally outperform the vanilla GNN models (e.g. GCN and GAT) since they have learned better structures to facilitate message passing. 3) Due to the majority of GSL algorithms not explicitly accounting for heterogeneity, they may exhibit poorer performance on heterogeneous graph datasets. 4) Some datasets (e.g. Yelp) exhibit stronger heterogeneity, and on such datasets, models that consider heterogeneity (e.g. HAN, GTN, and HGSL) perform significantly better.
Table 2: Accuracy ± STD comparison (%) under the standard setting of transductive node classification task in the Topology Refinement (TR) scenario, which means the original graph structure is available for each method. Performance is averaged from 10 independent repetitions. The highest results are highlighted with bold, while the second highest results are marked with underline. "OOM" denotes out of memory.
Edge Hom.	Cora 0.81	Citeseer 0.74	Pubmed 0.80	ogbn-arxiv 0.65	Cornell 0.12	Texas 0.06	Wisconsin 0.18	Actor 0.22
GCN	81.46±0.58	71.36±0.31	79.18±0.29	70.77±0.19	47.84±5.55	57.83±2.76	57.45±4.30	30.01±0.77
GAT	81.41±0.77	70.69±0.58	77.85±0.42	69.90±0.25	46.22±6.33	54.05±7.35	57.65±7.75	28.91±0.83
GPRGNN	83.66±0.77	71.64±0.49	75.99±1.63	50.80±0.29	76.76±5.30	85.14±3.68	83.33±3.42	34.09±1.09
LDS	83.01±0.41	73.55±0.54	OOM	OOM	47.87±7.14	58.92±4.32	61.70±3.58	31.05±1.31
GRCN	83.87±0.49	72.43±0.61	78.92±0.39	OOM	54.32±8.24	62.16±7.05	56.08±7.19	29.97±0.71
ProGNN	80.30±0.57	68.51±0.52	OOM	OOM	54.05±6.16	48.37±12.17	62.54±7.56	22.35±0.88
IDGL	83.88±0.42	72.20±1.18	80.00±0.38	OOM	50.00±8.98	62.43±6.09	59.41±4.11	28.16±1.41
GEN	80.21±1.72	71.15±1.81	78.91±0.69	OOM	57.02±7.19	65.94±1.38	66.07±3.72	27.21±2.05
CoGSL	81.76±0.24	73.09±0.42	OOM	OOM	52.16±3.21	59.46±4.36	58.82±1.52	32.95±1.20
SUBLIME	83.40±0.42	72.30±1.09	80.90±0.94	71.75±0.36	70.54±5.98	77.03±4.23	78.82±6.55	33.57±0.68
STABLE	80.20±0.68	68.91±1.01	OOM	OOM	44.03±4.05	55.24±6.04	53.00±5.27	30.18±1.00
NodeFormer	80.28±0.82	71.31±0.98	78.21±1.43	55.40±0.23	42.70±5.51	58.92±4.32	48.43±7.02	25.51±1.77
GSR	82.48±0.43	71.10±0.25	78.09±0.53	OOM	44.32±2.16	60.81±4.87	56.86±1.24	30.23±0.38
Table 3: Accuracy ± STD comparison (%) under the standard setting of transductive node classification task in the Topology Inference (TI) scenario, which means the original graph structure is not available for each method.
Edge Hom.	Cora 0.81	Citeseer 0.74	Pubmed 0.80	ogbn-arxiv 0.65	Cornell 0.12	Texas 0.06	Wisconsin 0.18	Actor 0.22
MLP	58.55±0.80	59.52±0.64	73.00±0.30	55.21±0.11	71.35±6.19	80.27±5.93	84.71±3.14	35.49±1.04
GCNknn	66.10±0.44	68.33±0.89	69.23±0.49	55.21±0.22	75.14±2.65	75.95±4.43	84.12±3.97	32.98±0.49
GATknn	64.62±1.04	68.05±1.12	68.76±0.80	55.92±0.30	74.05±5.16	76.49±4.99	82.16±4.06	31.67±1.19
GPRGNNknn	69.27±0.62	70.29±0.54	68.19±1.19	51.39±0.13	75.68±2.70	81.08±4.18	84.12±3.22	34.71±1.51
LDS	69.87±0.41	72.43±0.61	OOM	OOM	72.65±3.86	70.20±5.07	78.14±4.50	32.39±0.79
GRCNknn	69.48±0.66	68.41±0.50	68.96±0.85	OOM	71.08±6.84	74.32±5.02	78.63±4.92	30.83±0.76
ProGNNknn	67.11±0.56	64.55±0.95	OOM	OOM	71.35±4.04	71.89±5.69	72.94±7.93	31.56±1.14
IDGLknn	69.74±0.57	66.33±0.84	74.01±0.64	OOM	72.70±4.75	75.40±4.75	79.21±3.94	33.07±1.37
GENknn	66.95±1.40	67.29±1.17	69.76±1.53	OOM	71.08±5.54	74.59±3.46	81.76±2.91	31.28±1.06
CoGSLknn	66.65±0.37	68.72±0.84	OOM	OOM	70.27±3.42	72.70±4.26	76.96±5.25	34.52±1.56
GSRknn	66.28±0.59	66.77±0.62	68.49±1.49	OOM	70.27±3.62	74.86±3.63	78.62±5.91	33.73±1.12
SLAPS	72.28±0.97	70.71±1.13	74.50±1.47	55.19±0.21	74.59±3.67	79.19±4.99	81.96±3.26	37.16±0.91
SUBLIME	72.74±1.91	72.63±0.60	75.08±0.55	55.57±0.18	72.35±3.57	75.51±5.08	82.14±2.62	32.20±1.02
NodeFormer	54.35±5.33	45.90±5.42	59.83±6.50	55.37±0.23	42.70±5.51	58.92±4.32	48.24±6.63	29.24±1.68
HES-GSL	73.68±1.04	70.12±1.11	77.08±0.78	56.46±0.27	66.22±6.19	74.05±6.42	79.61±5.28	36.73±0.76
4.4 Performance of GSL algorithms on graph-level tasks
In this section, we conduct graph classification experiments on four social datasets (i.e., IMDB-B, RDT-B, COLLAB, and IMDB-M) and two biological datasets (i.e., MUTAG and PROTEINS). Table 5 shows the experimental results of average accuracy and the standard deviation of 10-fold cross-validation. We can observe that HGP-SL (with GCN as the encoder) consistently outperforms GCN on all datasets. However, we find that VIB-GSL exhibits strong instability across different random seeds. And due to the absence of training scripts in the official code<sup>2</sup>, we performed hyperparameter tuning based on the parameter search space  $(\beta \in \{10^{-1}, 10^{-2}, 10^{-3}, 10^{-4}, 10^{-5}, 10^{-6}\})$  provided in the paper, but we are unable to surpass the performance of the baseline models consistently. Lastly, we conducted an analysis of graph-level GSL algorithms on long-range graph dataset [8]. For detailed information, please refer to Appendix B.
$^{2}$ https://github.com/RingBDStack/VIB-GSL