SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Zhiyao Zhou $^{1}$ , Sheng Zhou $^{2*}$ , Bochao Mao $^{2}$ , Xuanyi Zhou $^{1}$ , Jiawei Chen $^{1}$ , Qiaoyu Tan $^{3}$ , Daochen Zha $^{4}$ , Yan Feng $^{1}$ , Chun Chen $^{1}$ , Can Wang $^{1}$
<sup>1</sup>College of Computer Science, Zhejiang University, Hangzhou, China
$^{2}$ School of Software Technology, Zhejiang University, Ningbo, China
$^{3}$ New York University Shanghai  $^{4}$ Rice University [zjucszzy, zhousheng_zju, bcmao, zxy2004 eepyhunt, fengyan, chenc, wcan]@zju.edu.cn
qiaoyu.tan@nyu.edu daochen.zha@rice.edu
Graph Neural Networks (GNNs) have emerged as the de facto standard for representation learning on graphs, owing to their ability to effectively integrate graph topology and node attributes. However, the inherent suboptimal nature of node connections, resulting from the complex and contingent formation process of graphs, presents significant challenges in modeling them effectively. To tackle this issue, Graph Structure Learning (GSL), a family of data-centric learning approaches, has garnered substantial attention in recent years. The core concept behind GSL is to jointly optimize the graph structure and the corresponding GNN models. Despite the proposal of numerous GSL methods, the progress in this field remains unclear due to inconsistent experimental protocols, including variations in datasets, data processing techniques, and splitting strategies. In this paper, we introduce OpenGL, the first comprehensive benchmark for GSL, aimed at addressing this gap. OpenGL enables a fair comparison among state-of-the-art GSL methods by evaluating them across various popular datasets using uniform data processing and splitting strategies. Through extensive experiments, we observe that existing GSL methods do not consistently outperform vanilla GNN counterparts. We also find that there is no significant correlation between the homophily of the learned structure and task performance, challenging the common belief. Moreover, we observe that the learned graph structure demonstrates a strong generalization ability across different GNN models, despite the high computational and space consumption. We hope that our open-sourced library will facilitate rapid and equitable evaluation and inspire further innovative research in this field. The code of the benchmark can be found in https://github.com/OpenGSL/OpenGSL.
Graph Neural Networks (GNNs) [6, 18, 13, 41] have emerged as the dominant approach for learning on graph-structured data, thanks to their exceptional ability to leverage both the graph topology structure and node attributes [44]. Considerable endeavors have been recently made to enhance the performance of GNNs by refining the architectures of GNN models, such as neural message passing [41, 46, 45, 11, 3] and transformer based methods [19, 48, 35]. However, these model-centric methods overlook the potential flaws of the underlying graph structure, which can result in suboptimal performance. Notably, extensive evidence from previous studies [5] confirms that real-world
Figure 1: Timeline of GSL research. Existing GSL methods are categorized into three groups based on the training procedure. Bottom left corner illustrates the key difference on the interaction between two components.
graphs often exhibit suboptimal characteristics, such as the absence of valuable links and the presence of spurious connections among nodes.
To improve the graph quality, Graph Structure Learning (GSL) [58], a family of data-centric graph learning methods [51, 50, 30], has recently attracted considerable research interest. These methods target optimizing the graph structure and the corresponding GNN representations jointly [26, 22, 16, 9, 42]. By refining the graph structure, GSL methods can potentially empower GNNs to learn better representations with improved performance. GSL has been successfully applied to disease analysis [4] and protein structure prediction [17].
Despite the plethora of GSL methods proposed in recent years, as illustrated in Figure 1, there is no comprehensive benchmark for GSL, which significantly impedes the understanding and progress of GSL in several aspects. i) The use of different datasets, data processing approaches and data splitting strategies in previous works makes many of the results incomparable. ii) There is a lack of understanding of the learned structure itself, particularly regarding its homophily and generalizability to GNN models other than GCN. iii) Apart from accuracy, understanding each method's computation and memory costs is imperative, yet often overlooked in the literature.
To bridge this gap, we introduce OpenGL, the first comprehensive benchmark for GSL. OpenGL implements a wide range of GSL algorithms through unified APIs, while also adopting consistent data processing and data splitting approaches for fair comparisons. Through benchmarking the existing GSL methods on various datasets, we make the following contributions:
- Comprehensive benchmark. OpenGL enables a fair comparison among thirteen state-of-the-art GSL methods by unifying the experimental settings across ten popular datasets of diverse characteristics. Surprisingly, the empirical results reveal that GSL methods do not consistently outperform the vanilla GNNs on all datasets.
- Multi-dimensional analysis. We conduct a systematic analysis of GSL methods from various dimensions, encompassing the homophily of the learned structure, the generalizability of the learned structure across GNN models, and the time and memory efficiency of the existing methods. Our key findings: i) Contrary to the common belief in the homophily assumption, increasing the homophily of the structure does not necessarily translate into improved performance. ii) The learned structures by GSL methods exhibit strong generalizability. iii) Most GSL methods are time- and memory-inefficient, some of which require orders of magnitudes more resources than vanilla GNNs, highlighting the pressing need for more efficient GSL approaches.
- Open-sourced benchmark library and future directions: We have made our benchmark library publicly available on GitHub, aiming to facilitate future research endeavors. We have also outlined potential future directions based on our benchmark findings to inspire further investigations.
To summarize, in this paper, we aim to create a comprehensive benchmark that facilitates the fair evaluation of graph structure learning algorithms, encourages new research, and ultimately advances the progress of the field as a whole.
2 Formulations and Background
Notations. Let  $\mathcal{G} = (\mathcal{V},\mathcal{E},\mathbf{A},\mathbf{X})$  be a graph, where  $\mathcal{V}$  is the set of  $N$  nodes and  $\mathbf{A}\in \mathbb{R}^{N\times N}$  is the adjacency matrix.  $\mathcal{E}$  denotes the edge set and  $\mathbf{X}\in \mathbb{R}^{N\times d}$  represents the corresponding feature matrix with dimension  $d$ . Typically, a GNN model is often parameterized by a mapping function  $f:(\mathbf{A},\mathbf{X})\to \mathbf{H}\in \mathbb{R}^{N\times l}$ , which maps each node  $v\in \mathcal{V}$  into a  $l$ -dimensional embedding vector  $\mathbf{h}_v$ . In the semi-supervised setting, some nodes  $v_{i}\in \mathcal{V}_{train}$  are often associated with labels  $y_{i}$  to guide the training. Please note that in the traditional GNNs, the adjacent matrix  $\mathbf{A}$  only serves as input and is not updated along with the training of GNNs. In the GSL methods, a gradually updated structure  $\mathbf{S}\in \mathbb{R}^{N\times N}$  is learned to replace the original structure  $\mathbf{A}$  during training process. This is the major difference between traditional GNNs and GSL methods.
Timeline of GSL methods through the lens of training procedures. To provide a global understanding of the literature, we provide a high-level overview of the existing GSL methods based on their training procedure. Specifically, we identify two key components in GSL methods: structure learning component and GNN component. Based on the interaction between these two components, we partition existing GSL methods into three categories, depicted in bottom left corner of Figure 1: (a) pre-training, (b) co-training, and (c) iter-training. Pre-training involves a two-stage learning process, where the structure is learned through pre-training and then used to train GNNs in downstream tasks [26, 22, 52]. In co-training methods [10, 2, 28], neural networks that generate the graph structure are optimized together with GNNs. The iterative methods [42, 59, 38] involve training the two components iteratively; they learn the structure from predictions or representations generated by an optimized GNN and use it to train a new GNN model for the subsequent iteration. A similar taxonomy can be found in [1]. Our taxonomy differs by introducing the pre-training category and combining two categories (named joint learning and adaptive learning in [1]) into one named co-training.
Homophily and Heterophily. Homophily [31] and heterophily are two mutually exclusive measurements based on similarity between connected node pairs, where two nodes are considered similar if they share the same node label. The homophily of graph  $\mathcal{G}$  can be formulated as:
$$
\operatorname {h o m o} (\mathcal {G}) = \frac {1}{| \mathcal {E} |} \left| \left\{\left(v _ {i}, v _ {j}\right) \mid \left(v _ {i}, v _ {j}\right) \in \mathcal {E}, y _ {i} = y _ {j} \right\} \right| \tag {1}
$$
where  $|\mathcal{E}|$  is the number of observed edges. Correspondingly, the heterophily of graph  $\mathcal{G}$  is defined as  $1 - \mathrm{homo}(\mathcal{G})$ . The homophily of graphs has been widely assumed to be the key motivation for designing GNNs, while a few recent works [29, 27] have argued on it. Although some GSL methods [53, 42] also claim to learn a graph structure with high homophily, the homophily of the graph structure learned by GSL methods has not been studied.
3 Benchmark Design
We begin by introducing the datasets utilized in our benchmarking process, along with the algorithm implementations. Then, we outline the research questions that guide our benchmarking study.
3.1 Datasets and Implementations
Datasets. In order to provide a comprehensive evaluation of existing GSL methods, we collect 10 graph node classification datasets that have been widely used in the GSL literature. These selected datasets come from different domains and exhibit different characteristics, enabling us to evaluate the generalizability of existing methods across a range of scenarios. Specifically, we use three classic citation datasets [36], namely Cora, Citeseer, Pubmed, as well as two representative social network datasets BlogCatalog and Flickr [15]. Additionally, we include five datasets that have been proposed recently in [34], due to their ability to overcome the drawbacks of the commonly used heterophilous datasets [32]. Table 1 shows the statistics of these datasets, which are divided into two groups according to whether the edge homophily is higher than 0.5. Note that in the five homophilous graph
Table 1: Overview of the datasets used in this study.
group	Dataset	# Nodes	# Edges	#Feat.	Avg.# degree	#Classes	#Homophily
homophilous	Cora	2,708	5,278	1,433	3.9	7	0.81
CiteSeer	3,327	4,552	3,703	2.7	6	0.74
Pubmed	19,717	44,324	500	4.5	3	0.80
Questions	48,921	153,540	301	6.3	2	0.84
Minesweeper	10,000	39,402	7	7.9	2	0.68
heterophilous	BlogCatalog	5,196	171,743	8,189	66.1	6	0.40
Flickr	7,575	239,738	12,047	63.3	9	0.24
Amazon-ratings	24,492	93,050	300	7.6	5	0.38
Roman-empire	22,662	32,927	300	2.9	18	0.05
Wiki-cooc	10,000	2,243,042	100	448.6	5	0.34
datasets, Questions and Minesweeper are binary classification datasets with highly imbalanced class distributions. We present more detailed introductions in Appendix A.1, along with other common graph statistics and newly proposed metrics [33] that can better capture the characteristics of graph datasets.
The data splitting methods in different GSL works are not consistent, bringing difficulties in conducting fair comparisons. We investigate various GSL works and choose the data splits that are most commonly used. For three citation datasets, we use the classic split from [47, 18]. For BlogCatalog and Flickr, we follow the split in [15, 53]. For five datasets from [34], we follow the original split in [34].
**Implementations.** We consider a collection of state-of-the-art algorithms, including LDS [10], ProGNN [16], IDGL [2], GRCN [49], GAug [53], SLAPS² [9], GEN [42], WSGNN [20], Nodeformer [43], CoGSL [24], SUBLIME [26], STABLE [22], and SEGSL [59]. We rigorously reproduced all methods according to their papers and source codes. To ensure a fair evaluation, we perform hyperparameter tuning with the same search budget on the same dataset for all methods. More details on these algorithms and implementations can be found in Appendix A.2.
3.2 Research Questions
We carefully design the OpenGL to systematically evaluate existing methods and inspire future research. Specifically, we aim to answer the following research questions.
RQ1: How much progress has been made by existing GSL methods?
Motivation. Previous research in GSL has been hindered by the use of different data preprocessing, and splits, which make it difficult to fairly evaluate and compare the performance of different methods. Given the fair comparison environment provided by OpenGL, the first research question is to revisit how much progress has been made by existing GSL methods. By answering this question, we aim to gain a deeper understanding of the strengths and weaknesses of existing methods, and identify areas that offer potential for further enhancements.
Experiment Design. Following the experimental setting of most existing GSL methods, we conduct the node classification experiments on all the datasets. For each method and dataset, we report the mean performance and standard deviation of 10 runs. For binary classification datasets, we report ROC AUC metric, while for other datasets, we report accuracy metric. Since most of the implemented GSL methods have used GCN as backbone, we compare the performances of GSL methods with vanilla GCN to verify the enhancement of learning structures. Additional results using other backbones can be found in Appendix D.5.
RQ2: Does GSL benefit from learning graph structures with higher homophily?
Motivation. The homophily assumption has been a fundamental motivation of modern GNNs' designs, which has also been brought to the GSL scenarios. More specifically, some existing GSL methods have attempted to learn the structure with higher homophily by introducing explicit
${}^{2}$  SLAPS does not use the original graph structure, thus being worse in most cases. Results of all methods without accessing the graph structure are provided in Appendix D.7.
homophily-oriented objectives [53, 42]. However, these claims are questionable since they have been evaluated only on a limited set of toy datasets [42], with little examination of their validity on real datasets. As researchers have started to question the homophily assumption on GNNs [29], it becomes imperative to re-evaluate the significance of GSL methods in learning more homophilous graph structures.
Experiment Design. To answer this question, we first compare the homophily of the structure learned by GSL methods with the original one. Then we determine whether the reported performance improvements stem from a more homophilous graph structure by examining the correlation between the homophily and node classification performance.
RQ3: Can the learned structures generalize to other GNN models?
Motivation. Node classification tasks have been a popular means of evaluating GNNs jointly optimized with GSL methods, the quality of the learned graph structure, however, has not been thoroughly evaluated. While GSL has demonstrated improvements in certain cases, it remains uncertain whether these improvements can be attributed to learning superior structures. Furthermore, it is unclear whether the learned structures have the capability to generalize effectively to other GNN models. Therefore, it is crucial to conduct experiments to evaluate the generalizability of the learned graph structures.
Experiment Design. To answer this research question, we use the learned structure and original features to create a new graph data  $\mathcal{G}^{\prime} = (\mathbf{S},\mathbf{X})$ , and train a new GNN method on the new graph. This differs from the setting in RQ1, as we treat each GSL method as a pre-processing step and train a GNN model from scratch using the learned structure, rather than using a GNN jointly optimized with the structure. By comparing the performance on the original graph and the new graph, we can evaluate the generalizability of the learned structure. To further verify the effectiveness of learned structure, we also include two non-GNN methods, Label Propagation [57, 55] and LINK [54], that only take graph structure as input without using node features for node classification.
RQ4: Are existing GSL methods efficient in terms of time and space?
Motivation. As the GSL methods simultaneously optimize the GNNs and graph structure, they naturally consume more computational complexity and space than the GNNs. However, the efficiency of GSL methods have been largely overlooked by existing methods. Although introducing the structure learning may benefit the GNNs, the extra computational consumption has posed significant requirements of trade-off between performance and efficiency. It is critical to understand the trade-off for deploying the GSL in the practical applications.
Experiment Design. To answer this research question, we evaluate the efficiency of each GSL method in terms of time and space. Specifically, we record the wall clock running time and peak GPU memory consumption during each method's training process. For a fair comparison, all experiments in this part were conducted on a single NVIDIA A800 GPU.
4 Experiment Results and Analyses
4.1 Performance Comparison (RQ1)
We present the performance of all methods on 10 datasets in Table 2 and Table 3. Below are the key findings from these tables.
① For homophilous graphs, many GSL methods work well in datasets with balanced classes, while they cannot handle highly imbalanced situations. Table 2 demonstrates that most GSL methods outperform vanilla GCN on balanced homophilous graphs such as Cora, Citeseer, and Pubmed. Out of the 12 methods tested, 7 methods exceeded GCN on at least two datasets, and 9 methods outperformed GCN on at least one dataset. However, some methods were unable to surpass vanilla GCN, suggesting that structure learning might even degrade GNN performance. This result highlights the need to investigate the general effectiveness of GSL methods further. In contrast, the results were vastly different on datasets such as Questions and Minesweeper, where only a few methods show an advantage over GCN. The imbalanced nature of these datasets limits the power of GSL, indicating that their effectiveness is constrained on such types of data. This fact suggests
In our experiments we focus on methods that learn a unique structure for simplicity.
Table 2: Node classification results on Cora, Citeseer, Pubmed, Questions and Minesweeper. Shown is the mean ± s.d. of 10 runs with different random seeds. Highlighted are the top first, second, and third results. "−" denotes out of memory or time limit of 24 hours exceeded.
Model	Cora	Citeseer	Pubmed	Questions	Minesweeper
GCN	81.95 ± 0.62	71.34 ± 0.48	78.98 ± 0.35	75.80 ± 0.51	78.28 ± 0.44
LDS	84.13 ± 0.52	75.16 ± 0.43	-	-	-
ProGNN	80.27 ± 0.48	71.35 ± 0.42	79.39 ± 0.29	-	51.43 ± 2.22
IDGL	84.19 ± 0.61	73.26 ± 0.53	82.78 ± 0.44	50.00 ± 0.00	50.00 ± 0.00
GRCN	84.61 ± 0.34	72.34 ± 0.73	79.30 ± 0.34	74.50 ± 0.84	72.57 ± 0.49
GAug	83.43 ± 0.53	72.79 ± 0.86	78.73 ± 0.77	-	77.93 ± 0.64
SLAPS	72.29 ± 1.01	70.00 ± 1.29	70.96 ± 0.99	-	50.89 ± 1.72
WSGNN	83.66 ± 0.30	71.15 ± 1.01	79.78 ± 0.35	-	67.91 ± 3.11
Nodeformer	78.81 ± 1.21	70.39 ± 2.04	78.38 ± 1.94	72.61 ± 2.29	77.29 ± 1.71
GEN	81.66 ± 0.91	73.21 ± 0.62	78.49 ± 3.98	-	79.56 ± 1.09
CoGSL	81.46 ± 0.88	72.94 ± 0.71	78.38 ± 0.41	-	-
SEGSL	81.04 ± 1.07	71.57 ± 0.40	79.26 ± 0.67	-	-
SUBLIME	83.33 ± 0.73	72.44 ± 0.89	80.56 ± 1.32	67.21 ± 0.99	49.93 ± 1.36
STABLE	83.25 ± 0.86	70.99 ± 1.19	81.46 ± 0.78	-	70.78 ± 0.27
Table 3: Node classification results on BlogCatalog, Flickr, Amazon-ratings, Roman-empire and Wiki-cooc. Shown is the mean ± s.d. of 10 runs with different random seeds. Highlighted are the top first, second, and third results. "−" denotes out of memory or time limit of 24 hours exceeded.
Model	BlogCatalog	Flickr	Amazon-ratings	Roman-empire	Wiki-cooc
GCN	76.12 ± 0.42	61.60 ± 0.49	45.24 ± 0.29	70.41 ± 0.47	92.03 ± 0.19
LDS	77.10 ± 0.27	-	-	-	-
ProGNN	73.38 ± 0.30	52.88 ± 0.76	-	56.21 ± 0.58	89.07 ± 5.59
IDGL	89.68 ± 0.24	86.03 ± 0.25	45.87 ± 0.58	47.10 ± 0.65	90.18 ± 0.27
GRCN	76.08 ± 0.27	59.31 ± 0.46	50.06 ± 0.38	44.41 ± 0.41	90.59 ± 0.37
GAug	76.92 ± 0.34	61.98 ± 0.67	48.42 ± 0.39	52.74 ± 0.48	91.30 ± 0.23
SLAPS	91.73 ± 0.40	83.92 ± 0.63	40.97 ± 0.45	65.35 ± 0.45	89.09 ± 0.54
WSGNN	92.30 ± 0.32	89.90 ± 0.19	42.36 ± 1.03	57.33 ± 0.69	90.10 ± 0.28
Nodeformer	44.53 ± 22.62	67.14 ± 6.77	41.33 ± 1.25	56.54 ± 3.73	54.83 ± 4.43
GEN	90.48 ± 0.99	84.84 ± 0.81	49.17 ± 0.68	-	91.15 ± 0.49
CoGSL	83.96 ± 0.54	75.10 ± 0.47	40.82 ± 0.13	46.52 ± 0.48	-
SeGSL	75.03 ± 0.28	60.59 ± 0.54	-	-	-
SUBLIME	95.29 ± 0.26	88.74 ± 0.29	44.49 ± 0.30	63.93 ± 0.27	76.10 ± 1.12
STABLE	71.84 ± 0.56	51.36 ± 1.24	48.36 ± 0.21	41.00 ± 1.18	80.46 ± 2.44
that as many real-world graphs are imbalanced, evaluating the effectiveness of GSLs on imbalanced datasets should receive more attention in the future research.
② GSL methods can be effective on specific heterophilous graphs. Table 3 reveals that some GSL methods, including IDGL, GAug, GEN, and SUBLIME, have the potential to exceed vanilla GCN on heterophilous graphs such as BlogCatalog, Flickr, and Amazon-ratings. This finding is intriguing because homophily is not well-preserved on these graphs, inspiring us to investigate the correlation between homophily and structure learning. However, the results are different on Roman-empire and Wiki-cooc datasets, where none of the methods demonstrate an improvement over GCN. This observation suggests that some heterophilous datasets such as Roman-empire and Wiki-cooc may have informative structural patterns that current GSL methods targeting homophily undermine. For more in-depth analysis, please refer to Section 4.2.
4.2 Homophily in Structure Learning (RQ2)
To analyze how different methods behave in terms of homophily and whether the performance is correlated with homophily of the learned structure, we plot the homophily of the learned structure and the node classification performance in the same figure. Figure 2 and Figure 3 show the results, from which we have the following observations:
Figure 2: Homophily of learned structures and performances on homophilous datasets. The methods are ordered by homophily.
Figure 3: Homophily of learned structures and performances on heterophilous datasets. The methods are ordered by homophily.
Table 4: Pearson's correlation coefficient between homophily of learned structures and accuracy for different methods on various datasets.
Datasets	Cora	Citeseer	Pubmed	Minesweeper	BlogCatalog	Flickr	Roman-empire	Amazon-ratings	Wiki-cooc
Pearson	0.29	-0.50	0.62	0.50	0.86	0.87	-0.25	-0.11	-0.84