SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$③$  The homophily of the learned structures varies on homophilous and heterophilous datasets. The results presented in Figure 2 are intriguing. They demonstrate that on homophilous datasets, the homophily of the learned structures is scarcely different from the original structure, and in some cases, even lower. This is counterintuitive, considering that methods like GEN explicitly aim to increase the homophily of the learned structures. However, on heterophilous datasets in Figure 3, the homophily of the learned structures is significantly improved in most cases. This distinction may be attributed to the starting level of graph homophily: As GSL methods are trained with limited supervision signals, the number of edges that can be recovered or removed by GSL is limited. Thus, on heterophilous datasets where most edges do not fit the homophily assumption, these limited edges are more likely to be updated. On the other hand, they are already satisfied on homophilous datasets, explaining the lack of improvement here.
$④$  Homophily is not always a proper guidance for structure learning. As shown in Figure 2, Figure 3 and Table 4, homophily is only significantly positively correlated with performance on two datasets, i.e., BlogCatalog and Flickr. In most cases, we do not observe positive correlation between the performance and the homophily, in some cases even negatively correlated (on Citeseer and Wiki-cooc). These findings suggest that homophily of structure is not a proper guidance for GSL in all cases, which may break the common assumption and call for novel learning objective. This observation can be possibly explained by the viewpoint that certain heterophilous structural patterns can be leveraged by GNNs [29, 27], thereby guiding structure learning with homophily given limited supervision may not generate a sufficiently homophilous structure, but break these patterns and leads to suboptimal results.
Table 5: Results on Cora. GNNs are trained on structures learned from different GSL methods.
Structure source	GCN	SGC	JKNet	APPNP	GPRGNN	LPA	LINK
Original Structure	81.95	80.00	80.40	83.33	83.51	60.30	50.06
ProGNN	82.58 (↑0.63)	82.10 (↑2.10)	82.24 (↑1.84)	83.38 (↑0.05)	83.36 (↓0.15)	75.85 (↑15.55)	78.64 (↑28.58)
IDGL	83.01 (↑1.06)	83.44 (↑3.44)	82.22 (↑1.82)	84.34 (↑1.01)	84.76 (↑1.25)	77.31 (↑17.01)	75.27 (↑25.21)
GRCN	83.98 (↑2.03)	84.66 (↑4.66)	84.09 (↑3.69)	83.35 (↑0.02)	84.41 (↑0.90)	79.03 (↑18.73)	81.83 (↑31.77)
GEN	81.74 (↓0.21)	81.82 (↑1.82)	81.92 (↑1.52)	82.21 (↓1.12)	82.16 (↓1.35)	80.97 (↑20.67)	79.12 (↑29.06)
SUBLIME	82.69 (↑0.74)	82.32 (↑2.32)	81.98 (↑1.58)	82.79 (↓0.54)	83.59 (↑0.08)	78.42 (↑18.12)	81.25 (↑31.19)
Table 6: Results on Blogcatalog. GNNs are trained on structures learned from different GSL methods.
Structure source	GCN	SGC	JKNet	APPNP	GPRGNN	LPA	LINK
Original Structure	76.12	75.37	74.17	93.72	93.82	57.53	64.47
ProGNN	71.73 (↓4.39)	75.20 (↓0.17)	73.73 (↓0.44)	93.71 (↓0.01)	94.70 (↑0.88)	53.66 (↓3.87)	66.00 (↑1.53)
IDGL	89.35 (↑13.23)	75.20 (↓0.17)	89.77 (↑15.60)	94.90 (↑1.18)	95.39 (↑1.57)	73.84 (↑16.31)	81.25 (↑16.78)
GRCN	75.69 (↓0.43)	74.77 (↓0.60)	75.62 (↑1.45)	93.45 (↓0.27)	93.53 (↓0.29)	59.68 (↑2.15)	66.76 (↑2.29)
GEN	90.01 (↑13.89)	89.31 (↑13.94)	90.23 (↑16.06)	90.37 (↓3.35)	90.40 (↓3.42)	86.04 (↑28.51)	86.29 (↑21.82)
SUBLIME	95.01 (↑18.89)	94.95 (↑19.58)	73.73 (↓0.44)	95.35 (↑1.63)	94.51 (↑0.69)	89.04 (↑31.51)	86.45 (↑21.98)
4.3 Generalizability (RQ3)
We show the performance of several GNN models and simple non-GNN models on Cora and BlogCatalog using the structures learned by GSL methods as inputs in Table 5 and 6. Results on other datasets can be found in Appendix D.2. We have the following observation based on the results:
⑤ The structures learned by GSL methods exhibit strong generalizability. The results in Table 5 and 6 demonstrate that many GNN models show improved performance on the learned structure, marked in green, as compared to the original structure. This observation underscores the generalizability of the learned structure and its potential to enhance numerous GNN methods. Additionally, we observed that the structures learned by GSL methods also help to improve the performance of two simple non-GNN methods - LPA and LINK - and in some instances, they even outperform GNNs. The promising results strengthen the notion of the learned structure's generalizability, even without considering node features as input. In conclusion, the experimental results provide strong evidence for the generalizability of the learned structure and call for further exploration and applications of GSL methods.
4.4 Time and Memory Efficiency (RQ4)
Figure 4: Time and space consumption of different methods on Cora
The efficiency of all methods on Cora are presented in Figure 4. For complete statistics on other datasets, please refer to Appendix D.3.
⑥ Most GSL methods have large time and space consumptions. Figure 4 clearly demonstrate that the current state-of-the-art GSL methods is struggling to achieve a satisfactory balance between performance and efficiency. In particular, most existing GSL methods suffer from significant efficiency issues, with many taking up to ten times longer to run than the GCN method. According to the results, ProGNN is the slowest, requiring 190 times longer than GCN. Likewise, most GSL methods consume an excessive amount of memory, with CoGSL consuming up to 66 times more.
The efficiency problem of GSL methods is especially pronounced on larger datasets, as discussed in Appendix D.3. According to Table 2, several GSL methods run out of memory when performing graph structure learning on the Questions dataset. Given these findings, it is vital to address the efficiency problem to ensure that GSL methods can be deployed successfully in a wide spectrum of real-world scenarios.
Drawing upon our empirical analyses, we point out some promising future directions for GSL.
Rethinking the necessity of homophily in GSL. While current GSL methods commonly pursue homophily within the refined graph structure, observations  $③$  and  $④$  suggest that performance improvements in GSL do not necessarily originate from increased homophily. Consequently, it is essential to rethink the necessity of homophily in GSL and explore other factors contributing to the effectiveness of GSL.
Designing adaptive GSL methods for diverse datasets. Observations ① and ② indicate that current GSL method do not universally work well across diverse datasets. Thus, there is an evident opportunity for creating innovative GSL methods that can adapt to diverse datasets. To achieve this goal, two critical questions arise: 1) What characteristics should learn structures exhibit for disparate datasets? Our observation ④ highlights that a sole focus on homophily may not lead to substantial improvements in certain datasets, suggesting the need to explore more reliable properties. 2) How can we incorporate these characteristics into the structure learning? Some properties might be hard to evaluate or optimize, warranting further investigation.
Developing task-agnostic GSL methods. Existing research efforts on GSL are mainly task-motivated. However, real-world scenarios often necessitate the refinement of a graph structure without accessing the downstream task. Although there are a few preliminary works on task-agnostic GSL [26, 22, 52], they exhibit certain limitations on preserving the naive characteristics such as proximity which is not sufficient in downstream tasks like graph clustering [12]. The core challenge is to extract semantic information from the graph data and to define optimality of the structure in the absence of explicit labels.
Improving the efficiency of GSL methods. Observation ⑥ exposes the issue of efficiency in GSL, requiring further attention in GSL research. Some GSL methods may run out of memory or exceed the time limit, even though the datasets we use are fairly small by today's standards [14]. The practical utility of current GSL methodologies is often hampered by these efficiency issues. Although some attempts have been made to address this problem, including limiting the weights on pre-existing edges [9] or employing anchor points [2], they commonly compromise the expressiveness of GSL. Drawing inspiration from the successful adoption of sampling strategies in Graph Neural Networks (GNNs) for acceleration [25], it would be promising to devise sophisticated sampling methodologies specifically tailored for GSL.
This paper introduces a comprehensive benchmark for graph structure learning (GSL), OpenGSL, by reimplementing and comparing thirteen cutting-edge methods across a diverse set of datasets. The fair comparison and comprehensive analysis unearth several key findings on this promising research topic. Firstly, we observe that GSL methods do not consistently outperform vanilla GNNs across diverse datasets. Secondly, we revisit the correlation between homophily and structure learning, encouraging innovative learning objectives other than homophily. Thirdly, we empirically demonstrate the generalizability of the learned structure. Lastly, we highlight the efficiency problem of the existing methods, emphasizing the need to develop scalable GSL methods. We believe that this benchmark will have a positive impact on this emerging research domain. We have made our code publicly available and welcome further contributions of new datasets and methods.
We plan to further enhance OpenGL in the following ways. To begin with, we plan to broaden our dataset coverage to ensure a more comprehensive evaluation. Specifically, we will look into heterogeneous networks that are used widely in practical applications. In view of the efficiency limitations in current GSL methods, we will explore ways to apply structure learning on large-scale datasets such as OGB [14]. Lastly, in this study we only focus on node classification, which is
widely adopted in existing GSL research. As there have been attempts to extend GSL to other tasks (e.g., graph classification [39], clustering [12]), we plan to expand the support of OpenGL to other graph learning tasks for a more comprehensive coverage. We will update our repository to reflect all the potential new tasks, datasets, as well as any improvement or correction. We are also open to suggestions and welcome any comments to improve our benchmark and elevate its usability.
This work is supported by the Starry Night Science Fund of Zhejiang University Shanghai Institute for Advanced Study, China (Grant No: SN-ZJU-SIAS-001), the National Natural Science Foundation of China (621062219, 62372399), Zhejiang Provincial Natural Science Foundation of China (Grant No: LTGG23F030005), Ningbo Natural Science Foundation (Grant No: 2022J183) and the advanced computing resources provided by the Supercomputing Center of Hangzhou City University.
[1] Y. Chen and L. Wu. Graph neural networks: Graph structure learning. In L. Wu, P. Cui, J. Pei, and L. Zhao, editors, Graph Neural Networks: Foundations, Frontiers, and Applications, pages 297-321. Springer Singapore, Singapore, 2022.
[2] Y. Chen, L. Wu, and M. Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems, 33:19314-19326, 2020.
[3] E. Chien, J. Peng, P. Li, and O. Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations.
[4] L. Cosmo, A. Kazi, S.-A. Ahmadi, N. Navab, and M. M. Bronstein. Latent patient network learning for automatic diagnosis.(2020). 2020.
[5] H. Dai, H. Li, T. Tian, X. Huang, L. Wang, J. Zhu, and L. Song. Adversarial attack on graph structured data. In International conference on machine learning, pages 1115-1124. PMLR, 2018.
[6] M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. Advances in neural information processing systems, 29, 2016.
[7] Y. Deshpande, S. Sen, A. Montanari, and E. Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems, 31, 2018.
[8] V. P. Dwivedi, C. K. Joshi, A. T. Luu, T. Laurent, Y. Bengio, and X. Bresson. Benchmarking graph neural networks. Journal of Machine Learning Research, 24(43):1-48, 2023.
[9] B. Fatemi, L. El Asri, and S. M. Kazemi. Slaps: Self-supervision improves structure learning for graph neural networks. Advances in Neural Information Processing Systems, 34:22667-22681, 2021.
[10] L. Franceschi, M. Niepert, M. Pontil, and X. He. Learning discrete structures for graph neural networks. In International conference on machine learning, pages 1972-1982. PMLR, 2019.
[11] J. Gasteiger, A. Bojchevski, and S. Gunnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations.
[12] M. Gu, G. Yang, S. Zhou, N. Ma, J. Chen, Q. Tan, M. Liu, and J. Bu. Homophily-enhanced structure learning for graph clustering. arXiv preprint arXiv:2308.05309, 2023.
[13] W. L. Hamilton, R. Ying, and J. Leskovec. Inductive representation learning on large graphs. In Proceedings of the 31st International Conference on Neural Information Processing Systems, NIPS'17, page 1025-1035, Red Hook, NY, USA, 2017. Curran Associates Inc.
[14] W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems, 33:22118-22133, 2020.
[15] X. Huang, J. Li, and X. Hu. Label informed attributed network embedding. In Proceedings of the tenth ACM international conference on web search and data mining, pages 731-739, 2017.
[16] W. Jin, Y. Ma, X. Liu, X. Tang, S. Wang, and J. Tang. Graph structure learning for robust graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining, pages 66-74, 2020.
[59] D. Zou, H. Peng, X. Huang, R. Yang, J. Li, J. Wu, C. Liu, and P. S. Yu. Se-gsl: A general and effective graph structure learning framework through structural entropy optimization. In Proceedings of the ACM Web Conference 2023, pages 499-510, 2023.
[60] D. Zügner and S. Gunnemann. Adversarial attacks on graph neural networks via meta learning. In International Conference on Learning Representations (ICLR), 2019.
A Additional Details on Benchmark
Table 7: Statistics of the datasets used in the benchmark.  $\overline{CC_l}$  denotes average local clustering coefficient.  $CC_{g}$  denotes global clustering coefficient.  $h_{edge}$  denotes edge homophily.  $h_{adj}$  and "LI" denotes adjusted homophily and label informativeness proposed in [33], respectively.
Nodes	Edges	Feat	Avg degree	Classes	\( \overline{CC_l} \)	\( CC_g \)	\( h_{edge} \)	\( h_{adj} \)	LI
Cora	2,708	5,278	1,433	3.9	7	0.24	0.09	0.81	0.77	0.59
Citeseer	3,327	4,552	3,703	2.7	6	0.14	0.13	0.74	0.67	0.45
Pubmed	19,717	44,324	500	4.5	3	0.06	0.05	0.80	0.69	0.41
Questions	48,921	153,540	301	6.3	2	0.06	0.02	0.84	0.02	0.00
Minesweeper	10,000	39,402	7	7.9	2	0.44	0.43	0.68	-0.05	0.00
BlogCatalog	5,196	171,743	8,189	66.1	6	0.12	0.08	0.40	0.27	0.09
Flickr	7,575	239,738	12,047	63.3	9	0.33	0.10	0.24	0.14	0.03
Amazon-ratings	24,492	93,050	300	7.6	5	0.58	0.32	0.38	0.14	0.04
Roman-empire	22,662	32,927	300	2.9	18	0.39	0.29	0.05	-0.05	0.11
Wiki-cooc	10,000	2,243,042	100	448.6	5	0.55	0.23	0.34	-0.03	0.02
Cora, Citeseer and Pubmed [36] are three citation networks commonly used in prior GSL works [2, 16, 53, 42, 43, 26], with nodes representing papers and edges representing papers' citation relationships. Node features are bag-of-words feature vectors. The label of each node is its category of research topic.
BlogCatalog [15] is a social network created from an online community where bloggers can follow each other. The features associated with each user are generated based on the keywords in their blog description, while the labels are chosen from a set of predefined categories based on interests of bloggers.
Flickr [15] is a platform for sharing images and videos where users can follow each other, forming a social network. In this dataset, user-specified interest tags are used to generate features, with the groups they have joined serving as labels.
Five datasets from [34] are listed below.
Amazon-ratings is based on the Amazon product co-purchasing network dataset [21]. In this dataset, nodes represent products while edges connect products that are frequently bought together. Node features are the mean of FastText embeddings for words present in the product description. The ratings of products are divided into five distinct classes as labels.
Roman-empire is based on the Roman Empire article from English Wikipedia. In this dataset, each node corresponds to a single word in the text. When two words are consecutive in the text or are connected by a dependency tree, an edge is created between them. Node features are FastText word embeddings. The label of a node is its syntactic role.
Questions is based on a question-answering website. In this dataset nodes represent users, and two users are connected if one user answered the other user's question during a given time interval. Node features are the mean of FastText embeddings for words in the user's description. The label of a node is whether the user remained active on the website. Notably, the dataset is highly imbalanced with  $97\%$  of users belonging to the active class.
Minesweeper is a synthetic dataset based on the Minesweeper game. Each node represents a square in a  $100 \times 100$  grid. Two nodes are connected if they are adjacent in the grid.  $20\%$  of the nodes are randomly designated as mines for prediction. Node features are one-hot-encoded numbers of neighboring mines, while for  $50\%$  of the selected nodes the features are set unknown to increase task difficulty, indicated by a separate binary feature. Notably, the dataset is also highly imbalanced.
Wiki-cooc is based on the English Wikipedia. In this dataset, nodes represent unique words and edges connect frequently co-occurring words. Node features are FastText word embeddings. The label of a node is its part of speech.
A.2 Algorithms
Here we introduce all the GSL algorithms implemented in OpenGSL.
LDS [10] assumes the structure is sampled from mutually independent Bernoulli distributions with a total of  $n^2$  parameters. This paper proposes a bilevel optimization problem, where the inner problem optimizes GNN parameters on the training set, while the outer problem optimizes the structure parameters on the validation set. A meta-learning-based approach is adopted to optimize the structure parameters.
ProGNN [16] optimizes the adjacency matrix directly, which is set as an  $n \times n$  parameter matrix. Three properties, namely sparsity, low-rankness, and smoothness, are introduced to guide structure learning.
IDGL [2] models the structure as a weighted cosine function of node representations. To increase efficiency for larger graphs, the paper proposes an anchor-based structure learning method.
GRCN [49] incorporates two GNNs, one for node classification and the other one for computing node representations that are used to derive structure via a metric function. Both GNNs are optimized simultaneously to minimize the task loss.
GAug [53] shares a similar architecture with GRCN, differing in that GAug employs a graph autoencoder to learn the structure, while structure learning is guided by an edge-prediction loss in addition to the task loss.
SLAPS [9] explores the scenario where original structure is absent. It leverages a MLP to obtain node representations to generate structure through a metric function. The overall architecture of SLAPS is similar to GRCN, with an additional auto-denoising loss to guide structure learning. Specifically, the corrupted features and the learned structure are fed into another GNN, and the output is expected to reconstruct the original features.
GEN [42] assumes that the optimal structure is generated by an SBM model, and further assumes that the similarity matrices of the node representations at different levels are observations of the optimal structure. The EM algorithm is employed to learn the expected optimal structure given a well-optimized GNN. GEN performs structural learning and task learning in an iterative way.
Nodeformer [43] is a model that allows for layer-wise edge reweighting on all node pairs by utilizing a kernelized Gumbel-Softmax operator, which reduces complexity from quadratic to linear with respect to the number of nodes. To guide structure learning, an extra edge-level regularization is implemented.
CoGSL [24] extracts two fundamental views from the original graph and refines them using a view estimator. An adaptive fusion strategy is employed to obtain the final view. CoGSL maintains the performance of three views while reducing the mutual information between every two views to achieve a "minimal sufficient structure."
SUBLIME [26] presents an approach for unsupervised structure learning. It proposes a structure bootstrapping contrastive learning framework, where an anchor structure is set up to provide supervision signals for the learner structure. Specifically, SUBLIME utilizes a GNN-based encoder to obtain node representations from both views and optimizes the GNN encoder through a node-level contrastive loss. During the training process, the anchor structure is updated every several epochs as the interpolation between the anchor structure and the learner structure.
WSGNN [20] is a probabilistic generative model that utilizes variational inference to jointly learn node labels and graph structure. It employs a two-branch model architecture to collectively refine node embeddings and latent structure. A composite loss function is derived from the underlying data distribution, effectively capturing the interplay between observed data and missing data.
STABLE [22] obtains reliable node representations leveraging contrastive learning. The new structure is computed as the similarity matrix of node representations. In addition, an advanced GCN is proposed to enhance the robustness of the ordinary GCN.
SEGSL [59] introduces the concept of graph structural entropy. It starts by enhancing the structure guided by the one-dimensional structural entropy maximization strategy. Then, an encoding tree is constructed to capture the hierarchical information of the graph structure. Finally, SEGSL reconstructs
Figure 5: Code structure of OpenGL. For a specified dataset and configuration, a solver will return the learned structure and performance on the task. Solver serves as a foundational class that supports different implemented methods. It includes two main modules, GraphLearner and GNN, and controls the training and testing processes. GraphLearner consists of four components, each of which can be easily replaced (as shown on the right).
the graph structure from the encoding tree. The method performs structure learning and task learning in an iterative way.
A.3 Discussion on Related GNN Benchmarks
There have been considerable efforts benchmarking GNNs [37, 14, 8, 23]. To name a few, Open Graph Benchmark (OGB) [14] released a diverse set of large-scale and challenging datasets covering various graph machine learning tasks. [8] introduces a collection of medium-sized datasets to quantify progress and support GNN research.