SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Our work differs from existing benchmarks in several aspects. Firstly, to the best of our knowledge, we are the first to propose a benchmark tailored for Graph Structure Learning (GSL). We have made significant efforts to carefully reimplement a wide range of cutting-edge GSL algorithms. OpenGL provides a fair evaluation of these algorithms, and a user-friendly platform for conducting experiments, which is not attainable with existing benchmarks. Secondly, we consider both homophilous and heterophilous datasets when evaluating GSL—a dimension that has been largely overlooked in existing benchmarks for GNNs. Lastly, in addition to node classification (Section 4.1), we include some experimental settings that are specific to GSL, such as experiments on homophily/heterophily (Section 4.2), generalizability of learned structures (Section 4.3), and results in the absence of original structures (Appendix D.7). These experimental settings allow us to delve deeper into GSL, based on which we have provided valuable insights and promising research directions. We believe that OpenGL, as a benchmark targeting GSL research, can have a positive impact on this emerging research field.
B Package
We have developed a open-sourced package Open Graph Structure Learning (OpenGSL)<sup>4</sup>, which offers a comprehensive and unbiased platform for evaluating GSL algorithms and facilitating future research in this domain.
As shown in Figure 5, the code structure is well organized to ensure fair experimental settings across algorithms, easy reproduction of the experimental results and convenient trials on flexibly assembled models. OpenGL consists of the following key modules. The Config module includes the files that define the necessary hyperparameters and settings. The Dataset module is used to load datasets. The Solver consisting of GraphLearner and GNN controls the process of training and evaluation, which serves as the base class for various reproduced methods.
GraphLearner is composed of four elements, each of which supports multiple design choices. Given
Figure 6: OpenGSL
<sup>4</sup>https://github.com/OpenGSL/OpenGSL
Table 8: Hyper-parameter search space of all implemented methods.
Algorithm	Hyper-parameter	Search Space
General Settings	learning rate	1e-1, 1e-2, 1e-3, 1e-4
weight decay	5e-4, 5e-5, 5e-6, 5e-7, 0
GCN [18]	number of layers	2, 3, 4, 5
hidden size	16, 32, 64, 128
dropout	0, 0.2, 0.5, 0.8
GRCN [49]	K for nearest neighbors	5, 50, 100, 200
learning rate for graph	1e-1, 1e-2, 1e-3, 1e-4
SLAPS [9]	learning rate of DAE	1e-2, 1e-3
dropout_adj1	0.25, 0.5
dropout_adj2	0.25, 0.5
k for nearest neighbors	10, 15, 20, 30
λ	0.1, 1, 10, 100, 500
ratio	1,5,10
nr	1, 5
WSGNN [20]	hidden size	16, 32, 64, 128
graph skip connection β for q model	0.5, 0.6, 0.7, 0.8, 0.9
number of heads in graph learner	1, 2, 4, 6, 8
CoGSL [24]	ve_lr	1e-1, 1e-2, 1e-3, 1e-4
ve_dropout	0, 0.2, 0.5, 0.8
τ	0, 0.2, 0.5, 0.8
ε	0.1, 1
λ	0, 0.2, 0.5, 0.8, 1
Nodeformer [43]	number of layers	2, 3, 4, 5
hidden size	16, 32, 64, 128
number of heads	1, 2, 4, 8
dropout	0, 0.2, 0.5, 0.8
number of samples for gumbel softmax sampling	5, 10, 20
weight for edge reg loss	1, 0.1, 0.01
GEN [42]	k for nearest neighbors	7, 8, 9, 10
tolerance for EM	1e-2, 1e-3, 1e-4
threshold for adding edges	0.5, 0.6, 0.7, 0.8
SEGSL [59]	K	2, 3, 4
se	2, 3, 4
LDS [10]	τ	5, 10, 15
ProGNN [16]	learning rate for adj	1e-1, 1e-2, 1e-3, 1e-4
GAug [51]	α for interpolation	0, 0, 0.1, 0.3, 0.5, 0.7, 0.9, 1
temperature	0.3, 0.6, 0.9, 1.2
epochs for warm up	0, 10, 20
IDGL [2]	number of anchors	300, 500, 700
number of heads in structure learning	2, 4, 6, 8
λ1 for interpolation	0.7, 0.8, 0.9
λ2 for interpolation	0.1, 0.2, 0.3
SUBLIME [26]	dropout rate for edge	0, 0.25, 0.5
τ for bootstrapping	0.99, 0.999, 0.9999
STABLE [22]	threshold for consine similarity	0.1, 0.2, 0.3
k for nearest neighbors	1, 3, 5, 7
specific dataset and config file, a solver will return the learned structure and the task performance. For more details and updated features, please refer to our Github repository.
C Additional Details on Experimental Settings
Running Experiments. Our experiments are mostly conducted on a Linux server with an Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz, 125 GB RAM, and an NVIDIA GTX 2080 Ti GPU (12GB). However, as some methods may exceed GPU memory, we run partial experiments (including all
experiments related to efficiency) on another Linux server with an Intel(R) Xeon(R) Platinum 8358 CPU @ 2.60GHz, 1008 GB RAM, and an NVIDIA A800 GPU (80GB).
General Experimental Settings. We strive to adhere to the original implementation of various GSL methods provided in their respective papers or source codes. To achieve this, we have integrated different options into a standardized framework as shown in Figure. We choose not to augment the models with LayerNorm as in [34], since this technique has not been adopted by GSL algorithms. Consequently, there may be some discrepancies observed in the results for the five datasets from [34].
Hyperparameter. We conduct comprehensive hyperparameter tuning to ensure a thorough and impartial evaluation of these GSL methods. When the paper and source code of a particular method do not provide information on hyperparameter settings, we tune the corresponding hyperparameters. Hyperparameter tuning is conducted via bayesian search using  $\mathrm{NNI}^5$ . The hyperparameter search spaces of all methods are presented in Table 8. For detailed meanings of these hyperparameters please refer to their original papers. More complete hyperparameter settings for all the methods can be found in our Github repository  $^{6}$ .
D.1 Additional Results on the Properties of Learned Structure
In Section 4.2, we analyze the correlation between homophily and performance. The results show that homophily is not always positively correlated with performance and therefore not a suitable guidance in all scenarios. Here, we present additional results on two new measures named "adjusted homophily" and "label informativeness", introduced by [33].
Adjusted homophily is proposed to overcome the sensitivity of commonly used homophily measures to the number of classes and their balance. Specifically, the adjusted homophily of a graph  $\mathcal{G}$  is defined as follows:
$$
\operatorname {a d j} _ {-} \operatorname {h o m o} (\mathcal {G}) = \frac {\operatorname {h o m o} (\mathcal {G}) - \sum_ {k = 1} ^ {C} p (k) ^ {2}}{1 - \sum_ {k = 1} ^ {C} p (k) ^ {2}} \tag {2}
$$
where  $p(k) = \frac{\sum_{v_i : y_i = k} d(v_i)}{2|E|}$ ,  $d(v_i)$  is the degree of node  $v_i$ .
Label informativeness measures the overall informativeness of neighbor's labels for central node's labels in a graph. This metric, which goes beyond the homophily-heterophily dichotomy, can further describe the degree of regularity in the connectivity patterns of heterophilous graphs. The label informativeness of a graph  $\mathcal{G}$  is defined as follows:
$$
\operatorname {L I} (\mathcal {G}) = 2 - \frac {\sum_ {c _ {1} , c _ {2}} p \left(c _ {1} , c _ {2}\right) \log p \left(c _ {1} . c _ {2}\right)}{\sum_ {c} p (c) \log p (c)} \tag {3}
$$
where  $p(c_1, c_2) = \frac{|(u,v)| (u,v) \in \mathcal{E}, y_u = c_1, y_v = c_2|}{2|\mathcal{E}|}$ .
The adjusted homophily and label informativeness of structures learned by GSL methods on various datasets are presented in Figure 7 and Figure 8, respectively.
The figures reveal a similarity in the patterns of adjusted homophily, label informativeness and homophily for structures learned by GSL methods. Notably, adjusted homophily and label informativeness show significant positive correlations with performance only on BlogCatalog and Flickr, which is the same as homophily. These results imply that the newly proposed measures are still unable to effectively gauge structure quality and unsuitable for guiding structure learning on real data, despite that they are found to be effective on synthetic data [33]. Further exploration is needed in this aspect.
<sup>5</sup>https://github.com/microsoft/nni/
$^{6}$ https://github.com/Open GSL/Open GSL/tree/main/opengsl/config
Figure 7: Adjusted homophily of learned structures and performances on homophilous datasets. The methods are ordered by homophily. Left to right on Minesweeper: ProGNN, original, SUBLIME, GEN, GRCN.
Figure 8: Adjusted homophily of learned structures and performances on heterophilous datasets. The methods are ordered by homophily. Left to right on Wiki-cooc: ProGNN, original, GRCN, GEN, SUBLIME.
Figure 9: Label informativeness of learned structures and performances on homophilous datasets. The methods are ordered by homophily. Left to right on Minesweeper: ProGNN, original, SUBLIME, GEN, GRCN.
Figure 10: Label informativeness of learned structures and performances on heterophilous datasets. The methods are ordered by homophily.
D.2 Additional Results on Generalizability
In addition to Section 4.3, we also assessed the generalizability of structures learned by various GSL methods on other datasets where GSL methods have made progress. The results are presented in Tables 9 - 11.
These additional results are consistent with the conclusions drawn in Section 4.3. In most cases, the learned structures exhibit strong generalizability. These learned structures frequently outperform the original structures and impart additional advantages to various GNN models.
Table 9: Generalizability on Citeseer. Improvements over the original structure are marked green.
Structure source	GCN	SGC	JKNet	APPNP	GPRGNN	LPA	LINK
Original Structure	71.34	71.20	68.26	71.86	71.14	26.20	34.70
ProGNN	70.00 (↓1.34)	69.02 (↓2.18)	68.33 (↑0.07)	71.14 (↓0.72)	70.45 (↓0.69)	64.46 (↑38.26)	57.12 (↑22.42)
IDGL	73.03 (↑1.69)	73.20 (↑2.00)	72.21 (↑3.95)	73.82 (↑1.96)	72.96 (↑1.82)	64.17 (↑37.97)	64.87 (↑30.17)
GRCN	70.90 (↓0.44)	71.44 (↑0.24)	71.03 (↑2.77)	73.33 (↑1.47)	73.25 (↑2.11)	70.43 (↑44.23)	69.10 (↑34.40)
GEN	72.83 (↑1.49)	73.02 (↑1.82)	72.67 (↑4.41)	73.24 (↑1.38)	73.27 (↑2.13)	65.55 (↑39.35)	68.20 (↑33.50)
SUBLIME	71.22 (↓0.12)	70.76 (↓0.44)	68.17 (↓0.09)	72.60 (↑0.74)	71.64 (↑0.50)	53.50 (↑27.30)	55.00 (↑20.30)
Table 10: Generalizability on Pubmed. Improvements over the original structure are marked green.
Structure source	GCN	SGC	JKNet	APPNP	GPRGNN	LPA	LINK
Original Structure	78.98	78.96	78.00	80.10	79.05	24.70	44.71
GEN	79.50 (↑0.52)	79.23 (↑0.27)	79.14 (↑1.14)	79.83 (↓0.27)	79.76 (↑0.71)	51.61 (↑26.91)	64.79 (↑20.08)
ProGNN	78.74 (↓0.24)	78.70 (↓0.26)	76.97 (↓1.03)	80.33 (↑0.23)	79.10 (↑0.05)	24.80 (↑0.10)	44.23 (↓0.48)
GRCN	79.15 (↑0.17)	79.12 (↑0.16)	77.92 (↓0.08)	80.28 (↑0.18)	79.41 (↑0.36)	44.82 (↑20.12)	44.57 (↓0.14)
SUBLIME	79.24 (↑0.26)	79.67 (↑0.71)	78.90 (↑0.90)	80.99 (↑0.89)	80.49 (↑1.44)	30.60 (↑5.90)	49.58 (↑4.87)
Table 11: Generalizability on Flickr. Improvements over the original structure are marked green.
Structure source	GCN	SGC	JKNet	APPNP	GPRGNN	LPA	LINK
Original Structure	61.60	60.72	57.69	83.58	82.31	39.22	42.60
ProGNN	47.28 (↓14.32)	60.79 (↑0.07)	61.63 (↑3.94)	84.16 (↑0.58)	86.14 (↑3.83)	40.14 (↑0.92)	50.15 (↑7.55)
GRCN	61.14 (↓0.46)	58.86 (↓1.86)	60.52 (↑2.83)	83.14 (↓0.44)	82.11 (↓0.20)	44.95 (↑5.73)	50.79 (↑8.19)
IDGL	85.98 (↑24.38)	86.43 (↑25.71)	84.97 (↑27.28)	87.98 (↑4.40)	88.75 (↑6.44)	71.18 (↑31.96)	80.69 (↑38.09)
GEN	85.94 (↑24.34)	85.93 (↑25.21)	85.86 (↑28.17)	84.16 (↑0.58)	85.92 (↑3.61)	84.15 (↑44.93)	84.47 (↑41.87)
SUBLIME	88.65 (↑27.05)	88.20 (↑27.48)	88.00 (↑30.31)	89.55 (↑5.97)	89.55 (↑7.24)	77.74 (↑38.52)	71.86 (↑29.26)
D.3 Additional Results on Efficiency
We record the efficiency of existing GSL methods on more datasets, and the results are displayed in Figures 11- 15.
Based on these results, it is evident that the current GSL methods faces challenges in efficiency, and the problem is more severe on large-scale graphs. Henceforth, there is a pressing demand for the development of more efficient GSL methods capable of processing large-scale graph data.
Figure 11: Time and space consumption of different methods on CiteSeer
Figure 12: Time and space consumption of different methods on Pubmed
Figure 13: Time and space consumption of different methods on BlogCatalog
Figure 14: Time and space consumption of different methods on Minesweeper
Figure 15: Time and space consumption of different methods on Amazon-ratings
D.4 Additional Results on Robustness
GSL is capable of refining the original suboptimal graph structure, which theoretically gives it superior robustness compared to GCN. We conduct experiments to evaluate the robustness of some GSL methods. Following [56, 22], we consider two types of attacks: Metattack and Random. In Metattack, we use GCN as the surrogate model and apply Metattack [60] to perturb the graph structure to varying degrees. In Random, we randomly add noisy edges to the graph structure to introduce varying degrees of homophily. The performances of various GSL methods under these two attacks are shown below. Figure 16-17 show that most GSL methods have a slower rate of decline than GCN, indicating their better robustness compared with GCN. Among them, SUBLIME demonstrates excellent robustness when facing different attacks.
Figure 16: Robustness of GSL methods on Cora and Citeseer under Metattack.
Figure 17: Robustness of GSL methods on Cora and Citeseer under Random attack.
D.5 Additional Results on Different Backbones
The results presented in previous parts are all obtained from GSL methods using GCN as the backbone. Here, we consider two additional backbones, namely APPNP and GIN, and evaluate the performance of GSL methods with these backbones. Figures 18-20 show the experimental results on Cora, BlogCatalog, and Roman-armpire.
The figures clearly illustrate that, when compared to vanilla GNN backbones, most GSL methods exhibit improvements on Cora and BlogCatalog, while do not work on Roman-empire regardless of the backbone used. This is consistent with the observations in Section 4.1. Moreover, when comparing the performance across different backbones, it is evident that GSL's performance can be further enhanced with more advanced backbones. These results indicate that GSL and GNN backbone, which are two parallel research fields, can benefit from each other.
Figure 18: GSL methods with different backbones on Cora. '—' in the left stands for the vanilla backbones.
Figure 19: GSL methods with different backbones on BlogCatalog. '—' in the left stands for the vanilla backbones.
Figure 20: GSL methods with different backbones on Roman-empire. '—' in the left stands for the vanilla backbones. Some methods are not included due to OOM or exceeding time limit.
One clarification we need to make is the distinction between this experiment and the one conducted in Section 4.3. In both cases, we combine GSL with various GNNs. However, the difference lies in that the performances here are obtained from GNNs that are jointly optimized with structure, while in the other setting, we treat each GSL method as a pre-processing step and train a GNN model from scratch using the learned structure. The purpose of the former experiment is to investigate the impact of the choices on backbones, while the latter aims to examine whether the improvement of GSL stems from learning superior structures.
D.6 Additional Results on Synthetic Graphs
In addition to the experiments conducted on real-world datasets, it is essential to include synthetic datasets as well. Synthetic datasets offer the advantage of enabling precise control over the homophily of the generated graph, facilitating more fine-grained empirical studies. Here, we adopt the CSBM model to construct synthetic datasets [7]. Following [3], we vary the connectivity probabilities to generate graph data with different structural patterns, while ensuring that the generated data satisfies the information-theoretic threshold. In the generated dataset, structures with high homophily and low homophily are both informative, which is different from the setting described in Appx. C.4.
Figure 21 shows the performance of GSL methods on synthetic datasets with different levels of homophily. We can observe that most methods perform well under high homophily conditions but poorly under low homophily conditions. However, some methods, such as SUBLIME and GEN, perform well even under low homophily. We can also see that the performance of some methods is better at 0.2 than at 0.4. These observations indicate that these methods capture informative heterophilious structures. Further exploration is needed to address real-world data, which is relatively more complex.
Figure 21: GSL methods on synthetic data with varying homophily.
D.7 Additional Results without Original Structures
A promising direction for GSL is to apply GNN models to data without graph structure, thereby enabling the use of GNNs for a wider range of machine learning tasks. Traditional approaches involve using GNN on a pre-defined KNN graph. Intuitively, GSL methods should better capture the relationships between samples through joint optimization of the structure and GNN model. In this context, we conduct some preliminary experiments. Specifically, we replace the original structure of graph datasets with an identity matrix or the KNN matrix computed from node features, in which case the models have zero knowledge on the graph structures. The results are presented in Table 12.
It is evident that most GSL methods exhibit improvements compared with GCN on Cora and BlogCatalog, demonstrating the potential of GSL to extend to non-graph data. However, few methods surpass GCN on Roman-empire. Further exploration is required to gain a deeper understanding in this regard.
Table 12: Performance of GSL methods without knowledge on the structure."I" denotes the use of the identity matrix, while "K" denotes the use of the feature-based KNN matrix. "-" denotes out of memory or time limit of 24 hours exceeded.
Model	Cora(I)	Cora(K)	BlogCatalog(I)	BlogCatalog(K)	Roman-empire(I)	Roman-empire(K)
GCN	59.00 ± 0.68	54.79 ± 0.85	85.09 ± 0.51	81.86 ± 0.22	65.98 ± 0.29	65.29 ± 0.36
LDS	68.53 ± 0.63	65.56 ± 1.33	88.26± 0.49	86.72 ± 0.31	-	-
ProGNN	53.06 ± 2.98	53.50 ± 2.63	85.45 ± 0.63	81.50 ± 0.25	-	-
IDGL	66.54 ± 1.13	61.38 ± 0.53	91.21 ± 0.29	87.85 ± 0.29	64.85 ± 0.35	55.17 ± 9.54
GRCN	70.23 ± 0.43	62.03 ± 1.02	89.68 ± 0.48	82.79 ± 0.47	64.86 ± 0.28	63.86 ± 0.29
GAug	-	58.98 ± 0.84	-	81.92 ± 0.40	-	61.17 ± 0.51
SLAPS	72.29 ± 1.01	72.29 ± 1.01	91.73 ± 0.40	91.73 ± 0.40	65.35 ± 0.45	65.35 ± 0.45
GEN	64.60 ± 0.95	63.87 ± 1.03	88.50 ± 0.63	85.15 ± 0.42	-	-
WSGNN	67.27 ± 1.22	64.73 ± 1.03	90.30 ± 0.37	88.87 ± 0.27	65.78 ± 0.19	65.19 ± 0.30
Nodeformer	51.38 ± 1.56	57.73 ± 1.72	53.43 ± 11.32	39.60 ± 12.26	51.18 ± 3.12	53.76 ± 1.89
CoGSL	59.68 ± 1.44	61.93 ± 0.92	83.68 ± 0.19	85.07 ± 0.45	63.94 ± 0.14	63.49 ± 0.10
SUBLIME	69.72 ± 0.97	69.06 ± 0.97	90.91 ± 0.40	82.78 ± 0.76	64.77 ± 0.41	64.38 ± 0.19
STABLE	51.22 ± 2.12	57.79 ± 2.54	74.93 ± 0.93	84.08 ± 0.26	-	-
SEGSL	58.82 ± 0.76	61.93 ± 0.92	86.07 ± 0.45	80.12 ± 0.16	-	-
All experimental results in OpenGL are easily reproducible. We provide more detailed information on reproducibility in the following aspects.
Accessibility. All the datasets, algorithm implementations, and experimental configurations can be found in our open-sourced project at https://github.com/OpenGSL/OpenGSL without personal request.
Datasets. All the datasets are publicly available. Cora, Citeseer and Pubmed are publicly available online<sup>7</sup>, licensed under Creative Commons 4.0. BlogCatalog and Flickr were first released in [40] and further processed in [15], both to the best of our knowledge without a license. Five datasets from [34] are available in their repository<sup>8</sup> with the MIT license. All of the above datasets are consented by the authors for academic usage. None of these datasets contains personally identifiable information or offensive content.
Documentation and uses. We have made a concerted effort to offer users a documentation<sup>9</sup>, ensuring their seamless use of our library. We have also added necessary comments to ensure code readability. In addition, we provide the necessary files to reproduce the experimental results, while also serving as examples on how to use the library<sup>10</sup>. To run the code, the users simply need to run .py files with specifies arguments such as data, method andgpu.
License We use an MIT license for our open-sourced project.
Code maintenance. We are committed to continuously updating our code while proactively addressing user issues and feedback on a regular basis. Furthermore, we enthusiastically welcome contributions from the community to enhance our library and benchmark algorithms. Nonetheless, we will implement strict version control measures to ensure reproducibility during the maintenance.
<sup>7</sup>https://linqs.org/datasets/
$^{8}$ https://github.com/yandex-research/heterophilous-graphs
<sup>9</sup>https://opengsl.readthedocs.io/en/latest/index.html
10https://github.com/OpenGSL/OpenGSL/tree/main/paper