SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 4: Macro-F1 and Micro-F1 ± STD comparison (%) under the standard setting of heterogeneous node classification task.
Method	ACM	DBLP	Yelp
Macro-F1	Micro-F1	Macro-F1	Micro-F1	Macro-F1	Micro-F1
GCN	90.27±0.59	90.18±0.61	90.01±0.32	90.99±0.28	78.01±1.89	81.03±1.81
GAT	91.52±0.62	91.46±0.62	90.22±0.37	91.13±0.40	82.12±1.47	84.43±1.56
HAN	91.67±0.39	91.47±0.22	90.53±0.24	91.47±0.22	88.49±1.73	88.78±1.40
LDS	92.35±0.43	92.05±0.26	88.11±0.86	88.74±0.85	75.98±2.35	78.14±1.98
GRCN	93.04±0.17	92.94±0.18	88.33±0.47	89.43±0.44	76.05±1.05	80.68±0.96
IDGL	91.69±1.24	91.63±1.24	89.65±0.60	90.61±0.56	76.98±5.78	79.15±5.06
ProGNN	90.57±1.03	90.50±1.29	83.13±1.56	84.83±1.36	51.76±1.46	58.39±1.25
GEN	87.91±2.78	87.88±2.61	89.74±0.69	90.65±0.71	80.43±3.78	82.68±2.84
STABLE	83.54±4.20	83.38±4.51	75.18±1.95	76.42±1.95	71.48±4.71	76.62±2.75
GEN	87.91±2.78	87.88±2.61	89.74±0.69	90.65±0.71	80.43±3.78	82.68±2.84
SUBLIME	92.42±0.16	92.13±0.37	90.98±0.37	91.82±0.27	79.68±0.79	82.99±0.82
NodeFormer	91.33±0.77	90.60±0.95	79.54±0.78	80.56±0.62	91.69±0.65	90.59±1.21
GSR	92.14±1.08	92.11±0.99	76.59±0.45	77.69±0.42	83.85±0.76	85.73±0.54
GTN	92.04±0.38	91.94±0.39	90.52±0.45	91.48±0.39	92.98±0.52	92.44±0.46
HGSL	93.23±0.50	93.13±0.51	91.58±0.40	92.49±0.35	92.79±0.44	92.24±0.48
Table 5: Accuracy  $\pm$  STD comparison (%) under the setting of graph-level classification task.
Method	IMDB-B	RDT-B	COLLAB	IMDB-M	MUTAG	PROTEINS
GCN	73.20±4.29	70.10±5.80	76.96±2.28	49.85±3.84	73.92±8.84	67.52±6.71
VIB-GSL (GCN)	71.90±4.48	68.95±2.66	77.14±1.59	49.05±5.52	68.63±5.15	65.68±8.53
HGP-SL (GCN)	74.10±4.55	OOM	78.06±2.17	51.07±2.00	78.07±10.85	70.80±4.25
GAT	72.30±2.26	73.55±4.76	79.08±1.36	48.90±2.98	78.71±7.51	68.63±6.24
VIB-GSL (GAT)	72.10±5.69	OOM	77.54±1.85	49.06±4.55	77.13±9.95	67.09±8.43
SAGE	72.60±3.69	70.20±4.11	75.58±2.04	48.55±2.03	68.65±4.31	64.47±7.15
VIB-GSL (SAGE)	73.00±4.78	65.75±3.17	77.74±1.52	48.79±5.06	72.81±11.41	66.61±4.48
HGP-SL (SAGE)	71.50±5.24	OOM	78.64±1.47	49.67±3.09	77.13±3.29	73.32±2.06
GIN	73.00±2.67	71.70±5.01	79.86±1.64	50.30±3.52	87.19±8.05	69.07±5.62
VIB-GSL (GIN)	69.90±3.90	75.85±3.63	77.25±2.34	49.97±3.65	85.18±10.11	75.15±5.72
HGP-SL (GIN)	73.50±6.25	OOM	80.14±1.51	48.67±2.58	73.92±6.24	69.37±3.95
4.5 Robustness analysis of GSL algorithms
To investigate the robustness of GSL algorithms, we primarily focus on three aspects: structure robustness, feature robustness, and supervision signal robustness. Due to limited space, we predominantly investigate the transductive node classification task in our paper. Nevertheless, researchers can utilize our GSLB library to efficiently and conveniently conduct experiments on other tasks as well.
Robustness analysis with respect to different supervision signals. We have discovered that GSL maintains surprising performance in scenarios with a scarcity of labeled samples. We varied the number of labels per class in the Cora and Cite-seer datasets and selected two baseline models, DAGNN [25] and Self-Training [22], that performed well in scenarios with limited labels. As shown in Fig
Figure 2: Performance of baselines and GSL algorithms with respect to different numbers of labels per class on Cora and Citeseer.
Table 6: Accuracy ± STD comparison (%) with respect to different perturbation rates. Jaccard and SimPGCN are representative state-of-the-art defense GNNs.
Dataset	Ptb Rate	GCN	Jaccard	SimPGCN	IDGL	GRCN	ProGNN	STABLE	SUBLIME
Cora	0%	83.68±0.37	83.78±0.50	82.66±0.48	84.69±1.13	84.43±0.26	84.53±0.89	83.70±0.30	83.84±0.28
5%	80.61±0.39	81.44±0.48	80.35±0.82	82.56±0.24	81.34±0.50	81.47±0.44	81.52±0.85	79.93±0.58
10%	74.38±0.59	75.90±0.64	76.50±1.12	78.06±0.62	77.12±0.38	72.61±0.73	78.64±1.82	78.71±0.46
15%	65.17±0.99	77.14±0.70	73.77±1.88	76.88±0.44	73.74±0.61	65.68±1.97	79.70±1.71	79.34±0.61
20%	61.98±1.23	70.71±0.91	69.08±2.78	67.19±0.69	69.54±0.58	61.07±0.61	76.44±2.47	75.25±1.08
Citeseer	0%	76.56±0.36	74.34±0.26	74.35±0.74	73.87±0.70	76.34±0.11	73.36±1.52	72.65±1.36	73.34±1.17
5%	72.51±0.30	70.01±0.79	72.99±1.05	72.46±0.47	74.66±0.27	71.46±0.47	69.66±0.95	72.63±0.50
10%	71.92±0.68	70.28±1.30	72.68±0.54	69.72±0.59	74.06±0.43	69.03±0.60	72.79±0.71	73.02±0.29
15%	64.44±0.53	67.13±1.28	71.74±1.46	62.83±1.28	66.46±1.12	65.42±1.20	70.98±0.61	73.90±0.52
20%	57.51±1.03	67.82±0.74	70.06±1.86	61.16±0.99	69.42±1.14	57.51±0.36	71.90±1.12	72.55±0.62
Polblogs	0%	95.62±0.69	94.93±0.28	94.50±0.43	94.83±0.20	95.65±0.28	94.84±0.19	95.63±0.32	95.27±0.51
5%	80.57±0.66	78.17±0.55	76.02±1.14	79.62±0.65	93.70±0.18	92.36±0.42	89.41±1.63	93.24±1.50
10%	71.83±2.37	71.86±1.34	70.12±1.10	74.54±0.69	87.99±1.56	84.66±0.52	89.87±0.82	93.62±0.50
15%	66.38±2.17	69.93±0.66	64.19±1.55	75.53±0.83	71.85±1.58	77.38±0.51	89.94±0.89	94.29±0.27
20%	68.19±2.24	69.22±0.34	63.64±1.41	71.63±0.62	71.73±1.58	73.57±0.29	87.42±0.69	92.60±0.72
Figure 3: Analysis of robustness when injecting random noise on Cora and CiteSeer.
ure 2, we can observe that GSL algorithms (for the sake of brevity, we opted to select only three models: GRCN, IDGL, and SUBLIME) achieve the best results in scenarios with fewer labels available. We speculate that this may be because the learned graph structure is denser and exhibits cleaner community boundaries. As a result, the supervision signals can propagate more effectively within such a structure.
Robustness analysis with respect to random noise. We randomly remove edges from or add edges to the original graph structures of Cora and CiteSeer, then evaluated the performance of GSL algorithms on the corrupted graphs. We change the ratios of modified edges from 0 to 0.9 to simulate different attack intensities. As shown in Figure 3, as the noise intensity increases, the models' performance generally exhibits a downward trend. And we can observe that GSL algorithms commonly demonstrate a certain degree of robustness, as they tend to exhibit more stable performance than GCN when random noise is injected. Besides, we also found that, due to variations in the graph modeling process, different algorithms display varying levels of robustness when facing edge deletion and edge addition scenarios. For example, GRCN demonstrates strong robustness in edge deletion scenarios. However, in the edge addition scenarios, it only exhibits slight performance improvements compared to GCN. On the contrary, STABLE exhibits strong robustness in the edge deletion scenario, while showing the opposite trend in edge addition.
Robust analysis with respect to graph topology attack. Following [21, 55], we conduct robust analysis on three graph datasets, i.e., Cora, Citeseer, and Polblogs. First, we select the largest connected component in the graph, and utilize Mettack [58], a non-targeted adversarial topology attack method, to generate perturbed graphs. We select the perturbation rate from  $0\%$  to  $20\%$ . Table 6 shows the performance of GSL algorithms on three datasets with respect to various perturbation rates. Surprisingly, we can observe that most GSL algorithms exhibit strong robustness against graph topology attacks, even better than state-of-the-art defense GNNs (e.g., Jaccard [41] and SimPGCN [19]). GSL can effectively remove the newly added adversarial edges, and recover important edges to promote message passing. As mentioned in Li et al. [21], optimizing graph structures based on either features or supervised signals might not be reliable. We found that self-supervised graph structure modeling methods (e.g., STABLE and SUBLIME) show excellent
Figure 4: Analysis of robustness when injecting random feature noise on Cora and CiteSeer.
Figure 5: Training time and space analysis on Cora, Citeseer and Pubmed.
performance on corrupted graph structure datasets, which means unsupervised representation learning might produce more reliable and high-quality representations to conduct structure modeling.
Robust analysis with respect to feature noise. On the basis of exploring structural robustness, we also study the feature robustness of GSL. We randomly mask a certain proportion of node features by filling them with zeros, to investigate the performance of GSL algorithms when node features are subjected to varying degrees of damage. As shown in Figure 4, we can observe that: 1) the node features play a more critical role than the structure on certain datasets. Under the same noise degree, feature noise brings more performance degradation compared with structure noise; 2) Interestingly, while most existing GSL methods rely on feature similarity between pairs of nodes to learn graph structure, they still exhibit good robustness when facing noisy node features; 3) edge-oriented algorithms (e.g., ProGNN) show stronger feature robustness, because they optimize adjacency matrix directly, and have less dependence on pairs of node features.
4.6 Efficiency and scalability analysis
In this section, we analyze the efficiency and scalability of GSL algorithms on Cora, Citeseer, and Pubmed datasets. For time efficiency, we evaluate the efficiency of the algorithms by measuring the time it takes for them to converge, i.e., achieve the best performance on the validation set. For scalability, we set all models to their dense version to ensure a fair comparison. As shown in Figure 5, GSL algorithms generally have higher time and space complexity compared to GCN. This limitation restricts the application of GSL on large-scale graphs. We can observe that some algorithms (e.g., GRCN) can achieve relatively good performance improvement with less complexity increase. Besides, although some algorithms (e.g., IDGL, LDS, and SUBLIME) achieve remarkable effectiveness improvement, they largely increase the complexity of time and space.
5 Conclusion and Future Directions
In this paper, we give a brief introduction and overview of graph structure learning. Then we present the first Graph Structure Learning Benchmark (GSLB) consisting of 16 algorithms and 20 datasets
for various tasks. Based on GSLB, we conducted extensive experiments to reveal and analyze the performance of GSL algorithms in different scenarios and tasks. Through our comparative study, we find that GSL achieves promising results in heterophily, robustness, etc. The goal of this work is to understand the current state of development of GSL and provide insights for future research.
Notwithstanding the promising results that have been made, there are still some critical challenges and research directions worthy of future investigation.
- Insufficient scalability. Most existing works model the existence probability of edges based on node pairs, with a complexity of  $O(N^2)$ . This makes it challenging to employ GSL in large-scale graphs in real-world applications. Future work should focus on overcoming the limitations of GSL in terms of complexity.
- Surprising performance with few labels. We have observed that GSL learns denser and more distinct graph structures, which facilitates the propagation of supervision signals. Most existing GNNs that address few label problems are based on deep GNNs [25, 13] or semi-supervised approaches [6, 35, 20], without refining the graph structure. In the future, it would be worth exploring the combination of increasing the supervision signals and making the graph structure more suitable for propagating those signals.
- Excellent performance of unsupervised GSL in robustness. Some algorithms using self-supervised methods for learning graph structures exhibit excellent performance in robustness, which may be attributed to the avoidance of unreliable supervision signals. In the future, further exploration can be done to utilize unsupervised structure learning for designing defense models.
- Hard to apply on incomplete graphs. Most existing algorithms rely on pairwise node embeddings to generate the probability of edge existence. The underlying assumption is that all attributes of nodes on the graph are complete. However, it is common in practice that some nodes or all nodes have no features. Future research should address the challenges of structure learning on incomplete graphs.
This work was partially supported by the Research Grants Council of Hong Kong, No. 14202919 and No. 14205520.
[1] Tian Bian, Xi Xiao, Tingyang Xu, Peilin Zhao, Wenbing Huang, Yu Rong, and Junzhou Huang. Rumor detection on social media with bi-directional graph convolutional networks. In Proceedings of the AAAI conference on artificial intelligence, volume 34, pages 549-556, 2020. 1
[2] Karsten M Borgwardt, Cheng Soon Ong, Stefan Schonauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(suppl_1):i47-i56, 2005. 2, 4
[3] Chen Cai and Yusu Wang. A simple yet effective baseline for non-attributed graph classification. arXiv preprint arXiv:1811.03508, 2018. 2, 4
[4] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in neural information processing systems, 33:19314-19326, 2020. 2, 4, 16
[5] Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2): 786-797, 1991. 2, 4
[6] Kaize Ding, Jianling Wang, James Caverlee, and Huan Liu. Meta propagation networks for graph few-shot semi-supervised learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 6524-6531, 2022. 10
[7] Paul D Dobson and Andrew J Doig. Distinguishing enzyme structures from non-enzymes without alignments. Journal of molecular biology, 330(4):771-783, 2003. 15