SlowGuess/NeurIPS_2023_OCR huggingface dataset:
The p-value can provide a statistical threshold value, such as 0.05, to indicate whether  $H$  is significantly better than  $X$  for node classification. As seen in Table I, KR and GNB based metrics significantly outperform the existing homophily metrics, reducing the errors from at least 5 down to just 1 out of 18 cases. Besides, we only need a small set of the labels to calculate the p-value, which makes it better for sparse label scenario. Table 2 summarizes its advantages over the existing metrics. (See Appendix H for more details on classifier-based performance metrics, experiments on synthetic datasets, more detailed comparisons on small-scale and large-scale datasets, discrepancy between linear and non-linear models, results for symmetric renormalized affinity matrix and running time.)
In this paper, we provide a complete understanding of homophily by studying intra- and inter-class ND together. To theoretically investigate ND, we study the PBE and  $D_{\mathrm{NGJ}}$  of the proposed CSBM-H and analyze how graph filters, class variances and node degree distributions will influence the PBE and  $D_{\mathrm{NGJ}}$  curves and the FP, LP, HP regimes. We extend the investigation to broader settings with weaker assumptions and theoretically prove that ND is indeed affected by both intra- and inter-class ND. We also discover that the effect of HP filter depends on the relative center distance.
Empirically, through hypothesis testing, we corroborate that the performance of GNNs versus NNs is closely related to whether intra-class node embedding "distance" is smaller than inter-class node embedding "distance". We find that the p-value is a much more effective performance metric beyond homophily metrics on revealing the advantage and disadvantage of GNNs. Based on this observation, we propose classifier-based performance metric, which is a non-linear feature-based metric and can provide statistical threshold value.
Performance Metrics	Linear or Non-linear	Feature Dependency	Sparse Labels	Statistical Threshold
Hnode	linear	X	X	X
Hedge	linear	X	X	X
Hclass	linear	X	X	X
Hagg	linear	X	✓	X
HGE	linear	✓	✓	X
Hadj	linear	X	X	X
LI	linear	X	X	X
Classifier	both	✓	✓	✓
Table 2: Property comparisons of performance metrics
<sup>12</sup>We discuss this special case in Appendix H.4, together with some similar inconsistency instances found on large-scale datasets.
6 Reproducibility and Blogs
Code: https://github.com/SitaoLuan/When-Do-GNNs-Help.
- Blog in English (on Medium): https://medium.com/SitaoLuan/when-should-we-use-graph-neural-networks-for-node-classification-8ce77a772085.
- Blog in Chinese (on Zhihu): https://zhuanlan.zhihu.com/p/653631858
This work was partially supported by the Natural Sciences and Engineering Research Council of Canada (NSERC) Grant RGPIN-2023-04125, RGPIN 2389 and Canadian Institute for Advanced Research (CIFAR) Grant CIFAR FS20-126, CIFAR 10450. Minkai Xu thanks the generous support of Sequoia Capital Stanford Graduate Fellowship.
[1] S. Abu-El-Haija, B. Perozzi, A. Kapoor, N. Alipourfard, K. Lerman, H. Harutyunyan, G. Ver Steeg, and A. Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning, pages 21–29. PMLR, 2019.
[2] S. Arora, S. S. Du, W. Hu, Z. Li, R. R. Salakhutdinov, and R. Wang. On exact computation with an infinitely wide neural net. Advances in neural information processing systems, 32, 2019.
[3] A. Baranwal, K. Fountoulakis, and A. Jagannath. Graph convolution for semi-supervised classification: Improved linear separability and out-of-distribution generalization. arXiv preprint arXiv:2102.06966, 2021.
[4] P. W. Battaglia, J. B. Hamrick, V. Bapst, A. Sanchez-Gonzalez, V. Zambaldi, M. Malinowski, A. Tacchetti, D. Raposo, A. Santoro, R. Faulkner, et al. Relational inductive biases, deep learning, and graph networks. arXiv preprint arXiv:1806.01261, 2018.
[5] D. Bo, X. Wang, C. Shi, and H. Shen. Beyond low-frequency information in graph convolutional networks. arXiv preprint arXiv:2101.00797, 2021.
[6] J. Chen, S. Chen, J. Gao, Z. Huang, J. Zhang, and J. Pu. Exploiting neighbor effect: Convag-. agnostic gnn framework for graphs with heterophily. IEEE Transactions on Neural Networks and Learning Systems, 2023.
[7] E. Chien, J. Peng, P. Li, and O. Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations. https://openreview.net/forum, 2021.
[8] I. Csiszár. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146-158, 1975.
[9] R. B. Davies. Numerical inversion of a characteristic function. Biometrika, 60(2):415-417, 1973.
[10] R. B. Davies. Algorithm as 155: The distribution of a linear combination of  $\chi$  2 random variables. Applied Statistics, pages 323-333, 1980.
[11] Y. Deshpande, S. Sen, A. Montanari, and E. Mossel. Contextual stochastic block models. Advances in Neural Information Processing Systems, 31, 2018.
[12] L. Devroye, L. Györfi, and G. Lugosi. A probabilistic theory of pattern recognition, volume 31. Springer Science & Business Media, 2013.
[13] D. Dowson and B. Landau. The frechet distance between multivariate normal distributions. Journal of multivariate analysis, 12(3):450-455, 1982.
[14] V. N. Ekambaram. Graph structured data viewed through a fourier lens. University of California, Berkeley, 2014.
[15] A. Faragó and G. Lugosi. Strong universal consistency of neural network classifiers. IEEE Transactions on Information Theory, 39(4):1146-1151, 1993.
[40] P. Massart. Concentration inequalities and model selection: Ecole d'Eté de Probabilités de Saint-Flour XXXIII-2003. Springer, 2007.
[41] A. G. d. G. Matthews, M. Rowland, J. Hron, R. E. Turner, and Z. Ghahramani. Gaussian process behaviour in wide deep neural networks. arXiv preprint arXiv:1804.11271, 2018.
[42] M. McPherson, L. Smith-Lovin, and J. M. Cook. Birds of a feather: Homophily in social networks. Annual review of sociology, 27(1):415-444, 2001.
[43] M. Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning. MIT press, 2018.
[44] I. Olkin and F. Pukelsheim. The distance between two random vectors with given dispersion matrices. Linear Algebra and its Applications, 48:257-263, 1982.
[45] L. Pardo. Statistical inference based on divergence measures. CRC press, 2018.
[46] H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287, 2020.
[47] O. Platonov, D. Kuznedelev, A. Babenko, and L. Prokhorenkova. Characterizing graph datasets for node classification: Beyond homophily-heterophily dichotomy. arXiv preprint arXiv:2209.06177, 2022.
[48] J. Tanton. Encyclopedia of mathematics. Facts On File, Inc, 2005.
[49] A. Tsitsulin, B. Rozemberczki, J. Palowitch, and B. Perozzi. Synthetic graph generation to benchmark graph learning. arXiv preprint arXiv:2204.01376, 2022.
[50] P. Velickovic, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv, abs/1710.10903, 2017.
[51] Y. Wang, K. Yi, X. Liu, Y. G. Wang, and S. Jin. Acmp: Allen-cahn message passing for graph neural networks with particle phase transition. arXiv preprint arXiv:2206.05437, 2022.
[52] R. Wei, H. Yin, J. Jia, A. R. Benson, and P. Li. Understanding non-linearity in graph neural networks from the bayesian-inference perspective. arXiv preprint arXiv:2207.11311, 2022.
[53] F. Wu, T. Zhang, A. H. d. Souza Jr, C. Fifty, T. Yu, and K. Q. Weinberger. Simplifying graph convolutional networks. arXiv preprint arXiv:1902.07153, 2019.
[54] Y. Yan, M. Hashemi, K. Swersky, Y. Yang, and D. Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. arXiv preprint arXiv:2102.06462, 2021.
[55] J. Zhu, R. A. Rossi, A. Rao, T. Mai, N. Lipka, N. K. Ahmed, and D. Koutra. Graph neural networks with heterophily. arXiv preprint arXiv:2009.13566, 2020.
[56] J. Zhu, R. A. Rossi, A. Rao, T. Mai, N. Lipka, N. K. Ahmed, and D. Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pages 11168-11176, 2021.
[57] J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems, 33, 2020.