SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Here  $0 < \delta < 1$  is a parameter that controls the ratio of negative edge weights.  $\langle \cdot ,\cdot \rangle$  denotes the inner product of two vectors. Although this strategy appears simple, it is highly effective. The critical point here is that we do not need to know the exact classes of the two nodes and we just care about whether they belong to the same class. Let  $p$  be the accuracy of the estimator and assume that there is a uniform distribution chance of being any incorrect class, namely  $\frac{1 - p}{c - 1}$
The estimator has a probability of  $\frac{(1 - p)(pc + c - 2)}{(c - 1)^2}$  to identify any two heterophilous nodes as homophilous nodes. This probability is less than  $1 - p$ , provided that  $c \geq 3$  and  $p < 1$ . For instance, when the accuracy of the estimator is  $50\%$  and  $c = 5$ , for heterophilous  $(i,j)$ , the probability of assigning them different labels is about  $82.8\%$ , which is significantly higher than  $50\%$ . In our experiments, we use simple estimators including GCN and MLP for fairness.
The succeeding section discusses the design of  $\tau_{i,j}$ , which can be expressed as  $\tau_{i,j} = c_{i,j} \cdot \tau$ , where
$0 < c_{i,j} < 1$  is a parameter. We adopt a widely accepted definition of outliers [Yeh, 2007],
Edge Hom.	#Nodes	#Edges	#Features	#Classes
Texas	0.21	183	295	1,703	5
Wisconsin	0.11	251	466	1,703	5
Cornell	0.30	183	280	1,703	5
Actor	0.22	7,600	26,752	931	5
Squirrel	0.22	5,201	198,493	2,089	5
Chameleon	0.23	2,277	31,421	2,325	5
Cora	0.81	2,708	5,278	1,433	6
Citeseer	0.74	3,327	4,676	3,703	7
Pubmed	0.80	19,717	44,327	500	3
Penn94	0.47	41,554	1,362,229	5	2
arXiv-year	0.22	169,343	1,166,243	128	5
genius	0.61	421,961	984,979	12	2
Table 1: Dataset statistics.
$$
\tau = q _ {3} + 1. 5 \times \left(q _ {3} - q _ {1}\right), \tag {12}
$$
Texas	Wiscon.	Cornell	Actor	Squir.	Chanel.	Cora	Citeseer	Pubmed
MLP	80.81±4.75	85.29±3.31	81.89±6.40	36.53±0.70	28.77±1.56	46.21±2.99	75.69±2.00	74.02±1.90	87.16±0.37
GCN	55.14±5.16	51.76±3.06	60.54±5.30	27.32±1.10	53.43±2.01	64.82±2.24	86.98±1.27	76.50±1.36	88.42±0.50
GAT	52.16±6.63	49.41±4.09	61.89±5.05	27.44±0.89	40.72±1.55	60.26±2.50	87.30±1.10	76.55±1.23	86.33±0.48
H2GCN	84.86±7.23	87.65±4.98	82.70±5.28	35.70±1.00	36.48±1.86	60.11±2.15	87.87±1.20	77.11±1.57	89.49±0.38
GPR-GNN	78.38±4.36	82.94±4.21	80.27±8.11	34.63±1.22	31.61±1.24	46.58±1.71	87.95±1.18	77.13±1.67	87.54±0.38
WRGAT	83.62±5.50	86.98±3.78	81.62±3.90	36.53±0.77	48.85±0.78	65.24±0.87	88.20±2.26	76.81±1.89	88.52±0.92
GloGNN++	84.05±4.90	88.04±3.22	85.95±5.10	37.70±1.40	57.88±1.76	71.21±1.84	88.33±1.09	77.22±1.78	89.24±0.39
GGCN	84.86±4.55	86.86±3.29	85.68±6.63	37.54±1.56	55.17±1.58	71.14±1.84	87.95±1.05	77.14±1.45	89.15±0.37
ACM-GCN	87.84±4.40	88.43±3.22	85.14±6.07	36.28±1.09	54.40±1.88	66.93±1.85	87.91±0.95	77.32±1.70	90.00±0.52
LINKX	74.60±8.37	75.49±5.72	77.84±5.81	36.10±1.55	61.81±1.80	68.42±1.38	84.64±1.13	73.19±0.99	87.86±0.77
OGNN	86.22±4.12	88.04±3.63	87.03±4.73	37.99±1.00	62.44±1.96	72.28±2.29	88.37±0.75	77.31±1.73	90.15±0.38
LRGNN	90.27±4.49	88.23±3.54	86.48±5.65	37.34±1.78	74.38±1.96	79.16±2.05	88.33±0.89	77.53±1.31	90.24±0.64
Table 2: Node classification accuracy (%) on small real-world benchmark datasets. The best results are highlighted in bold, whereas runner-up results are underlined. Each experiment is executed 10 times.
GCN	GAT	GPR-GNN	GloGNN++	ACM-GCN	LINKX	OGNN	LRGNN
Penn94	82.47±0.27	81.53 ±0.55	81.38±0.16	85.74±0.42	82.52±0.96	84.71±0.52	83.31±0.54	86.48 ± 0.52
arXiv-year	46.02±0.26	46.05±0.51	45.07±0.21	54.79±0.25	47.37±0.59	56.00±1.34	54.49±0.29	55.68 ± 0.35
genius	87.42±0.37	55.80±0.87	90.05±0.31	90.91±0.13	80.33±3.91	90.77±0.27	88.52±0.45	91.13 ± 0.12
Table 3: Node classification accuracy (%) on large-scale datasets.
where  $q_{1}$  and  $q_{3}$  are the first and third quartiles, respectively. Mathematically,  $c_{i,j}$  is defined as
$$
c _ {i, j} = a _ {1} \left(\| \hat {\mathbf {Y}} _ {i,:} \| _ {2} ^ {2} - \frac {1}{c}\right) \left(\| \hat {\mathbf {Y}} _ {j,:} \| _ {2} ^ {2} - \frac {1}{c}\right) + a _ {2} \log \left(e - 1 + | \tilde {\mathbf {A}} _ {i, j} |\right) + a _ {3} \text {S i g m o i d} \left(a ^ {T} \operatorname {R e L U} \left([ h _ {i} ^ {(0)} | | h _ {j} ^ {(0)} ]\right)\right), \tag {13}
$$
where  $[a_1, a_2, a_3] = \operatorname{Softmax}(\mathbf{W}_a), \mathbf{W}_a \in \mathbb{R}^{1 \times 3}$ ,  $||$  denotes vector concatenation, and  $e$  is the natural logarithm.
The Euclidean norm term measures the quality of the pseudo label since a uniformly distributed  $\| \hat{\mathbf{Y}}_{i,:}\| _2^2$  suggests that the estimator has no confidence in the class of  $v_{i}$ . The absolute value term takes into account that extreme values are typically more reliable. For example, a zero  $\tilde{\mathbf{A}}_{i,j}$  indicates that the relationship between  $v_{i}$  and  $v_{j}$  is ambiguous. The third term employs graph attention [Velickovic et al., 2018, Brody et al., 2022] to adapt  $c_{i,j}$  accordingly.
This section evaluates the performance of LRGNN. We put the experimental results w.r.t. ablation study and robustness in the Supplement due to space limitation.
Datasets. We use three homophilous datasets including Cora, Citeseer and Pubmed [Yang et al., 2016], along with 6 heterophilous datasets released in Pei et al. [2020] and Rozemberczki et al. [2021], and three large-scale heterophilous graphs [Lim et al., 2021]. The training/validation/testing splits used in this paper are the same as [Li et al., 2022].
Baselines. We compare LRGNN with 11 baselines, including (1) classic GNN models: vanilla GCN [Kipf and Welling, 2017], and GAT [Velickovic et al., 2018]. (2) models specifically designed to handle heterophily:  $\mathrm{H}_2\mathrm{GCN}$  [Zhu et al., 2020], GPR-GNN [Chien et al., 2021], WRGAT [Suresh et al., 2021], LINKX [Lim et al., 2021], GGCN [Yan et al., 2021], ACM-GCN [Luan et al., 2021], GloGNN++ [Li et al., 2022], and Ordered GNN (OGNN) [Song et al., 2023]. (3) 2-layer MLP. We specifically choose GloGNN++ and ACM-GCN as they generally outperform the other variants proposed in their respective papers.
Node classification results. Table 2 and Table 3 provide a summary of the test accuracy of the tested methods on 12 datasets with diverse homophily ratios and scales. Some baselines experience out-of-memory errors on large datasets, thus the corresponding results are excluded. From the tables, we can
Figure 3: Recovery error. The lowest point is associated with a vertical line. The shaded region corresponds to a  $95\%$  confidence interval.
Figure 4: Visualization result of  $\mathbf{Z}$ . The entries have been grouped into two categories based on the label relationship between the two nodes.
draw several observations. Firstly, MLP is a strong baseline for heterophilous datasets, outperforming GCN and GAT on Texas, Wisconsin, and Cornell datasets. Secondly, methods designed specifically for heterophilous graphs generally perform better than MLP and traditional GNNs. Thirdly,  $\mathrm{H}_2\mathrm{GCN}$ , WRGAT, GGCN, ACM-GCN, GloGNN++, and LRGNN are the top performers on small datasets. However,  $\mathrm{H}_2\mathrm{GCN}$ , WRGAT, and GGCN lack scalability, thereby limiting their widespread adoption for large datasets. Fourthly, LRGNN performs the best in terms of the average rank, emerging as the winner or runner-up on all datasets except for Actor.
The results suggest that LRGNN consistently offers superior performance on both homophilous and heterophilous graphs. Notably, LRGNN achieved the highest score on Squirrel, with approximately  $19\%$  improvement over the runner-up score achieved by OGNN. The excellent performance on Squirrel and Chameleon could be attributed to the relatively large average node degrees of these two datasets, providing more observations to recover the label relationship matrix. Moreover, the performance of LRGNN on these three large datasets demonstrates the scalability of our model.
Sign inference accuracy w.r.t. operating rank  $q$ . We investigate the effect of the operating rank  $q$  on the recovery loss:  $err = \frac{1}{2n^2}\sum_{i,j}|sign((\mathbf{U}_*\mathbf{V}_*^T)_{i,j}) - sign(\hat{\mathbf{Z}}_{i,j})|$ . 3 shows that the error decreases initially when  $q$  increases, then it increases gradually after reaching the minimum point. The best result is consistently achieved when  $q$  is approximately 5, the number of classes. Empirically, this supports the use of low-rank approximation to predict the label relationship matrix: the appropriate dimensionality for the factors can be determined based on the number of classes. In a standard LRMF problem, the rank of the matrix of interest is inaccessible. Therefore, we have to select a sufficiently large  $q$  to ensure that  $q$  surpasses the rank, which guarantees that the solution to LRMF problem provides a solution to the LRMC problem. Nevertheless, selecting excessively large  $q$  can substantially increase the number of iterations required to converge.
Visualizing the predicted label relationship matrix  $\mathbf{Z}$ . There are two main desirable traits of  $\hat{\mathbf{Z}}$  we want  $\mathbf{Z}$  to possess. The first is that within-class entries are positive while between-class entries are negative. This ensures that nodes are embedded close to their homophilous nodes and distant from the heterophilous nodes in the embedding space. The second is that  $\hat{\mathbf{Z}}_{\mathrm{i},:}$  and  $\hat{\mathbf{Z}}_{\mathrm{j},:}$  are identical if they share the same label, so that  $\hat{\mathbf{Z}}_{\mathrm{i},:}\mathbf{X} = \hat{\mathbf{Z}}_{\mathrm{j},:}\mathbf{X}$ . Consequently, we visualize  $\mathbf{Z}$  in two different styles.
First, as shown in Figure 4, dividing the entries into two groups reveals that the distribution is similar to a Gaussian distribution. Although most entries converge around zero, within-class entries and between-class entries are positioned on opposite sides of the zero.
To illustrate the second trait, we visualize  $\mathbf{Z}$  with reordered rows and columns based on ground truth labels. As shown in Figure 5, the matrices exhibit the structure of a block diagonal matrix. But here the entries of sub-matrices along the diagonal are positive, whereas off-diagonal sub-matrices contain
Figure 5: Visualization result of  $\mathbf{Z}$ . Both row and column indices have been reordered based on the ground truth labels.
negative entries. On closer inspection of the figure, we notice that nodes in the same class share a high degree of similarity, with the rows' pixels showing comparable patterns. In conclusion,  $\mathbf{Z}$  exhibits both desirable traits of  $\hat{\mathbf{Z}}$ . This helps explain the effectiveness of LRGNN.
Figure 6: The first three subfigures are results on synthetic graphs and the last one is the result on corrupted Texas. Error bars indicate  $95\%$  confidence interval.
Results on synthetic graphs. To comprehensively evaluate the performance of LRGNN, we leverage random partition graphs [Kim and Oh, 2021] generated by the stochastic block model. The node features are sampled from Gaussian distributions where the centers of clusters are vertices of a hypercube. Notably, the distance between Gaussian distribution means is small, thus, it is challenging to differentiate between node features of distinct classes. This is evident by MLP's unsatisfactory performance on the synthetic graphs. We use 15 synthetic random graphs with varying node-level homophily ratios and average degrees. More information about the synthetic graphs can be found in the Supplement.
It can be observed from Figure 6 that, when the homophily ratio or node degree is low, LRGNN is the only model that outperforms MLP. Also, LRGNN performs significantly better than other models when the homophily ratio is below 0.5. These results suggest that LRGNN performs particularly well on random graphs, which we attribute to two reasons. Firstly, the observed entries of these graphs are uniformly distributed, meeting one of the conditions for exact recovery. Secondly, to derive aggregation coefficients, low-rank approximation methods utilize only the observed entries of  $\hat{\mathbf{Z}}$ , while neural network based methods rely on the raw features. However, in this case, node features are not that informative since the means of Gaussian distributions are close. To validate the second reason, we use the Texas dataset and degrad the quality of its features by adding Gaussian variables. We obtain a degraded feature matrix by  $\mathbf{X}'_{i,j} = \mathbf{X}_{i,j} + \epsilon_j$ , with  $\epsilon_j$  i.i.d. sampled from a Gaussian distribution  $\mathcal{N}(0,\sigma^2)$ . Note that the original features will be overwhelmed by the Gaussian random variables if a large  $\sigma$  is applied. We compare the performance of representative models that included GloGNN++, H2GCN, and MLP with that of LRGNN. It can be observed that, as the features get less informative, the performance of GloGNN++, H2GCN, and MLP decreases significantly, while LRGNN's accuracy remains above  $80\%$ . In addition to that, a large  $\sigma$  also makes their training process erratic, which is accompanied by significant error bars. These empirical results confirm our previous analysis.
This paper presents a method for extending GNNs to heterophilous graphs through the use of a label relationship matrix. In order to utilize the low-rank properties of weakly-balanced graphs, we propose a robust low-rank matrix approximation technique for the prediction of the label relationship matrix. The proposed LRGNN has been thoroughly evaluated through extensive experiments. We conduct extensive experiments to evaluate our proposed LRGNN. Our results indicate that LRGNN outperforms other baseline methods on both synthetic and real-world graphs. Notably, when the node degree distribution of the graph conforms to a uniform distribution, LRGNN exhibits significantly superior performance over other baseline methods.
The main theoretical support for our approach assumes a uniform distribution of node degrees. However, in practice, node degree distributions frequently follow a power-law distribution that is non-uniform. Although the main results of this paper are obtained using real-world graphs with non-uniform node degree distributions, it is unclear how this affects the theoretical results. Additionally, it is acknowledged that LRGNN's performance is affected by the accuracy of the generated pseudo labels. The supplementary material provides the corresponding experiment that indicates LRGNN demonstrates a non-sensitive response to this impact. However, further exploration and discussion are helpful to investigate the theoretical implications fully.
This work was partially supported by the National Key Research and Development Program of China (No. 2018AAA0100204), a key program of fundamental research from Shenzhen Science and Technology Innovation Commission (No. JCYJ20200109113403826), the Major Key Project of PCL (No. 2022ZD0115301), and an Open Research Project of Zhejiang Lab (NO.2022RC0AB04).
Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In ICML, volume 97 of Proceedings of Machine Learning Research, pages 21-29. PMLR, 2019.
Dzmitry Bahdanau, Kyunghyun Cho, and Yoshua Bengio. Neural machine translation by jointly learning to align and translate. In Yoshua Bengio and Yann LeCun, editors, ICLR 2015, 2015.
Deyu Bo, Xiao Wang, Chuan Shi, and Huawei Shen. Beyond low-frequency information in graph convolutional networks. In AAAI, pages 3950-3957. AAAI Press, 2021.
Shaked Brody, Uri Alon, and Eran Yahav. How attentive are graph attention networks? In ICLR. OpenReview.net, 2022.
A. M. Buchanan and Andrew W. Fitzgibbon. Damped newton algorithms for matrix factorization with missing data. In CVPR (2), pages 316-322. IEEE Computer Society, 2005.
Emmanuel J. Candès and Yaniv Plan. Matrix completion with noise. Proc. IEEE, 98(6):925-936, 2010. doi: 10.1109/JPROC.2009.2035722.
Emmanuel J. Candès and Terence Tao. The power of convex relaxation: near-optimal matrix completion. IEEE Trans. Inf. Theory, 56(5):2053-2080, 2010. doi: 10.1109/TIT.2010.2044061.
Dorwin Cartwright and Frank Harary. Structural balance: a generalization of heider's theory. Psychological review, 63(5):277, 1956.
Pei Chen. Optimization algorithms on subspaces: Revisiting missing data problem in low-rank matrix. Int. J. Comput. Vis., 80(1):125-142, 2008.
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. In ICLR (Poster). OpenReview.net, 2018.
T Wiberg. Computation of principal components when data are missing. In Proc. of Second Symp. Computational Statistics, pages 229-236, 1976.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In 7th International Conference on Learning Representations, ICLR 2019. OpenReview.net, 2019a.
Zenglin Xu, Feng Yan, and Yuan (Alan) Qi. Bayesian nonparametric models for multiway data analysis. IEEE Trans. Pattern Anal. Mach. Intell., 37(2):475-487, 2015.
Zenglin Xu, Bin Liu, Shandian Zhe, Haoli Bai, Zihan Wang, and Jennifer Neville. Variational random function model for network modeling. IEEE Trans. Neural Networks Learn. Syst., 30(1):318-324, 2019b.
Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. CoRR, abs/2102.06462, 2021.
Liang Yang, Mengzhe Li, Liang Liu, Bingxin Niu, Chuan Wang, Xiaochun Cao, and Yuanfang Guo. Diverse message passing for attribute with heterophily. In NeurIPS, pages 4751-4763, 2021.
Zhilin Yang, William W. Cohen, and Ruslan Salakhutdinov. Revisiting semi-supervised learning with graph embeddings. In ICML, volume 48 of JMLR Workshop and Conference Proceedings, pages 40-48. JMLR.org, 2016.
Arthur B. Yeh. A modern introduction to probability and statistics. Technometrics, 49(3):359, 2007. doi: 10.1198/tech.2007.s502.
Xin Zheng, Yixin Liu, Shirui Pan, Miao Zhang, Di Jin, and Philip S. Yu. Graph neural networks for graphs with heterophily: A survey. CoRR, abs/2202.07082, 2022.
Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. In NeurIPS, 2020.
Jiong Zhu, Ryan A. Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K. Ahmed, and Danai Koutra. Graph neural networks with heterophily. In AAAI, pages 11168-11176. AAAI Press, 2021.