SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection, we conduct a rigorous analysis elucidating primary causes for performance disparity across different node subgroups with distinct structural patterns. Drawing inspiration from the discriminative metric described in Section 3.2, we identify two key factors for satisfying test performance: (1) test node  $u$  should have a close feature distance  $\min_{v \in V_{\mathrm{tr}}} \| \mathbf{f}_u - \mathbf{f}_v \|$  to training nodes  $V_{\mathrm{tr}}$ , indicating that test nodes can be greatly influenced by training nodes. (2) With identifying the closest training node  $v$ , nodes  $u$  and  $v$  should be more likely to share the same class, where  $\sum_{c_i \in \mathcal{C}} |\mathbf{P}(y_u = c_i | \mathbf{f}_u) - \mathbf{P}(y_v = c_i | \mathbf{f}_v)|$  is required to be small. The second factor, focusing on whether two close nodes are in the same class, is dependent on the homophily ratio difference  $|h_u - h_v|$ , as shown in Lemma 1. Notably, since training nodes are randomly sampled, their structural patterns are likely to be the majority one. Therefore, training nodes will show a smaller homophily ratio difference with majority test nodes sharing the same majority pattern than minority test nodes, resulting in the performance disparity in distinct structural patterns. We substantiate the above intuitions with controllable synthetic experiments in Appendix B.
To rigorously examine the role of aggregated feature distance and homophily ratio difference in performance disparity, we derive a non-i.i.d. PAC-Bayesian GNN generalization bound, based on the Subgroup Generalization bound of Deterministic Classifier [46]. We begin by stating key assumptions on graph data and GNN model to clearly delineate the scope of our theoretical analysis. All remaining assumptions, proof details, and background on PAC-Bayes analysis can be found in Appendix F. Moreover, a comprehensive introduction on the generalization ability on GNN can be found in A.
Definition 2 (Generalized CSBM-S model). Each node subgroup  $V_{m}$  follows the CSBM distribution  $V_{m} \sim \text{CSBM}(\pmb{\mu}_{1}, \pmb{\mu}_{2}, p^{(i)}, q^{(i)})$ , where different subgroups share the same class mean but different intra-class and inter-class probabilities  $p^{(i)}$  and  $q^{(i)}$ . Moreover, node subgroups also share the same degree distribution as  $p^{(i)} + q^{(i)} = p^{(j)} + q^{(j)}$ .
Instead of CSBM-S model with one homophilic and heterophilic pattern, we take the generalized CSBM-S model assumption, allowing more structural patterns with different levels of homophily.
Assumption 1 (GNN model). We focus on SGC [16] with the following components: (1) a one-hop mean aggregation function  $g$  with  $g(X, G)$  denoting the output. (2) MLP feature transformation  $f(g_{i}(X, G); W_{1}, W_{2}, \dots, W_{L})$ , where  $f$  is a ReLU-activated L-layer MLP with  $W_{1}, \dots, W_{L}$  as parameters for each layer. The largest width of all the hidden layers is denoted as  $b$ .
Notably, despite analyzing simple GNN architecture theoretically, similar with [46, 13, 47], our theory analysis could be easily extended to the higher-order case with empirical success across different GNN architectures shown in Section 3.4.
Our main theorem is based on the PAC-Bayes analysis which typically aims to bound the generalization gap between the expected margin loss  $\mathcal{L}_m^0$  on test subgroup  $V_{m}$  for a margin 0 and the empirical margin loss  $\widehat{\mathcal{L}}_{\mathrm{tr}}^{\gamma}$  on train subgroup  $V_{\mathrm{tr}}$  for a margin  $\gamma$ . Those losses are generally utilized in PAC-Bayes analysis[48-51]. More details are found in Appendix F. The formulation is shown as follows:
Theorem 1 (Subgroup Generalization Bound for GNNs). Let  $\tilde{h}$  be any classifier in the classifier family  $\mathcal{H}$  with parameters  $\{\widetilde{W_l}\}_{l = 1}^L$ . For any  $0 < m \leq M$ ,  $\gamma \geq 0$ , and large enough number of the training nodes  $N_{tr} = |V_{tr}|$ , there exist  $0 < \alpha < \frac{1}{4}$  with probability at least  $1 - \delta$  over the sample of
$$
y ^ {t r} := \left\{y _ {i} \right\} _ {i \in V _ {t r}}, w e h a v e:
$$
$$
\mathcal {L} _ {m} ^ {0} (\tilde {h}) \leq \widehat {\mathcal {L}} _ {t r} ^ {\gamma} (\tilde {h}) + O \left(\underbrace {\frac {K \rho}{\sqrt {2 \pi} \sigma} (\epsilon_ {m} + | h _ {t r} - h _ {m} | \cdot \rho)} _ {(a)} + \underbrace {\frac {b \sum_ {l = 1} ^ {L} \| \widetilde {W} _ {l} \| _ {F} ^ {2}}{(\gamma / 8) ^ {2 / L} N _ {t r} ^ {\alpha}} (\epsilon_ {m}) ^ {2 / L}} _ {(b)} + \mathbf {R}\right) \quad (3)
$$
The bound is related to three terms: (a) describes both large homophily ratio difference  $|h_{\mathrm{tr}} - h_m|$  and large aggregated feature distance  $\epsilon = \max_{j \in bV_m} \min_{i \in V_{\mathrm{tr}}} \| g_i(X, G) - g_j(X, G) \|_2$  between test node subgroup  $V_m$  and training nodes  $V_{\mathrm{tr}}$  lead to large generalization error.  $\rho = \| \pmb{\mu}_1 - \pmb{\mu}_2 \|$  denotes the original feature separability, independent of structure.  $K$  is the number of classes. (b) further strengthens the effect of nodes with the aggregated feature distance  $\epsilon$ , leads to a large generalization error. (c)  $\mathbf{R}$  is a term independent with aggregated feature distance and homophily ratio difference, depicted as  $\frac{1}{N_{\mathrm{tr}}^{1-2\alpha}} + \frac{1}{N_{\mathrm{tr}}^{2\alpha}} \ln \frac{LC(2B_m)^{1/L}}{\gamma^{1/L}\delta}$ , where  $B_m = \max_{i \in V_{\mathrm{tr}} \cup V_m} \| g_i(X, G) \|_2$  is the maximum feature norm.  $\mathbf{R}$  vanishes as training size  $N_0$  grows. Proof details are in Appendix F
Our theory suggests that both homophily ratio difference and aggregated feature distance to training nodes are key factors contributing to the performance disparity. Typically, nodes with large homophily ratio difference and aggregated feature distance to training nodes lead to performance degradation.
3.4 Performance Disparity Across Node Subgroups on Real-World Datasets
(b) Ogbn-arxiv  $(h = 0.63)$
Figure 5: Test accuracy disparity across node subgroups by aggregated-feature distance and homophily ratio difference to training nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. A clear performance decrease tendency can be found from subgroups 1 to 5 with increasing differences to training nodes.
(d) Squirrel  $(h = 0.25)$
(a) PubMed  $(h = 0.79)$
Figure 6: Test accuracy disparity across node subgroups by aggregated-feature distance to train nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. A clear performance decrease tendency can be found from subgroups 1 to 5 with increasing differences to training nodes.
To empirically examine the effects of theoretical analysis, we compare the performance on different node subgroups divided with both homophily ratio difference and aggregated feature distance to training nodes with popular GNN models including GCN [2], SGC [16], GAT [11], GCNII [52], and GPRGNN [53]. Typically, test nodes are partitioned into subgroups based on their disparity scores to the training set in terms of both 2-hop homophily ratio  $h_i^{(2)}$  and 2-hop aggregated features  $\mathbf{F}^{(2)}$  obtained by  $\mathbf{F}^{(2)} = (\tilde{\mathbf{D}}^{-1}\tilde{\mathbf{A}})^{2}\mathbf{X}$ , where  $\tilde{\mathbf{A}} = \mathbf{A} + \mathbf{I}$  and  $\tilde{\mathbf{D}} = \mathbf{D} + \mathbf{I}$ . For a test node  $i$ , we measure the node disparity by (1) selecting the closest training node  $v = \arg \min_{v\in V_0}||\mathbf{F}_u^{(2)} - \mathbf{F}_v^{(2)}||$  (2) then calculating the disparity score  $s_u = ||\mathbf{F}_u^{(2)} - \mathbf{F}_v^{(2)}||_2 + |h_u^{(2)} - h_v^{(2)}|$ , where the first and the second terms correspond to the aggregated-feature distance and homophily ratio differences,
Figure 7: Test accuracy disparity across node subgroups by homophily ratio difference to train nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. A clear performance decrease tendency can be found from subgroups 1 to 5 with increasing differences to training nodes.
respectively. We then sort test nodes in terms of the disparity score and divide them into 5 equal-binned subgroups accordingly. Performance on different node subgroups is presented in Figure 5 with the following observations. Obs.1: We note a clear test accuracy degradation with respect to the increasing differences in aggregated features and homophily ratios. Furthermore, we investigate on the individual effect of aggregated feature distance and homophily ratio difference in Figure 6 and 7, respectively. An overall trend of performance decline with increasing disparity score is evident though some exceptions are present. Obs.2: When only considering the aggregated feature distance, there is no clear trend among groups 1, 2, and 3 on GCN, SGC, and GAT on heterophilic datasets. Obs.3: When only considering the homophily ratio difference, there is no clear trend among groups 1, 2, and 3 across four datasets. These observations underscore the importance of both aggregated-feature distance and homophily ratio differences in shaping GNN performance disparity. Combining these factors together provides a more comprehensive and accurate understanding of the reason for GNN performance disparity. For a more comprehensive analysis, we further substantiate our finding involving higher-order information and a wider array of datasets in Appendix J.1.
Summary In this section, we study GNN performance disparity on nodes with distinct structural patterns and uncover its underlying causes. We primarily investigate the impact of aggregation, the key component in GNNs, on nodes with different structural patterns in Sections 3.1 and 3.2. We observe that aggregation effects vary across nodes with different structural patterns, notably enhancing the discriminative ability on majority nodes. These observed performance disparities inspire us to identify crucial factors contributing to GNN performance disparities across nodes with a non-i.i.d PAC-Bayes bound in Section 3.3. The theoretical analysis indicates that test nodes with larger aggregated feature distances and homophily ratio differences with training nodes experience performance degradation. We substantiate our findings on real-world datasets in Section 3.4.
4 Implications of graph structural disparity
In this section, we illustrate the significance of our findings on structural disparity via (1) elucidating the effectiveness of existing deeper GNNs (2) unveiling an over-looked aspect of distribution shift on graph out-of-distribution (OOD) problem, and introducing a new OOD scenario accordingly. Experimental details and discussions on more implications are in Appendix G and O, respectively.
4.1 Elucidate the effectiveness of Deeper GNNs
Deeper GNNs [53, 52, 15, 54-57] enable each node to capture more complex higher-order graph structure than vanilla GCN, via reducing the over-smoothing problem [58-60, 55, 61] Deeper GNNs empirically exhibit overall performance improvement, as demonstrated in Appendix G.2. Nonetheless, which structural patterns deeper GNNs can exceed and the reason for its effectiveness remain unclear. To investigate this problem, we compare vanilla GCN with different deeper GNNs, including GPRGNN[53], APPNP[15], and GCNII[52], on node subgroups with varying homophily ratios, adhering the same setting with Figure 3. Experimental results are shown in Figure 8. We can observe that deeper GNNs primarily surpass GCN on minority node subgroups with slight performance trade-offs on the majority node subgroups. We conclude that the effectiveness of deeper GNNs majorly contributes to improved discriminative ability on minority nodes. Additional results on more datasets and significant test are in Appendix I and M.
(b) Ogbn-arxiv  $(h = 0.63)$
Figure 8: Performance comparison between GCN and deeper GNNs. Each bar represents the accuracy gap on a specific node subgroup exhibiting homophily ratio within range specified on x-axis.
Having identified where deeper GNNs excel, reasons why effectiveness primarily appears in the minority node group remain elusive. Since the superiority of deeper GNNs stems from capturing higher-order information, we further investigate how higher-order homophily ratio differences vary on the minority nodes, denoted as,  $|h_u^{(k)} - h_v^{(k)}|$ , where node  $u$  is the test node, node  $v$  is the closest train node to test node  $u$ . We concentrate on analyzing these minority nodes  $V_{\mathrm{mi}}$  in terms of default one-hop homophily ratio  $h_u$  and examine how  $\sum_{u \in V_{\mathrm{mi}}} |h_u^{(k)} - h_v^{(k)}|$  varies with different  $k$  orders. Experimental results are shown in Figure 9, where a decreasing trend of homophily ratio difference is observed along with more neighborhood hops. The smaller homophily ratio difference leads to smaller generalization errors with better performance. This observation is consistent with [20], where heterophilic nodes in heterophilic graphs exhibit large higher-order homophily ratios, implicitly leading to a smaller homophily ratio difference.
Figure 9: Multiple hop homophily ratio differences between training and minority test nodes.
4.2 A new graph out-of-distribution scenario
The graph out-of-distribution (OOD) problem refers to the underperformance of GNN due to distribution shifts on graphs. Many Graph OOD scenarios [62-65, 24, 66, 46, 67], e.g., biased training labels, time shift, and popularity shift, have been extensively studied. These OOD scenarios can be typically categorized into covariate shift with  $\mathbf{P}^{\mathrm{train}}(\mathbf{X}) \neq \mathbf{P}^{\mathrm{test}}(\mathbf{X})$  and concept shift [68-70] with  $\mathbf{P}^{\mathrm{train}}(\mathbf{Y}|\mathbf{X}) \neq \mathbf{P}^{\mathrm{test}}(\mathbf{Y}|\mathbf{X})$ .  $\mathbf{P}^{\mathrm{train}}(\cdot)$  and  $\mathbf{P}^{\mathrm{test}}(\cdot)$  denote train and test distributions, respectively. Existing graph concept shift scenarios [62, 66] introduce different environment variables  $e$  resulting in  $\mathbf{P}(\mathbf{Y}|\mathbf{X}, e_{\mathrm{train}}) \neq \mathbf{P}(\mathbf{Y}|\mathbf{X}, e_{\mathrm{test}})$  leading to spurious correlations. To address existing concept shifts, algorithms [62, 71] have been developed to capture the environment-invariant relationship  $\mathbf{P}(\mathbf{Y}|\mathbf{X})$ . Nonetheless, existing concept shift settings overlook the scenario where there is not a unique environment-invariant relationship  $\mathbf{P}(\mathbf{Y}|\mathbf{X})$ . For instance,  $\mathbf{P}(\mathbf{Y}|\mathbf{X}_{\mathrm{homo}})$  and  $\mathbf{P}(\mathbf{Y}|\mathbf{X}_{\mathrm{hete}})$  can be different, indicated in Section 3.1.  $\mathbf{X}_{\mathrm{homo}}$  and  $\mathbf{X}_{\mathrm{hete}}$  correspond to features of nodes in homophilic and heterophilic patterns. Notably, homophilic and heterophilic patterns are crucial task knowledge that cannot be recognized as the irrelevant environmental variable. Consequently, we find that homophily ratio difference between train and test sets could be an important factor leading to an overlook concept shift, namely, graph structural shift. Notably, structural patterns cannot be considered as environment variables given their integral role in node classification task. The practical implications of this concept shift are substantiated by the following scenarios: (1) graph structural shift frequently occurs in most graphs, with a performance degradation in minority nodes, as depicted in Figure 3. (2) graph structural shift hides secretly in existing graph OOD scenarios. For instance, the FaceBook-100 dataset [62] reveals a substantial homophily ratio difference between train and test sets, averaging 0.36. This discrepancy could be the primary cause of OOD performance deterioration since the exist OOD algorithms [62, 72] that neglect such a concept shift can only attain a minimal average performance gain of  $0.12\%$ . (3) graph structural shift is a recurrent phenomenon in numerous real-world applications where new nodes in graphs may exhibit distinct structural patterns. For example, in a recommendation system, existing users with rich data can receive well-personalized recommendations in the exploitation stage (homophily), while new users with less data may receive diverse recommendations during the exploration stage (heterophily).
Given the prevalence and importance of the graph structural shift, we propose a new graph OOD scenario emphasizing this concept shift. Specifically, we introduce a new data split on existing datasets, namely Cora, CiteSeer, PubMed, Ogbn-Arxiv, Chameleon, and Squirrel, where majority nodes are selected for train and validation, and minority ones for test. This data split strategy highlights the
Table 1: Performance (Accuracy) on the proposed OOD split.
Pubmed	Ogbn-Arxiv	Squirrel	Chameleon
GCN(i.i.d)	89.18±0.15	72.99±0.14	58.09±0.71	75.09±0.79
GCN	51.04±0.16	34.94±0.07	32.13±4.93	43.35±3.47
MLP	68.38±0.43	33.17±0.37	24.57±0.77	34.78±4.97
GLNN	67.51±0.25	35.89±0.14	31.51±0.70	47.01±1.09
GCNII	67.76±0.36	36.81±0.14	37.15±1.39	41.25±2.03
GPRGNN	57.24±0.18	34.95±0.43	42.43±7.71	35.27±7.67
SRGNN	57.91±0.10	40.37±1.65	37.62±1.74	42.09±0.43
EERM	65.37±1.35	34.23±0.46	40.93±0.57	45.84±1.05
EERM(II)	67.59±0.91	40.28±0.84	44.31±0.40	48.59±0.78
homophily ratio difference and the corresponding concept shift. To better illustrate the challenges posed by our new scenario, we conduct experiments on models including GCN, MLP, GLNN, GPRGNN, and GCNII. We also include graph OOD algorithms, SRGNN [63] and EERM [62], with GCN encoders. EERM(II) is a variant of EERM with a GCNII encoder For a fair comparison, we show GCN performance on an i.i.d. random split, GCN(i.i.d.), sharing the same node sizes for train, validation, and test. Results are shown in Table 1 while additional ones are in Appendix G.4. Following observations can be made: Obs.1: The performance degradation can be found by comparing OOD setting with i.i.d. one across four datasets, confirming OOD issue existence. Obs.2: MLP-based models and deeper GNNs generally outperform vanilla GCN, demonstrating the superiority on minority nodes. Obs.3: Graph OOD algorithms with GCN encoders struggle to yield good performance across datasets, indicating a unique challenge over other Graph OOD scenarios. This primarily stems from the difficulty in learning both accurate relationships on homophilic and heterophilic nodes with distinct  $\mathbf{P}(\mathbf{Y}|\mathbf{X})$ . Nonetheless, it can be alleviated by selecting a deeper GNN encoder, as the homophily ratio difference may vanish in higher-order structure information, with reduced concept shift. Obs.4: EERM(II), EERM with GCNII, outperforms the one with GCN. Observations suggest that GNN architecture plays an indispensable role in addressing graph OOD issues, highlighting the new direction.
5 Conclusion & Discussion
In conclusion, this work provides crucial insights into GNN performance meeting structural disparity, common in real-world scenarios. We recognize that aggregation exhibits different effects on nodes with structural disparity, leading to better performance on majority nodes than those in minority. The understanding also serves as a stepping stone for multiple graph applications.
Our exploration majorly focuses on common datasets with clear majority structural patterns while real-world scenarios, offering more complicated datasets, posing new challenges. Additional experiments are conducted on Actor, IGB-tiny, Twitch-gamer, and Amazon-ratings. Dataset details and experiment results are in Appendix G.3 and Appendices H-K, respectively. Despite our understanding is empirically effective on most datasets, further research and more sophisticated analysis are still necessary. Discussions on the limitation and broader impact are in Appendix N and O, respectively.
We want to thank Xitong Zhang, He Lyu, Wenzhuo Tang at Michigan State University, Yuanqi Du at Cornell University, Haonan Wang at the National University of Singapore, and Jianan Zhao at Mila for their constructive comments on this paper.
This research is supported by the National Science Foundation (NSF) under grant numbers CNS 2246050, IIS1845081, IIS2212032, IIS2212144, IIS2153326, IIS2212145, IOS2107215, DUE 2234015, DRL 2025244 and IOS2035472, the Army Research Office (ARO) under grant number W911NF-21-1-0198, the Home Depot, Cisco Systems Inc, Amazon Faculty Award, Johnson&Johnson, JP Morgan Faculty Award and SNAP.
[1] Yao Ma and Jiliang Tang. Deep learning on graphs. Cambridge University Press, 2021.
A Related work 19
B Investigation on the effectiveness of homophily ratio difference with targeted synthetic edge addition algorithm. 20
B.1 Targeted heterophilic & homophilic edge addition algorithms 20
B.2 Detailed experiment results 21
B.3 Details on the generated graph 22
C Instance-level discriminative analysis 23
D Effects of aggregation on nodes in different classes with structural disparity 25
D.1 Linear separability analysis based on CSBM model 25
D.2 Linear separability experiment on synthetic CSBM dataset 27
D.3 Proof details of linear seperability within the same pattern 27
D.4 Proof details of linear separability between different structural patterns 29
E Proof details of the conditional probability difference for nodes with the same feature but different structural patterns 30
F Proof details of PAC-Bayes Bound 33
F.1 Background Knowledge on PAC-Bayes Analysis 33
F.2 PAC-Bayesian Analysis on Subgroup Generalization bound of Deterministic classifier 33
F.3 Proof details of Expected Loss Discrepancy 34
F.4 Proof details of subgroup generalization Bound for GNNs 38
GExperiment details 40
G.1 Hardware & Software Environment 40
G.2 Model details & hyperparameter settings & results 40
G.3 Dataset details 41
G.4 OOD dataset statistics and details 43
H Additional results on performance comparison between GCN and MLP-based models 46
I Additional results on performance comparison between GCN and deeper GNN models 47
J Additional Experiment on performance disparity across subgroups 47
J.1 Additional investigation on higher-order neighborhood 49
J.2 Additional results on more datasets 49
K Additional investigation on the discriminative ability of GCN 50
L Significant test between GCN and MLP-based models 51
M Significant test between GCN and deeper GNN models 51
N Limitation 52
O Broader Impact 53
P Future work 54
Graph Neural Networks (GNNs) have emerged as a powerful technique in Deep Learning, specifically designed for graph-structured data. They address the limitations of traditional neural networks in dealing with irregular data structures. GNNs learn node representations by aggregating neighborhood and transforming features recursively. The node representation can then be successfully utilized to a wide range of graph-related downstream tasks [2, 5, 3, 6-8, 73, 74].
The aggregation mechanism in GNNs is often viewed as feature smoothing [61, 75, 76]. This perspective leads some recent studies [26, 53, 20, 77] claiming that GNN models are overly reliant on homophilic patterns and unsuited to capturing heterophilic patterns. To accommodate heterophilic graphs, recent works propose carefully designed GNN architectures, e.g., CPGNN [78], GGNN [79], GPRGNN [53], GCNII [52] GBK-GNN [27], ACM-GNN [19], Bernnet [80].
More recent analyses on GNNs [18, 13, 19] indicate that even GCN [2], a vanilla GNN, can deliver strong performance on certain heterophilic graphs. According to these findings, new metrics and understandings [81, 82, 21, 19, 13, 28] have been proposed to further expose the remaining weaknesses of GNNs.
There is a concurrent work [28] investigates when Graph Neural Networks help with node classification with a comparison between GNN and MLP. Nonetheless, our work is distinct from [28] in two primary ways: