SlowGuess/NeurIPS_2023_OCR huggingface dataset:
In this subsection, we conduct more experimental results on the datasets with more complicated with more diverse structural patterns, including Cora, CiteSeer, IGB-tiny, Twitch-gamers, Amazon-ratings, and Actor. A comprehensive discussion can be found in Appendix G.3. The performance on different node subgroups is presented in Figure 19. A consistent observation can be found with the expected test accuracy degradation trend in terms of larger feature distance and homophily ratio. the difference in most datasets, except for the Actor. The potential reason is that its structure could hardly provide useful information. The key evidence is that the vanilla MLP model outperforms all the GNN models shown in Table 11. The actor dataset shows entirely different properties with other datasets. Furthermore, we investigate on the individual effect of aggregated feature distance and homophily ratio difference in Figure 20 and 21, respectively. An overall trend of performance decline with increasing disparity score is evident though some exceptions are present. When only considering the aggregated feature distance to training nodes, the accuracy in Twitch-gamers exhibits an opposite increasing tendency along with the increasing difference to train nodes. Notably, when considering only the homophily ratio difference, the decreasing tendency is clear across datasets exception Actor. We can observe even better phenomenon on the Amazon-ratings dataset than combining both homophily ratio difference and the aggregated feature distance. It indicates that the homophily ratio difference w.r.t structural disparity significantly contributes to those datasets with more diverse structural patterns. We make the above initial observations showing the practicability of our theoretical analysis on more datasets with more diverse structural patterns. Moreover, exception can still be found on the Actor dataset. We leave a more comprehensive discussion on the Actor, as the future work.
(c) IGB-Tiny  $(h = 0.57)$
(e) Twitch-gamers  $(h = 0.56)$
(f) Amazon-ratings  $(h = 0.38)$
Figure 19: Test accuracy disparity across node subgroups by aggregated-feature distance and homophily ratio differences to training nodes on more datasets. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing difference to training set. A clear performance decrease tendency can be found from subgroups 1 to 5 with increasing differences to the training set except the Actor dataset.
(b) CiteSeer
(c) IGB-Tiny
(d) Actor
Figure 20: Test accuracy disparity across node subgroups by aggregated-feature distance to training nodes on more datasets. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing differences to the training set.
(e) Twitch-gamers
(f) Amazon-ratings
K Additional investigation on the discriminative ability of GCN
In Section 3.2, we examine discriminative ratios on input features and multiple hops aggregated features for homophilic and heterophilic test nodes, respectively. Nonetheless, those experiments only focus on feature aggregation while ignoring the feature transformation and the training procedure. In this section, we further explore the discriminative ratios on well-trained GCNs with both feature aggregation and transformation. Typically, we utilize the hidden representation trained with the best hyperparameter as shown in Appendix G. Experimental results can be found in Figure 22. Observations on GCN are consistent with those in Section 3.2 focusing on aggregation, where majority nodes generally show a larger discriminative improvement than the minority nodes along with more aggregation layers, indicating a better discriminative ability on majority nodes than the minority ones. Those observations further substantiate our conclusions in a more general setting.
(d) Actor
Figure 21: Test accuracy disparity across node subgroups by homophily ratio differences to training nodes on more datasets. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing differences to the training set.
Figure 22: Illustration of the change on relative discriminative ratio along with the aggregation. The x-axis represents the number of aggregations and the y-axis represents the relative discriminative ratio.
L Significant test between GCN and MLP-based models
We conduct the performance comparison between GCN with MLP-based models, on test nodes with different homophily ratios, as shown in Fig 3. In the corresponding, we include the p-values of a paired t-test and confidence intervals between GCN and MLP-base models in Table 23, to determine whether the performance difference is statistically significant. The  $x$ -axis corresponds to the node homophily ratio range. The  $y$ -axis corresponds to p-values of whether the performance of GCN is significantly better or worse than the corresponding MLP model. Typically,  $p < 0.05$  indicates statistical significance.
M Significant test between GCN and deeper GNN models
We conduct the performance comparison between GCN with deeper GNNs, on test nodes with different homophily ratios, as shown in Fig 8. In the corresponding, we include the p-values of a paired t-test and confidence intervals between GCN and deeper GNN models in Table 24, to determine
(c) PubMed
(d) Chameleon
(e) Squirrel
(f) Actor
(g) Twitch-gamers
Figure 23: p-value of the paired t-test between GCN and MLP-based models with respect to accuracy on nodes with different homophily ratios.
(h) Amazon-ratings
whether the performance difference is statistically significant. The  $x$ -axis corresponds to the node homophily ratio range. The  $y$ -axis corresponds to p-values of whether the performance of GCN is significantly better or worse than the corresponding deeper model. Typically,  $p < 0.05$  indicates statistical significance.
N Limitation
It is important to acknowledge certain limitations in our work despite our research providing valuable insights into the effectiveness of GNN on structural disparity. In order to facilitate a more feasible theoretical analysis, we have made several assumptions, the most significant of which is the use of the generalized CSBM-S model assumption in our theoretical analysis. This CSBM assumption, although prevalent in graph analysis, is subject to a notable drawback as the CSBM assumption postulates that feature dimensions are independent and identically distributed. It restricts the generality of our analysis.
Our current findings lay a strong foundation for further exploration. It is worth mentioning that our theoretical analysis predominantly focuses on simple aggregation techniques. We hope that these efforts will contribute significantly to the Graph Neural Network research community. Moving forward, we aspire to broaden our findings to encompass more advanced message-passing neural networks, with particular emphasis on deeper GNN models. Currently, there is only empirical evidence showing the effectiveness of the deeper GNN models. Besides, the node homophily ratio is calculated on the undirected graph while the one on the directed graph is still under exploration. More comprehensive analyses on the higher-order aggregated feature and higher-order homophily ratio are still under-explored. Moreover, We aim to utilize our understanding to mitigate the performance difference induced by the structural disparity in the future.
(g) Twitch-gamers
Figure 24: p-value of the paired t-test between GCN and deep GNN models with respect to accuracy on nodes with different homophily ratios.
O Broader Impact
Graph Neural Networks (GNNs) have emerged as a powerful architecture for modeling graph-structured data across a wide range of practical applications [2, 5, 3, 6-8, 73, 107]. A key aspect of GNNs is the inductive bias on leveraging neighborhood information. However, such characteristics may lead to a biased test prediction, particularly when majority neighborhood patterns are prevalent. For example, a scenario in which we are required to predict the gender of individuals with a dating network. GNN may excessively rely on the pattern that when all neighboring nodes are female, the center node should be male. This approach disregards the minority group of individuals who are more likely to date others of the same gender. This overlooked drawback in GNN may lead to ethical concerns.
In this study, we reveal the potential for performance disparity inherent in Graph Neural Networks and propose a novel OOD scenario for exploring methods to mitigate such issues. It is important to emphasize that our research offers an understanding of these limitations rather than introducing a new methodology or approach. We only identify the problem and show the potential solution to
address this issue. Consequently, we do not foresee any negative broader impacts stemming from our findings. We expect our work to contribute significantly to the ongoing research efforts aimed at enhancing the versatility and fairness of GNN models when applied to diverse data settings.
Our findings can also help to boost different graph domains. In section 4, we show how our understanding can help to elucidate the effectiveness of deeper GNN, reveal an overlooked OOD factor, and propose a new Graph out-of-distribution scenario. It is worth noting that our findings can derive new understandings of more graph applications and inspire us to identify new problems in graph domains. We further illustrate more understanding on the robustness of GNN and more initial understanding on Graph out-of-distribution problem as follows.
Graph Robustness. Recent studies have exposed the susceptibility of GNNs to graph adversarial attacks [108-112], where small perturbations can mislead GNNs into making incorrect predictions. Several literature [113-115] reveals that increasing the heterophily of the homophily graph could be a key factor contributing to a successful attack. More recently, [116] observes that (1) learning-based attacks [114] tend to increase the heterophily in the smaller of train and test sets. (2) Directed attacks on particular nodes can outperform the undirected attack on the whole graph. Those observations are aligned with the understanding in our paper that (1) Increasing the heterophily of the homophilic graph enlarges the homophily ratio difference, leading to performance degradation. (2) Perturbations lying in the smaller set can efficiently enlarge the homophily ratio difference between training and test nodes with minimal perturbation budgets. (3) Undirected attack on the whole graph may not lead to a larger homophily ratio difference between the train and test nodes with permutation on both train and test nodes, leading to performance degradation.
Understanding on Graph out-of-distribution problem. Despite the new proposed Graph Out-Of-Distribution (OOD) scenario, our findings can further inspire a new understanding of existing graph OOD problems. We provide an initial discussion as follows. The uniqueness challenge of the OOD problem in the graph domain is that the discrepancy in graph structure can also lead to test performance degradation, in addition to node feature differences. There exists various graph OOD scenarios[62-64, 24, 66, 46], e.g., graph density, biased training labels, time shift, and popularity shift. Nonetheless, despite various graph OOD scenarios with different qualitative concepts, it still lacks of quantitative graph metrics to measure how the distribution shift happens, causing performance degradation. Inspired by our finding, We can roughly recognize the main reasons as graph covariate shift and concept shift [68-70, 117] in a graph context: (1) Graph covariate shift [118] is defined as  $P^{\mathrm{train}}(X) \neq P^{\mathrm{test}}(X)$  where  $P^{\mathrm{train}}(\cdot)$  and  $P^{\mathrm{test}}(\cdot)$  are training and test distribution, respectively. Such graph covariate shift corresponds to the large feature distance  $\| \mathbf{f}_u - \mathbf{f}_v\|$  between train and test nodes. (2) Graph concept shift is defined as  $P^{\mathrm{train}}(Y|X) \neq P^{\mathrm{test}}(Y|X)$ . Such graph concept shift corresponds to the homophily ratio difference  $|h_u - h_v|$  between train and test nodes. We leave a more detailed discussion and a careful design on the graph OOD quantitative metrics as a further work.
P Future work
Further investigation on Deeper GNN In section 4, we conduct experiments on the effectiveness of deeper GNNs, indicating the performance gain is majorly from the improved discriminative ability on nodes in minor patterns. It could be a good direction to further explain and understand why such phenomenon happens. A potential reference for conducting discriminative analysis of deeper GNN is [39], aims to theoretically quantify the discriminative ability on deeper GNNs with more aggregations. Nonetheless, their analysis uses the vanilla CSBM model as the data assumption, denoted as  $\mathrm{CSBM}(\mu_1,\mu_2,p,q)$ . The CSBM model presumes that all nodes follow either homophilic with  $p > q$  or heterophilic patterns with  $p < q$  exclusively. However, this assumption conflicts with real-world scenarios as homophilic and heterophilic patterns coexist across different graphs, as shown in Figure 2. As far as we can see, conducting a similar discriminative analysis as [1] on our proposed CSBM-M model considering both homophily and heterophily patterns could be a good solution.
Other factors on performance disparity Existing literature [63, 46, 86, 87] shows that some other structural information, e.g., degree, geodesic distance to the training node, and Personal Pagerank score, could lead to a performance disparity. Nonetheless, all those analyses and conclusions are conducted on homophilic graphs, e.g., PubMed and ogbn-arxiv, while ignoring the heterophilic graphs e.g., Chameleon and Squirrel, which also broadly exist in the graph domain. It could be a good new research direction to see how the above factors, focus on the homophilic pattern in the context
of both homophilic and heterophilic properties. Another one could be how to find other important factors on both homophilic and heterophilic graphs.
Solutions on performance disparity In this paper, we propose and analyze the importance of the performance disparity problem without no explicit solution. We provide two potential solutions as follows:
- Combining MLP and GNN in an adaptive approach since MLPs can achieve better performance on minority nodes and GNNs better on majority nodes. Ideally, we can adaptively select MLP for minority nodes and GNN for majority nodes, using an adaptively gated function to control the proportion of MLP and GNN. Our findings suggest the homophilic/heterophilic pattern selection can serve as a guide for learning the gate function.
- Utilizing global structural information, as the global pattern is more robust, showing less disparity than the local structural pattern. Empirical evidence can be found in Figure 9 that the higher-order homophily ratio differences are smaller than the local structure disparity.