SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Haitao Mao $^{1}$ , Zhikai Chen $^{1}$ , Wei Jin $^{4}$ , Haoyu Han $^{1}$ , Yao Ma $^{3}$ , Tong Zhao $^{2}$ , Neil Shah $^{2}$ , Jiliang Tang $^{1}$
<sup>1</sup>Michigan State University <sup>2</sup>Snap Inc <sup>3</sup>Rensselaer Polytechnic Institute. <sup>4</sup> Emory University {haitaoma, chenzh85,hanhaoy1,tangjili} @msu.edu, {tzhao,nshah} @snap.com may13@rpi.edu, wei.jin@emory.edu
Recent studies on Graph Neural Networks(GNNs) provide both empirical and theoretical evidence supporting their effectiveness in capturing structural patterns on both homophilic and certain heterophilic graphs. Notably, most real-world homophilic and heterophilic graphs are comprised of a mixture of nodes in both homophilic and heterophilic structural patterns, exhibiting a structural disparity. However, the analysis of GNN performance with respect to nodes exhibiting different structural patterns, e.g., homophilic nodes in heterophilic graphs, remains rather limited. In the present study, we provide evidence that Graph Neural Networks(GNNs) on node classification typically perform admirably on homophilic nodes within homophilic graphs and heterophilic nodes within heterophilic graphs while struggling on the opposite node set, exhibiting a performance disparity. We theoretically and empirically identify effects of GNNs on testing nodes exhibiting distinct structural patterns. We then propose a rigorous, non-i.i.d PAC-Bayesian generalization bound for GNNs, revealing reasons for the performance disparity, namely the aggregated feature distance and homophily ratio difference between training and testing nodes. Furthermore, we demonstrate the practical implications of our new findings via (1) elucidating the effectiveness of deeper GNNs; and (2) revealing an over-looked distribution shift factor on graph out-of-distribution problem and proposing a new scenario accordingly.
Graph Neural Networks (GNNs) [1-4] are a powerful technique for tackling a wide range of graph-related tasks [5, 3, 6-10], especially node classification [2, 4, 11, 12], which requires predicting unlabeled nodes based on the graph structure, node features, and a subset of labeled nodes. The success of GNNs can be ascribed to their ability to capture structural patterns through the aggregation mechanism that effectively combines feature information from neighboring nodes [13].
GNNs have been widely recognized for their effectiveness on homophilic graphs [14, 2, 11, 4, 15-17]. In homophilic graphs, connected nodes tend to share the same label, which we refer to as homophilic patterns. An example of the homophilic pattern is depicted in the upper part of Figure 1, where node features and node labels are denoted by colors (i.e., blue and red) and numbers (i.e., 0 and 1), respectively. We can observe that all connected nodes exhibit homophilic patterns and share the same label 0. Recently, several studies have demonstrated that GNNs can also perform well on certain heterophilic graphs [18, 13, 19]. In heterophili
Figure 1: Examples of homophilic and heterophilic patterns. Colors/numbers indicate node features/terms.
tend to have different labels, which we refer to as heterophilic patterns. The example in the lower part of Figure 1 shows the heterophilic patterns. Based on this example, we intuitively illustrate how GNNs can work on such heterophilic patterns (lower right): after averaging features over all neighboring nodes, nodes with label 0 completely switch from their initial blue color to red, and vice versa; despite this feature alteration, the two classes remain easily distinguishable since nodes with the same label (number) share the same color (features).
However, existing studies on the effectiveness of GNNs [14, 18, 13, 19] only focus on either homophilic or heterophilic patterns solely and overlook the fact that real-world graphs typically exhibit a mixture of homophilic and heterophilic patterns. Recent studies [20, 21] reveal that many heterophilic graphs, e.g., Squirrel and Chameleon [22], contain over  $20\%$  homophilic nodes. Similarly, our preliminary study depicted in Figure 2 demonstrates that heterophilic nodes are consistently present in many homophilic graphs, e.g., PubMed [23] and Ogbn-arxiv [24]. Hence, real-world homophilic graphs predominantly consist of homophilic nodes as the majority structural pattern and heterophilic nodes in the minority one, while heterophilic graphs exhibit an opposite phenomenon with heterophilic nodes in the majority and homophilic ones in the minority.
To provide insights aligning the real-world scenario with structural disparity, we revisit the toy example in Figure 1, considering both homophilic and heterophilic patterns together. Specifically, for nodes labeled 0, both homophilic and heterophilic node features appear in blue before aggregation. However, after aggregation, homophilic and heterophilic nodes in label 0 exhibit different features, appearing blue and red, respectively. Such differences may lead to performance disparity between nodes in majority and minority patterns. For instance, in a homophilic graph with the majority pattern being homophilic, GNNs are more likely to learn the association between blue features and class 0 on account of more supervised signals in majority. Consequently, nodes in the majority structural pattern can perform well, while nodes in the minority structural pattern may exhibit poor performance, indicating an over-reliance on the majority structural pattern. Inspired by insights from the above toy example, we focus on answering following questions systematically in this paper: How does a GNN behave when encountering the structural disparity of homophilic and heterophilic nodes within a dataset? and Can one GNN benefit all nodes despite structural disparity?
Present work. Drawing inspiration from above intuitions, we investigate how GNNs exhibit different effects on nodes with structural disparity, the underlying reasons, and implications on graph applications. Our study proceeds as follows: First, we empirically verify the aforementioned intuition by examining the performance of testing nodes w.r.t. different homophily ratios, rather than the overall performance across all test nodes as in [13, 14, 19]. We show that GCN [2], a vanilla GNN, often underperforms MLP-based models on nodes with the minority pattern while outperforming them on the majority nodes. Second, we examine how aggregation, the key mechanism of GNNs, shows different effects on homophilic and heterophilic nodes. We propose an understanding of why GNNs exhibit performance disparity with a non-i.i.d PAC-Bayesian generalization bound, revealing that both feature distance and homophily ratio differences between train and test nodes are key factors leading to performance disparity. Third, we showcase the significance of these insights by exploring implications for (1) elucidating the effectiveness of deeper GNNs and (2) introducing a new graph out-of-distribution scenario with an over-looked distribution shift factor. Codes are available at here.
2 Prelimaries
Semi-Supervised Node classification (SSNC). Let  $G = (V, E)$  be an undirected graph, where  $V = \{v_1, \dots, v_n\}$  is the set of  $n$  nodes and  $E \subseteq V \times V$  is the edge set. Nodes are associated with node features  $\mathbf{X} \in \mathbb{R}^{n \times d}$ , where  $d$  is the feature dimension. The number of class is denoted as  $K$ . The adjacency matrix  $\mathbf{A} \in \{0, 1\}^{n \times n}$  represents graph connectivity where  $\mathbf{A}[i, j] = 1$  indicates an edge between nodes  $i$  and  $j$ .  $\mathbf{D}$  is a degree matrix and  $\mathbf{D}[i, i] = d_i$  with  $d_i$  denoting degree of node  $v_i$ . Given a small set of labeled nodes,  $V_{\mathrm{tr}} \subseteq V$ , SSNC task is to predict on unlabeled nodes  $V \setminus V_{\mathrm{tr}}$ .
Node homophily ratio is a common metric to quantify homophilic and heterophilic patterns. It is calculated as the proportion of a node's neighbors sharing the same label as the node [25, 26, 20]. It is formally defined as  $h_i = \frac{|\{u \in \mathcal{N}(v_i) : y_u = y_v\}|}{d_i}$ , where  $\mathcal{N}(v_i)$  denotes the neighbor node set of  $v_i$  and  $d_i = |\mathcal{N}(v_i)|$  is the cardinality of this set. Following [20, 27, 25], node  $i$  is considered to be homophilic when more neighbor nodes share the same label as the center node with  $h_i > 0.5$ . We
define the graph homophily ratio  $h$  as the average of node homophily ratios  $h = \frac{\sum_{i\in V}h_i}{|V|}$ . Moreover, this ratio can be easily extended to higher-order cases  $h_i^{(k)}$  by considering  $k$ -order neighbors  $\mathcal{N}_k(v_i)$ . Node subgroup refers to a subset of nodes in the graph sharing similar properties, typically homophilic and heterophilic patterns measured with node homophily ratio. Training nodes are denoted as  $V_{\mathrm{tr}}$ . Test nodes  $V_{\mathrm{te}}$  can be categorized into  $M$  node subgroups,  $V_{\mathrm{te}} = \bigcup_{m=1}^{M} V_m$ , where nodes in the same subgroup  $V_m$  share similar structural pattern.
3 Effectiveness of GNN on nodes with different structural properties
In this section, we explore the effectiveness of GNNs on different node subgroups exhibiting distinct structural patterns, specifically, homophilic and heterophilic patterns. It is different from previous studies [13, 14, 18, 28, 19] that primarily conduct analysis on the whole graph and demonstrate effectiveness with an overall performance gain. These studies, while useful, do not provide insights into the effectiveness of GNNs on different node subgroups, and may even obscure scenarios where GNNs fail on specific subgroups despite an overall performance gain. To accurately gauge the effectiveness of GNNs, we take a closer examination on node subgroups with distinct structural patterns. The following experiments are conducted on two common homophilic graphs, Ogbnarxiv [24] and Pubmed [23], and two heterophilic graphs, Chameleon and Squirrel [22]. These datasets are chosen since GNNs can achieve better overall performance than MLP. Experiment details and related work on GNN disparity are in Appendix G and A, respectively.
(a) PubMed  $(h = 0.79)$
Figure 2: Node homophily ratio distributions. All graphs exhibit a mixture of homophilic and heterophilic nodes despite various graph homophily ratio  $h$ .
(b) Ogbn-arxiv  $(h = 0.63)$
(c) Chameleon  $(h = 0.22)$
(d) Squirrel  $(h = 0.25)$
Existence of structural pattern disparity within a graph is to recognize real-world graphs exhibiting different node subgroups with diverse structural patterns, before investigating the GNN effectiveness on them. We demonstrate node homophily ratio distributions on the aforementioned datasets in Figure 2. We can have the following observations. Obs.1: All four graphs exhibit a mixture of both homophilic and heterophilic patterns, rather than a uniform structural patterns. Obs.2: In homophilic graphs, the majority of nodes exhibit a homophilic pattern with  $h_i > 0.5$ , while in heterophilic graphs, the majority of nodes exhibit the heterophilic pattern with  $h_i \leq 0.5$ . We define nodes in majority structural pattern as majority nodes, e.g., homophilic nodes in a homophilic graph.
(c) Chameleon  $(h = 0.22)$
(d) Squirrel  $(h = 0.25)$
Figure 3: Performance comparison between GCN and MLP-based models. Each bar represents the accuracy gap on a specific node subgroup exhibiting a homophily ratio within the range specified on the x-axis. MLP-based models often outperform GCN on heterophilic nodes in homophilic graphs and homophilic nodes in heterophilic graphs with a positive value.
Examining GCN performance on different structural patterns. To examine the effectiveness of GNNs on different structural patterns, we compare the performance of GCN [2] a vanilla GNN, with two MLP-based models, vanilla MLP and Graphless Neural Network (GLNN) [29], on testing nodes with different homophily ratios. It is evident that the vanilla MLP could have a large performance gap compared to GCN (i.e.,  $20\%$  in accuracy) [13, 29, 2]. Consequently, an under-trained vanilla MLP comparing with a well-trained GNN leads to an unfair comparison without rigorous conclusion. Therefore, we also include an advanced MLP model GLNN. It is trained in an advanced manner via distilling GNN predictions and exhibits performance on par with GNNs. Notably, only GCN has the ability to leverage structural information during the inference phase while both vanilla MLP and GLNN models solely rely on node features as input. This comparison ensures a fair study on the effectiveness of GNNs in capturing different structural patterns with mitigating the effects of node features. Experimental results on four datasets are presented in Figure 3. In the figure, y-axis corresponds to the accuracy differences between GCN and MLP-based models where positive indicates MLP models can outperform GCN; while x-axis represents different node subgroups with nodes in the subgroup satisfying homophily ratios in the given range, e.g., [0.0-0.2]. Based on experimental results, the following key observations can be made: Obs.1: In homophilic graphs, both GLNN and MLP demonstrate superior performance on the heterophilic nodes with homophily ratios in [0-0.4] while GCN outperforms them on homophilic nodes. Obs.2: In heterophilic graphs, MLP models often outperform on homophilic nodes yet underperform on heterophilic nodes. Notably, vanilla MLP performance on Chameleon is worse than that of GCN across different subgroups. This can be attributed to the training difficulties encountered on Chameleon, where an unexpected similarity in node features from different classes is observed [30]. Our observations indicate that despite the effectiveness of GCN suggested by [13, 19, 14], GCN exhibits limitations with performance disparity across homophilic and heterophilic graphs. It motivates investigation why GCN benefits majority nodes, e.g., homophilic nodes in homophilic graphs, while struggling with minority nodes. Moreover, additional results on more datasets and significant test results are shown in Appendix H and L.
Organization. In light of the above observations, we endeavor to understand the underlying causes of this phenomenon in the following sections by answering the following research questions. Section 3.1 focuses on how aggregation, the fundamental mechanism in GNNs, affects nodes with distinct structural patterns differently. Upon identifying differences, Section 3.2 further analyzes how such disparities contribute to superior performance on the majority nodes as opposed to minority nodes. Building on these observations, Section 3.3 recognizes the key factors driving performance disparities on different structural patterns with a non-i.i.d. PAC-Bayes bound. Section 3.4 empirically corroborates the validity of our theoretical analysis with real-world datasets.
3.1 How does aggregation affect nodes with structural disparity differently?
In this subsection, we examine how aggregation reveals different effects on nodes with structural disparity, serving as a precondition for performance disparity. Specifically, we focus on the discrepancy between nodes from the same class but with different structural patterns.
For a controlled study on graphs, we adopt the contextual stochastic block model (CSBM) with two classes. It is widely used for graph analysis, including generalization [13, 14, 31, 18, 32, 33], clustering [34], fairness [35, 36], and GNN architecture design [37-39]. Typically, nodes in CSBM model are generated into two disjoint sets  $\mathcal{C}_1$  and  $\mathcal{C}_2$  corresponding to two classes,  $c_{1}$  and  $c_{2}$ , respectively. Each node with  $c_{i}$  is associated with features  $x\in \mathbb{R}^d$  sampling from  $N(\pmb {\mu}_i,I)$ , where  $\mu_{i}$  is the feature mean of class  $c_{i}$  with  $i\in \{1,2\}$ . The distance between feature means in different classes  $\rho = \| \pmb {\mu}_1 - \pmb {\mu}_2\|$ , indicating the classification difficulty on node features. Edges are then generated based on intra-class probability  $p$  and inter-class probability  $q$ . For instance, nodes with class  $c_{1}$  have probabilities  $p$  and  $q$  of connecting with another node in class  $c_{1}$  and  $c_{2}$ , respectively. The CSBM model, denoted as  $\mathrm{CSBM}(\pmb {\mu}_1,\pmb {\mu}_2,p,q)$ , presumes that all nodes follow either homophilic with  $p > q$  or heterophilic patterns  $p < q$  exclusively. However, this assumption conflicts with real-world scenarios, where graphs often exhibit both patterns simultaneously, as shown in Figure 2. To mirror such scenarios, we propose a variant of CSBM, referred to as CSBM-Structure (CSBM-S), allowing for the simultaneous description of homophilic and heterophilic nodes.
Definition 1 (CSBM-S  $(\mu_1,\mu_2,(p^{(1)},q^{(1)}),(p^{(2)},q^{(2)}),\mathrm{Pr}(\mathrm{homo}))$  ). The generated nodes consist of two disjoint sets  $\mathcal{C}_1$  and  $\mathcal{C}_2$  . Each node feature  $x$  is sampled from  $N(\mu_i,I)$  with  $i\in \{1,2\}$  Each set  $\mathcal{C}_i$  consists of two subgroups:  $\mathcal{C}_i^{(1)}$  for nodes in homophilic pattern with intra-class and
inter-class edge probability  $p^{(1)} > q^{(1)}$  and  $\mathcal{C}_i^{(2)}$  for nodes in heterophilic pattern with  $p^{(2)} < q^{(2)}$ .  $\operatorname*{Pr}(\text{homo})$  denotes the probability that the node is in homophilic pattern.  $\mathcal{C}_i^{(j)}$  denotes node in class  $i$  and subgroup  $j$  with  $(p^{(j)}, q^{(j)})$ . We assume nodes follow the same degree distribution with  $p^{(1)} + q^{(1)} = p^{(2)} + q^{(2)}$ .
Based on the neighborhood distributions, the mean aggregated features  $\mathbf{F} = \mathbf{D}^{-1}\mathbf{A}\mathbf{X}$  obtained follow Gaussian distributions on both homophilic and heterophilic subgroups.
$$
\mathbf {f} _ {i} ^ {(j)} \sim N \left(\frac {p ^ {(j)} \boldsymbol {\mu} _ {1} + q ^ {(j)} \boldsymbol {\mu} _ {2}}{p ^ {(j)} + q ^ {(j)}} \cdot \frac {\mathbf {I}}{\sqrt {d _ {i}}}\right), \text {f o r} i \in \mathcal {C} _ {1} ^ {(j)}; \mathbf {f} _ {i} ^ {(j)} \sim N \left(\frac {q ^ {(j)} \boldsymbol {\mu} _ {1} + p ^ {(j)} \boldsymbol {\mu} _ {2}}{p ^ {(j)} + q ^ {(j)}} \cdot \frac {\mathbf {I}}{\sqrt {d _ {i}}}\right), \text {f o r} i \in \mathcal {C} _ {2} ^ {(j)} \tag {1}
$$
Where  $\mathcal{C}_i^{(j)}$  is the node subgroups with structural pattern with  $(p^{(j)},q^{(j)})$  in label  $i$ . Our initial examination of different effects on aggregation focuses on the aggregated feature distance between homophilic and heterophilic node subgroups within class  $c_{1}$ .
Proposition 1. The aggregated feature mean distance between homophilic and heterophilic node subgroups within class  $c_{1}$  is  $\left\| \frac{p^{(1)}\pmb{\mu}_1 + q^{(1)}\pmb{\mu}_2}{p^{(1)} + q^{(1)}} - \frac{p^{(2)}\pmb{\mu}_1 + q^{(2)}\pmb{\mu}_2}{p^{(2)} + q^{(2)}} \right\| > 0$ , indicating the aggregated feature of homophilic and heterophilic subgroups are from different feature distributions, with a mean distance larger than 0 distance before aggregation, since original node features draw from the same distribution, regardless of different structural patterns.
Notably, the distance between original features is regardless of the structural pattern. This proposition suggests that aggregation results in a distance gap between different patterns within the same class.
In addition to node feature differences with the same class, we further examine the discrepancy between nodes  $u$  and  $v$  with the same aggregated feature  $\mathbf{f}_u = \mathbf{f}_v$  but different structural patterns. We examine the discrepancy with the probability difference of nodes  $u$  and  $v$  in class  $c_1$ , denoted as  $|\mathbf{P}_1(y_u = c_1|\mathbf{f}_u) - \mathbf{P}_2(y_v = c_1|\mathbf{f}_v)|$ .  $\mathbf{P}_1$  and  $\mathbf{P}_2$  are the conditional probability of  $y = c_1$  given the feature  $\mathbf{f}$  on structural patterns  $(p^{(1)}, q^{(1)})$  and  $(p^{(2)}, q^{(2)})$ , respectively.
Lemma 1. With assumptions (1) A balance class distribution with  $\mathbf{P}(Y = 1) = \mathbf{P}(Y = 0)$  and (2) aggregated feature distribution shares the same variance  $\sigma$ . When nodes  $u$  and  $v$  have the same aggregated features  $\mathbf{f}_u = \mathbf{f}_v$  but different structural patterns,  $(p^{(1)}, q^{(1)})$  and  $(p^{(2)}, q^{(2)})$ , we have:
$$
\left| \mathbf {P} _ {1} \left(y _ {u} = c _ {1} \mid \mathbf {f} _ {u}\right) - \mathbf {P} _ {2} \left(y _ {v} = c _ {1} \mid \mathbf {f} _ {v}\right) \right| \leq \frac {\rho^ {2}}{\sqrt {2 \pi} \sigma} \left| h _ {u} - h _ {v} \right| \tag {2}
$$
Notably, above assumptions are not strictly necessary but employed for elegant expression. Lemma 1 implies that nodes with a small homophily ratio difference  $|h_1 - h_2|$  are likely to share the same class, and vice versa. Proof details and additional analysis on between-class effects are in Appendix D.
3.2 How does Aggregation Contribute to Performance Disparity?
We have established that aggregation can affect nodes with distinct structural patterns differently. However, it remains to be elucidated how such disparity contributes to performance improvement predominantly on majority nodes as opposed to minority nodes. It should be noted that, notwithstanding the influence on features, test performance is also profoundly associated with training labels. Performance degradation may occur when the classifier is inadequately trained with biased training labels.
We then conduct an empirical discriminative analysis taking both mean aggregated features and training labels into consideration. Drawing inspiration from existing literature [40-43], we describe the discriminative ability with the distance between train class prototypes [44, 45], i.e., feature mean of each class, and the corresponding test class prototype within the same class  $i$ . For instance, it can be denoted as  $\left|\left|\mu_i^{\mathrm{tr}} - \mu_i^{\mathrm{ma}}\right|\right|$ , where  $\mu_i^{\mathrm{tr}}$  and  $\mu_i^{\mathrm{ma}}$  are the prototype of class  $i$  on train nodes and test majority nodes, respectively. A smaller value suggests that majority test nodes are close to train nodes within
Figure 4: Illustration on discriminative ratio variation along with aggregation. x-axis denotes the number of aggregations and y-axis denotes the discriminative ratio.
the same class, thus implying superior discriminative ability. A relative discriminative ratio is then proposed to compare the discriminative ability between majority and minority nodes. It can be denoted as:  $r = \sum_{i=1}^{K} \frac{||\pmb{\mu}_i^{\mathrm{tr}} - \pmb{\mu}_i^{\mathrm{ma}}||}{||\pmb{\mu}_i^{\mathrm{tr}} - \pmb{\mu}_i^{\mathrm{mi}}||}$  where  $\mu_i^{\mathrm{mi}}$  corresponds to the prototype on minority test nodes. A lower relative discriminative ratio suggests that majority nodes are easier to be predicted than minority nodes.
The relative discriminative ratios are then calculated on different hop aggregated features and original features denote as 0-hop. Experimental results are presented in Figure 4, where the discriminative ratio shows an overall decrease tendency as the number of aggregations increases across four datasets. This indicates that majority test nodes show better discriminative ability than the minority test nodes along with more aggregation. We illustrate more results on GCN in Appendix K. Furthermore, instance-level experiments other than class prototypes are in Appendix C.
3.3 Why does Performance Disparity Happen? Subgroup Generalization Bound for GNNs