SlowGuess/NeurIPS_2023_OCR huggingface dataset:
- [28] is majorly grounded on the CSBM-H model focusing on different feature variances for different classes, from a feature perspective. In contrast, our work examines the homophilic and heterophilic patterns from a structural perspective instead of sharing the same node homophily ratio across the graph.
- [28] proposes a new metric to identify whether GNN can outperform graph-agnostic MLP on particular datasets. Nonetheless, it still focuses on all the nodes in the whole graph together. In contrast, our work reveals the scenario where GNNs show performance degradation on certain node subgroup across most homophilic and heterophilic graph datasets. Typically, we focus on a node subgroup perspective, rather than all nodes in the whole graph. Our paper highlights the drawback of GNNs in a more general case.
Fairness on Graph Although recent years have seen a satisfying performance from Graph Neural Networks (GNNs), risks can be found. GNNs may unintentionally learn and perpetuate biases present in the training data, potentially resulting in unfair outcomes for certain populations. Such risks raise concerns about biases and discriminatory behaviors when GNNs are used in human-centric applications.
Fairness issues on graphs can be roughly categorized into attribute bias and structural bias, based on the source of the bias. Several works [35, 83-85] focus on fairness issues originating from sensitive node attributes, e.g., gender or underrepresented ethnic groups. Other literature [63, 46, 86-88] focuses on addressing fairness issues arising from different structural information, e.g., degree, geodesic distance to the training node, and Personal Pagerank score. Our work aligns closely with structural bias, showing that a larger homophily ratio difference between training and test nodes may lead to performance degradation. To the best of our knowledge, we are the first to propose this performance disparity induced by the homophily ratio difference.
Generalization ability analysis on Graph Neural Network The generalization ability analysis on Graph Neural Networks aims to develop theoretical understandings of GNNs with a focus on the uniqueness of the graph structure. Recent research progress reveals the generalization ability [89, 47, 90–92] of GNN across different tasks and settings. Our work typically studies the generalization ability on the transductive node classification task indicating relationship with [93, 14, 18, 13]. Typically, [93] employs the transductive uniform stability [94] to understand why deeper GNNs generalize better than the vanilla GCN. Several studies [14, 18, 13] investigate the generalization of GCN under the CSBM model assumption, while they focus on either homophilic or heterophilic patterns rather than consider both patterns together. [46] provides the first non-i.i.d. PAC-Bayes generalization bound on GNNs, serving as the basis of our theory 1. However, there are key differences between our work and [46] detailed as follows:
- Assumption difference. [46] assumes the existence of  $c$ -Lipschitz continuous functions on the conditional probability of  $y_{i} = k$  given the aggregated feature  $g_{i}(X,G)$ , denoted as  $\mathbf{P}(y_i = k|g_i(X,G))$ . The assumption suggests that nodes with a closer distance are likely to belong to the same class. However, this assumption may not align with the real-world scenario where nodes with closer distances may belong to different classes when they exhibit a large homophily ratio difference, as shown in Lemma 1. In contrast, we utilize a different assumption considering the influence of the structural pattern on the conditional probability, enabling analyses on the scenario that nodes with a closer distance but from different classes. This scenario happens frequently when the graph is a mixture of homophilic and heterophilic patterns. An intuitive illustration can be found in Figure 1.
- Application scope difference. [46] is primarily focused on the homophilic graphs and does not easily extend to the heterophilic ones. Empirical evidence can be found in Section 3.4 and Appendix J. In contrast, our theory can be established on most datasets except for the Actor dataset which structural information hardly helps.
Disclaimer: Structural disparity is different from distribution shift between train and test data. The structural disparity is that homophilic and heterophilic patterns exist simultaneously in a single node set. Such disparity consistently exists in all graphs, as shown in Figure 2. If randomly sample train and test data, the distribution shift between train and test will not happen. Both train and test sets will have homophilic and heterophilic patterns, indicating structural disparity happens on both the train and test sets.
B Investigation on the effectiveness of homophily ratio difference with targeted synthetic edge addition algorithm.
In Section 3.4, we intuitively show that a larger homophily ratio difference could be a key reason for performance disparity. We demonstrate the effectiveness of intuition with both theoretical analysis and experiments conducted on real-world datasets. To further verify the influence of the homophily ratio difference, we conduct more controllable synthetic experiments adopted from [13], adding different amounts of synthetic edges on real-world datasets, to further evaluate how the performance of GCN, a vanilla GNN, changes on varied homophily ratio differences. Typically, we are to manually make heterophilic nodes more homophilic and make homophilic nodes more heterophilic with the targeted homophilic edge addition and the targeted heterophilic edge addition algorithms, respectively. Notably, despite synthetic edges added, we utilize the real-world dataset serves as the entrance to keep the analysis more approach to the real-world scenario. The following subsections are organized as follows. We first introduce the targeted heterophilic and homophilic edge addition algorithms and how it leads to larger homophily ratio difference in Appendix B.1. Detailed experiment analysis can be found in Appendix B.2. Further experiment details can be found in the Appendix B.3.
B.1 Targeted heterophilic & homophilic edge addition algorithms
Both targeted heterophilic and homophilic edge addition algorithms commence with standard, real-world benchmark graphs, modifying the structure by adding synthetic edges to manipulate the homophily ratio on either train nodes or test nodes. For example, we can add synthetic heterophilous edges on the targeted test nodes in a homophily graph. Consequently, a homophily ratio difference between the training and testing nodes can be observed.
We first introduce the homophilic edge addition algorithm, as shown in Algorithm 1. Specially, we introduce a total of  $K$  edges on either training or targeted test nodes on heterophilic dataset, denoted as  $\mathcal{V}_{\mathrm{targeted}}$ . For each edge addition, a node  $u$  is uniformly sampled from the targeted node set,  $\mathcal{V}_{\mathrm{targeted}}$ , and obtains its corresponding label,  $y_{u}$ . Another targeted node  $v$  sharing the same label,  $y_{v}$  is selected, and a homophilic edge is added between them, resulting in a homophily ratio decrease. Consequently, we can observe a homophily ratio decrease on both target nodes  $u$  and  $v$  with the new homophily edge added. Notably, we only add synthetic edges among targeted nodes, ensuring the homophily ratio unchanged on the other nodes.
The heterophilic edge addition algorithm, as shown in the Algorithm 2, is analogous to the homophilic one. The only difference is the selection of a newly added heterophilic edge instead of the homophilic one. While homophilic edges can be readily added by connecting nodes within the same label, adding
cross-label heterophilic edges pose a new challenge that the target node  $u$  should be connected to which label, other than the target label  $y_{u}$ . Typically, we follow the principle [13] that nodes with the same label should share similar neighborhood label distributions. This principle aligns with the CSBM assumption, discussed in Section 3. Specifically, given a real-world graph  $G$ , we first define a discrete neighborhood target label distribution  $\mathcal{D}_c$  for each label  $c \in \mathcal{C}$ . Examples of  $\mathcal{D}_c$  can be found in Appendix B.3. Specifically, we will first randomly select a label  $c$  based on distribution  $\mathcal{D}_c$ . Another node  $v$  in the target set with label  $c$  is then selected, and a heterophilic edge is added between them. This process enables us to add heterophilic edges on target nodes following similar neighborhood label distributions.
Algorithm 1: Targeted Homophilic Edge Addition
input  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$ $\mathcal{V}_{targeted}\in \mathcal{V}$ $K$  , and  $\{\mathcal{V}_c\}_{c = 0}^{|C| - 1}$  output:  $\mathcal{G}' = \{\mathcal{V},\mathcal{E}'\}$    
1 Initialize  $\mathcal{E}' = \mathcal{E},k = 1$  .   
2 while  $1\leq k\leq K$  do   
3 do   
4 Sample node  $i\sim$  Uniform(Vtargeted);   
5 Obtain the label,  $y_{i}$  of node i;   
6 Sample node  $j\sim$  Uniform  $(\mathcal{V}_{y_i}\cap \mathcal{V}_{targeted})$  .   
7 while  $(i,j)\in \mathcal{E}'$  .   
8 Update edge set  $\mathcal{E}' = \mathcal{E}'\cup \{(i,j)\}$  .   
9  $k\gets k + 1$    
10 return  $\mathcal{G}' = \{\mathcal{V},\mathcal{E}'\}$
Algorithm 2: Targeted Heterophilous Edge Addition
input  $\mathcal{G} = \{\mathcal{V},\mathcal{E}\}$ $\mathcal{V}_{targeted}\in \mathcal{V},K,\{\mathcal{D}_c\}_{c = 0}^{|C| - 1}$  and  $\{\mathcal{V}_c\}_{c = 0}^{|C| - 1}$  output:  $\mathcal{G}' = \{\mathcal{V},\mathcal{E}'\}$    
1 Initialize  $\mathcal{E}' = \mathcal{E},k = 1$  .   
2 while  $1\leq k\leq K$  do   
3 Sample node  $i\sim$  Uniform(Vtargeted);   
4 Obtain the label,  $y_{i}$  of node i;   
5 Sample a number  $r\sim$  Uniform(0,1);   
6 Sample a label  $c\sim \mathcal{D}_{y_i}$    
7 Sample node  $j\sim$  Uniform  $(\mathcal{V}_c\cap \mathcal{V}_{targeted})$    
8 Update edge set  $\mathcal{E}' = \mathcal{E}'\cup \{(i,j)\}$    
9  $k\gets k + 1$    
10 return  $\mathcal{G}' = \{\mathcal{V},\mathcal{E}'\}$
B.2 Detailed experiment results
In this section, we conduct experiments on the homophilic graph, Cora, employing the targeted homophilic edge addition algorithm, and the heterophilic graph, Squirrel, employing the targeted heterophilic edge addition algorithm. For each dataset, we apply the synthetic edge addition algorithm to training nodes and a subset of test nodes, respectively, resulting in homophily ratio difference between training and test sets. The synthetic edges are added gradually until reaching the predefined maximum budget  $K_{\mathrm{max}}$ , resulting in multiple synthetic graphs generated. More experimental details on the synthetic graphs can be found in Appendix B.3. The GCN, a vanilla GNN model, is trained from the sketch on each synthetic graph. The experimental results are illustrated in Figure 10, where the  $x$ -axis represents the edge permutation ratio calculated as  $\frac{K}{K_{\mathrm{max}}}$ , where  $K$  is the number of added edges on the  $i$ -th synthetic graph. The  $y$ -axis represents the test performance when adding synthetic edge on training nodes. When adding synthetic edges on the targeted test subset, the  $y$ -axis represents the performance on the targeted test node subset. A clear degradation in test performance is observed with more and more synthetic edges added to test nodes and training nodes exclusively. The above observations clearly demonstrate that the GCN test performance will degrade with a larger homophilic ratio difference between train and test nodes.
(a) Synthetic graphs generated from Cora with the targeted heterophilic edge algorithm
(b) Synthetic graphs generated from Squirrel with the targeted homophilious edge algorithm
Figure 10: The performance of GCN on synthetic graphs with various edge permutation ratios. The  $y$ -axis represents the test performance on all nodes when adding synthetic edge on training nodes. When adding synthetic edges on the targeted test subset, The  $y$ -axis represents the performance on the targeted test node subset. Test performance generally decrease with more synthetic edges added with a large homophily ratio difference.
B.3 Details on the generated graph
In this subsection, we elaborate on the details of the synthetic graphs generated in Appendix B.2. Specifically, we provide more details regarding the distributions  $\mathcal{D}_c$ ,  $c \in \mathcal{C}$  employed in the Cora and Squirrel datasets. We adopted circulant matrix-like designs for simplicity and straightforward implementation. We also show details on the number of added edges, and the corresponding graph homophily ratio on the Cora and Squirrel datasets.
Cora We utilize the targeted heterophilic edge addition algorithm on the Cora dataset. The Cora dataset has seven labels that we represent as  $0, 1, 2, 3, 4, 5, 6$ , and the neighborhood distributions  $\mathcal{D}_c, c \in \mathcal{C}$  for adding heterophilic synthetic edges are presented as follows.
$\mathcal{D}_0$  :Categorical([0,0.5,0,0,0,0,0.5]),
$\mathcal{D}_1$  :Categorical([0.5,0,0.5,0,0,0,0]),
$\mathcal{D}_2$  :Categorical([0,0.5,0,0.5,0,0,0]),
$\mathcal{D}_3$  :Categorical([0,0,0.5,0,0.5,0,0]),
$\mathcal{D}_4$  :Categorical([0,0,0,0.5,0,0.5,0]),
$\mathcal{D}_5$  :Categorical([0,0,0,0,0.5,0,0.5]),
$\mathcal{D}_6$  :Categorical([0.5,0,0,0,0,0.5,0]).
The number of added edges  $K$  and the corresponding homophily ratio  $h$  on the targeted test nodes and the train nodes are presented in Table 2 and 3, respectively.
Table 2:  $K$  and  $h$  values for graphs with synthetic edges added targeted test nodes on Cora
K	100	200	300	400	500	600	700	800	900
h	0.645	0.483	0.382	0.321	0.279	0.249	0.223	0.206	0.194
Table 3:  $K$  and  $h$  values for graphs with synthetic edges added targeted training nodes on Cora
K	200	400	600	800	1000	1200	1400	1600	1800
h	0.620	0.507	0.424	0.365	0.324	0.292	0.264	0.243	0.224
Squirrel We utilize the targeted homophilic edge addition algorithm on the Squirrel dataset. We first sample the label from discrete uniform distributions. Then, we randomly select two nodes with the same label and do not have an edge between them. The number of added edges  $K$  and the corresponding homophily ratio  $h$  on the targeted test nodes and train nodes are presented in Table 4 and 5, respectively.
Table 4:  $K$  and  $h$  values for graphs with synthetic edges added targeted test nodes on Squirrel
K	100	200	300	400	500	600	700	800	900
h	0.237	0.345	0.412	0.467	0.505	0.536	0.562	0.585	0.606
Table 5:  $K$  and  $h$  values for graphs with synthetic edges added targeted train nodes on Squirrel
K	1500	3000	4500	6000	7500	9000	10500	12000	13500
h	0.261	0.299	0.330	0.356	0.379	0.399	0.417	0.434	0.448
C Instance-level discriminative analysis
In section 3.2, we conduct a discriminative analysis considering the distance between the feature means from different classes from a global perspective. In this section, we further investigate the discriminability from an instance-level perspective. Specifically, we focus on the feature distance between different nodes feature rather than the feature mean.
To qualify the discriminative difficulty on the feature of each individual test node, we propose the two metrics, local agreement ratio, and local accuracy difference. The local agreement ratio aims to measure the local clustering property in a KNN manner. It can be calculated with the following steps: (1) Given a particular node, find the top-  $k$  feature-close train nodes. We set  $k = 9$  in our experiment, and L2 distance is utilized as the feature distance metric. (2) Given the top-  $k$  closest train nodes, we examine the number of nodes on each label. If there exists a particular label  $c$  with the number of nodes over  $\frac{k}{2}$ , it indicates that over half of neighborhood nodes reach an agreement on the class  $c$ . (3) The local agreement ratio can then be calculated as the proportion of nodes reaching an agreement in the test set, denoted as:
$$
r = \frac {\sum_ {v \in V _ {\mathrm {t e}}} \mathbb {1} \left[ \exists c \in \mathcal {C} , \left| \mathcal {M} _ {v} ^ {c} \right| > \frac {k}{2} \right]}{\left| V _ {\mathrm {t e}} \right|} \tag {4}
$$
where  $r$  is the local agreement ratio,  $V_{\mathrm{te}}$  and  $\mathcal{C}$  are the test node set and class set, respectively.  $\mathcal{M}_v$  is the node set including top-  $k$  nearest training of the node  $v$ .  $|\mathcal{M}_v^c|$  is the number of nearest training nodes in class  $c$ . A larger agreement ratio on the test set indicates the data are more clustered with a close distance between train and test nodes. Nonetheless, a higher agreement ratio does not naturally lead to a better discriminative ability. Despite the top-  $k$  feature-close nodes reaching an agreement on a particular class, the test node may have a different class from the agreement. It indicates that the center node misaligns with the top-  $k$  feature-close nodes.
The local accuracy is then proposed to identify whether the category of the center node aligns with the agreement from top- $k$  feature-close nodes. Concretely speaking, the local agreement accuracy is the proportion of agreement nodes in the test set. It can be calculated as:
$$
\operatorname {A c c} _ {\text {l o c a l}} = \frac {\sum_ {v \in V _ {\text {a g r e e}}} \mathbb {1} \left[ c _ {v} = c _ {\mathcal {N} _ {v}} \right]}{\left| V _ {\text {a g r e e}} \right|} \tag {5}
$$
where  $V_{\mathrm{agree}}$  is the test node set reaching top-  $k$  feature-close nodes agreement.  $c_v$  is the category of node  $v$ .  $c_{\mathcal{M}_v}$  is the agreement category from feature-close nodes. A higher local agreement accuracy indicates that most agreement nodes are aligned with the same category.
Similar to the relative discriminative ratio proposed in Section 3.2, we illustrate the local agreement accuracy improvement on the majority nodes over minority nodes. Experiments are conducted on four homophilic datasets, Cora, CiteSeer, PubMed and Ogbn-arxiv, and two heterophilic datasets, Chameleon and Squirrel. The experimental setting details can be found in Section G. Experiment
(a) Cora  $(h = 0.81)$
(b)CiteSeer  $(h = 0.71)$
(c) PubMed  $(h = 0.79)$
(d) Ogbn-arxiv  $(h = 0.63)$
(e) Chameleon  $(h = 0.25)$
(f)Squirrel  $(h = 0.22)$
Figure 11: Illustration of the change on local agreement ratio along with the aggregation. The x-axis represents the number of aggregations and the y-axis represents the local agreement ratio. For homophilic graphs, the local agree ratio generally increases along with more aggregations, indicating better cluster effects. For heterophilic graphs, the local agreement ratio shows an opposite phenomenon, which decreases consistently.
results on local agreement ratio and local agreement accuracy difference are illustrated in Figure 11 and 12, respectively.
The observations can be found as follows: (1) For homophilic graphs, the local agree ratio generally increases along with more aggregations, indicating better cluster effects. Meanwhile, the relative accuracy improvement on the majority nodes also increases, further indicating the disparity effects on different nodes group with more improvement on the majority nodes. (2) For heterophilic graphs, the local agreement ratio shows an opposite phenomenon, which decreases consistently. Meanwhile, the relative accuracy improvement on the majority nodes only increases on the first two hops and decrease and decline from the third hop. The potential reason is that, despite a general global trend, the heterophilic patterns on each individual node may still be quite complicated, with a local pattern shift disparity. We leave the discussion on more complex local structure patterns as the future work.
(a) Homophilic graphs
(b) Heterophilic graphs
Figure 12: Illustration of the change on local agreement accuracy difference between the majority and minority patterns along with the aggregation. The x-axis represents the number of aggregations and the y-axis represents the majority local agreement accuracy minus the minority local agreement accuracy. For homophilic graphs, the relative accuracy improvement on the majority nodes also increases, further, while the relative accuracy improvement on the majority nodes only increases on the first two hops and decrease and decline from the third hop.
D Effects of aggregation on nodes in different classes with structural disparity
In Section 3.1, we examine the behavior difference between nodes from the same class but with different structural patterns. In this section, we further provide a more complicated analysis focusing on the between-class patterns, i.e., linear separability. Notably, linear separability is a good indicator of feature differences in different classes, where features with better linear separability can be easier to distinguish with a suitable linear classifier.
D.1 Linear separability analysis based on CSBM model
To ease the analysis, we utilize the CSBM-S model as the data assumption showing as follows:
Definition 1 (CSBM-S( $\mu_1, \mu_2$ ,  $(p^{(1)}, q^{(1)})$ ,  $(p^{(2)}, q^{(2)})$ ,  $\operatorname{Pr}(\text{homo}))$ ). The generated nodes consist of two disjoint sets  $\mathcal{C}_1$  and  $\mathcal{C}_2$ . Each node feature  $x$  is sampled from  $N(\mu_i, I)$  with  $i \in \{1, 2\}$ . Each set  $\mathcal{C}_i$  consists of two subgroups:  $\mathcal{C}_i^{(1)}$  for nodes in homophilic pattern with intra-class and inter-class edge probability  $p^{(1)} > q^{(1)}$  and  $\mathcal{C}_i^{(2)}$  for nodes in heterophilic pattern with  $p^{(2)} > q^{(2)}$ .  $\operatorname{Pr}(\text{homo})$  denotes the probability that the node is in the homophilic pattern.  $\mathcal{C}_i^{(j)}$  denotes node in class  $i$  and subgroup  $j$  with  $(p^{(j)}, q^{(j)})$ . We assume nodes follow the same degree distribution with  $p^{(1)} + q^{(1)} = p^{(2)} + q^{(2)}$ .
The original node features follow the Gaussian distribution:
$$
\mathbf {x} _ {i} \sim N \left(\boldsymbol {\mu} _ {1}, \mathbf {I}\right) \text {f o r} i \in \mathcal {C} _ {1}; \text {a n d} \mathbf {x} _ {i} \sim N \left(\boldsymbol {\mu} _ {2}, \mathbf {I}\right) \text {f o r} i \in \mathcal {C} _ {2} \tag {6}
$$
Where  $\mathcal{C}_i$  corresponds to the node set corresponding to class  $i$ .
$$
\mathbf {f} _ {i} ^ {(j)} \sim N \left(\frac {p ^ {(j)} \boldsymbol {\mu} _ {1} + q ^ {(j)} \boldsymbol {\mu} _ {2}}{p ^ {(j)} + q ^ {(j)}} \cdot \frac {\mathbf {I}}{\sqrt {d _ {i}}}\right), \text {f o r} i \in \mathcal {C} _ {1} ^ {(j)}; \mathbf {f} _ {i} ^ {(j)} \sim N \left(\frac {q ^ {(j)} \boldsymbol {\mu} _ {1} + p ^ {(j)} \boldsymbol {\mu} _ {2}}{p ^ {(j)} + q ^ {(j)}} \cdot \frac {\mathbf {I}}{\sqrt {d _ {i}}}\right), \text {f o r} i \in \mathcal {C} _ {2} ^ {(j)}
$$
Where  $\mathcal{C}_i^{(j)}$  is the node subgroups with structural pattern with  $(p^{(j)},q^{(j)})$  in label  $i$ . To more accurately assess the effectiveness of the aggregation mechanism, we execute the largest-margin linear classifiers on nodes and evaluate their performance, to illustrate the linear separability. Notably, linear separability depends on the distance between the mean features of different classes as well as the standard deviations within each class. Typically, we focus on examining the linear separability on (1) nodes from different classes with the same structural patterns (2) nodes from different classes with different structural patterns, i.e., homophilic and heterophilic patterns.
We first examine the linear separability for nodes from different classes within the same structural pattern. We can summarize the proposition as follows:
Lemma 2 (Linear separability on nodes with the same structural patterns). Considering mean aggregated features are from the same structural pattern  $\mathbf{f}_i^{(j)}$ , for  $i \in \{1,2\}$ . For any node  $i$ , the largest-margin linear classifier on  $\mathbf{f}_i^{(j)}$  will have a lower probability to misclassify than  $\mathbf{x}_i$ , when  $d_i > \frac{(p^{(1)} + q^{(1)})^2}{(p^{(1)} - q^{(1)})^2}$