Title: Demystifying Structural Disparity in Graph Neural Networks: Can One Size Fit All?
Venue: NeurIPS 2023
Accepted: yes
Keywords: Graph Neural Network
Abstract: Recent studies on Graph Neural Networks(GNNs) provide both empirical and theoretical evidence supporting their effectiveness in capturing structural patterns on both homophilic and certain heterophilic graphs. Notably, most real-world homophilic and heterophilic graphs are comprised of a mixture of nodes in both homophilic and heterophilic structural patterns, exhibiting a structural disparity. However, the analysis of GNN performance with respect to nodes exhibiting different structural patterns, e.g., homophilic nodes in heterophilic graphs, remains rather limited. In the present study, we provide evidence that Graph Neural Networks(GNNs) on node classification typically perform admirably on homophilic nodes within homophilic graphs and heterophilic nodes within heterophilic graphs while struggling on the opposite node set, exhibiting a performance disparity. We theoretically and empirically identify effects of GNNs on testing nodes exhibiting distinct structural patterns. We then propose a rigorous, non-i.i.d PAC-Bayesian generalization bound for GNNs, revealing reasons for the performance disparity, namely the aggregated feature distance and homophily ratio difference between training and testing nodes. Furthermore, we demonstrate the practical implications of our new findings via (1) elucidating the effectiveness of deeper GNNs; and (2) revealing an over-looked distribution shift factor on graph out-of-distribution problem and proposing a new scenario accordingly.

Demystifying Structural Disparity in Graph Neural

Abstract

Recent studies on Graph Neural Networks(GNNs) provide both empirical and theoretical evidence supporting their effectiveness in capturing structural patterns on both homophilic and certain heterophilic graphs. Notably, most real-world homophilic and heterophilic graphs are comprised of a mixture of nodes in both homophilic and heterophilic structural patterns, exhibiting a structural disparity. However, the analysis of GNN performance with respect to nodes exhibiting different structural patterns, e.g., homophilic nodes in heterophilic graphs, remains rather limited. In the present study, we provide evidence that Graph Neural Networks(GNNs) on node classification typically perform admirably on homophilic nodes within homophilic graphs and heterophilic nodes within heterophilic graphs while struggling on the opposite node set, exhibiting a performance disparity. We theoretically and empirically identify effects of GNNs on testing nodes exhibiting distinct structural patterns. We then propose a rigorous, non-i.i.d PAC-Bayesian generalization bound for GNNs, revealing reasons for the performance disparity, namely the aggregated feature distance and homophily ratio difference between training and testing nodes. Furthermore, we demonstrate the practical implications of our new findings via (1) elucidating the effectiveness of deeper GNNs; and (2) revealing an over-looked distribution shift factor on graph out-of-distribution problem and proposing a new scenario accordingly.

1 Introduction

Graph Neural Networks (GNNs) [1-4] are a powerful technique for tackling a wide range of graph-related tasks [5, 3, 6-10], especially node classification [2, 4, 11, 12], which requires predicting unlabeled nodes based on the graph structure, node features, and a subset of labeled nodes. The success of GNNs can be ascribed to their ability to capture structural patterns through the aggregation mechanism that effectively combines feature information from neighboring nodes [13].

Aggregation 0 1 0 1 0 1 0 1 Figure 1: Examples of homophilic and heterophilic patterns. Colors/numbers indicate node features/labels. GNNs have been widely recognized for their effectiveness on homophilic graphs [14, 2, 11, 4, 15-17]. In homophilic graphs, connected nodes tend to share the same label, which we refer to as homophilic patterns . An example of the homophilic pattern is depicted in the upper part of Figure 1, where node features and node labels are denoted by colors (i.e., blue and red) and numbers (i.e., 0 and 1), respectively. We can observe that all connected nodes exhibit homophilic patterns and share the same label 0. Recently, several studies have demonstrated that GNNs can also perform well on certain heterophilic graphs [18, 13, 19]. In heterophilic graphs, connected nodes

Aggregation

Homophily

Heterophily

0

0

0

0

0

0

0

0

0

1

0

1

tend to have different labels, which we refer to as heterophilic patterns . The example in the lower part of Figure 1 shows the heterophilic patterns. Based on this example, we intuitively illustrate how GNNs can work on such heterophilic patterns (lower right): after averaging features over all neighboring nodes, nodes with label 0 completely switch from their initial blue color to red, and vice versa; despite this feature alteration, the two classes remain easily distinguishable since nodes with the same label (number) share the same color (features).

However, existing studies on the effectiveness of GNNs [14, 18, 13, 19] only focus on either homophilic or heterophilic patterns solely and overlook the fact that real-world graphs typically exhibit a mixture of homophilic and heterophilic patterns. Recent studies [20, 21] reveal that many heterophilic graphs, e.g., Squirrel and Chameleon [22], contain over 20% homophilic nodes. Similarly, our preliminary study depicted in Figure 2 demonstrates that heterophilic nodes are consistently present in many homophilic graphs, e.g., PubMed [23] and Ogbn-arxiv [24]. Hence, real-world homophilic graphs predominantly consist of homophilic nodes as the majority structural pattern and heterophilic nodes in the minority one, while heterophilic graphs exhibit an opposite phenomenon with heterophilic nodes in the majority and homophilic ones in the minority.

To provide insights aligning the real-world scenario with structural disparity, we revisit the toy example in Figure 1, considering both homophilic and heterophilic patterns together. Specifically, for nodes labeled 0, both homophilic and heterophilic node features appear in blue before aggregation. However, after aggregation, homophilic and heterophilic nodes in label 0 exhibit different features, appearing blue and red, respectively. Such differences may lead to performance disparity between nodes in majority and minority patterns. For instance, in a homophilic graph with the majority pattern being homophilic, GNNs are more likely to learn the association between blue features and class 0 on account of more supervised signals in majority. Consequently, nodes in the majority structural pattern can perform well, while nodes in the minority structural pattern may exhibit poor performance, indicating an over-reliance on the majority structural pattern. Inspired by insights from the above toy example, we focus on answering following questions systematically in this paper: How does a GNN behave when encountering the structural disparity of homophilic and heterophilic nodes within a dataset? and Can one GNN benefit all nodes despite structural disparity?

Present work . Drawing inspiration from above intuitions, we investigate how GNNs exhibit different effects on nodes with structural disparity, the underlying reasons, and implications on graph applications. Our study proceeds as follows: First , we empirically verify the aforementioned intuition by examining the performance of testing nodes w.r.t. different homophily ratios, rather than the overall performance across all test nodes as in [13, 14, 19]. We show that GCN [2], a vanilla GNN, often underperforms MLP-based models on nodes with the minority pattern while outperforming them on the majority nodes. Second , we examine how aggregation, the key mechanism of GNNs, shows different effects on homophilic and heterophilic nodes. We propose an understanding of why GNNs exhibit performance disparity with a non-i.i.d PAC-Bayesian generalization bound, revealing that both feature distance and homophily ratio differences between train and test nodes are key factors leading to performance disparity. Third , we showcase the significance of these insights by exploring implications for (1) elucidating the effectiveness of deeper GNNs and (2) introducing a new graph out-of-distribution scenario with an over-looked distribution shift factor. Codes are available at here.

2 Prelimaries

Semi-Supervised Node classification (SSNC). Let G = ( V, E ) be an undirected graph, where V = { v 1 , · · · , v n } is the set of n nodes and E ⊆ V × V is the edge set. Nodes are associated with node features X ∈ R n × d , where d is the feature dimension. The number of class is denoted as K . The adjacency matrix A ∈ { 0 , 1 } n × n represents graph connectivity where A [ i, j ] = 1 indicates an edge between nodes i and j . D is a degree matrix and D [ i, i ] = d i with d i denoting degree of node v i . Given a small set of labeled nodes, V tr ⊆ V , SSNC task is to predict on unlabeled nodes V \ V tr .

Node homophily ratio is a common metric to quantify homophilic and heterophilic patterns. It is calculated as the proportion of a node's neighbors sharing the same label as the node [25, 26, 20]. It is formally defined as h i = |{ u ∈N ( v i ): y u = y v }| d i , where N ( v i ) denotes the neighbor node set of v i and d i = |N ( v i ) | is the cardinality of this set. Following [20, 27, 25], node i is considered to be homophilic when more neighbor nodes share the same label as the center node with h i > 0 . 5 . We

define the graph homophily ratio h as the average of node homophily ratios h = ∑ i ∈ V h i | V | . Moreover, this ratio can be easily extended to higher-order cases h ( k ) i by considering k -order neighbors N k ( v i ) .

Node subgroup refers to a subset of nodes in the graph sharing similar properties, typically homophilic and heterophilic patterns measured with node homophily ratio. Training nodes are denoted as V tr. Test nodes V te can be categorized into M node subgroups, V te = ⋃ M m =1 V m , where nodes in the same subgroup V m share similar structural pattern.

3 Effectiveness of GNN on nodes with different structural properties

In this section, we explore the effectiveness of GNNs on different node subgroups exhibiting distinct structural patterns, specifically, homophilic and heterophilic patterns. It is different from previous studies [13, 14, 18, 28, 19] that primarily conduct analysis on the whole graph and demonstrate effectiveness with an overall performance gain. These studies, while useful, do not provide insights into the effectiveness of GNNs on different node subgroups, and may even obscure scenarios where GNNs fail on specific subgroups despite an overall performance gain. To accurately gauge the effectiveness of GNNs, we take a closer examination on node subgroups with distinct structural patterns. The following experiments are conducted on two common homophilic graphs, Ogbnarxiv [24] and Pubmed [23], and two heterophilic graphs, Chameleon and Squirrel [22]. These datasets are chosen since GNNs can achieve better overall performance than MLP. Experiment details and related work on GNN disparity are in Appendix G and A, respectively.

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

4

5

Density

(a) PubMed (

h

=0.79)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

Density

(b) Ogbn-arxiv (

h

=0.63)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

h

=0.22)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

Density

h

=0.25)

Existence of structural pattern disparity within a graph is to recognize real-world graphs exhibiting different node subgroups with diverse structural patterns, before investigating the GNN effectiveness on them. We demonstrate node homophily ratio distributions on the aforementioned datasets in Figure 2. We can have the following observations. Obs.1: All four graphs exhibit a mixture of both homophilic and heterophilic patterns, rather than a uniform structural patterns. Obs.2: In homophilic graphs, the majority of nodes exhibit a homophilic pattern with h i > 0 . 5 , while in heterophilic graphs, the majority of nodes exhibit the heterophilic pattern with h i ≤ 0 . 5 . We define nodes in majority structural pattern as majority nodes, e.g., homophilic nodes in a homophilic graph.

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

-0.2

-0.1

0.0

0.1

0.2

Difference in acc

model - GCN

MLP

GLNN

(a) PubMed (

h

=0.79)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.5

0.4

0.3

0.2

0.1

0.0

0.1

Difference in acc

model - GCN

MLP

GLNN

(b) Ogbn-arxiv (

h

=0.63)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.3

0.2

0.1

0.0

0.1

Difference in acc

model - GCN

MLP

GLNN

h

=0.22)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.1

0.0

0.1

0.2

0.3

Difference in acc

model - GCN

MLP

GLNN

h

=0.25)

Examining GCN performance on different structural patterns. To examine the effectiveness of GNNs on different structural patterns, we compare the performance of GCN [2] a vanilla GNN, with two MLP-based models, vanilla MLP and Graphless Neural Network (GLNN) [29], on testing nodes with different homophily ratios. It is evident that the vanilla MLP could have a large performance gap compared to GCN (i.e., 20% in accuracy) [13, 29, 2]. Consequently, an under-trained vanilla MLP comparing with a well-trained GNN leads to an unfair comparison without rigorous conclusion. Therefore, we also include an advanced MLP model GLNN. It is trained in an advanced manner via distilling GNN predictions and exhibits performance on par with GNNs. Notably, only GCN has the ability to leverage structural information during the inference phase while both vanilla MLP and GLNN models solely rely on node features as input. This comparison ensures a fair study on the effectiveness of GNNs in capturing different structural patterns with mitigating the effects of node features. Experimental results on four datasets are presented in Figure 3. In the figure, y-axis corresponds to the accuracy differences between GCN and MLP-based models where positive indicates MLP models can outperform GCN; while x-axis represents different node subgroups with nodes in the subgroup satisfying homophily ratios in the given range, e.g., [0.0-0.2]. Based on experimental results, the following key observations can be made: Obs.1: In homophilic graphs, both GLNN and MLP demonstrate superior performance on the heterophilic nodes with homophily ratios in [0-0.4] while GCN outperforms them on homophilic nodes. Obs.2: In heterophilic graphs, MLP models often outperform on homophilic nodes yet underperform on heterophilic nodes. Notably, vanilla MLP performance on Chameleon is worse than that of GCN across different subgroups. This can be attributed to the training difficulties encountered on Chameleon, where an unexpected similarity in node features from different classes is observed [30]. Our observations indicate that despite the effectiveness of GCN suggested by [13, 19, 14], GCN exhibits limitations with performance disparity across homophilic and heterophilic graphs. It motivates investigation why GCN benefits majority nodes, e.g., homophilic nodes in homophilic graphs, while struggling with minority nodes. Moreover, additional results on more datasets and significant test results are shown in Appendix H and L.

Organization. In light of the above observations, we endeavor to understand the underlying causes of this phenomenon in the following sections by answering the following research questions. Section 3.1 focuses on how aggregation, the fundamental mechanism in GNNs, affects nodes with distinct structural patterns differently. Upon identifying differences, Section 3.2 further analyzes how such disparities contribute to superior performance on the majority nodes as opposed to minority nodes. Building on these observations, Section 3.3 recognizes the key factors driving performance disparities on different structural patterns with a non-i.i.d. PAC-Bayes bound. Section 3.4 empirically corroborates the validity of our theoretical analysis with real-world datasets.

3.1 How does aggregation affect nodes with structural disparity differently?

In this subsection, we examine how aggregation reveals different effects on nodes with structural disparity, serving as a precondition for performance disparity. Specifically, we focus on the discrepancy between nodes from the same class but with different structural patterns.

For a controlled study on graphs, we adopt the contextual stochastic block model (CSBM) with two classes. It is widely used for graph analysis, including generalization [13, 14, 31, 18, 32, 33], clustering [34], fairness [35, 36], and GNN architecture design [37-39]. Typically, nodes in CSBM model are generated into two disjoint sets C 1 and C 2 corresponding to two classes, c 1 and c 2 , respectively. Each node with c i is associated with features x ∈ R d sampling from N ( µ i , I ) , where µ i is the feature mean of class c i with i ∈ { 1 , 2 } . The distance between feature means in different classes ρ = ∥ µ 1 -µ 2 ∥ , indicating the classification difficulty on node features. Edges are then generated based on intra-class probability p and inter-class probability q . For instance, nodes with class c 1 have probabilities p and q of connecting with another node in class c 1 and c 2 , respectively. The CSBM model, denoted as CSBM ( µ 1 , µ 2 , p, q ) , presumes that all nodes follow either homophilic with p > q or heterophilic patterns p < q exclusively. However, this assumption conflicts with real-world scenarios, where graphs often exhibit both patterns simultaneously, as shown in Figure 2. To mirror such scenarios, we propose a variant of CSBM, referred to as CSBM-Structure (CSBM-S), allowing for the simultaneous description of homophilic and heterophilic nodes.

Definition 1 (CSBM-S ( µ 1 , µ 2 , ( p (1) , q (1) ) , ( p (2) , q (2) ) , Pr( homo )) ) . The generated nodes consist of two disjoint sets C 1 and C 2 . Each node feature x is sampled from N ( µ i , I ) with i ∈ { 1 , 2 } . Each set C i consists of two subgroups: C (1) i for nodes in homophilic pattern with intra-class and

inter-class edge probability p (1) > q (1) and C (2) i for nodes in heterophilic pattern with p (2) < q (2) . Pr( homo ) denotes the probability that the node is in homophilic pattern. C ( j ) i denotes node in class i and subgroup j with ( p ( j ) , q ( j ) ) . We assume nodes follow the same degree distribution with p (1) + q (1) = p (2) + q (2) .

Based on the neighborhood distributions, the mean aggregated features F = D -1 AX obtained follow Gaussian distributions on both homophilic and heterophilic subgroups.

Where C ( j ) i is the node subgroups with structural pattern with ( p ( j ) , q ( j ) ) in label i . Our initial examination of different effects on aggregation focuses on the aggregated feature distance between homophilic and heterophilic node subgroups within class c 1 .

Proposition 1. The aggregated feature mean distance between homophilic and heterophilic node subgroups within class c 1 is ∥ ∥ ∥ p (1) µ 1 + q (1) µ 2 p (1) + q (1) -p (2) µ 1 + q (2) µ 2 p (2) + q (2) ∥ ∥ ∥ > 0 , indicating the aggregated feature of homophilic and heterophilic subgroups are from different feature distributions, with a mean distance larger than 0 distance before aggregation, since original node features draw from the same distribution, regardless of different structural patterns.

Notably, the distance between original features is regardless of the structural pattern. This proposition suggests that aggregation results in a distance gap between different patterns within the same class.

In addition to node feature differences with the same class, we further examine the discrepancy between nodes u and v with the same aggregated feature f u = f v but different structural patterns. We examine the discrepancy with the probability difference of nodes u and v in class c 1 , denoted as | P 1 ( y u = c 1 | f u ) -P 2 ( y v = c 1 | f v ) | . P 1 and P 2 are the conditional probability of y = c 1 given the feature f on structural patterns ( p (1) , q (1) ) and ( p (2) , q (2) ) , respectively.

Lemma 1. With assumptions (1) A balance class distribution with P ( Y = 1) = P ( Y = 0) and (2) aggregated feature distribution shares the same variance σ . When nodes u and v have the same aggregated features f u = f v but different structural patterns, ( p (1) , q (1) ) and ( p (2) , q (2) ) , we have:

Notably, above assumptions are not strictly necessary but employed for elegant expression. Lemma 1 implies that nodes with a small homophily ratio difference | h 1 -h 2 | are likely to share the same class, and vice versa. Proof details and additional analysis on between-class effects are in Appendix D.

3.2 How does Aggregation Contribute to Performance Disparity?

We have established that aggregation can affect nodes with distinct structural patterns differently. However, it remains to be elucidated how such disparity contributes to performance improvement predominantly on majority nodes as opposed to minority nodes. It should be noted that, notwithstanding the influence on features, test performance is also profoundly associated with training labels. Performance degradation may occur when the classifier is inadequately trained with biased training labels.

We then conduct an empirical discriminative analysis taking both mean aggregated features and training labels into consideration. Drawing inspiration from existing literature [40-43], we describe the discriminative ability with the distance between train class prototypes [44, 45], i.e., feature mean of each class, and the corresponding test class prototype within the same class i . For instance, it can be denoted as || µ tr i -µ ma i || , where µ tr i and µ ma i are the prototype of class i on train nodes and test majority nodes, respectively. A smaller value suggests that majority test nodes are close to train nodes within

0

1

2

3

4

5

#aggregation

0.00

0.25

0.50

0.75

1.00

1.25

1.50

Ratio

Chameleon

Arxiv

PubMed

the same class, thus implying superior discriminative ability. A relative discriminative ratio is then proposed to compare the discriminative ability between majority and minority nodes. It can be denoted as: r = ∑ K i =1 || µ tr i -µ ma i || || µ tr i -µ mi i || where µ mi i corresponds to the prototype on minority test nodes. A lower relative discriminative ratio suggests that majority nodes are easier to be predicted than minority nodes.

The relative discriminative ratios are then calculated on different hop aggregated features and original features denote as 0-hop. Experimental results are presented in Figure 4, where the discriminative ratio shows an overall decrease tendency as the number of aggregations increases across four datasets. This indicates that majority test nodes show better discriminative ability than the minority test nodes along with more aggregation. We illustrate more results on GCN in Appendix K. Furthermore, instance-level experiments other than class prototypes are in Appendix C.

3.3 Why does Performance Disparity Happen? Subgroup Generalization Bound for GNNs

In this subsection, we conduct a rigorous analysis elucidating primary causes for performance disparity across different node subgroups with distinct structural patterns. Drawing inspiration from the discriminative metric described in Section 3.2, we identify two key factors for satisfying test performance: (1) test node u should have a close feature distance min v ∈ V tr ∥ f u -f v ∥ to training nodes V tr, indicating that test nodes can be greatly influenced by training nodes. (2) With identifying the closest training node v , nodes u and v should be more likely to share the same class, where ∑ c i ∈C | P ( y u = c i | f u ) -P ( y v = c i | f v ) | is required to be small. The second factor, focusing on whether two close nodes are in the same class, is dependent on the homophily ratio difference | h u -h v | , as shown in Lemma 1. Notably, since training nodes are randomly sampled, their structural patterns are likely to be the majority one. Therefore, training nodes will show a smaller homophily ratio difference with majority test nodes sharing the same majority pattern than minority test nodes, resulting in the performance disparity in distinct structural patterns. We substantiate the above intuitions with controllable synthetic experiments in Appendix B.

To rigorously examine the role of aggregated feature distance and homophily ratio difference in performance disparity, we derive a non-i.i.d. PAC-Bayesian GNN generalization bound, based on the Subgroup Generalization bound of Deterministic Classifier [46]. We begin by stating key assumptions on graph data and GNN model to clearly delineate the scope of our theoretical analysis. All remaining assumptions, proof details, and background on PAC-Bayes analysis can be found in Appendix F. Moreover, a comprehensive introduction on the generalization ability on GNN can be found in A.

Definition 2 (Generalized CSBM-S model) . Each node subgroup V m follows the CSBM distribution V m ∼ CSBM ( µ 1 , µ 2 , p ( i ) , q ( i ) ) , where different subgroups share the same class mean but different intra-class and inter-class probabilities p ( i ) and q ( i ) . Moreover, node subgroups also share the same degree distribution as p ( i ) + q ( i ) = p ( j ) + q ( j ) .

Instead of CSBM-S model with one homophilic and heterophilic pattern, we take the generalized CSBM-S model assumption, allowing more structural patterns with different levels of homophily.

Assumption 1 (GNN model) . We focus on SGC [16] with the following components: (1) a one-hop mean aggregation function g with g ( X,G ) denoting the output. (2) MLP feature transformation f ( g i ( X,G ); W 1 , W 2 , · · · , W L ) , where f is a ReLU-activated L -layer MLP with W 1 , · · · , W L as parameters for each layer. The largest width of all the hidden layers is denoted as b .

Notably, despite analyzing simple GNN architecture theoretically, similar with [46, 13, 47], our theory analysis could be easily extended to the higher-order case with empirical success across different GNN architectures shown in Section 3.4.

Our main theorem is based on the PAC-Bayes analysis which typically aims to bound the generalization gap between the expected margin loss L 0 m on test subgroup V m for a margin 0 and the empirical margin loss ̂ L γ tr on train subgroup V tr for a margin γ . Those losses are generally utilized in PAC-Bayes analysis[48-51]. More details are found in Appendix F. The formulation is shown as follows:

Theorem 1 (Subgroup Generalization Bound for GNNs) . Let ˜ h be any classifier in the classifier family H with parameters { ˜ W l } L l =1 . for any 0 < m ≤ M , γ ≥ 0 , and large enough number of the training nodes N tr = | V tr | , there exist 0 < α < 1 4 with probability at least 1 -δ over the sample of

y tr := { y i } i ∈ V tr , we have:

The bound is related to three terms: (a) describes both large homophily ratio difference | h tr -h m | and large aggregated feature distance ϵ = max j ∈ bV m min i ∈ V tr ∥ g i ( X,G ) -g j ( X,G ) ∥ 2 between test node subgroup V m and training nodes V tr lead to large generalization error. ρ = ∥ µ 1 -µ 2 ∥ denotes the original feature separability, independent of structure. K is the number of classes. (b) further strengthens the effect of nodes with the aggregated feature distance ϵ , leads to a large generalization error. (c) R is a term independent with aggregated feature distance and homophily ratio difference, depicted as 1 N 1 -2 α tr + 1 N 2 α tr ln LC (2 B m ) 1 /L γ 1 /L δ , where B m = max i ∈ V tr ∪ V m ∥ g i ( X,G ) ∥ 2 is the maximum feature norm. R vanishes as training size N 0 grows. Proof details are in Appendix F

Our theory suggests that both homophily ratio difference and aggregated feature distance to training nodes are key factors contributing to the performance disparity. Typically, nodes with large homophily ratio difference and aggregated feature distance to training nodes lead to performance degradation.

3.4 Performance Disparity Across Node Subgroups on Real-World Datasets

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

1.0

ACC

(a) PubMed (

h

=0.79)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv (

h

=0.63)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

h

=0.22)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

h

=0.25)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(a) PubMed (

h

=0.79)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv (

h

=0.63)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

h

=0.22)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

h

=0.25)

To empirically examine the effects of theoretical analysis, we compare the performance on different node subgroups divided with both homophily ratio difference and aggregated feature distance to training nodes with popular GNN models including GCN [2], SGC [16], GAT [11], GCNII [52], and GPRGNN [53]. Typically, test nodes are partitioned into subgroups based on their disparity scores to the training set in terms of both 2-hop homophily ratio h (2) i and 2-hop aggregated features F (2) obtained by F (2) = ( ˜ D -1 ˜ A ) 2 X , where ˜ A = A + I and ˜ D = D + I . For a test node i , we measure the node disparity by (1) selecting the closest training node v = arg min v ∈ V 0 || F (2) u -F (2) v || (2) then calculating the disparity score s u = || F (2) u -F (2) v || 2 + | h (2) u -h (2) v | , where the first and the second terms correspond to the aggregated-feature distance and homophily ratio differences,

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

1.0

ACC

(a) PubMed (

h

=0.79)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv (

h

=0.63)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

h

=0.22)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

h

=0.25)

respectively. We then sort test nodes in terms of the disparity score and divide them into 5 equalbinned subgroups accordingly. Performance on different node subgroups is presented in Figure 5 with the following observations. Obs.1: We note a clear test accuracy degradation with respect to the increasing differences in aggregated features and homophily ratios. Furthermore, we investigate on the individual effect of aggregated feature distance and homophily ratio difference in Figure 6 and 7, respectively. An overall trend of performance decline with increasing disparity score is evident though some exceptions are present. Obs.2: When only considering the aggregated feature distance, there is no clear trend among groups 1, 2, and 3 on GCN, SGC, and GAT on heterophilic datasets. Obs.3: When only considering the homophily ratio difference, there is no clear trend among groups 1, 2, and 3 across four datasets. These observations underscore the importance of both aggregated-feature distance and homophily ratio differences in shaping GNN performance disparity. Combining these factors together provides a more comprehensive and accurate understanding of the reason for GNN performance disparity. For a more comprehensive analysis, we further substantiate our finding involving higher-order information and a wider array of datasets in Appendix J.1.

Summary In this section, we study GNN performance disparity on nodes with distinct structural patterns and uncover its underlying causes. We primarily investigate the impact of aggregation, the key component in GNNs, on nodes with different structural patterns in Sections 3.1 and 3.2. We observe that aggregation effects vary across nodes with different structural patterns, notably enhancing the discriminative ability on majority nodes. These observed performance disparities inspire us to identify crucial factors contributing to GNN performance disparities across nodes with a non-i.i.d PAC-Bayes bound in Section 3.3. The theoretical analysis indicates that test nodes with larger aggregated feature distances and homophily ratio differences with training nodes experience performance degradation. We substantiate our findings on real-world datasets in Section 3.4.

4 Implications of graph structural disparity

In this section, we illustrate the significance of our findings on structural disparity via (1) elucidating the effectiveness of existing deeper GNNs (2) unveiling an over-looked aspect of distribution shift on graph out-of-distribution (OOD) problem, and introducing a new OOD scenario accordingly. Experimental details and discussions on more implications are in Appendix G and O, respectively.

4.1 Elucidate the effectiveness of Deeper GNNs

Deeper GNNs [53, 52, 15, 54-57] enable each node to capture more complex higher-order graph structure than vanilla GCN, via reducing the over-smoothing problem [58-60, 55, 61] Deeper GNNs empirically exhibit overall performance improvement, as demonstrated in Appendix G.2. Nonetheless, which structural patterns deeper GNNs can exceed and the reason for its effectiveness remain unclear. To investigate this problem, we compare vanilla GCN with different deeper GNNs, including GPRGNN[53], APPNP[15], and GCNII[52], on node subgroups with varying homophily ratios, adhering the same setting with Figure 3. Experimental results are shown in Figure 8. We can observe that deeper GNNs primarily surpass GCN on minority node subgroups with slight performance trade-offs on the majority node subgroups. We conclude that the effectiveness of deeper GNNs majorly contributes to improved discriminative ability on minority nodes. Additional results on more datasets and significant test are in Appendix I and M.

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

-0.2

-0.1

0.0

0.1

0.2

Difference in acc

model - GCN

GPRGNN

APPNP

(a) PubMed (

h

=0.79)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.04

0.02

0.00

0.02

0.04

Difference in acc

model - GCN

GPRGNN

APPNP

(b) Ogbn-arxiv (

h

=0.63)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.2

0.1

0.0

0.1

0.2

Difference in acc

model - GCN

GPRGNN

APPNP

h

=0.22)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.2

0.1

0.0

0.1

0.2

0.3

Difference in acc

model - GCN

GPRGNN

APPNP

h

=0.25)

Having identified where deeper GNNs excel, reasons why effectiveness primarily appears in the minority node group remain elusive. Since the superiority of deeper GNNs stems from capturing higher-order information, we further investigate how higher-order homophily ratio differences vary on the minority nodes, denoted as, | h ( k ) u -h ( k ) v | , where node u is the test node, node v is the closest train node to test node u . We concentrate on analyzing these minority nodes V mi in terms of default one-hop homophily ratio h u and examine how ∑ u ∈ V mi | h ( k ) u -h ( k ) v | varies with different k orders. Experimental results are shown in Figure 9, where a decreasing trend of homophily ratio difference is observed along with more neighborhood hops. The smaller homophily ratio difference leads to smaller generalization errors with better performance. This observation is consistent with [20], where heterophilic nodes in heterophilic graphs exhibit large higher-order homophily ratios, implicitly leading to a smaller homophily ratio difference.

4.2 A new graph out-of-distribution scenario

̸

The graph out-of-distribution (OOD) problem refers to the underperformance of GNN due to distribution shifts on graphs. Many Graph OOD scenarios [62-65, 24, 66, 46, 67], e.g., biased training labels, time shift, and popularity shift, have been extensively studied. These OOD scenarios can be typically categorized into covariate shift with P train ( X ) = P test ( X ) and concept shift [68-70] with P train ( Y | X ) = P test ( Y | X ) . P train ( · ) and P test ( · ) denote train and test distributions, respectively. Existing graph concept shift scenarios [62, 66] introduce different environment variables e resulting in P ( Y | X , e train ) = P ( Y | X , e test ) leading to spurious correlations. To address existing concept shifts, algorithms [62, 71] have been developed to capture the environment-invariant relationship P ( Y | X ) . Nonetheless, existing concept shift settings overlook the scenario where there is not a unique environment-invariant relationship P ( Y | X ) . For instance, P ( Y | X homo ) and P ( Y | X hete ) can be different, indicated in Section 3.1. X homo and X hete correspond to features of nodes in homophilic and heterophilic patterns. Notably, homophilic and heterophilic patterns are crucial task knowledge that cannot be recognized as the irrelevant environmental variable. Consequently, we find that homophily ratio difference between train and test sets could be an important factor leading to an overlook concept shift, namely, graph structural shift. Notably, structural patterns cannot be considered as environment variables given their integral role in node classification task. The practical implications of this concept shift are substantiated by the following scenarios: (1) graph structural shift frequently occurs in most graphs, with a performance degradation in minority nodes, as depicted in Figure 3. (2) graph structural shift hides secretly in existing graph OOD scenarios. For instance, the FaceBook-100 dataset [62] reveals a substantial homophily ratio difference between train and test sets, averaging 0.36. This discrepancy could be the primary cause of OOD performance deterioration since the exist OOD algorithms [62, 72] that neglect such a concept shift can only attain a minimal average performance gain of 0.12%. (3) graph structural shift is a recurrent phenomenon in numerous real-world applications where new nodes in graphs may exhibit distinct structural patterns. For example, in a recommendation system, existing users with rich data can receive well-personalized recommendations in the exploitation stage (homophily), while new users with less data may receive diverse recommendations during the exploration stage (heterophily).

1

2

3

4

5

#hop

0.0

0.2

0.4

0.6

Ratio difference

PubMed

Arxiv

Chameleon

̸

̸

Given the prevalence and importance of the graph structural shift, we propose a new graph OOD scenario emphasizing this concept shift. Specifically, we introduce a new data split on existing datasets, namely Cora, CiteSeer, PubMed, Ogbn-Arxiv, Chameleon, and Squirrel, where majority nodes are selected for train and validation, and minority ones for test. This data split strategy highlights the

homophily ratio difference and the corresponding concept shift. To better illustrate the challenges posed by our new scenario, we conduct experiments on models including GCN, MLP, GLNN, GPRGNN, and GCNII. We also include graph OOD algorithms, SRGNN [63] and EERM [62], with GCN encoders. EERM(II) is a variant of EERM with a GCNII encoder For a fair comparison, we show GCN performance on an i.i.d. random split, GCN(i.i.d.), sharing the same node sizes for train, validation, and test. Results are shown in Table 1 while additional ones are in Appendix G.4. Following observations can be made: Obs.1: The performance degradation can be found by comparing OOD setting with i.i.d. one across four datasets, confirming OOD issue existence. Obs.2: MLP-based models and deeper GNNs generally outperform vanilla GCN, demonstrating the superiority on minority nodes. Obs.3: Graph OOD algorithms with GCN encoders struggle to yield good performance across datasets, indicating a unique challenge over other Graph OOD scenarios. This primarily stems from the difficulty in learning both accurate relationships on homophilic and heterophilic nodes with distinct P ( Y | X ) . Nonetheless, it can be alleviated by selecting a deeper GNN encoder, as the homophily ratio difference may vanish in higher-order structure information, with reduced concept shift. Obs.4: EERM(II), EERM with GCNII, outperforms the one with GCN. Observations suggest that GNN architecture plays an indispensable role in addressing graph OOD issues, highlighting the new direction.

5 Conclusion & Discussion

In conclusion, this work provides crucial insights into GNN performance meeting structural disparity, common in real-world scenarios. We recognize that aggregation exhibits different effects on nodes with structural disparity, leading to better performance on majority nodes than those in minority. The understanding also serves as a stepping stone for multiple graph applications.

Our exploration majorly focuses on common datasets with clear majority structural patterns while realworld scenarios, offering more complicated datasets, posing new challenges Additional experiments are conducted on Actor, IGB-tiny, Twitch-gamer, and Amazon-ratings. Dataset details and experiment results are in Appendix G.3 and Appendices H-K, respectively. Despite our understanding is empirically effective on most datasets, further research and more sophisticated analysis are still necessary. Discussions on the limitation and broader impact are in Appendix N and O, respectively.

References

Yao Ma and Jiliang Tang. Deep learning on graphs . Cambridge University Press, 2021.

Thomas Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ArXiv , abs/1609.02907, 2016.

Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations .

Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. Advances in neural information processing systems , 31, 2018.

Xiangnan He, Kuan Deng, Xiang Wang, Yan Li, Yongdong Zhang, and Meng Wang. Lightgcn: Simplifying and powering graph convolution network for recommendation. In Proceedings of the 43rd International ACM SIGIR conference on research and development in Information Retrieval , pages 639-648, 2020.

Oliver Wieder, Stefan Kohlbacher, Mélaine Kuenemann, Arthur Garon, Pierre Ducrot, Thomas Seidel, and Thierry Langer. A compact review of molecular property prediction with graph neural networks. Drug Discovery Today: Technologies , 37:1-12, 2020.

Zhaocheng Zhu, Zuobai Zhang, Louis-Pascal Xhonneux, and Jian Tang. Neural bellman-ford networks: A general graph neural network framework for link prediction. Advances in Neural Information Processing Systems , 34:29476-29490, 2021.

Juanhui Li, Harry Shomer, Haitao Mao, Shenglai Zeng, Yao Ma, Neil Shah, Jiliang Tang, and Dawei Yin. Evaluating graph neural networks for link prediction: Current pitfalls and new benchmarking. arXiv preprint arXiv:2306.10453 , 2023.

Yanci Zhang, Yutong Lu, Haitao Mao, Jiawei Huang, Cien Zhang, Xinyi Li, and Rui Dai. Company competition graph. arXiv preprint arXiv:2304.00323 , 2023.

Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, Yoshua Bengio, et al. Graph attention networks. stat , 1050(20):10-48550, 2017.

Tong Zhao, Gang Liu, Daheng Wang, Wenhao Yu, and Meng Jiang. Learning from counterfactual links for link prediction. In International Conference on Machine Learning , pages 26911-26926. PMLR, 2022.

Yao Ma, Xiaorui Liu, Neil Shah, and Jiliang Tang. Is homophily a necessity for graph neural networks? ArXiv , abs/2106.06134, 2021.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Graph convolution for semisupervised classification: Improved linear separability and out-of-distribution generalization. arXiv preprint arXiv:2102.06966 , 2021.

Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. In International Conference on Learning Representations , 2018.

Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Weinberger. Simplifying graph convolutional networks. In International conference on machine learning , pages 6861-6871. PMLR, 2019.

Tong Zhao, Yozen Liu, Leonardo Neves, Oliver Woodford, Meng Jiang, and Neil Shah. Data augmentation for graph neural networks. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11015-11023, 2021.

Aseem Baranwal, Kimon Fountoulakis, and Aukosh Jagannath. Effects of graph convolutions in multi-layer networks. In The Eleventh International Conference on Learning Representations , 2023.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Mingde Zhao, Shuyuan Zhang, Xiao-Wen Chang, and Doina Precup. Revisiting heterophily for graph neural networks. In Advances in Neural Information Processing Systems .

Xiang Li, Renyu Zhu, Yao Cheng, Caihua Shan, Siqiang Luo, Dongsheng Li, and Wei Qian. Finding global homophily in graph neural networks when meeting heterophily. In International Conference on Machine Learning , 2022.

Derek Lim, Felix Hohne, Xiuyu Li, Sijia Linda Huang, Vaishnavi Gupta, Omkar Bhalerao, and Ser Nam Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , 34:20887-20902, 2021.

Benedek Rozemberczki, Carl Allen, and Rik Sarkar. Multi-scale attributed node embedding. Journal of Complex Networks , 9(2):cnab014, 2021.

Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina EliassiRad. Collective classification in network data. AI magazine , 29(3):93-93, 2008.

Weihua Hu, Matthias Fey, Marinka Zitnik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00687 , 2020.

Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gcn: Geometric graph convolutional networks. In International Conference on Learning Representations .

Jiong Zhu, Yujun Yan, Lingxiao Zhao, Mark Heimann, Leman Akoglu, and Danai Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33:7793-7804, 2020.

Lun Du, Xiaozhou Shi, Qiang Fu, Xiaojun Ma, Hengyu Liu, Shi Han, and Dongmei Zhang. Gbk-gnn: Gated bi-kernel graph neural networks for modeling both homophily and heterophily. In Proceedings of the ACM Web Conference 2022 , pages 1550-1558, 2022.

Sitao Luan, Chenqing Hua, Minkai Xu, Qincheng Lu, Jiaqi Zhu, Xiao-Wen Chang, Jie Fu, Jure Leskovec, and Doina Precup. When do graph neural networks help with node classification: Investigating the homophily principle on node distinguishability. arXiv preprint arXiv:2304.14274 , 2023.

Shichang Zhang, Yozen Liu, Yizhou Sun, and Neil Shah. Graph-less neural networks: Teaching old mlps new tricks via distillation. ArXiv , abs/2110.08727, 2021.

Sitao Luan, Chenqing Hua, Qincheng Lu, Jiaqi Zhu, Xiao-Wen Chang, and Doina Precup. When do we need gnn for node classification? arXiv preprint arXiv:2210.16979 , 2022.

Haonan Wang, Jieyu Zhang, Qi Zhu, and Wei Huang. Augmentation-free graph contrastive learning. arXiv preprint arXiv:2204.04874 , 2022.

Haonan Wang, Jieyu Zhang, Qi Zhu, and Wei Huang. Can single-pass contrastive learning work for both homophilic and heterophilic graph? arXiv preprint arXiv:2211.10890 , 2022.

Kimon Fountoulakis, Amit Levi, Shenghao Yang, Aseem Baranwal, and Aukosh Jagannath. Graph attention retrospective. arXiv preprint arXiv:2202.13060 , 2022.

Santo Fortunato and Darko Hric. Community detection in networks: A user guide. Physics reports , 659:1-44, 2016.

Zhimeng Jiang, Xiaotian Han, Chao Fan, Zirui Liu, Na Zou, Ali Mostafavi, and Xia Hu. Fmp: Toward fair graph message passing against topology bias. arXiv preprint arXiv:2202.04187 , 2022.

Zhimeng Jiang, Xiaotian Han, Chao Fan, Zirui Liu, Xiao Huang, Na Zou, Ali Mostafavi, and Xia Hu. Topology matters in fair graph learning: a theoretical pilot study.

Rongzhe Wei, Haoteng Yin, Junteng Jia, Austin R Benson, and Pan Li. Understanding nonlinearity in graph neural networks from the bayesian-inference perspective. In Advances in Neural Information Processing Systems .

Yoonhyuk Choi, Jiho Choi, Taewook Ko, and Chong-Kwon Kim. Is signed message essential for graph neural networks? arXiv preprint arXiv:2301.08918 , 2023.

Xinyi Wu, Zhengdao Chen, William Wang, and Ali Jadbabaie. A non-asymptotic analysis of oversmoothing in graph neural networks. arXiv preprint arXiv:2212.10701 , 2022.

Tomer Galanti, András György, and Marcus Hutter. On the role of neural collapse in transfer learning. arXiv preprint arXiv:2112.15121 , 2021.

Tomer Galanti, András György, and Marcus Hutter. Improved generalization bounds for transfer learning via neural collapse. In First Workshop on Pre-training: Perspectives, Pitfalls, and Paths Forward at ICML 2022 , 2022.

Mayee Chen, Daniel Y Fu, Avanika Narayan, Michael Zhang, Zhao Song, Kayvon Fatahalian, and Christopher Ré. Perfectly balanced: Improving transfer and robustness of supervised contrastive learning. In International Conference on Machine Learning , pages 3090-3122. PMLR, 2022.

Tomer Galanti, András György, and Marcus Hutter. Generalization bounds for transfer learning with pretrained classifiers. arXiv preprint arXiv:2212.12532 , 2022.

Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems , 29, 2016.

Jake Snell, Kevin Swersky, and Richard Zemel. Prototypical networks for few-shot learning. Advances in neural information processing systems , 30, 2017.

Jiaqi Ma, Junwei Deng, and Qiaozhu Mei. Subgroup generalization and fairness of graph neural networks. Advances in Neural Information Processing Systems , 34:1048-1061, 2021.

Vikas Garg, Stefanie Jegelka, and Tommi Jaakkola. Generalization and representational limits of graph neural networks. In International Conference on Machine Learning , pages 3419-3430. PMLR, 2020.

David McAllester. Simplified pac-bayesian margin bounds. In Learning Theory and Kernel Machines: 16th Annual Conference on Learning Theory and 7th Kernel Workshop, COLT/Kernel 2003, Washington, DC, USA, August 24-27, 2003. Proceedings , pages 203-215. Springer, 2003.

Andreas Maurer. A note on the pac bayesian theorem. arXiv preprint cs/0411099 , 2004.

Eugenio Clerico, George Deligiannidis, and Arnaud Doucet. Wide stochastic networks: Gaussian limit and pac-bayesian training. In International Conference on Algorithmic Learning Theory , pages 447-470. PMLR, 2023.

Gintare Karolina Dziugaite, Kyle Hsu, Waseem Gharbieh, Gabriel Arpino, and Daniel Roy. On the role of data in pac-bayes bounds. In International Conference on Artificial Intelligence and Statistics , pages 604-612. PMLR, 2021.

Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In International conference on machine learning , pages 1725-1735. PMLR, 2020.

Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations .

Sami Abu-El-Haija, Bryan Perozzi, Amol Kapoor, Nazanin Alipourfard, Kristina Lerman, Hrayr Harutyunyan, Greg Ver Steeg, and Aram Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning , pages 21-29. PMLR, 2019.

Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 338-348, 2020.

Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgcns: Can gcns go as deep as cnns? In Proceedings of the IEEE/CVF international conference on computer vision , pages 9267-9276, 2019.

Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International conference on machine learning , pages 5453-5462. PMLR, 2018.

Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. arXiv preprint arXiv:1905.10947 , 2019.

T Konstantin Rusch, Michael M Bronstein, and Siddhartha Mishra. A survey on oversmoothing in graph neural networks. arXiv preprint arXiv:2303.10993 , 2023.

Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI conference on artificial intelligence , volume 34, pages 3438-3445, 2020.

Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Proceedings of the AAAI conference on artificial intelligence , volume 32, 2018.

Qitian Wu, Hengrui Zhang, Junchi Yan, and David Wipf. Handling distribution shifts on graphs: An invariance perspective. arXiv preprint arXiv:2202.02466 , 2022.

Qi Zhu, Natalia Ponomareva, Jiawei Han, and Bryan Perozzi. Shift-robust gnns: Overcoming the limitations of localized graph training data. Advances in Neural Information Processing Systems , 34:27965-27977, 2021.

Yizhou Zhang, Guojie Song, Lun Du, Shuwen Yang, and Yilun Jin. Dane: Domain adaptive network embedding. International Joint Conference on Artificial Intelligence , abs/1906.00684:4362-4368, 2019.

Hongrui Liu, Binbin Hu, Xiao Wang, Chuan Shi, Zhiqiang Zhang, and Jun Zhou. Confidence may cheat: Self-training on graph neural networks under distribution shift. In Proceedings of the ACM Web Conference 2022 , pages 1248-1258, 2022.

Shurui Gui, Xiner Li, Limei Wang, and Shuiwang Ji. Good: A graph out-of-distribution benchmark. In Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track .

Haitao Mao, Lun Du, Yujia Zheng, Qiang Fu, Zelin Li, Xu Chen, Shi Han, and Dongmei Zhang. Source free unsupervised graph domain adaptation. arXiv preprint arXiv:2112.00955 , 2021.

Joaquin Quinonero-Candela, Masashi Sugiyama, Anton Schwaighofer, and Neil D Lawrence. Dataset shift in machine learning . Mit Press, 2008.

Jose G Moreno-Torres, Troy Raeder, Rocío Alaiz-Rodríguez, Nitesh V Chawla, and Francisco Herrera. A unifying view on dataset shift in classification. Pattern recognition , 45(1):521-530, 2012.

Gerhard Widmer and Miroslav Kubat. Learning in the presence of concept drift and hidden contexts. Machine learning , 23:69-101, 1996.

Shengyu Zhang, Kun Kuang, Jiezhong Qiu, Jin Yu, Zhou Zhao, Hongxia Yang, Zhongfei Zhang, and Fei Wu. Stable prediction on graphs with agnostic distribution shift. arXiv preprint arXiv:2110.03865 , 2021.

Wei Jin, Tong Zhao, Jiayuan Ding, Yozen Liu, Jiliang Tang, and Neil Shah. Empowering graph representation learning with test-time graph transformation. In The Eleventh International Conference on Learning Representations , 2023.

Wenqi Fan, Yao Ma, Qing Li, Yuan He, Eric Zhao, Jiliang Tang, and Dawei Yin. Graph neural networks for social recommendation. In The world wide web conference , pages 417-426, 2019.

Ruichao Yang, Xiting Wang, Yiqiao Jin, Chaozhuo Li, Jianxun Lian, and Xing Xie. Reinforcement subgraph reasoning for fake news detection. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2253-2262, 2022.

Yao Ma, Xiaorui Liu, Tong Zhao, Yozen Liu, Jiliang Tang, and Neil Shah. A unified view on graph neural networks as graph signal denoising. In Proceedings of the 30th ACM International Conference on Information & Knowledge Management , pages 1202-1211, 2021.

Meiqi Zhu, Xiao Wang, Chuan Shi, Houye Ji, and Peng Cui. Interpreting and unifying graph neural networks with an optimization framework. In Proceedings of the Web Conference 2021 , pages 1215-1226, 2021.

Jonathan Halcrow, Alexandru Mosoi, Sam Ruth, and Bryan Perozzi. Grale: Designing networks for graph learning. In Proceedings of the 26th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2523-2532, 2020.

Jiong Zhu, Ryan A Rossi, Anup Rao, Tung Mai, Nedim Lipka, Nesreen K Ahmed, and Danai Koutra. Graph neural networks with heterophily. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11168-11176, 2021.

Yujun Yan, Milad Hashemi, Kevin Swersky, Yaoqing Yang, and Danai Koutra. Two sides of the same coin: Heterophily and oversmoothing in graph convolutional neural networks. In 2022 IEEE International Conference on Data Mining (ICDM) , pages 1287-1292. IEEE, 2022.

Mingguo He, Zhewei Wei, Hongteng Xu, et al. Bernnet: Learning arbitrary graph spectral filters via bernstein approximation. Advances in Neural Information Processing Systems , 34:14239-14251, 2021.

Sannat Singh Bhasin, Vaibhav Holani, and Divij Sanjanwala. What do graph convolutional neural networks learn? arXiv preprint arXiv:2207.01839 , 2022.

Oleg Platonov, Denis Kuznedelev, Artem Babenko, and Liudmila Prokhorenkova. Characterizing graph datasets for node classification: Beyond homophily-heterophily dichotomy. arXiv preprint arXiv:2209.06177 , 2022.

Tahleen Rahman, Bartlomiej Surma, Michael Backes, and Yang Zhang. Fairwalk: Towards fair graph embedding. 2019.

Indro Spinelli, Simone Scardapane, Amir Hussain, and Aurelio Uncini. Fairdrop: Biased edge dropout for enhancing fairness in graph representation learning. IEEE Transactions on Artificial Intelligence , 3(3):344-354, 2021.

Ziqian Zeng, Rashidul Islam, Kamrun Naher Keya, James Foulds, Yangqiu Song, and Shimei Pan. Fair representation learning for heterogeneous information networks. In Proceedings of the International AAAI Conference on Web and Social Media , volume 15, pages 877-887, 2021.

Xianfeng Tang, Huaxiu Yao, Yiwei Sun, Yiqi Wang, Jiliang Tang, Charu Aggarwal, Prasenjit Mitra, and Suhang Wang. Investigating and mitigating degree-related biases in graph convoltuional networks. In Proceedings of the 29th ACM International Conference on Information & Knowledge Management , pages 1435-1444, 2020.

Yushun Dong, Ninghao Liu, Brian Jalaian, and Jundong Li. Edits: Modeling and mitigating data bias for graph neural networks. In Proceedings of the ACM Web Conference 2022 , pages 1259-1269, 2022.

Alan Mislove, Massimiliano Marcon, Krishna P Gummadi, Peter Druschel, and Bobby Bhattacharjee. Measurement and analysis of online social networks. In Proceedings of the 7th ACM SIGCOMM conference on Internet measurement , pages 29-42, 2007.

Simon S Du, Kangcheng Hou, Russ R Salakhutdinov, Barnabas Poczos, Ruosong Wang, and Keyulu Xu. Graph neural tangent kernel: Fusing graph neural networks with graph kernels. Advances in neural information processing systems , 32, 2019.

Renjie Liao, Raquel Urtasun, and Richard Zemel. A pac-bayesian approach to generalization bounds for graph neural networks. arXiv preprint arXiv:2012.07690 , 2020.

Saurabh Verma and Zhi-Li Zhang. Stability and generalization of graph convolutional neural networks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining , pages 1539-1548, 2019.

Franco Scarselli, Ah Chung Tsoi, and Markus Hagenbuchner. The vapnik-chervonenkis dimension of graph and recursive neural networks. Neural Networks , 108:248-259, 2018.

Weilin Cong, Morteza Ramezani, and Mehrdad Mahdavi. On provable benefits of depth in training graph convolutional networks. Advances in Neural Information Processing Systems , 34:9936-9949, 2021.

Ran El-Yaniv and Dmitry Pechyony. Stable transductive learning. In Learning Theory: 19th Annual Conference on Learning Theory, COLT 2006, Pittsburgh, PA, USA, June 22-25, 2006. Proceedings 19 , pages 35-49. Springer, 2006.

Gintare Karolina Dziugaite and Daniel M. Roy. Computing nonvacuous generalization bounds for deep (stochastic) neural networks with many more parameters than training data. In Proceedings of the 33rd Annual Conference on Uncertainty in Artificial Intelligence (UAI) , 2017.

Nan Ding, Xi Chen, Tomer Levinboim, Soravit Changpinyo, and Radu Soricut. Pactran: Pac-bayesian metrics for estimating the transferability of pretrained models to classification tasks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part XXXIV , pages 252-268. Springer, 2022.

Anthony Sicilia, Xingchen Zhao, Anastasia Sosnovskikh, and Seong Jae Hwang. Pac bayesian performance guarantees for deep (stochastic) networks in medical imaging. In Medical Image Computing and Computer Assisted Intervention-MICCAI 2021: 24th International Conference, Strasbourg, France, September 27-October 1, 2021, Proceedings, Part III 24 , pages 560-570. Springer, 2021.

Zifan Wang, Nan Ding, Tomer Levinboim, Xi Chen, and Radu Soricut. Improving robust generalization by direct pac-bayesian bound minimization. arXiv preprint arXiv:2211.12624 , 2022.

Joel A Tropp et al. An introduction to matrix concentration inequalities. Foundations and Trends® in Machine Learning , 8(1-2):1-230, 2015.

Behnam Neyshabur, Srinadh Bhojanapalli, and Nathan Srebro. A pac-bayesian approach to spectrally-normalized margin bounds for neural networks. arXiv preprint arXiv:1707.09564 , 2017.

Cheng Yang, Jiawei Liu, and Chuan Shi. Extract the knowledge of graph neural networks and go beyond it: An effective knowledge distillation framework. In Proceedings of the web conference 2021 , pages 1227-1237, 2021.

Jie Tang, Jimeng Sun, Chi Wang, and Zi Yang. Social influence analysis in large-scale networks. In Proceedings of the 15th ACM SIGKDD international conference on Knowledge discovery and data mining , pages 807-816, 2009.

Jure Leskovec and Andrej Krevl. Snap datasets: Stanford large network dataset collection, 2014.

Oleg Platonov, Denis Kuznedelev, Michael Diskin, Artem Babenko, and Liudmila Prokhorenkova. A critical look at the evaluation of GNNs under heterophily: Are we really making progress? In The Eleventh International Conference on Learning Representations , 2023.

Arpandeep Khatua, Vikram Sharma Mailthody, Bhagyashree Taleka, Tengfei Ma, Xiang Song, and Wen-mei Hwu. Igb: Addressing the gaps in labeling, features, heterogeneity, and size of public graph datasets for deep learning research. arXiv preprint arXiv:2302.13522 , 2023.

Arthur Gretton, Karsten M Borgwardt, Malte J Rasch, Bernhard Schölkopf, and Alexander Smola. A kernel two-sample test. The Journal of Machine Learning Research , 13(1):723-773, 2012.

Zhikai Chen, Haitao Mao, Hang Li, Wei Jin, Hongzhi Wen, Xiaochi Wei, Shuaiqiang Wang, Dawei Yin, Wenqi Fan, Hui Liu, et al. Exploring the potential of large language models (llms) in learning on graphs. arXiv preprint arXiv:2307.03393 , 2023.

Daniel Zügner, Amir Akbarnejad, and Stephan Günnemann. Adversarial attacks on neural networks for graph data. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining , pages 2847-2856, 2018.

Yiwei Sun, Suhang Wang, Xianfeng Tang, Tsung-Yu Hsieh, and Vasant Honavar. Adversarial attacks on graph neural networks via node injections: A hierarchical reinforcement learning approach. In Proceedings of the Web Conference 2020 , pages 673-683, 2020.

Kaidi Xu, Hongge Chen, Sijia Liu, Pin Yu Chen, Tsui Wei Weng, Mingyi Hong, and Xue Lin. Topology attack and defense for graph neural networks: An optimization perspective. In 28th International Joint Conference on Artificial Intelligence, IJCAI 2019 , pages 3961-3967. International Joint Conferences on Artificial Intelligence, 2019.

Simon Geisler, Tobias Schmidt, Hakan ¸ Sirin, Daniel Zügner, Aleksandar Bojchevski, and Stephan Günnemann. Robustness of graph neural networks at scale. Advances in Neural Information Processing Systems , 34:7637-7649, 2021.

Wei Jin, Yaxing Li, Han Xu, Yiqi Wang, Shuiwang Ji, Charu Aggarwal, and Jiliang Tang. Adversarial attacks and defenses on graphs. ACM SIGKDD Explorations Newsletter , 22(2):1934, 2021.

Marcin Waniek, Tomasz P Michalak, Michael J Wooldridge, and Talal Rahwan. Hiding individuals and communities in a social network. Nature Human Behaviour , 2(2):139-147, 2018.

Daniel Zügner and Stephan Günnemann. Adversarial attacks on graph neural networks via meta learning. In International Conference on Learning Representations .

Jiong Zhu, Junchen Jin, Donald Loveland, Michael T Schaub, and Danai Koutra. How does heterophily impact the robustness of graph neural networks? theoretical connections and practical implications. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2637-2647, 2022.

Kuan Li, Yang Liu, Xiang Ao, and Qing He. Revisiting graph adversarial attack and defense from a data distribution perspective. In The Eleventh International Conference on Learning Representations , 2023.

Yu Song and Donglin Wang. Learning on graphs with out-of-distribution nodes. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 1635-1645, 2022.

Shai Ben-David, John Blitzer, Koby Crammer, Alex Kulesza, Fernando Pereira, and Jennifer Wortman Vaughan. A theory of learning from different domains. Machine learning , 79:151-175, 2010.

Appendix

Table of Contents

A Related work

Graph Neural Networks (GNNs) have emerged as a powerful technique in Deep Learning, specifically designed for graph-structured data. They address the limitations of traditional neural networks in dealing with irregular data structures. GNNs learn node representations by aggregating neighborhood and transforming features recursively. The node representation can then be successfully utilized to a wide range of graph-related downstream tasks [2, 5, 3, 6-8, 73, 74].

The aggregation mechanism in GNNs is often viewed as feature smoothing [61, 75, 76]. This perspective leads some recent studies [26, 53, 20, 77] claiming that GNN models are overly reliant on homophilic patterns and unsuited to capturing heterophilic patterns To accommodate heterophilic graphs, recent works propose carefully designed GNN architectures, e.g., CPGNN [78], GGNN [79], GPRGNN [53], GCNII [52] GBK-GNN [27], ACM-GNN [19], Bernnet [80].

More recent analyses on GNNs [18, 13, 19] indicate that even GCN [2], a vanilla GNN, can deliver strong performance on certain heterophilic graphs. According to these findings, new metrics and understandings [81, 82, 21, 19, 13, 28] have been proposed to further expose the remaining weaknesses of GNNs.

There is a concurrent work [28] investigates when Graph Neural Networks help with node classification with a comparison between GNN and MLP. Nonetheless, our work is distinct from [28] in two primary ways:

[28] is majorly grounded on the CSBM-H model focusing on different feature variances for different classes, from a feature perspective. In contrast, our work examines the homophilic and heterophilic patterns from a structural perspective instead of sharing the same node homophily ratio across the graph.

[28] proposes a new metric to identify whether GNN can outperform graph-agnostic MLP on particular datasets. Nonetheless, it still focuses on all the nodes in the whole graph together. In contrast, our work reveals the scenario where GNNs show performance degradation on certain node subgroup across most homophilic and heterophilic graph datasets. Typically, we focus on a node subgroup perspective, rather than all nodes in the whole graph. Our paper highlights the drawback of GNNs in a more general case.

Fairness on Graph Although recent years have seen a satisfying performance from Graph Neural Networks (GNNs), risks can be found. GNNs may unintentionally learn and perpetuate biases present in the training data, potentially resulting in unfair outcomes for certain populations. Such risks raise concerns about biases and discriminatory behaviors when GNNs are used in human-centric applications.

Fairness issues on graphs can be roughly categorized into attribute bias and structural bias, based on the source of the bias. Several works [35, 83-85] focus on fairness issues originating from sensitive node attributes, e.g., gender or underrepresented ethnic groups. Other literature [63, 46, 86-88] focuses on addressing fairness issues arising from different structural information, e.g., degree, geodesic distance to the training node, and Personal Pagerank score. Our work aligns closely with structural bias, showing that a larger homophily ratio difference between training and test nodes may lead to performance degradation. To the best of our knowledge, we are the first to propose this performance disparity induced by the homophily ratio difference.

Generalization ability analysis on Graph Neural Network The generalization ability analysis on Graph Neural Networks aims to develop theoretical understandings of GNNs with a focus on the uniqueness of the graph structure. Recent research progress reveals the generalization ability [89, 47, 90-92] of GNN across different tasks and settings. Our work typically studies the generalization ability on the transductive node classification task indicating relationship with [93, 14, 18, 13]. Typically, [93] employs the transductive uniform stability [94] to understand why deeper GNNs generalize better than the vanilla GCN. Several studies [14, 18, 13] investigate the generalization of GCN under the CSBM model assumption, while they focus on either homophilic or heterophilic patterns rather than consider both patterns together. [46] provides the first non-i.i.d. PAC-Bayes generalization bound on GNNs, serving as the basis of our theory 1. However, there are key differences between our work and [46] detailed as follows:

Assumption difference. [46] assumes the existence of c -Lipschitz continuous functions on the conditional probability of y i = k given the aggregated feature g i ( X,G ) , denoted as P ( y i = k | g i ( X,G )) . The assumption suggests that nodes with a closer distance are likely to belong to the same class. However, this assumption may not align with the real-world scenario where nodes with closer distances may belong to different classes when they exhibit a large homophily ratio difference, as shown in Lemma 1. In contrast, we utilize a different assumption considering the influence of the structural pattern on the conditional probability, enabling analyses on the scenario that nodes with a closer distance but from different classes. This scenario happens frequently when the graph is a mixture of homophilic and heterophilic patterns. An intuitive illustration can be found in Figure 1.

Application scope difference. [46] is primarily focused on the homophilic graphs and does not easily extend to the heterophilic ones. Empirical evidence can be found in Section 3.4 and Appendix J. In contrast, our theory can be established on most datasets except for the Actor dataset which structural information hardly helps.

Disclaimer: Structural disparity is different from distribution shift between train and test data. The structural disparity is that homophilic and heterophilic patterns exist simultaneously in a single node set. Such disparity consistently exists in all graphs, as shown in Figure 2. If randomly sample train and test data, the distribution shift between train and test will not happen. Both train and test sets will have homophilic and heterophilic patterns, indicating structural disparity happens on both the train and test sets.

B Investigation on the effectiveness of homophily ratio difference with targeted synthetic edge addition algorithm.

In Section 3.4, we intuitively show that a larger homophily ratio difference could be a key reason for performance disparity. We demonstrate the effectiveness of intuition with both theoretical analysis and experiments conducted on real-world datasets. To further verify the influence of the homophily ratio difference, we conduct more controllable synthetic experiments adopted from [13], adding different amounts of synthetic edges on real-world datasets, to further evaluate how the performance of GCN, a vanilla GNN, changes on varied homophily ratio differences. Typically, we are to manually make heterophilic nodes more homophilic and make homophilic nodes more heterophilic with the targeted homophilic edge addition and the targeted heterophilic edge addition algorithms, respectively. Notably, despite synthetic edges added, we utilize the real-world dataset serves as the entrance to keep the analysis more approach to the real-world scenario. The following subsections are organized as follows. We first introduce the targeted heterophilic and homophilic edge addition algorithms and how it leads to larger homophily ratio difference in Appendix B.1. Detailed experiment analysis can be found in Appendix B.2. Further experiment details can be found in the Appendix B.3.

B.1 Targeted heterophilic & homophilic edge addition algorithms

Both targeted heterophilic and homophilic edge addition algorithms commence with standard, realworld benchmark graphs, modifying the structure by adding synthetic edges to manipulate the homophily ratio on either train nodes or test nodes. For example, we can add synthetic heterophilous edges on the targeted test nodes in a homophily graph. Consequently, a homophily ratio difference between the training and testing nodes can be observed.

We first introduce the homophilic edge addition algorithm, as shown in Algorithm 1. Specially, we introduce a total of K edges on either training or targeted test nodes on heterophilic dataset, denoted as V targeted. For each edge addition, a node u is uniformly sampled from the targeted node set, V targeted, and obtains its corresponding label, y u . Another targeted node v sharing the same label, y v , is selected, and a homophilic edge is added between them, resulting in a homophily ratio decrease. Consequently, we can observe a homophily ratio decrease on both target nodes u and v with the new homophily edge added. Notably, we only add synthetic edges among targeted nodes, ensuring the homophily ratio unchanged on the other nodes.

The heterophilic edge addition algorithm, as shown in the Algorithm 2, is analogous to the homophilic one. The only difference is the selection of a newly added heterophilic edge instead of the homophilic one. While homophilic edges can be readily added by connecting nodes within the same label, adding

cross-label heterophilic edges poses a new challenge that the target node u should be connected to which label, other than the target label y u . Typically, we follow the principle [13] that nodes with the same label should share similar neighborhood label distributions. This principle aligns with the CSBM assumption, discussed in Section 3. Specifically, given a real-world graph G , we first define a discrete neighborhood target label distribution D c for each label c ∈ C . Examples of D c can be found in Appendix B.3. Specifically, we will first randomly select a label c based on distribution D c . Another node v in the target set with label c is then selected, and a heterophilic edge is added between them. This process enables us to add heterophilic edges on target nodes following similar neighborhood label distributions.

B.2 Detailed experiment results

In this section, we conduct experiments on the homophilic graph, Cora, employing the targeted homophilic edge addition algorithm, and the heterophilic graph, Squirrel, employing the targeted heterophilic edge addition algorithm. For each dataset, we apply the synthetic edge addition algorithm to training nodes and a subset of test nodes, respectively, resulting in homophily ratio difference between training and test sets. The synthetic edges are added gradually until reaching the predefined maximum budget K max, resulting in multiple synthetic graphs generated. More experimental details on the synthetic graphs can be found in Appendix B.3. The GCN, a vanilla GNN model, is trained from the sketch on each synthetic graph. The experimental results are illustrated in Figure 10, where the x -axis represents the edge permutation ratio calculated as K K max , where K is the number of added edges on the i -th synthetic graph. The y -axis represents the test performance when adding synthetic edge on training nodes. When adding synthetic edges on the targeted test subset, The y -axis represents the performance on the targeted test node subset. A clear degradation in test performance is observed with more and more synthetic edges added to test nodes and training nodes exclusively. The above observations clearly demonstrate that the GCN test performance will degrade with a larger homophilic ratio difference between train and test nodes.

0.0

0.2

0.4

0.6

0.8

1.0

Edge ratio

0.2

0.4

0.6

0.8

1.0

Test acc

train

test

0.0

0.2

0.4

0.6

0.8

1.0

Edge ratio

0.2

0.3

0.4

0.5

0.6

Test acc

train

test

(a) Synthetic graphs generated from Cora with the targeted heterophilic edge algorithm (b) Synthetic graphs generated from Squirrel with the targeted homophilious edge algorithm

Figure 10: The performance of GCN on synthetic graphs with various edge permutation ratios. The y -axis represents the test performance on all nodes when adding synthetic edge on training nodes. When adding synthetic edges on the targeted test subset, The y -axis represents the performance on the targeted test node subset. Test performance generally decrease with more synthetic edges added with a large homophily ratio difference.

B.3 Details on the generated graph

In this subsection, we elaborate on the details of the synthetic graphs generated in Appendix B.2. Specifically, we provide more details regarding the distributions D c , c ∈ C employed in the Cora and Squirrel datasets. We adopted circulant matrix-like designs for simplicity and straightforward implementation. We also show details on the number of added edges, and the corresponding graph homophily ratio on the Cora and Squirrel datasets.

Cora We utilize the targeted heterophilic edge addition algorithm on the Cora dataset. The Cora dataset has seven labels that we represent as 0 , 1 , 2 , 3 , 4 , 5 , 6 , and the neighborhood distributions D c , c ∈ C for adding heterophilic synthetic edges are presented as follows.

D 0 : Categorical ([0 , 0 . 5 , 0 , 0 , 0 , 0 , 0 . 5]) ,

D 1 : Categorical ([0 . 5 , 0 , 0 . 5 , 0 , 0 , 0 , 0]) ,

D 2 : Categorical ([0 , 0 . 5 , 0 , 0 . 5 , 0 , 0 , 0]) ,

D 3 : Categorical ([0 , 0 , 0 . 5 , 0 , 0 . 5 , 0 , 0]) ,

D 4 : Categorical ([0 , 0 , 0 , 0 . 5 , 0 , 0 . 5 , 0]) ,

D 5 : Categorical ([0 , 0 , 0 , 0 , 0 . 5 , 0 , 0 . 5]) ,

D 6 : Categorical ([0 . 5 , 0 , 0 , 0 , 0 , 0 . 5 , 0]) .

The number of added edges K and the corresponding homophily ratio h on the targeted test nodes and the train nodes are presented in Table 2 and 3, respectively.

Squirrel We utilize the targeted homophilic edge addition algorithm on the Squirrel dataset. We first sample the label from discrete uniform distributions. Then, we randomly select two nodes with the same label and do not have an edge between them. The number of added edges K and the corresponding homophily ratio h on the targeted test nodes and train nodes are presented in Table 4 and 5, respectively.

C Instance-level discriminative analysis

In section 3.2, we conduct a discriminative analysis considering the distance between the feature means from different classes from a global perspective. In this section, we further investigate the discriminability from an instance-level perspective. Specifically, we focus on the feature distance between different nodes feature rather than the feature mean.

To qualify the discriminative difficulty on the feature of each individual test node, we propose the two metrics, local agreement ratio, and local accuracy difference. The local agreement ratio aims to measure the local clustering property in a KNN manner. It can be calculated with the following steps: (1) Given a particular node, find the topk feature-close train nodes. We set k = 9 in our experiment, and L2 distance is utilized as the feature distance metric. (2) Given the topk closest train nodes, we examine the number of nodes on each label. If there exists a particular label c with the number of nodes over k 2 , it indicates that over half of neighborhood nodes reach an agreement on the class c . (3) The local agreement ratio can then be calculated as the proportion of nodes reaching an agreement in the test set, denoted as:

where r is the local agreement ratio, V te and C are the test node set and class set, respectively. M v is the node set including topk nearest training of the node v . |M c v | is the number of nearest training nodes in class c . A larger agreement ratio on the test set indicates the data are more clustered with a close distance between train and test nodes. Nonetheless, a higher agreement ratio does not naturally lead to a better discriminative ability. Despite the topk feature-close nodes reaching an agreement on a particular class, the test node may have a different class from the agreement. It indicates that the center node misaligns with the topk feature-close nodes.

The local accuracy is then proposed to identify whether the category of the center node aligns with the agreement from topk feature-close nodes. Concretely speaking, the local agreement accuracy is the proportion of agreement nodes in the test set. It can be calculated as:

where V agree is the test node set reaching topk feature-close nodes agreement. c v is the category of node v . c M v is the agreement category from feature-close nodes. A higher local agreement accuracy indicates that most agreement nodes are aligned with the same category.

Similar to the relative discriminative ratio proposed in Section 3.2, we illustrate the local agreement accuracy improvement on the majority nodes over minority nodes. Experiments are conducted on four homophilic datasets, Cora, CiteSeer, PubMed and Ogbn-arxiv, and two heterophilic datasets, Chameleon and Squirrel. The experimental setting details can be found in Section G. Experiment

0

1

2

3

4

5

#aggregation

0.0

0.2

0.4

0.6

0.8

1.0

Ratio

hete

homo

h

=0.81)

0

1

2

3

4

5

#aggregation

0.0

0.2

0.4

0.6

0.8

1.0

Ratio

hete

homo

(b) CiteSeer (

h

=0.71)

0

1

2

3

4

5

#aggregation

0.4

0.6

0.8

1.0

Ratio

hete

homo

(c) PubMed (

h

=0.79)

0

1

2

3

4

5

#aggregation

0.2

0.4

0.6

0.8

1.0

1.2

Ratio

hete

homo

(d) Ogbn-arxiv (

h

=0.63)

0

1

2

3

4

5

#aggregation

0.2

0.4

0.6

0.8

1.0

1.2

Ratio

hete

homo

h

=0.25)

0

1

2

3

4

5

#aggregation

0.0

0.2

0.4

0.6

0.8

1.0

Ratio

hete

homo

h

=0.22)

results on local agreement ratio and local agreement accuracy difference are illustrated in Figure 11 and 12, respectively.

The observations can be found as follows: (1) For homophilic graphs, the local agree ratio generally increases along with more aggregations, indicating better cluster effects. Meanwhile, the relative accuracy improvement on the majority nodes also increases, further indicating the disparity effects on different nodes group with more improvement on the majority nodes. (2) For heterophilic graphs, the local agreement ratio shows an opposite phenomenon, which decreases consistently. Meanwhile, the relative accuracy improvement on the majority nodes only increases on the first two hops and decrease and decline from the third hop. The potential reason is that, despite a general global trend, the heterophilic patterns on each individual node may still be quite complicated, with a local pattern shift disparity. We leave the discussion on more complex local structure patterns as the future work.

0

1

2

3

4

5

#aggregation

0.1

0.2

0.3

0.4

Acc

CiteSeer

PubMed

Arxiv

(a) Homophilic graphs

0

1

2

3

4

5

#aggregation

0.2

0.1

0.0

0.1

0.2

0.3

0.4

0.5

Acc

Chameleon

(b) Heterophilic graphs

D Effects of aggregation on nodes in different classes with structural disparity

In Section 3.1, we examine the behavior difference between nodes from the same class but with different structural patterns. In this section, we further provide a more complicated analysis focusing on the between-class patterns, i.e., linear separability. Notably, linear separability is a good indicator of feature differences in different classes, where features with better linear separability can be easier to distinguish with a suitable linear classifier.

D.1 Linear separability analysis based on CSBM model

To ease the analysis, we utilize the CSBM-S model as the data assumption showing as follows:

Definition 1 (CSBM-S ( µ 1 , µ 2 , ( p (1) , q (1) ) , ( p (2) , q (2) ) , Pr( homo )) ) . The generated nodes consist of two disjoint sets C 1 and C 2 . each node feature x is sampled from N ( µ i , I ) with i ∈ { 1 , 2 } . Each set C i consists of two subgroups: C (1) i for nodes in homophilic pattern with intra-class and inter-class edge probability p (1) > q (1) and C (2) i for nodes in heterophilic pattern with p (2) > q (2) . Pr( homo ) denotes the probability that the node is in the homophilic pattern. C ( j ) i denotes node in class i and subgroup j with ( p ( j ) , q ( j ) ) . We assume nodes follow the same degree distribution with p (1) + q (1) = p (2) + q (2) .

The original node features follow the Gaussian distribution:

Where C i corresponds to the node set corresponding to class i .

Based on the neighborhood distributions, the mean aggregated features F = D -1 AX obtained follow Gaussian distributions on both homophilic and heterophilic subgroups.

Where C ( j ) i is the node subgroups with structural pattern with ( p ( j ) , q ( j ) ) in label i . To more accurately assess the effectiveness of the aggregation mechanism, we execute the largest-margin linear classifiers on nodes and evaluate their performance, to illustrate the linear separability. Notably, linear separability depends on the distance between the mean features of different classes as well as the standard deviations within each class. Typically, we focus on examining the linear separability on (1) nodes from different classes with the same structural patterns (2) nodes from different classes with different structural patterns, i.e., homophilic and heterophilic patterns.

We first examine the linear separability for nodes from different classes within the same structural pattern. We can summarize the proposition as follows:

Lemma 2 (Linear separability on nodes with the same structural patterns ) . Considering mean aggregated features are from the same structural pattern f ( j ) i , for i ∈ { 1 , 2 } . For any node i , the largest-margin linear classifier on f ( j ) i will have a lower probability to misclassify than x i , when d i > ( p (1) + q (1) ) 2 ( p (1) -q (1) ) 2

The detailed proof can be found in Appendix D.3. The proposition suggests that aggregated features f have better linear separability than the original feature x when the node i satisfies d i > ( p (1) + q (1) ) 2 / ( p (1) -q (1) ) 2 . For instance, when p (1) = 0 . 9 and q (1) = 0 . 1 , the aggregated features are easier show better linear separability with d i > 1 . 75 , which are commonly met in real-world scenarios. This indicates that aggregation is likely to contribute to improved feature separability within the same node subgroup sharing a similar structural property.

We then examine the linear separability for nodes from different classes within different structural patterns, e.g., h (1) 1 and h (2) 2 . Before diving deep into the rigorous analysis, we first illustrate a special example to show how the structure can lead to worse linear separability. When p (1) = q (2) and q (1) = p (1) , we can identify that the h (1) 1 and h (2) 2 are exactly generated from the same distribution. Such distribution can hardly be linearly separable. We then conduct a more rigorous analysis showing a similar observation with the same pattern one. We can summarize the proposition as follows:

Lemma 3 (Linear separability on nodes with different structural patterns) . Consider features are from different structural patterns, where f (1) i for i ∈ C 1 and f (2) i for i ∈ C 2 . For any node i , the largest-margin linear classifier will have a lower probability to misclassify f (1) i for i ∈ C 1 and f (2) i for i ∈ C 2 than x i when d i > ( p (1) + q (1) ) 2 ( p (1) -q (2) ) 2

we can find that aggregated feature show better separability when d i > ( p (1) + q (1) ) 2 / | p (1) -q (2) | 2 . The detailed proof can be found in Appendix D.4. For instance, when p (1) = 0 . 9 , q (1) = 0 . 1 and p (2) = 0 . 2 , q (2) = 0 . 8 , the aggregated features can only show better linear separability with d i > 100 , which are hardly met in real-world scenarios. Notice that the only difference with the one in the same pattern one is that the denominator is | p (1) -q (2) | 2 , smaller than the | p (1) -q (1) | 2 since q (1) > q (2) . It indicates that only nodes with higher degrees. e.g., d i = 100 , can achieve improved linear separability.

Based on our analysis, we can draw the conclusion that when nodes are in the same pattern, aggregation can show improved linear separability on ( p (1) + q (1) ) 2 / | p (1) -q (1) | 2 < d i < ( p (1) + q (1) ) 2 / | p (1) -q (2) | 2 while the ones in the different patterns cannot. It indicates that aggregation can help when nodes are in the same pattern, however, show limitation when nodes are from different patterns. Notice that, such linear separability can be evidence for different behavior on nodes with different patterns. Nonetheless, it cannot directly indicate the performance disparity. Feature separability is conducted based on the ideal classifier with the largest margin. Better feature

separability does not necessarily result in better performance if the classifier is not well-trained with biased training data.

D.2 Linear separability experiment on synthetic CSBM dataset

In this section, we aim to empirically examine the validity of the theoretical analysis results in Section D.1. We generate graph with the CSBM model with the following detailed settings. The class mean distance ρ = ∥ µ 1 -µ 2 ∥ is set to 0 . 1 . The feature dimension d and the number of nodes n are set to 50, and 500, respectively. p and q correspond to probabilities of the intra-class and inter-class probability, respectively. The mean aggregation is defined as F = D -1 AX . It is the key to generating graphs with different structural properties. To generate the homophilic node subgroup with p > q , we utilize the following settings with ( p = 0 . 01 , q = 0 . 005) , ( p = 0 . 01 , q = 0 . 003) , and ( p = 0 . 01 , q = 0 . 001) . To generate the heterphilic node subgroup with p < q , we utilize the following settings with ( p = 0 . 001 , q = 0 . 005) , ( p = 0 . 001 , q = 0 . 003) , and ( p = 0 . 001 , q = 0 . 002) .

Experiments are conducted to examine how a linear model, logistic regression, fits effectively on the aggregated features. Higher performance on logistic regression indicates better linear separability of features. It is important to note that we focus on the fitting ability rather than the generalization performance. Consequently, we do not evaluate with a test set; instead, we provide all labels for training and assess the performance on all the nodes.

Experiments on the logistic regression are conducted on the homophilic nodes solely, heterophilic nodes solely, and a mixture of homophilic and heterophilic nodes. Typically, we aim to show the linear separability both within the same structural pattern and between homophilic and heterophilic patterns. Experimental results are presented in Table 6. The following observations can be made: (1) When considering nodes following the same structural pattern, either homophilic pattern or heterophilic pattern, better performance can be observed when the probability difference | p -q | between intra-class and inter-class probability is larger. (2) Comparing the performance considering both structural patterns with the one on a single structural pattern, we reveal a noticeable performance degradation on the one with mixture patterns. This suggests that the nodes with different structural patterns results in decreased linear separability. (3) Comparing the performance among different mixing structural patterns, it is evident that larger homophily ratio differences, e.g., (p=0.01, q=0.005) compared to (p=0.001, q=0.005), yield poorer performance than those with smaller homophily ratio differences. The above observations further supports the validity of Lemma 2 and 3, Theorem 1 .

D.3 Proof details of linear seperability within the same pattern

In this section, we provide detailed proof of the improved linear separability on aggregated features where nodes from different classes follow the same structural pattern. The following theoretical results indicate that it is more difficult to have improved linear separability when nodes follow different structural patterns. Notably, the following proof is derived based on [13]. For completeness, we show the previous proof as follows, the difference is that we consider a more complicated CSBM-S model with both homophilic and heterophilic patterns rather than the simple CSBM model only describing either homophilic or heterophilic pattern.

Lemma 2 (Linear separability on nodes with the same structural patterns ) . Considering mean aggregated features are from the same structural pattern f ( j ) i , for i ∈ { 1 , 2 } . For any node i , the largest-margin linear classifier on f ( j ) i will have a lower probability to misclassify than x i , when d i > ( p (1) + q (1) ) 2 ( p (1) -q (1) ) 2

Notably, we only show the case p (1) > q (1) in Appendix D.1 for simplicity. It can be easily extend to p (2) < q (2) . Proof details can be found as follows.

Consider the vanilla mean aggregation as F = D -1 AX , the aggregated features with the same structure patterns follow Gaussian distributions:

Then we denote the expectation of the original node features x in two classes as E c 1 [ x i ] and E c 2 [ x i ] . Similarly, we denote the expectation of the aggregated features as E c 1 [ f i ] and E c 2 [ f i ] . Proposition 2. ( E c 1 [ x i ] , E c 2 [ x i ]) and ( E c 1 [ f i ] , E c 2 [ f i ]) share the same middle point. E c 1 [ x i ] -E c 2 [ x i ] and E c 1 [ f i ] -E c 2 [ f i ] share the same direction.

The proposition can be calculated through direct calculations. If we consider the feature distributions of two classes, we can observe that both x and f exhibit a systematic relationship. As a consequence, we can determine the optimal linear classifier for both x i and f i . We then define decision boundary as the hyperplane that is orthogonal to the direction w = µ 1 -µ 2 || µ 1 -µ 2 || and passes through the middle point m = ( µ 1 + µ 2 ) / 2 as:

We then show the proof details on when f has a lower mis-classified probability than x with the same decision boundary, indicating better linear separability.

Proof. We provide the proof only for nodes belonging to class c 0 , as the case for nodes from class c 1 is symmetric and the proof follows the same logic. For a node i in C 1 , We have the following:

we can then scale the decision boundary without changing the original meaning. Then we can have P ( w ⊤ f i + b ≤ 0 ) = P ( √ d i w ⊤ f i + √ d i b ≤ 0 ) We denote the scaled version of f i as f ′ i = √ d i f i . Then, f ′ i follows:

Given the scale in Eq. equation 11, the decision boundary for f ′ i is consequently shifted to w ⊤ f ′ + √ d i b = 0 . Now, considering that x i and f ′ i have the same variance, we can compare the misclassification probabilities by merely comparing the distance from their expected values to their corresponding decision boundaries. Specifically, the distances can be described as follows:

The larger the distance from the expectation to the decision boundary indicates a smaller the misclassification probability. Therefore, when dis f ′ i > dis x i , f ′ i has a lower probability of being misclassified than x i . By comparing the two distances, we conclude that when d i > ( p ( j ) + q ( j ) ) 2 ( p ( j ) -q ( j ) ) 2 , the mean aggregated feature f ′ i exhibits a lower probability of misclassification compared to the original node feature x i .

D.4 Proof details of linear separability between different structural patterns

In this section, we provide detailed proof of the improved linear separability on aggregated features where nodes from different classes follow different structural patterns. The following theoretical results indicate that it is more difficult to have improved linear separability when nodes follow different structural patterns. Notably, the following proof is derived based on [13]. For completeness, we show the previous proof as follows, the difference is that we consider a more complicated CSBM-S model with both homophilic and heterophilic patterns rather than the simple CSBM model only describing either homophilic or heterophilic pattern

Lemma 3 (Linear separability on nodes with different structural patterns) . Consider features are from different structural patterns, where f (1) i for i ∈ C 1 and f (2) i for i ∈ C 2 . For any node i , the largest-margin linear classifier will have a lower probability to misclassify f (1) i for i ∈ C 1 and f (2) i for i ∈ C 2 than x i when d i > ( p (1) + q (1) ) 2 ( p (1) -q (2) ) 2

In the following discussion, we only focus on the scenario where nodes in class c 1 are in a homophilic pattern with p (1) > q (1) , and nodes in class c 2 are in a heterophilic pattern with p (2) < q (2) . The other scenario is symmetric and the proof follows the same logic.

Consider the vanilla mean aggregation as F = D -1 AX , the aggregated features with different structure patterns follow Gaussian distributions:

Similar to the notation in Proposition 2, we denote the expectation of the original node features x in two classes as E c 1 [ x i ] and E c 2 [ x i ] . Similarly, we denote the expectation of the aggregated features as E c 1 [ f i ] and E c 2 [ f i ] .

Proposition 3. ( E c 1 [ x i ] , E c 2 [ x i ]) and ( E c 1 [ f i ] , E c 2 [ f i ]) share the same middle point. When satisfying p (1) > q (2) , E c 1 [ x i ] -E c 2 [ x i ] and E c 1 [ f i ] -E c 2 [ f i ] share the opposite direction. Specifically, the middle point m and the shared direction w are as follows: m = ( µ 1 + µ 2 ) / 2 , and w = ( µ 1 -µ 2 ) / ∥ µ 1 -µ 2 ∥ 2 . When satisfying q (2) > p (1) , E c 1 [ x i ] -E c 2 [ x i ] and E c 1 [ h i ] -E c 2 [ h i ] are in same directions. It indicates that the largest-margin linear model still shares the same discriminative boundary which flips the prediction on class c 1 and c 2 .

Notably, with the assumption p (1) + q (1) = p (2) + q (2) , the linear classifier can be generally to two cases according to whether the decision boundary direction flips: (1) The decision boundary direction unchanged with q (2) > p (1) > p (2) > q (1) (2) The decision boundary direction flipped with p (1) > q (2) > q (1) > p (2) . Notably, the two cases are symmetric with the same conclusion and proof logic. We will focus on the unchanged case with q (2) > p (1) > p (2) > q (1) .

The proposition can be calculated through direct calculations. If we consider the feature distributions of two classes, we can observe that both x and f exhibit a systematic relationship. As a consequence, we can determine the optimal linear classifier for both x i and f i . We then define decision boundary as the hyperplane that is orthogonal to the direction w = µ 1 -µ 2 || µ 1 -µ 2 || and passes through the middle point m = ( µ 1 + µ 2 ) / 2 as:

We then show the proof details on when f has a lower mis-classified probability than x with the same decision boundary, indicating better linear separability.

Proof. We provide the proof only for nodes belonging to class c 0 , as the case for nodes from class c 1 is symmetric and the proof follows the same logic. For a node i in C 1 , We have the following:

we can then scale the decision boundary without changing the original meaning. Then we can have P ( w ⊤ f i + b ≤ 0 ) = P ( √ d i w ⊤ f i + √ d i b ≤ 0 ) We denote the scaled version of f i as f ′ i = √ d i f i . Then, f ′ i follows:

Given the scale in Eq. equation 17, the decision boundary for f ′ i is consequently shifted to w ⊤ f ′ + √ d i b = 0 . Now, considering that x i and f ′ i have the same variance, we can compare the misclassification probabilities by merely comparing the distance from their expected values to their corresponding decision boundaries. Specifically, the distances can be described as follows:

The larger the distance from the expectation to the decision boundary indicates a smaller the misclassification probability. Notably, q (1) -p (2) = p (1) -q (2) with the assumptin p (1) + p (2) = q (1) + q (2) Therefore, when dis f ′ i > dis x i , f ′ i has a lower probability of being misclassified than x i . By comparing the two distances, we conclude that when d i > ( p (1) + q (1) ) 2 ( p (1) -q (2) ) 2 , the mean aggregated feature f ′ exhibits a lower probability of misclassification compared to the original node feature x .

i i

which completes the proof.

A comparison between linear separability within classes and between classes Notice that the only difference with the one in the same pattern one is that the denominator is | p (1) -q (2) | 2 , smaller than the | p (1) -q (1) | 2 since q (1) > q (2) . It indicates that only nodes with higher degrees. For instance, when p (1) = 0 . 9 , q (1) = 0 . 1 and p (2) = 0 . 2 , q (2) = 0 . 8 , the aggregated features can only show better linear separability with d i > 100 when nodes are from different structural patterns. The above condition can be hardly met in real-world scenarios. Based on our analysis, we can draw the conclusion that when nodes are in the same pattern, aggregation can show improved linear separability on ( p (1) + q (1) ) 2 / | p (1) -q (1) | 2 < d i < ( p (1) + q (1) ) 2 / | p (1) -q (2) | 2 while the ones in the different patterns cannot. It indicates that aggregation can help more when nodes are in the same pattern, however, show limitation when nodes are from different patterns.

E Proof details of the conditional probability difference for nodes with the same feature but different structural patterns

In section 3.1, we examine the discrepancy between nodes 1 and 2 with the same aggregated feature f 1 = f 2 but different structural patterns. Typically, we examine the discrepancy with the probability difference of nodes 1 and 2 in class c 1 , denoted as | p 1 ( y u = c 1 | f u ) -p 2 ( y v = c 1 | f v ) | . p 1 and p 2 are the conditional probabilities of node labels in c given the feature f for nodes on homophilic and heterophilic structural patterns, respectively. The lemma is shown as follows:

Lemma 1. With assumptions (1) A balance class distribution with P ( Y = 1) = P ( Y = 0) and (2) aggregated feature distribution shares the same variance σ I . When nodes u and v have the same aggregated features f u = f v but different structural patterns, ( p (1) , q (1) ) and ( p (2) , q (2) ) , we can have:

ρ = ∥ µ 1 -µ 2 ∥ is the original feature separability, independent with structure. P 1 and P 2 are the conditional probability of y = c 1 given the feature f on structural patterns ( p (1) , q (1) ) and ( p (2) , q (2) ) ,

respectively. Lemma 1 implies that nodes with a small homophily ratio difference | h 1 -h 2 | are likely to share the same class, and vice versa.

We first remind the assumptions of the CSBM-S model. There are two assumptions on the models: (1) Nodes from different components share the same feature distribution. In other word, node features within the same class are sampled from the same Gaussian distribution, regardless of different structural patterns. (2) Nodes from different components share similar degree distribution with p (1) + q (1) = p (2) + q (2) . Notably, our conclusions are still valid without the above assumptions. Those assumptions are not strictly necessary but employed for the elegant expression.

Based on the neighborhood distributions, the mean aggregated features F = D -1 AX obtained follow Gaussian distributions on both homophilic and heterophilic node subgroups.

Where C ( j ) i is the node subgroups with structural pattern with ( p ( j ) , q ( j ) ) in label i . Notably, there are typically homophilic pattern with p (1) > q (1) and heterophilic pattern with p (2) < q (2) . To simply the assumption for an elegant expression, we denote the aggregated feature mean, p ( j ) µ 1 + q ( j ) µ 2 p ( j ) + q ( j ) as µ ′ ij , where i and j correspond to the class and the structural pattern, respectively. For instance, µ ′ 11 represents the mean of nodes in class c 1 with the homophilic pattern. We then show the proof details as follows:

Proof. The conditional probability of class c 1 given the aggregated feature f , P 1 ( y = c 1 | f 1 ) can be derived with the Bayes theorem:

where (a) we utilize the assumption P ( y = c 1 ) = P ( y = c 2 ) . Notably, such assumption aims to simplify for an elegant expression and better understanding, which is not necessary in our proof.

Hence, we have:

Notably, the denominator can be denoted as

where each exponential component is in exp ( -|| f i -µ ′ ij || 2 σ 2 ) ∈ [0 , 1] . Therefore, the denominator m is in the range [0 , 4] . We then denote m = exp( -A ) , where A is a constant, correspondingly. Hence,

we can have:

(25)

ρ = ∥ µ 1 -µ 2 ∥ is the original feature separability, independent with structure. | h 1 -h 2 | is the homophily ratio difference between node 1 and 2 . We explain the key steps as follows: (a) is derived from Lagrange's mean value theorem. From the Lagrange mean value theorem we have,

where ξ is between x and y . Therefore,

since exp( -ξ ) ≤ 1 .

Knowing that | P 1 ( y u = c 1 | f u ) -P 2 ( y v = c 1 | f v ) | < 1 , we can easily get

where exp( B ) > 0 , we can get ( f u -µ ′ 11 ) 2 +( f u -µ ′ 22 ) 2 σ 2 > A . Similarly, with | P 1 ( y u = c 1 | f u ) -P 2 ( y v = c 1 | f v ) | > 0 , we can get ( f 2 -µ ′ 12 ) 2 +( f 2 -µ ′ 21 ) 2 σ 2 > A . In conclusion, we have:

Let x = A -( f 2 -µ ′ 12 ) 2 +( f 2 -µ ′ 21 ) 2 σ 2 and y = A -( f 1 -µ ′ 11 ) 2 +( f 1 -µ ′ 22 ) 2 σ 2 in equation 27. The proof is complete.

̸

(c) we utilize f u = f v and h i = p ( i ) p ( i ) + q ( i ) . Notice that, (c) step can be easily generally to the case that f u = f v with ∥ f u -f v ∥ ≤ ϵ . We can easily proof with the same logis. The lemma is shown as follows:

Lemma 2. With assumptions (1) A balance class distribution with P ( Y = 1) = P ( Y = 0) and (2) aggregated feature distribution shares the same variance σI . When nodes have the same aggregated features ∥ f u -f v ∥ ≤ ϵ but different structural patterns, ( p (1) , q (1) ) and ( p (2) , q (2) ) , we can have:

ρ = ∥ µ 1 -µ 2 ∥ is the original feature separability, independent with structure. Lemma 2 implies that nodes with a small homophily ratio difference | h 1 -h 2 | are likely to share the same class, and vice versa.

F Proof details of PAC-Bayes Bound

In this section, we provide background knowledge, assumptions, and proof details on the PAC-Bayes analysis. Our PAC-Bayes bound is derived based on [46], which is the first PAC-Bayesian analysis for GNN on the semi-supervised node classification task. A details comparison between our bound and [46] can be found in Appendix A.

F.1 Background Knowledge on PAC-Bayes Analysis

PAC-Bayes Analysis The Probably Approximately Correct-Bayesian (PAC-Bayes) analysis [48] is a powerful theoretical framework utilizing Bayesian learning principles for analyzing the generalization ability of machine learning models. Typically, PAC-Bayesian Analysis is to connect and bound the difference between a training error of a machine learning model with its expected generalization error. Various PAC-Bayes bounds [49, 95, 50, 51] are proposed for advanced Deep Learning models in recent years. Such theoretical guidance shows practical success in many real-world applications including pretraining model [96], medical image [97], and robustness [98].

PAC-Bayes analysis on Graph Neural Networks [90] is the first PAC-Bayesian generalization bound for GNNs. Nonetheless, it focuses on the i.i.d. graph classification task, which is different from the non-i.i.d. node classification task. [46] provides the first PAC-Bayesian generalization bound for GNN on semi-supervised node classification task. Our following theorem is developed based on it and generalized into more generalized scenarios. A more detailed discussion on the difference of our work and [46] can be found in the Appendix A.

F.2 PAC-Bayesian Analysis on Subgroup Generalization bound of Deterministic classifier

In this section, we provide a more detailed discussion on the PAC-Bayesian analysis with the semisupervised subgroup Generalization bound of Deterministic classifiers proposed in [46], which serves as the basis of proof. Notably, in the main content, we denote the training node subgroup as V tr. As we focus on the relationship between the train nodes and test subgroups V m with 0 < m ≤ M , we re-denote the train nodes as V 0 . The PAC-Bayesian analysis on subgroup Generalization bound of Deterministic classifiers, focusing on the deterministic classifiers ˜ h , a classifier drawn in the predictor family H , learned on training nodes. The primary objective of the PAC-Bayesian analysis is to derive bounds on the generalization gap between the empirical margin loss of ˜ h denoting as ̂ L γ 0 ( ˜ h ) on the train node subgroup V 0 and the corresponding expected margin loss L 0 m ( ˜ h ) on the test node subgroup V m . The empirical marginal loss ̂ L γ 0 ( ˜ h ) on the train node subgroup V 0 is defined as:

̸

where ✶ [ · ] is the indicator function and γ is the margin. V 0 is the training node set. N 0 = | V 0 | is the number of the training node. y i is the label corresponding to node i ∈ V 0 . ˜ h i ( X,G ) ∈ R K is the predictor output, where ˜ h i ( X,G )[ k ] refers to the prediction probability of class k on sample i ∈ V 0 .

The expected margin loss is the expectation of L 0 m ( ˜ h ) on the test node subgroup V m of the corresponding empirical loss ̂ L 0 m ( ˜ h ) .

where Pr( y | g i ( X,G )) is the conditional distribution. g ( X,G ) : R N × D ×G N → R N × D ′ is the aggregation function, typically, g i ( X,G ) = 1 |N ( i ) ∑ j ∈N ( i ) X j as default. G n is the space for all undirected graphs with n nodes.

To bound the generalization gap between the expected margin loss L 0 m ( ˜ h ) on test subgroup V m and the empirical margin loss ̂ L γ 0 on train subgroup V 0 . The theorem is shown as follows:

Theorem 2 (Subgroup Generalization of Deterministic Classifiers [46]) . Let ˜ h be any classifier in H . For any 0 < m ≤ M , for any λ > 0 and γ ≥ 0 , for any 'prior' distribution P on H that is independent of the training data on V 0 , with probability at least 1 -δ over the sample of y 0 , for any Q on H such that Pr h ∼ Q ( max i ∈ V 0 ∪ V m ∥ h i ( X,G ) -˜ h i ( X,G ) ∥ ∞ < γ 8 ) > 1 2 , we have

The generalization bound is typically related to the following four terms: (a) D KL ( Q ∥ P ) := ∫ ln dQ dP dQ is the KL-divergence between the learned (posterior) predictor distribution Q and the predefined (prior) distribution P , independent of the training data. The generalization gap is bounded by the discrepancy between P and Q . It is a general term in PAC-Bayes analysis considering as the model complexity measurement. (2) λ 2 4 N 0 will vanish with larger number of training samples N 0 Notably, the above two common terms in PAC-Bayes analysis are not our focus. (3) ln 1 δ is the term related with the probability 1 -δ . (4) The expected loss Discrepancy between the training nodes V 0 and targeted test node subgroup V m is essential in our analysis, denoted as:

where a prior distribution P over H , γ is the correponding loss margin, λ > 0 is a parameter, reflects the concentration of the learning distribution Q . Intuitively speaking, the term D γ/ 2 m, 0 ( P ; λ ) signifies the difference in expected loss between V m and V 0 , evaluated in an expectation with respect to the prior distribution P .

The expected loss Discrepancy is typically for the non-i.i.d. semi-supervised setting, which trivially comes true in i.i.d. setting. In the i.i.d. case, all samples in V m and V 0 are i.i.d., where L γ/ 4 m ( h ) = L γ/ 4 0 ( h ) < L γ/ 2 0 ( h ) ≤ 0 for any classifier h , leading to a trivial upper bound D γ/ 2 m, 0 ( P ; λ ) ≤ 0 . To provide a meaningful upper bound, we are required to essentially assume there exists the relationship between train node subgroup V 0 and test node subgroup V m . Typically, we expect the expected margin loss L γ/ 4 m ( h ) on V m is not much larger than L γ/ 2 0 ( h ) on V 0 when the number of samples becomes large. Taking a further step, to bound the difference L γ/ 4 m ( h ) -L γ/ 2 0 ( h ) , we should find suitable assumptions and derive proof to bound on (1) the difference on P ( g ( X,G )) controlling with the feature distance between V 0 and V m . (2) The differencce on P ( Y | g ( X,G )) controlling with the homophily ratio as shown in Lemma 1.

F.3 Proof details of Expected Loss Discrepancy

To establish the generalization guarantee, it becomes necessary to provide an upper bound for the expected loss discrepancy, D γ m, 0 ( P ; λ ) , which is the main focus in the proof. It emerges that certain assumptions about the data are required to derive a meaningful and useful upper bound. The assumptions are shown as follows. Notably, the assumptions 2 and 3 are adapted from [46]. Remark: Notably, the following proof is derived based on [46]. For completeness, we show the previous proof as follows, the difference is that we consider a more complicated conditional probability P ( y i = k | g i ( X,G )) , correlated with the homophily ratio difference, rather than consider the conditional probability as c -Lipschitz continuous functions. In other words, our proof further strengthens the effect of the structure disparity.

Definition 1 (Generalized CSBM-S model ) . Each node subgroup V m follows the CSBM distribution V m ∼ CSBM ( µ 1 , µ 2 , p ( i ) , q ( i ) ) , where different subgroups share the same class mean but different intra-class and inter-class probabilities p ( i ) and q ( i ) . Moreover, node subgroups also share the same degree distribution as p ( i ) + q ( i ) = p ( j ) + q ( j ) .

Instead of only considering one homophilic and one heterophilic pattern in CSBM-S model, the generalized CSBM-S model allows more diverse structural patterns with different levels of homophily.

Assumption 1 (Data follows Generalized CSBM-S model assumption) . The graph data is generated from the Generalized CSBM-S model.

Assumption 2 (Equal-Sized and Disjoint Near Sets ) . For any 0 < m ≤ M , assume the near sets of each i ∈ V 0 with respect to V m are disjoint and have the same size s m ∈ N + .

Assumption 2 assumes that V m can be divided into equally sized partitions, each can be identified by the corresponding training samples. It assumes that test nodes are closely aligned with the respective training sample, while distant to the other training samples.

Definition 2 (Distance To Training Set and Near Set) . For each 0 < m ≤ M , define the distance from the subgroup V m to the training set V 0 as

Further, for each i ∈ V 0 , define the near set of i with respect to V m as

Assumption 3 (Concentrated Expected Loss Difference) . Let P be a distribution on H , defined by sampling the vectorized MLP parameters from N (0 , σ 2 I ) for some σ 2 ≤ ( γ/ 8 ϵ m ) 2 /L 2 b ( λN -α 0 +ln2 bL ) . For any L -layer GNN classifier h ∈ H with model parameters W h 1 , . . . , W h L , define T h := max l =1 ,...,L ∥ W h l ∥ 2 . Assume that there exists some 0 < α < 1 4 satisfying

Assumption 3 postulates that the expected margin loss on the test node subgroup, V m , is not significantly larger that on the train node subgroup, V 0 , as the number of training samples N 0 = | V 0 | becomes larger. More discussions about this assumption can be found in Appendix A.5 of [46].

Lemma 4 (Bound for D γ m, 0 ( P ; λ ) (Adaption of Lemma 6 in [46])) . Under Assumption 1, 2 and 3, for any 0 < m ≤ M , any 0 < λ ≤ N 2 α 0 and γ ≥ 0 , assume the 'prior' P on H is defined by sampling the vectorized MLP parameters from N (0 , σ 2 I ) for some σ 2 ≤ ( γ/ 8 ϵ m ) 2 /L 2 b ( λN -α +ln2 bL ) . We have

0

We first present the following Lemma 5 that bounds the difference between the margin loss on V m and that on V 0 .

Lemma 5 (Adaption of Lemma 5 in [46]) . Suppose an L -layer GNN classifier h is associated with model parameters W 1 , . . . , W L . Define T h := max l =1 ,...,L ∥ W l ∥ 2 . Under Assumption 1 and 2, for any 0 < m ≤ M and γ ≥ 0 , if ϵ m T L h ≤ γ 4 , then

Proof. For simplicity in this proof, for any i ∈ V 0 ∪ V m and k = 1 , . . . , K , we use h i to denote h i ( X,G ) and use η k ( i ) to denote Pr( y i = k | g i ( X,G )) . And define L γ ( h i , y i ) := ✶ [ h i [ y i ] ≤ γ +max k = y i h i [ k ]] . Then we can write

Clearly,

̸

where in the last step we have used Assumption 2. Therefore,

where the last inequality utilizes the facts that both η k ( j ) and L γ ( h i , k ) are upper-bounded by 1 . According to the lemma 2, we can get:

ρ = ∥ µ 0 -µ 1 ∥ denotes the original feature separability, independent with structure. Further, as h i = f ( g i ( X,G ); W 1 , . . . , W L ) where f is a ReLU-activated MLP, so

This implies that, for any k = 1 , . . . , K ,

Detailed proof can be found in Lemma 5. So we have

Then we can prove the Bound for D γ m, 0 ( P ; λ ) .

Lemma 3 (Bound for D γ m, 0 ( P ; λ ) ) . Under Assumption 1, 2 and 3, for any 0 < m ≤ M , any 0 < λ ≤ N 2 α 0 and γ ≥ 0 , assume the 'prior' P on H is defined by sampling the vectorized MLP parameters from N (0 , σ 2 I ) for some σ 2 ≤ ( γ/ 8 ϵ m ) 2 /L 2 b ( λN -α 0 +ln2 bL ) . We have

Proof. Recall that D γ/ 2 m, 0 ( P ; λ ) = ln E h ∼ P e λ ( L γ/ 4 m ( h ) -L γ/ 2 0 ( h ) ) . We prove the upper bound of D γ/ 2 m, 0 ( P ; λ ) by decomposing the space H into the two regimes: a regime with bounded spectral norms of the model parameters required by Lemma 5, and its complement. Following Lemma 5, for any classifier h with parameters W 1 , . . . , W L , we define T h := max l =1 ,...,L ∥ W l ∥ 2 .

We first prove an upper bound on the probability Pr ( T L h ϵ m > γ 8 ) over the drawing of h ∼ P . For any h , as its vectorized MLP parameters vec ( W l ) , for each l = 1 , . . . , L , is sampled from N (0 , σ 2 I ) , we have the following spectral norm bound [99], for any t > 0 ,

where b is the maximum width of all hidden layers of the MLP. Setting t = ( γ 8 ϵ m ) 1 /L and applying a union bound, we have that

where the last inequality utilizes the condition σ 2 ≤ ( γ/ 8 ϵ m ) 2 /L 2 b ( λN -α 0 +ln2 bL ) .

For any h satisfying T L h ϵ m ≤ γ 8 , by Lemma 5, we know that e λ ( L γ/ 4 m ( h ) -L γ/ 2 0 ( h ) ) ≤ e Kλρ √ 2 πσ ( ϵ + | h 0 -h m |· ρ ) . For all h such that T L h ϵ m > γ 8 , by Assumption 3, with probability at least 1 -e -N 2 α 0 ,

Also note that L γ/ 4 m ( h ) -L γ/ 2 0 ( h ) ≤ 1 trivially holds for any h . Therefore we have

̸

We need to proof from h j [ k ] ≤ γ 2 +max l = k h j [ l ] to h i [ k ] ≤ γ +max i = k h i [ l ]

̸

Lemma 5. Define L γ ( h i , y i ) = ✶ [ h i [ y i ] ≤ γ +max k = y i h i [ k ]] , if || h i -h j || ∞ ≤ γ 4 , for any k = 1 , 2 , · · · , K , L γ/ 2 ( h j , k ) ≤ L γ ( h i , k )

̸

̸

̸

̸

̸

The proof complete.

F.4 Proof details of subgroup generalization Bound for GNNs

With the bound for D γ m, 0 ( P ; λ ) , we can then derive the subgroup generalization Bound for GNNs. Remark: Notably, the following proof is derived based on [46]. For completeness, we show the previous proof as follows, the difference is that we consider a more complicated conditional probability P ( y i = k | g i ( X,G )) , correlated with the homophily ratio difference, rather than consider the conditional probability as c -Lipschitz continuous functions. In other words, our proof further strengthens the effect of the structure disparity.

Assumption 1 (GNN model) . We focus on SGC [16] with the following components: (1) a one-hop mean aggregation function g with g ( X,G ) denoting the output. (2) MLP feature transformation f ( g i ( X,G ); W 1 , W 2 , · · · , W L ) , where f is a ReLU-activated L -layer MLP with W 1 , · · · , W L as parameters for each layer. The largest width of all the hidden layers is denoted as b .

Despite analyzing simple GNN architecture theoretically, similar with [46, 13, 47], our theory analysis could be easily extended to the higher-order case with empirical success across different GNN architectures shown in Section 3.4. Notably, the following assumptions 2 and 3 are adapted from [46].

Assumption 2. Define B m := max i ∈ V 0 ∪ V m ∥ g i ( X,G ) ∥ 2 . For any classifier ˜ h ∈ H with parameters { ˜ W l } L l =1 , assume ∥ ˜ W l ∥ F ≤ C for l = 1 , . . . , L . Assume B m , C are constants with respect to N 0 .

Theorem 1 (Subgroup Generalization Bound for GNNs) . Let ˜ h be any classifier in H with parameters { ˜ W l } L l =1 . for any 0 < m ≤ M , γ ≥ 0 , and large enough N 0 , with probability at least 1 -δ over the sample of y 0 , we have

The proof of Theorem 1 relies on the combination of Theorem 2, Lemma 4, and an intermediate result of the Theorem 1 in [100] (which we state as Lemma 6 below).

Lemma 6 ([100]) . Let ˜ h be any classifier in H with parameters { ˜ W l } L l =1 . Define ˜ β = ( ∏ L l =1 ∥ ˜ W l ∥ 2 ) 1 /L . Let { U l } L l =1 be the random perturbation to be added to { ˜ W l } L l =1 and vec ( { U l } L l =1 ) ∼ N (0 , σ 2 I ) . Define B m := max i ∈ V 0 ∪ V m ∥ g i ( X,G ) ∥ 2 . If

and β is any constant satisfying | ˜ β -β | ≤ ˜ β L , then with respect to the random draw of { U l } L l =1 ,

Then we prove Theorem 1, a re-stata as Theorem 1 with replacing the notation of training node subgroup V train to V 0 .

̸

Then we need to proof 3 4 γ +max l = k h j [ l ] ≤ γ +max l = k h i [ l ]

̸

̸

̸

̸

̸

̸

Theorem 1 (Subgroup Generalization Bound for GNNs (Adaption of Theorem 3 in [46])) . Let ˜ h be any classifier in H with parameters { ˜ W l } L l =1 . Under Assumptions 1, 1, 2, and 3, for any 0 < m ≤ M , γ ≥ 0 , and large enough N 0 , with probability at least 1 -δ over the sample of y 0 , we have

Proof. Note that, the following proof is adopted from [46]. There are two main steps in the proof. In the first step, for a given constant β > 0 , we first define the 'prior' P and the 'posterior' Q on H in a way complying the conditions in Lemma 4 and Lemma 6. Then for all classifiers with parameters satisfying | ˜ β -β | ≤ ˜ β L , where ˜ β = ( ∏ L l =1 ∥ ˜ W l ∥ 2 ) 1 /L , we can derive a generalization bound by applying Theorem 2 and Lemma 4. In the second step, we investigate the number of β we need to cover all possible relevant classifier parameters and apply a union bound to get the final bound. The second step is essentially the same as [100] while the first step differs by the need of incorporating Lemma 4.

We first show the first step. Given a choice of β independent of the training data, let

Assume the 'prior' P on H is defined by sampling the vectorized MLP parameters from N (0 , σ 2 I ) ; and the 'posterior' Q on H is defined by first sampling a set of random perturbations { U l } L l =1 with vec ( { U l } L l =1 ) ∼ N (0 , σ 2 I ) and then adding them to { ˜ W l } L l =1 , the parameters of ˜ h . Then for any ˜ h with { ˜ W l } L l =1 satisfying | ˜ β -β | ≤ ˜ β L , by Lemma 6, we have

Therefore, by applying Theorem 2, we know the bound (33) holds for ˜ h , i.e., with probability at least 1 -δ ,

where in (43) we have applied Lemma 4 to bound D γ/ 2 m, 0 ( P ; λ ) under Assumptions 1, 2, and 3; and in (44) we have set λ = N 2 α 0 .

Moreover, since both P and Q are normal distributions, we know that

By Assumption 2, both B m and C are constant with respect to N 0 . Later we will show that we only need β ≤ C . Therefore, for large enough N 0 , we can have which implies,

and hence,

Therefore, with probability at least 1 -δ ,

Then we show the second step, i.e., finding out the number of β we need to cover all possible relevant classifier parameters. Similarly as [100], we will show that we only need to consider ( γ 2 B m ) 1 /L ≤ ˜ β ≤ C (recall that ∥ ˜ W l ∥ F ≤ C, l = 1 , . . . , L ). For any ˜ β outside this range, the bound (47) automatically holds. If ˜ β < ( γ 2 B m ) 1 /L , then for any node i ∈ V 0 , ∥ ˜ h i ( X,G ) ∥ ∞ < γ 2 , which implies ̂ L γ 0 ( ˜ h ) = 1 as the difference between any two output logits for any training node is smaller than γ . Also noticing that L 0 m ( ˜ h ) ≤ 1 by definition, so the bound (47) trivially holds. And for ˜ β in this range, | β -˜ β | ≤ 1 L ( γ 2 B m ) 1 /L is a sufficient condition for β to satisfy | ˜ β -β | ≤ ˜ β L , and we need at most LC (2 B m ) 1 /L γ 1 /L of β to cover all ˜ β in the above range. Taking a union bound on all such β , which is equivalent to replace δ with δ LC (2 Bm ) 1 /L γ 1 /L in (46), it gives us the final result: with probability at least 1 -δ ,

G Experiment details

G.1 Hardware & Software Environment

The experiments are performed on two Linux servers (CPU: Intel(R) Xeon(R) CPU E5-2690 v4 @ 2.60GHz, Operation system: Ubuntu 16.04.6 LTS). For GPU resources, four NVIDIA Tesla V100 cards are utilized The python libraries we use to implement our experiments are PyTorch 1.12.1 and PyG 2.1.0.post1. The maximum time for training one epoch is no more than one minute.

G.2 Model details & hyperparameter settings & results

In this section, we provide details for baseline methods, hyperparameter search space, and the accuracy results.

Details for baseline methods We provide details for baseline methods in this section. MLP, GCN [2], GAT [11], and SGC [16] are classical baseline models, we directly adopt the implementation of them from Pytorch Geometric. Notice that, SRGNN and EERM are two baseline method specifically for the Graph OOD scenario.

GLNN, as proposed by [29], address the scalability challenges faced by Graph Neural Networks (GNNs) in practical industry deployments due to their data dependency. By leveraging the strengths of both GNNs and MLPs through knowledge distillation (KD), the authors

demonstrate that MLP performance can be significantly improved. The authors' code can be found at https://github.com/snap-research/graphless-neural-networks . In this work, we adopt GCN as the teacher model.

APPNP, developed by [15], leverages the relationship between graph convolutional networks (GCN) and PageRank to develop an enhanced propagation scheme based on personalized PageRank. APPNP focuses on addressing the limitations of traditional neural messagepassing algorithms in graph-based semi-supervised classification, which only considers a small, fixed neighborhood of nodes. The author's code is available at https://github. com/gasteigerjo/ppnp .

GPRGNN, introduced by [53], aggregates features across multiple steps and subsequently combines them linearly, with the weights of the linear combination being learned during model training. The original features prior to aggregation are also incorporated into the combination. The authors' code is available at https://github.com/jianhao2016/ GPRGNN .

GCNII, developed by [52], is a deep learning method for graph-structured data that extends the traditional GCN model by incorporating two innovative yet straightforward techniques: "Initial residual" and "Identity mapping." These techniques effectively address the oversmoothing issue commonly encountered in shallow GCN models. The author's code is available at https://github.com/chennnM/GCNII .

SRGNN, introduced by [63], is a novel framework incorporating a shift-robust regularizer that encourages the learned representation to enhance the ood generalization capabilities of GNNs and overcome the limitations posed by localized graph training data. The author's code is available at https://github.com/GentleZhu/Shift-Robust-GNNs .

EERM, introduced by [62], proposes a principled methodology to identify and quantify these factors in graph data and incorporate a novel invariance-inducing regularization term into the GNN training objective. The author's code is available at https://github.com/ qitianwu/GraphOOD-EERM .

hyperparameter searching range For model-specific parameters, we use the same notations for the following models considering brevity. Details are shown as follows.

λ : For GLNN, λ refers to the weight of knowledge distillation loss. For GCNII, λ refers to the strength of identity mapping.

α : For APPNP and GPRGNN, α refers to the teleport probability. For GCNII, α refers to the strength of residual connection.

K : For SGC, APPNP, and GPRGNN, K refers to the number of transformation layers.

In hyperparameter searching, we utilize the common range of hyperparameters for most models. Notice that, there are particular models where the reported optimal parameters in their papers are quite specific. In order to achieve better implementation results, we expand the search range on top of the original reported set for these models.

Reimplementation experimental results We illustrate the experimental results with the above hyperparameter search in Table 10 and 11 on homophilic and heterophilic datasets, respectively.

G.3 Dataset details

In this paper, we majorly focus on two homophilic datasets, PubMed and Ogbn-Arxiv and two heterophilic datasets, Squirrel and Chameleon. Moreover, we consider more datasets for comprehensive

experimental results including Cora, CiteSeer, IGB-tiny, Actor, Amazon-rating, and Twitch-gamers. Dataset statistics details can be found in Tab. 13. The license of datasets can be found in Table 12.

We then provide a detailed description on those datasets and the corresponding data split as follows:

Planetoid datasets include Cora, CiteSeer, and PubMed. They consist of scientific publications as nodes. The citation links between the papers create a graph structure, with nodes representing papers and edges representing citations. We utilize the 10 fixed splits in [101]. For each split, we have 20 nodes per class for training, 30 nodes per class for validation, and all other nodes for testing. More details can be found on https://github.com/BUPT-GAMMA/CPF/tree/master/data/npz

Ogbn-Arxiv dataset is based on a large-scale scholarly paper citation network, where nodes represent papers and edges represent citation links between them. We utilize the default fix split provided by [24]. More details can be found on https://ogb.stanford.edu/ docs/nodeprop/#ogbn-arxiv .

Wikipedia datasets include Chameleon and Squirrel, which are a collection of web pages from Wikipedia. The node features correspond to several informative nouns in the respective Wikipedia pages. The edge represents the link between node. We utilize the 10 fixed splits in [25]. For each split, we have 48% nodes for training, 32% nodes for validation, and 20% for test. More details can be found on https://github.com/graphdml-uiuc-jlu/ geom-gcn

The Actor dataset [102] is a graph that represents the co-occurrence relationships between actors. We utilize the 10 fixed splits in [25]. For each split, we have 48% nodes for training, 32% nodes for validation, and 20% for test. More details can be found on https: //github.com/graphdml-uiuc-jlu/geom-gcn

The Twitch-gamers dataset represents the relationships between accounts on the streaming platform Twitch, where each node corresponds to a Twitch account, and edges are present between accounts that are mutual followers. We utilize the 5 fixed splits in [21]. For each split, we have 50% nodes for training, 25% nodes for validation, and 25% for test. More details can be found on https://github.com/CUAI/Non-Homophily-Large-Scale

Amazon-ratings a new dataset derived from the Amazon product co-purchasing network metadata dataset, which is part of the SNAP Datasets [103]. We utilize the 10 fixed splits in [104]. For each split, we have 50% nodes for training, 25% nodes for validation, and 25% for test. More details can be found on https://github.com/yandex-research/ heterophilous-graphs/tree/main/data

IGB-tiny is a recent new citation graph dataset. It utilizes two publicly available datasets: Microsoft Academic Graph (MAG) and SemanticScholar Corpus. We utilize the homogeneous setting with default fix split provide by [105]. More details can be found on https://github.com/IllinoisGraphBenchmark/IGB-Datasets .

Details on Data distribution The node homophily ratio distribution for different datasets are presented in Figure 13. The following observations can be made: (1) Cora, CiteSeer, PubMed, and Ogbn-Arxiv are four homophilic datasets where most nodes have a homophily ratio h > 0 . 5 . (2) Chamaleon, Squirrel, and Actor are three heterophilic datasets where most nodes have a homophily ratio h ≤ 0 . 5 . Nonetheless, the Actor dataset exhibits entirely different properties from Chamaleon and Squirrel where graph structure information is barely useful, leading to performance degradation. Empirical evidence can be found in Table 11 where the vanilla MLP outperforms all the GNN models. (3) IGB-tiny [105] and Twitch-gamers [21] are two representative datasets with no apparent majority structural pattern, exhibiting graph homophily ratios of 0.567 and 0.556, respectively. Specifically, IGB-tiny contains both large numbers of homophilic and heterophilic nodes, respectively, whereas, most nodes in the Twitch-gamers dataset do not show a clear homophilic nor heterophilic pattern with a homophily ratio close to 0.5. Notably, the IGB-tiny is still recognized as a homophilic graph [105] while Twitch-gamers is generally recognized as a homophilic graph [20, 21]. (4) Nodes in Amazonratings [104] exhibit diverse structural patterns with multiple peaks observed for homophily ratios 0.0, 0.2, 0.4, 0.6, and 0.8.

G.4 OODdataset statistics and details

We provide more analysis and statistics details about datasets with our proposed out-ofdistribution (OOD) data split for a more comprehensive analysis. The data split statistics can be found in Table 14. 20% of the majority nodes are selected as the validation set. Notably, we generally use nodes with h i < 0 . 5 and h i > 0 . 5 as the majority and minority on heterophilic graph.

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

4

Density

h

=0.81)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

(b) CiteSeer (

h

=0.71)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

4

5

Density

(c) PubMed (

h

=0.79)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

Density

(d) Ogbn-Arxiv (

h

=0.63)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

(e) IGB-tiny (

h

=0.57)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

h

=0.25)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

Density

h

=0.22)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

3

4

Density

(h) Actor (

h

=0.22)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

h

=0.56)

0.0

0.2

0.4

0.6

0.8

1.0

hnode

0

1

2

Density

(j) Amazon-ratings (

h

=0.38)

̸

Nonetheless, the Squirrel dataset only has 71 nodes with h i > 0 . 5 . To ensure enough test nodes, we adapt the nodes with h i > 0 . 4 as the minority nodes for test. We conduct experiments to measure the distributional shift between train and test nodes. To further analysis on the distribution shift, we employ Maximum Mean Discrepancy (MMD) [106] as a discrepancy metric. A large MMD distance indicates the covariate shift with P train ( X ) = P test ( X ) . MMDmeasures the distance between distributions as follows:

where ϕ is a kernel function that maps the input samples into a higher-dimensional feature space, and | || · || 2 denotes the L 2 norm. Using the Gaussian kernel, we can define the kernel matrices as:

where the kernel function is given by:

and σ is a parameter that controls the width of the kernel. In our experiment, we choose multiple σ including [0.01, 0.1, 1, 10, 100]. We average the MMD distance with different σ for a more robust distance measurement. The MMD distance can be computed using these kernel matrices:

̸

̸

where the first and third terms represent the within-set variations, and the second term represents the between-set variations. We then conduct experiments to determine where the source of distribution discrepancy from the covariate shift or concept shift. We measure the MMD distance between the train set and validation set as the i.i.d. distribution distance, and the MMD distance between the training set and test set as the OOD distribution distance. However, the in-distribution and out-distribution distances may not be directly comparable due to potential estimation errors introduced by different numbers of nodes in valid and test sets.

̸

To enable a fair comparison, we also measure in-distribution and out-distribution distances on the i.i.d. data split as a control group. The MMD distances are measured on the last hidden representation of a well-trained GCN. The experiment results are provided in Table 15. Focusing on the i.i.d setting, we observe an MMD distance difference on the validation set and test set to the training set, induced by the difference in node counts between the validation and test set. Comparing the OOD setting with the i.i.d one no significant MMD differences are found, except for PubMed. This suggests that the primary factor driving the distribution shift is not the covariate shift with P train ( X ) = P test ( X ) , but the concept shift with P train ( Y | X ) = P test ( Y | X ) . The above observations further indicate the existence of the concept shift in our proposed OOD scenario.

̸

Additional OOD results : We show additional experimental results on Cora and CiteSeer datasets in Table 16 with a consistent observation. Notice that, all experimental results are average with 10 random seeds including (1, 3, 5, 7, 11, 13, 17, 19, 23, 29).

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.3

0.2

0.1

0.0

0.1

Difference in acc

model - GCN

MLP

GLNN

h

=0.81)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.15

0.10

0.05

0.00

0.05

0.10

Difference in acc

model - GCN

MLP

GLNN

(b) CiteSeer (

h

=0.71)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.050

0.025

0.000

0.025

0.050

0.075

0.100

0.125

Difference in acc

model - GCN

MLP

GLNN

(c) IGB-tiny (

h

=0.58)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.100

0.075

0.050

0.025

0.000

0.025

0.050

0.075

0.100

Difference in acc

model - GCN

MLP

GLNN

(d) Actor (

h

=0.22)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.1

0.0

0.1

0.2

0.3

Difference in acc

model - GCN

MLP

GLNN

(e) twitch-gamers (

h

=0.56)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.1

0.0

0.1

0.2

0.3

Difference in acc

model - GCN

MLP

GLNN

(f) Amazon-ratings (

h

=0.38)

H Additional results on performance comparison between GCN and MLP-based models

In Section 3, we conduct a performance comparison between GCN with two MLP-based models, i.e., MLP and GLNN, on test nodes with different structural patterns. In this section, we provide more elaboration on why using GLNN and conduct additional experiments on more datasets.

We first provide more elaborations on why using GLNN. The experiment is to examine the effectiveness of GCN on utilizing different structural patterns. Therefore, we compare GCN and MLP

architectures as GCN utilizes graph structure during inference while MLP cannot, serving as a structure-agnostic baseline. When GCN surpasses MLP,it indicates GNN benefits from structural patterns effectively, and vice visa. Notably, GLNN can be viewed as a better-trained MLP model. GLNN also utilizes the same MLP model architecture as vanilla MLP, the only difference between vanilla MLP and GLNN is that GLNN is trained in an advanced distillation manner while vanilla MLP is trained with cross-entropy loss The reason why we utilize GLNN rather than only comparing GCN with vanilla MLP is that MLP meets optimization issue on training. Such an obstacle leads to a large performance gap (more than 20%) between under-trained vanilla MLP and well-trained GCN. Consequently, a large performance gap induced by training difficulty hinders the potential for MLP architecture. Contrastly, GLNN enjoys a better training process leads to a more clear comparison between well-trained GNN and well-trained MLP(GLNN) architecture with a convincing conclusion

Additional experiments are illustrated in Fig 14. Cora and CiteSeer are two Planetoid datasets, showing similar properties as PubMed. Observations on Cora and CiteSeer are consistent with those in Section 3, further substantiating our conclusions. It should be noted that our paper primarily focuses on homophilic graphs and heterophilic graphs with a clear major structural pattern. Nonetheless, there exists more complicated real-world graphs with more diverse structural properties, as demonstrated in Figure 13. We conduct further investigations on these datasets, including IGB-tiny, Twitchgamers, Amazon-ratings, and Actor. A comprehensive discussion on those datasets can be found in Appendix G.3. Experimental results are depicted in Figure 14. We make initial observations as follows: (1) IGB-tiny is a homophilic dataset exhibiting consistent observations in line with the main analysis in Section 3, where the MLP-based models outperform GCN on the heterophilic nodes and underperform on homophilic nodes. (2) Twitch-gamers and Amazon-ratings are two heterophilic datasets exhibiting consistent observations in line with the main analysis in Section 3, where the MLP-based models outperform GCN on the homophilic nodes while falling short on the heterophily nodes. (3) Actor is a heterophilic dataset where graph information is of limited utility. Empirical evidence shows that MLP-based models show an overall better performance than GCN. From a local perspective, the MLP-based models outperform GCN on the heterophilic nodes while underperforming on the homophily nodes. Those observations indicate our conclusion can be extended on most datasets with diverse structural patterns Nonetheless, it still remains a mystery on the Actor dataset. We leave a more comprehensive investigation especially on the Actor dataset as the future work.

I Additional results on performance comparison between GCN and deeper GNN models

In Section 4.1, we conduct a performance comparison between GCN with deeper GNN models, on test nodes with different structural patterns. In this section, we conduct additional experiments on more datasets, correspondingly. Cora and CiteSeer are two Planetoid datasets, showing similar properties as PubMed. Notably, deeper GNNs generally show better performance across different structural patterns, where the improvement still primarily lies in the minority groups, Observations on Cora and CiteSeer are consistent with those in Section 3, further substantiating our conclusions.

It should be noted that our paper primarily focuses on homophilic graphs and heterophilic graphs with a clear major structural pattern. Nonetheless, there exist more complicated real-world graphs with more diverse structural properties, as demonstrated in Figure 13. We conduct further investigations on these datasets, including IGB-tiny, Twitch-gamers, Amazon-ratings, and Actor. A comprehensive discussion on those datasets can be found in Appendix G.3. New experimental results are depicted in Figure 15. We observe that deeper GNNs generally outperform GCN on the heterophilic node while underperforming on the homophilic nodes on those datasets with diverse structural properties. We can conclude that the effectiveness of GNN on those datasets with no clear majority structural patterns primarily from the improvement from the heterophilic nodes. We leave a more comprehensive investigation as the future work.

J Additional Experiment on performance disparity across subgroups

In Section 3.4, we conduct experiments to empirically examine the effects of two-hop aggregatedfeature distance and the homophily ratio difference on the test performance of different GNN models. Despite the empirical success on the second hop, we conduct additional experiment results focusing

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.02

0.00

0.02

0.04

0.06

0.08

0.10

Difference in acc

model - GCN

GPRGNN

APPNP

h

=0.81)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.04

0.02

0.00

0.02

0.04

0.06

Difference in acc

model - GCN

GPRGNN

APPNP

(b) CiteSeer (

h

=0.71)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.02

0.00

0.02

0.04

0.06

0.08

0.10

Difference in acc

model - GCN

GPRGNN

APPNP

(c) IGB-tiny (

h

=0.57)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.15

0.10

0.05

0.00

0.05

0.10

Difference in acc

model - GCN

GPRGNN

APPNP

(d) Actor (

h

=0.22)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.10

0.05

0.00

0.05

0.10

Difference in acc

model - GCN

GPRGNN

APPNP

(e) twitch-gamers (

h

=0.56)

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

Homophily Ratio Range

0.3

0.2

0.1

0.0

0.1

Difference in acc

model - GCN

GPRGNN

APPNP

(f) Amazon-ratings (

h

=0.38)

on observations on higher-order homophily ratios, higher-order aggregated feature distance, and more datasets in this section. Notably, despite more complicated datasets, our theoretical analysis can still achieve empirical success.

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(a) PubMed

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(c) Chameleon

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(a) PubMed

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(c) Chameleon

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(a) PubMed

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) Ogbn-arxiv

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(c) Chameleon

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

J.1 Additional investigation on higher-order neighborhood

Our theory 1 in Section 3.3 focuses on the one-hop aggregation while it could be easily extended to the higher-order aggregation case in the real-world scenario. More experiments revolving on higher-order homophily ratio and aggregated features are shown in this subsection. In particular, the k -hop node homophily ratio can be calculated as:

where N k ( v i ) denotes the k -hop neighbor node set of v i . We then sort test nodes in terms of the disparity score and split them into 5 equal-size. Figures 16 and 17 illustrate results on higherorder (third hop) homophily ratio difference and higher-order aggregated feature distance (third hop), respectively while Figure 18 illustrates on the result on higher-order homophily ratio difference and higher-order aggregated feature distance.

A similar phenomenon can be found with the expected trend of test accuracy in terms of increasing differences on the aggregated feature distance and homophily ratio differences in different scenarios. Exceptional observation can be found in the following scenarios: (1) Squirrel dataset with higherorder aggregated feature. It may be because such higher-order feature aggregation leads to more distribution overlapping between different categories. (2) PubMed with both higher-order aggregated features and homophily ratio. The potential reason is that the PubMed dataset does not rely on higher-order information. Overall speaking, those results further indicate that for the generalization of our theory in the real-world scenario.

J.2 Additional results on more datasets

In this subsection, we conduct more experimental results on the datasets with more complicated with more diverse structural patterns, including Cora, CiteSeer, IGB-tiny, Twitch-gamers, Amazonratings, and Actor. A comprehensive discussion can be found in Appendix G.3. The performance on different node subgroups is presented in Figure 19. A consistent observation can be found with the expected test accuracy degradation trend in terms of larger feature distance and homophily ratio. the difference in most datasets, except for the Actor. The potential reason is that its structure could hardly provide useful information. The key evidence is that the vanilla MLP model outperforms all the GNN models shown in Table 11. The actor dataset shows entirely different properties with other datasets. Furthermore, we investigate on the individual effect of aggregated feature distance and homophily ratio difference in Figure 20 and 21, respectively. An overall trend of performance decline with increasing disparity score is evident though some exceptions are present. When only considering the aggregated feature distance to training nodes, the accuracy in Twitch-gamers exhibits an opposite increasing tendency along with the increasing difference to train nodes. Notably, when considering only the homophily ratio difference, the decreasing tendency is clear across datasets exception Actor. We can observe even better phenomenon on the Amazon-ratings dataset than combining both homophily ratio difference and the aggregated feature distance. It indicates that the homophily ratio difference w.r.t structural disparity significantly contributes to those datasets with more diverse structural patterns. We make the above initial observations showing the practicability of our theoretical analysis on more datasets with more diverse structural patterns. Moreover, exception can still be found on the Actor dataset. We leave a more comprehensive discussion on the Actor, as the future work.

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

h

=0.81)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) CiteSeer (

h

=0.71)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(c) IGB-Tiny (

h

=0.57)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

ACC

(d) Actor (

h

=0.22)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

h

=0.56)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(f) Amazon-ratings (

h

=0.38)

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) CiteSeer

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

(c) IGB-Tiny

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

ACC

(d) Actor

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

(e) Twitch-gamers

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

(f) Amazon-ratings

K Additional investigation on the discriminative ability of GCN

In Section 3.2, we examine discriminative ratios on input features and multiple hops aggregated features for homophilic and heterophilic test nodes, respectively. Nonetheless, those experiments only focus on feature aggregation while ignoring the feature transformation and the training procedure. In this section, we further explore the discriminative ratios on well-trained GCNs with both feature aggregation and transformation. Typically, we utilize the hidden representation trained with the best hyperparameter as shown in Appendix G. Experimental results can be found in Figure 22. Observations on GCN are consistent with those in Section 3.2 focusing on aggregation, where majority nodes generally show a larger discriminative improvement than the minority nodes along with more aggregation layers, indicating a better discriminative ability on majority nodes than the minority ones. Those observations further substantiate our conclusions in a more general setting.

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(b) CiteSeer

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

0.8

ACC

(c) IGB-Tiny

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

ACC

(d) Actor

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

(e) Twitch-gamers

12345

12345

12345

12345

12345

GCN

SGC

GAT

0.0

0.2

0.4

0.6

ACC

(f) Amazon-ratings

0

1

2

#aggregation

0.0

0.5

1.0

1.5

2.0

2.5

Ratio

PubMed

Arxiv

Chameleon

L Significant test between GCN and MLP-based models

We conduct the performance comparison between GCN with MLP-based models, on test nodes with different homophily ratios, as shown in Fig 3. In the corresponding, we include the p-values of a paired t-test and confidence intervals between GCN and MLP-base models in Table 23, to determine whether the performance difference is statistically significant. The x -axis corresponds to the node homophily ratio range. The y -axis corresponds to p-values of whether the performance of GCN is significantly better or worse than the corresponding MLP model. Typically, p < 0 . 05 indicates statistical significance.

M Significant test between GCN and deeper GNN models

We conduct the performance comparison between GCN with deeper GNNs, on test nodes with different homophily ratios, as shown in Fig 8. In the corresponding, we include the p-values of a paired t-test and confidence intervals between GCN and deeper GNN models in Table 24, to determine

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.14

0

0

0

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.14

0

0

0

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(b) CiteSeer

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.14

0

0

0

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(c) PubMed

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0

0

0.03

0

0

0.01

0.5

0

0.0

0.2

0.4

0.6

0.8

1.0

(d) Chameleon

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.83

0.01

0.86

0.01

0

0.08

0.01

0.24

0.0

0.2

0.4

0.6

0.8

1.0

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.67

0.19

0

0

0

0.04

0.26

0

0.0

0.2

0.4

0.6

0.8

1.0

(f) Actor

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0.97

0

0

0

0

0.83

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(g) Twitch-gamers

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

MLP

GLNN

0

0

0

0

0

0

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(h) Amazon-ratings

whether the performance difference is statistically significant. The x -axis corresponds to the node homophily ratio range. The y -axis corresponds to p-values of whether the performance of GCN is significantly better or worse than the corresponding deeper model. Typically, p < 0 . 05 indicates statistical significance.

N Limitation

It is important to acknowledge certain limitations in our work despite our research providing valuable insights into the effectiveness of GNN on structural disparity. In order to facilitate a more feasible theoretical analysis, we have made several assumptions, the most significant of which is the use of the generalized CSBM-S model assumption in our theoretical analysis. This CSBM assumption, although prevalent in graph analysis, is subject to a notable drawback as the CSBM assumption postulates that feature dimensions are independent and identically distributed. It restricts the generality of our analysis.

Our current findings lay a strong foundation for further exploration. It is worth mentioning that our theoretical analysis predominantly focuses on simple aggregation techniques. We hope that these efforts will contribute significantly to the Graph Neural Network research community. Moving forward, we aspire to broaden our findings to encompass more advanced message-passing neural networks, with particular emphasis on deeper GNN models. Currently, there is only empirical evidence showing the effectiveness of the deeper GNN models. Besides, the node homophily ratio is calculated on the undirected graph while the one on the directed graph is still under exploration. More comprehensive analyses on the higher-order aggregated feature and higher-order homophily ratio are still under-explored. Moreover, We aim to utilize our understanding to mitigate the performance difference induced by the structural disparity in the future.

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0

0

0

0.04

0

0.43

0.85

0.03

0.97

0.11

0.18

0.53

0.02

0.03

0.1

0.0

0.2

0.4

0.6

0.8

1.0

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0.31

0.01

0.81

0.61

0.46

0.08

0.03

0.04

0.53

0.19

0.11

0.87

0

0.01

0.03

0.0

0.2

0.4

0.6

0.8

1.0

(b) CiteSeer

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0

0

0

0

0

0

0

0

0

0.35

0

0.01

0

0

0.3

0.0

0.2

0.4

0.6

0.8

1.0

(c) PubMed

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0.47

0.89

0.1

0.59

0

0.84

0.87

0.01

0.09

0.49

0.09

0

0.14

0.38

0.8

0.0

0.2

0.4

0.6

0.8

1.0

(d) Chameleon

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0.01

0.01

0.05

0.28

0.44

0.1

0

0.21

0

0.57

0.03

0

0.03

0.65

0.12

0.0

0.2

0.4

0.6

0.8

1.0

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0

0

0.05

0.14

0

0

0

0.55

0.06

0

0

0

0.09

0.07

0

0.0

0.2

0.4

0.6

0.8

1.0

(f) Actor

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0

0.54

0

0

0

0.26

0.46

0.09

0.56

0.87

0

0

0.17

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(g) Twitch-gamers

0.0-0.2

0.2-0.4

0.4-0.6

0.6-0.8

0.8-1.0

GPRGNN

APPNP

0

0

0

0

0

0

0

0.01

0

0

0

0

0

0

0

0.0

0.2

0.4

0.6

0.8

1.0

(h) Amazon-ratings

O Broader Impact

Graph Neural Networks (GNNs) have emerged as a powerful architecture for modeling graphstructured data across a wide range of practical applications [2, 5, 3, 6-8, 73, 107]. A key aspect of GNNs is the inductive bias on leveraging neighborhood information. However, such characteristics may lead to a biased test prediction, particularly when majority neighborhood patterns are prevalent. For example, a scenario in which we are required to predict the gender of individuals with a dating network. GNN may excessively rely on the pattern that when all neighboring nodes are female, the center node should be male. This approach disregards the minority group of individuals who are more likely to date others of the same gender. This overlooked drawback in GNN may lead to ethical concerns.

In this study, we reveal the potential for performance disparity inherent in Graph Neural Networks and propose a novel OOD scenario for exploring methods to mitigate such issues. It is important to emphasize that our research offers an understanding of these limitations rather than introducing a new methodology or approach. We only identify the problem and show the potential solution to

address this issue. Consequently, we do not foresee any negative broader impacts stemming from our findings. We expect our work to contribute significantly to the ongoing research efforts aimed at enhancing the versatility and fairness of GNN models when applied to diverse data settings.

Our findings can also help to boost different graph domains. In section 4, we show how our understanding can help to elucidate the effectiveness of deeper GNN, reveal an overlooked OOD factor, and propose a new Graph out-of-distribution scenario. It is worth noting that our findings can derive new understandings of more graph applications and inspire us to identify new problems in graph domains. We further illustrate more understanding on the robustness of GNN and more initial understanding on Graph out-of-distribution problem as follows.

Graph Robustness. Recent studies have exposed the susceptibility of GNNs to graph adversarial attacks [108-112], where small perturbations can mislead GNNs into making incorrect predictions. several literature [113-115] reveals that increasing the heterophily of the homophily graph could be a key factor contributing to a successful attack. More recently, [116] observes that (1) learning-based attacks [114] tend to increase the heterophily in the smaller of train and test sets. (2) Directed attacks on particular nodes can outperform the undirected attack on the whole graph. Those observations are aligned with the understanding in our paper that (1) Increasing the heterophily of the homophilic graph enlarges the homophily ratio difference, leading to performance degradation. (2) Perturbations lying in the smaller set can efficiently enlarge the homophily ratio difference between training and test nodes with minimal perturbation budgets. (3) Undirected attack on the whole graph may not lead to a larger homophily ratio difference between the train and test nodes with permutation on both train and test nodes, leading to performance degradation.

̸

Understanding on Graph out-of-distribution problem. Despite the new proposed Graph OutOf-Distribution (OOD) scenario, our findings can further inspire a new understanding of existing graph OOD problems. We provide an initial discussion as follows. The uniqueness challenge of the OOD problem in the graph domain is that the discrepancy in graph structure can also lead to test performance degradation, in addition to node feature differences. There exists various graph OOD scenarios[62-64, 24, 66, 46], e.g., graph density, biased training labels, time shift, and popularity shift. Nonetheless, despite various graph OOD scenarios with different qualitative concepts, it still lacks of quantitative graph metrics to measure how the distribution shift happens, causing performance degradation. Inspired by our finding, We can roughly recognize the main reasons as graph covariate shift and concept shift [68-70, 117] in a graph context: (1) Graph covariate shift [118] is defined as P train ( X ) = P test ( X ) where P train ( · ) and P test ( · ) are training and test distribution, respectively. Such graph covariate shift corresponds to the large feature distance ∥ f u -f v ∥ between train and test nodes. (2) Graph concept shift is defined as P train ( Y | X ) = P test ( Y | X ) . Such graph concept shift corresponds to the homophily ratio difference | h u -h v | between train and test nodes. We leave a more detailed discussion and a careful design on the graph OOD quantative metrics as a further work.

P Future work

Further investigation on Deeper GNN In section 4, we conduct experiments on the effectiveness of deeper GNNs, indicating the performance gain is majorly from the improved discriminative ability on nodes in minor patterns. It could be a good direction to further explain and understand why such phenomenon happens. A potential reference for conducting discriminative analysis of deeper GNN is [39], aims to theoretically quantify the discriminative ability on deeper GNNs with more aggregations. Nonetheless, their analysis uses the vanilla CSBM model as the data assumption, denoted as CSBM ( µ 1 , µ 2 , p, q ) . The CSBM model presumes that all nodes follow either homophilic with p > q or heterophilic patterns with p < q exclusively. However, this assumption conflicts with real-world scenarios as homophilic and heterophilic patterns coexist across different graphs, as shown in Figure 2. As far as we can see, conducting a similar discriminative analysis as [1] on our proposed CSBM-M model considering both homophily and heterophily patterns could be a good solution.

Other factors on performance disparity Existing literature [63, 46, 86, 87] shows that some other structural information, e.g., degree, geodesic distance to the training node, and Personal Pagerank score, could lead to a performance disparity. Nonetheless, all those analyses and conclusions are conducted on homophilic graphs, e.g., PubMed and ogbn-arxiv, while ignoring the heterophilic graphs e.g., Chameleon and Squirrel, which also broadly exist in the graph domain. It could be a good new research direction to see how the above factors, focus on the homophilic pattern in the context

̸

of both homophilic and heterophilic properties. Another one could be how to find other important factors on both homophilic and heterophilic graphs.

Solutions on performance disparity In this paper, we propose and analyze the importance of the performance disparity problem without no explicit solution. We provide two potential solution as follows:

Combining MLP and GNN in an adaptive approach since MLPs can achieve better performance on minority nodes and GNNs better on majority nodes. Ideally, we can adaptively select MLP for minority nodes and GNN for majority nodes, using an adaptively gated function to control the proportion of MLP and GNN. Our findings suggest the homophilic/heterophilic pattern selection can serve as a guide for learning the gate function.

Utilizing global structural information, as the global pattern is more robust, showing less disparity than the local structural pattern. Empirical evidence can be found in Figure 9 that the higher-order homophily ratio differences are smaller than the local structure disparity.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates the effectiveness of GNNs on nodes with different structural patterns in real-world graphs and proposes a new method to identify the reasons for performance disparities. The authors found that GNNs tend to perform well on homophilic nodes within homophilic graphs, but struggle with the opposite node set, and they provide insights into the performance disparities by analyzing aggregated feature distance and homophily ratio difference between training and testing nodes.\n\nSTRENGTHS:\nThis paper provides both empirical and theoretical evidence to support its findings. It conducts rigorous analyses and derives a non-i.i.d. PAC-Bayesian generalization bound for GNNs, which adds credibility to the research. The authors substantiate their findings with controllable synthetic experiments and real-world datasets, further strengthening the quality of the paper.\n\nWEAKNESSES:\n1.\tWriting needs to improve.\n2.\tSome explanation needs more elaboration.\n\nQUESTIONS:\n1.\tLine 49-50, “while heterophilic graphs exhibit an opposite phenomenon with homophilic nodes in the majority and heterophilic ones in the minority.” Check this definition of heterophilic graphs.\n2.\tLine 55-56, “ Such differences may lead to performance disparity between nodes in majority and minority patterns.” If you randomly sample training and test data, the distribution of the majority and minority patterns should be the same for training and test nodes. Thus, the disparity shown in this example is not valid in practice.\n3.\t“Consequently, the performance disparity can be overwhelmed by such gap which renders the effect from structural patterns.” I don’t understand this sentence. Please elaborate the reason why you use GLNN here.\n4.\tAre the observations in Figure 3 also hold on other benchmark datasets?\n5.\tLine 170, why both homophilic and heterophilic patterns have p>q? In your definition, nodes from the subgroup but different classes have the same (p,q)?\n6.\tLine 179-181, check the English.\n7.\t“This proposition suggests that aggregation results in a distance gap between different patterns within the same class.” I cannot see how this proposition suggests the distance gap. Please elaborate it.\n8.\tIn Lemma 1, are nodes u,v from the same class? Why do you want to examine “discrepancy between nodes and with the same aggregated feature but different structural patterns.”? How does this relates to the proposed claim about majority and minority nodes?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on the performance disparity on homophily and heterophily nodes in node-level classification tasks. It claims that although GNNs have good performance on both pure homophily and pure heterophily graphs, GNNs cannot perform well when dealing with graphs with both these two types of nodes. The paper then analyses the reason for different impact on majority and minority nodes and illustrates the aggregation function's contribution on this. It delves into the theoretical analysis, deriving a non-i.i.d PAC-Bayesian generalization bound based on the Subgroup Generalization bound of Deterministic Classifier. The theoretical analysis indicates that test nodes with larger aggregated feature distances and homophily ratio differences from training nodes experience performance degradation. The practical implications of the findings are demonstrated on real-world datasets.\n\nSTRENGTHS:\n- The paper provides sufficient real-world experiments to show how the homophily ratio and the feature distance influence the GNN’s aggregation and the actual performance on both homophilic and heterophilic datasets. This could sever as a foundation for future GNN architecture design.\n-  The paper performs the theoretical analysis that reveals potential reasons for the performance disparity. The findings are supported by experimental results.\n- The paper is well structured with good clarity in presentation.\n\nWEAKNESSES:\n- The paper identifies the problem of performance disparity but does not provide a solution to address it. \n- The classification of nodes into homophily and heterophily classes using a hard threshold is a simple and ideal setting. This setting may fail to capture the complexity and nuances that may exist in real-world scenarios. Do the conclusions/findings still hold if we directly use the continuous homophily ratio rather than thresholding it?\n\nQUESTIONS:\nSee the weaknesses section.\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-04"}

comments: SUMMARY:
This paper provides a rigorous analysis of the effect of structural disparity on the performance of GNNs. The proposed CSBM-S model and the application of PAC-Bayes analysis, among others, show the different effects of aggregation on the performance of nodes with different structural disparity. The analysis further indicate subgroup generalization bound for GNN and elucidate the effectiveness of deeper GNNs.

STRENGTHS:
1.	The problem considered in this paper is extremely important. I agree with the authors that real-world graphs can’t be easily classified as homophilic or heterophilic. Therefore, analyzing how structural disparity influences the performance is essential to GNN design.
2.	It is well backed up by theoretical analysis and examples. The authors show several nice analysis methods, including CSBM-S model and subgroup generalization bound for GNNs.
3.	Extensive experiment enables the conclusions presented in this paper to be convincing

WEAKNESSES:
1.	The conclusions hold in this paper share the same promise that the aggregation operation is neighbourhood averaging, which may not generalize to a broad range of GCNs models
2.	The writing could be improved in terms of tone and phrases. There are also some grammar and spelling errors

QUESTIONS:
There are two assumptions that nodes from different subgroup share the same distribution and similar degree distribution. How does the conclusions are still valid without the above assumptions as stated in the supplementary material.

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-04

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tries to understand the effectiveness of GNNs w.r.t different structural disparities within a graph. Previous studies have focussed on GNN's effectiveness on overall graphs, but here the authors try to understand GNN's effectiveness w.r.t structural patterns such as homophilic and heterophilic nodes to provide deeper insight into GNN's performance. \n\nContributions include understanding why the performance of GNNs is rather good on homophilic nodes in homophilic graphs and heterophilic nodes in heterophilic graphs, but not on the opposite set. Specifically, how aggregated feature distances and homophily ratio differences in mixed graphs impact the GNN performance. Authors also present experimental results on how deeper GNNs perform better on minority node subgroups while also proposing a new data split strategy to where majority nodes are selected for train/validation and minority nodes for test.\n\nSTRENGTHS:\nGNN performance studies on datasets across difference structural subgroups is a crucial metric to understand the overall performance of the model. Quantitive understanding of the effect of aggregation and homophily ratio difference w.r.t train nodes is a significant metric to compare different GNN architectures. This provides new evaluation strategies for future GNN studies. The effect of new data splitting strategy is again a significant contribution and is useful to evaluate the overall GNN performance.\n\nWEAKNESSES:\nThe work by itself is strong and would have been even stronger if presented with more understanding of the effectiveness of Deeper GNNs on doing comparatively well on minority nodes.\n\nQUESTIONS:\n1. Is there a way to prove deeper GNNs improved discriminative ability on minority nodes? \n2. Are there any additional factors other than aggregation and homophily ratio difference that you thought about?\n\nLIMITATIONS:\nNone", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-24"}