SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Table 7: Hyperparameter searching range for baseline models on Cora, CiteSeer, PubMed
Hyperparameters for Baseline GNNs
Models\Hyperparameters	lr	weight Decay	dropout	hidden	# layers	Gat heads	λ	α	K
GCN	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{16, 32, 64}	2	-	-	-	-
SGC	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{16, 32, 64}	{1.2}	-	-	-	{1.2,3}
GAT	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.6, 0.8}	{8, 16, 32, 64}	2	{1,4,8}	-	-	-
GLNN	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{16, 32, 64}	{1,2}	-	{0.0-1.0}	-	-
APPNP	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{8, 16, 32, 64}	2	-	-	{0.0-1.0}	{5.6,7,8,9,10}
MLP	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{16, 32, 64}	2	-	-	-	-
GPRGNN	{1e-2, 5e-2, 5e-3}	{0, 1e-3, 1e-5, 1e-4, 5e-4, 5e-5}	{0, 0.2, 0.5, 0.8}	{16, 32, 64, 128}	{2.3}	-	-	{0.0-1.0}	{5.8,9,10}
GCNII	{1e-2, 5e-3, 1e-3}	{0, 5e-5, 5e-4}	{0, 0.1, 0.2, 0.4, 0.5, 0.8}	{16, 32, 64, 128, 256}	{2,4,8,16,32,64}	-	{0.0-1.0}	{0.0-1.0}	-
Reimplementation experimental results We illustrate the experimental results with the above hyperparameter search in Table 10 and 11 on homophilic and heterophilic datasets, respectively.
G.3 Dataset details
In this paper, we majorly focus on two homophilic datasets, PubMed and Ogbn-Arxiv and two heterophilic datasets, Squirrel and Chameleon. Moreover, we consider more datasets for comprehensive
Table 8: Hyperparameter searching range for baseline models on Chameleon, Squirrel, Actor, Amazon-ratings
Hyperparameters for Baseline GNNs
Models\Hyperparameters |	Ir	weight Decay	dropout	hidden	# layers	Gat heads	λ	α	K
GCN	{1e-2, 5e-2, 5e-3,}	{0, 5e-4, 5e-5}	{0, 0.2, 0.5, 0.8}	{16, 32, 64, 128, 256}	{2.3,4}	-	-	-	-
SGC	{1e-2, 5e-2, 5e-3, 3e-5}	{0, 5e-4, 5e-5}	{0, 0.2, 0.5, 0.8}	{16, 32, 64, 128, 256}	{2,3}	-	-	-	{1,2,3}
GAT	{1e-2, 5e-2, 5e-3, 3e-5}	{0, 3e-3, 3e-2}	{0, 0.2, 0.5, 0.6, 0.8}	{8, 32, 64, 128, 256}	{2,3}	{1,4,8}	-	-	-
GLNN	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-3}	{0, 0.2, 0.5, 0.8}	{16, 22, 64, 128, 256}	{1,2}	-	{0.0-1.0}	-	-
APPNP	{1e-2, 5e-2, 5e-3}	{0, 5e-3, 1e-2}	{0, 0.2, 0.5, 0.8}	{8, 16, 32, 64, 128}	2	-	-	{0.0-1.0}	{5,6,7,8,9,10}
MLP	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-3, 5e-5, 5e-8}	{0, 0.2, 0.5, 0.8}	{16, 32, 64, 128, 256}	2	-	-	-	-
GPRGNN	{1e-2, 5e-2, 5e-3}	{0, 1e-3, 1e-5, 1e-4, 5e-5}	{0, 0.2, 0.5, 0.8}	{16, 32, 64, 128}	{2,3}	-	-	{0.0-1.0}	{5,8,9,10}
GCNII	{1e-2, 5e-3, 1e-3}	{0, 5e-5, 5e-4}	{0., 0.1, 0.2, 0.4, 0.5, 0.8}	{16, 32, 64, 128, 256}	{2,4,8,16,32,64}	-	{0.0-1.0}	-	-
Table 9: Hyperparameter searching range for baseline models on OGB-Arxiv, Twitch-gamer, IGB-tiny
Hyperparameters for Baseline GNNs
Models\Hyperparameters	lr	weight Decay	dropout	hidden	# layers	Gat heads	λ	α	K
GCN	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{128, 256, 512}	{2, 3}	-	-	-	-
SGC	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{64, 128, 256, 512}	{2, 3}	-	-	-	{1,2,3}
GAT	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.6, 0.8}	{64, 128, 256, 512}	{2, 3}	{1,4,8}	-	-	-
GLNN	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{1024, 2048}	{2, 3}	-	{0.0-1.0}	-	-
APPNP	{1e-2, 5e-2, 5e-3}	{0, 5e-3, 1e-2, 5e-4, 5e-5}	{0, 0.2, 0.3, 0.5, 0.8}	{128, 256, 512}	{2, 3}	-	-	{0.0-1.0}	{5,6,7,8,9,10}
MLP	{1e-2, 5e-2, 5e-3}	{0, 5e-4, 5e-5}	{0, 0.5, 0.8}	{256, 512}	{2, 3}	-	-	-	-
GPRGNN	{1e-2, 5e-2, 5e-3}	{0, 1e-3, 1e-5, 1e-4, 5e-5}	{0, 0.2, 0.5, 0.8}	{128, 256, 512}	{2, 3}	-	-	{0.0-1.0}	{5,8,9,10}
GCNII	{1e-2, 5e-3, 1e-3}	{0, 5e-5, 5e-4}	{0., 0.1, 0.2, 0.4, 0.5, 0.8}	{64, 128, 256, 512}	{2, 4, 8, 16, 32, 64}	-	{0.0-1.0}	{0.0-1.0}	-
Table 10: The accuracy of GNN and MLP models on homophilic graphs
Dataset	Cora	Citeseer	Pubmed	Arxiv	IGB-tiny
MLP	61.1±1.2	60.0±1.4	69.0±2.3	54.0±0.1	73.2±0.1
GLNN	81.3±1.5	73.0±2.7	78.2±2.6	71.7±0.1	73.2±0.1
GCN	81.5±1.4	73.7±1.6	77.9±2.0	71.4±0.1	70.7±0.1
SGC	81.7±1.4	72.7±2.2	77.0±2.7	68.0±0.1	71.0±0.1
GAT	82.2±1.1	73.6±1.6	77.3±1.5	71.0±0.1	70.8±0.2
APPNP	83.1±1.3	75.0±1.1	79.6±1.3	70.3±0.5	71.2±0.1
GCNII	82.8±1.1	73.8±1.7	79.0±2.5	71.7±0.5	73.5±0.1
GPRGNN	82.9±1.4	72.4±1.8	78.3±2.1	72.3±0.3	73.9±0.1
Table 11: The accuracy of GNN and MLP models on heterophilic graphs
Dataset	Chameleon	Squirrel	Twitch-gamers	Actor	Amazon-ratings
MLP	49.0±2.4	30.1±1.7	60.7±0.2	37.0±0.7	45.9±0.8
GLNN	39.2±2.7	52.3±1.4	61.1±0.1	37.3±1.0	54.0±0.7
GCN	68.0±2.0	54.7±1.4	62.2±0.2	30.7±0.9	49.0±0.6
SGC	69.1±1.8	53.0±1.1	62.0±2.0	30.0±1.5	46.5±0.6
GAT	67.0±1.9	53.2±1.7	59.9±0.3	30.7±1.0	48.0±0.5
APPNP	56.7±2.5	42.4±1.9	59.7±0.1	37.0±1.3	44.9±0.8
GCNII	64.7±1.8	44.0±1.5	64.5±0.3	36.0±1.2	50.0±0.5
GPRGNN	68.5±1.4	53.8±1.4	61.9±0.2	36.5±1.4	49.8±0.5
experimental results including Cora, CiteSeer, IGB-tiny, Actor, Amazon-rating, and Twitch-gamers. Dataset statistics details can be found in Tab. 13. The license of datasets can be found in Table 12.
Table 12: The license of datasets.
Dataset	license
Cora	NLM license
CiteSeer	NLM license
PubMed	NLM license
Ogbn-arxiv	ODC-BY
IGB-tiny	ODC-By-1.0
Chameleon	MIT license
Squirrel	MIT license
Actor	MIT license
Amazon-ratings	MIT license
Twitch-gamers	MIT license
We then provide a detailed description on those datasets and the corresponding data split as follows:
- Planetoid datasets include Cora, CiteSeer, and PubMed. They consist of scientific publications as nodes. The citation links between the papers create a graph structure, with nodes representing papers and edges representing citations. We utilize the 10 fixed splits in [101]. For each split, we have 20 nodes per class for training, 30 nodes per class for validation, and all other nodes for testing. More details can be found on https://github.com/BUPT-GAMMA/CPF/tree/master/data/npz
- Ogbn-Arxiv dataset is based on a large-scale scholarly paper citation network, where nodes represent papers and edges represent citation links between them. We utilize the default fix split provided by [24]. More details can be found on https://ogb.stanford.edu/docs/nodeprop/#ogbn-arxiv.
- Wikipedia datasets include Chameleon and Squirrel, which are a collection of web pages from Wikipedia. The node features correspond to several informative nouns in the respective Wikipedia pages. The edge represents the link between node. We utilize the 10 fixed splits in [25]. For each split, we have  $48\%$  nodes for training,  $32\%$  nodes for validation, and  $20\%$  for test. More details can be found on https://github.com/graphhdl-uiuc-jlu/geom-gcn
- The Actor dataset [102] is a graph that represents the co-occurrence relationships between actors. We utilize the 10 fixed splits in [25]. For each split, we have  $48\%$  nodes for training,  $32\%$  nodes for validation, and  $20\%$  for test. More details can be found on https://github.com/graphhdl-uiuc-jlu/geom-gcn
- The Twitch-gamers dataset represents the relationships between accounts on the streaming platform Twitch, where each node corresponds to a Twitch account, and edges are present between accounts that are mutual followers. We utilize the 5 fixed splits in [21]. For each split, we have  $50\%$  nodes for training,  $25\%$  nodes for validation, and  $25\%$  for test. More details can be found on https://github.com/CUAI/Non-Homophily-Large-Scale
- Amazon-ratings a new dataset derived from the Amazon product co-purchasing network metadata dataset, which is part of the SNAP Datasets [103]. We utilize the 10 fixed splits in [104]. For each split, we have  $50\%$  nodes for training,  $25\%$  nodes for validation, and  $25\%$  for test. More details can be found on https://github.com/yandex-research/heterophilous-graphs/tree/main/data
- IGB-tiny is a recent new citation graph dataset. It utilizes two publicly available datasets: Microsoft Academic Graph (MAG) and SemanticScholar Corpus. We utilize the homogeneous setting with default fix split provide by [105]. More details can be found on https://github.com/IllinoisGraphBenchmark/IGB-Datasets.
Details on Data distribution The node homophily ratio distribution for different datasets are presented in Figure 13. The following observations can be made: (1) Cora, CiteSeer, PubMed, and Ogbn-Arxiv are four homophilic datasets where most nodes have a homophily ratio  $h > 0.5$ . (2) Chamaleon, Squirrel, and Actor are three heterophilic datasets where most nodes have a homophily ratio  $h \leq 0.5$ . Nonetheless, the Actor dataset exhibits entirely different properties from Chamaleon and Squirrel where graph structure information is barely useful, leading to performance degradation. Empirical evidence can be found in Table 11 where the vanilla MLP outperforms all the GNN models. (3) IGB-tiny [105] and Twitch-gamers [21] are two representative datasets with no apparent majority structural pattern, exhibiting graph homophily ratios of 0.567 and 0.556, respectively. Specifically, IGB-tiny contains both large numbers of homophilic and heterophilic nodes, respectively, whereas, most nodes in the Twitch-gamers dataset do not show a clear homophilic nor heterophilic pattern with a homophily ratio close to 0.5. Notably, the IGB-tiny is still recognized as a homophilic graph [105] while Twitch-gamers is generally recognized as a homophilic graph [20, 21]. (4) Nodes in Amazon-ratings [104] exhibit diverse structural patterns with multiple peaks observed for homophily ratios 0.0, 0.2, 0.4, 0.6, and 0.8.
G.4 OOD dataset statistics and details
We provide more analysis and statistics details about datasets with our proposed out-distribution (OOD) data split for a more comprehensive analysis. The data split statistics can be found in Table 14.  $20\%$  of the majority nodes are selected as the validation set. Notably, we generally use nodes with  $h_i < 0.5$  and  $h_i > 0.5$  as the majority and minority on heterophilic graph.
Table 13: Detailed dataset statistics
Dataset	Cora	CiteSeer	PubMed	Arxiv	IGB-tiny	Squirrel	Chameleon	Amazon-ratings	Actor	Twitch-gamers
Homo Ratio	0.814	0.714	0.792	0.631	0.567	0.216	0.247	0.376	0.221	0.556
Nodes	2485	2110	19717	169343	100000	5201	2277	24492	7600	249992
Edges	5069	3668	44324	583121	223538	198493	31421	93050	15009	93050
Features	1433	3703	500	128	1024	2089	2235	300	932	300
Classes	7	6	3	40	19	5	5	5	5	5
(d) Ogbn-Arxiv  $(h = 0.63)$
(e) IGB-tiny  $(h = 0.57)$
(f) Chameleon  $(h = 0.25)$
(g) Squirrel  $(h = 0.22)$
(h) Actor  $(h = 0.22)$
(i) Twitch-gamers  $(h = 0.56)$
Figure 13: Node homophily ratio distributions of different datasets. Both homophilic and heterophilic nodes exist across all graphs
(j) Amazon-ratings  $(h = 0.38)$
Table 14: the numbers of train, validation, test nodes on OOD data split
Dataset	Cora	CiteSeer	PubMed	Arxiv	Squirrel	Chameleon
#train	1599	1160	12466	85788	3709	1642
#valid	400	290	3117	21447	928	441
#test	486	660	4134	62108	564	564
Nonetheless, the Squirrel dataset only has 71 nodes with  $h_i > 0.5$ . To ensure enough test nodes, we adapt the nodes with  $h_i > 0.4$  as the minority nodes for test. We conduct experiments to measure the distributional shift between train and test nodes. To further analysis on the distribution shift, we employ Maximum Mean Discrepancy (MMD) [106] as a discrepancy metric. A large MMD distance indicates the covariate shift with  $P^{\mathrm{train}}(X) \neq P^{\mathrm{test}}(X)$ . MMD measures the distance between distributions as follows:
$$
\operatorname {M M D} ^ {2} (X, Y) = \left\| \frac {1}{n} \sum_ {i = 1} ^ {n} \phi \left(x _ {i}\right) - \frac {1}{m} \sum_ {j = 1} ^ {m} \phi \left(y _ {j}\right) \right\| ^ {2} \tag {48}
$$
where  $\phi$  is a kernel function that maps the input samples into a higher-dimensional feature space, and  $||\cdot ||_2$  denotes the  $L_{2}$  norm. Using the Gaussian kernel, we can define the kernel matrices as:
$$
K _ {x x, i j} = k \left(x _ {i}, x _ {j}\right), \quad K _ {y y, i j} = k \left(y _ {i}, y _ {j}\right), \tag {49}
$$
$$
K _ {x y, i j} = k \left(x _ {i}, y _ {j}\right), K _ {y x, i j} = k \left(y _ {i}, x _ {j}\right),
$$
where the kernel function is given by:
$$
k (x, y) = \exp \left(- \frac {\| x - y \| ^ {2}}{2 \sigma^ {2}}\right) \tag {50}
$$
and  $\sigma$  is a parameter that controls the width of the kernel. In our experiment, we choose multiple  $\sigma$  including [0.01, 0.1, 1, 10, 100]. We average the MMD distance with different  $\sigma$  for a more robust distance measurement. The MMD distance can be computed using these kernel matrices:
$$
\mathrm {M M D} ^ {2} (X, Y) = \frac {1}{n (n - 1)} \sum_ {i \neq j} K _ {x x, i j} - \frac {2}{n m} \sum_ {i, j} K _ {x y, i j} + \frac {1}{m (m - 1)} \sum_ {i \neq j} K _ {y y, i j}, \tag {51}
$$
where the first and third terms represent the within-set variations, and the second term represents the between-set variations. We then conduct experiments to determine where the source of distribution discrepancy from the covariate shift or concept shift. We measure the MMD distance between the train set and validation set as the i.i.d. distribution distance, and the MMD distance between the training set and test set as the OOD distribution distance. However, the in-distribution and out-distribution distances may not be directly comparable due to potential estimation errors introduced by different numbers of nodes in valid and test sets.
To enable a fair comparison, we also measure in-distribution and out-distribution distances on the i.i.d. data split as a control group. The MMD distances are measured on the last hidden representation of a well-trained GCN. The experiment results are provided in Table 15. Focusing on the i.i.d setting, we observe an MMD distance difference on the validation set and test set to the training set, induced by the difference in node counts between the validation and test set. Comparing the OOD setting with the i.i.d one no significant MMD differences are found, except for PubMed. This suggests that the primary factor driving the distribution shift is not the covariate shift with  $\mathbf{P}^{\mathrm{train}}(\mathbf{X})\neq \mathbf{P}^{\mathrm{test}}(\mathbf{X})$  but the concept shift with  $\mathbf{P}^{\mathrm{train}}(\mathbf{Y}|\mathbf{X})\neq \mathbf{P}^{\mathrm{test}}(\mathbf{Y}|\mathbf{X})$ . The above observations further indicate the existence of the concept shift in our proposed OOD scenario.
Additional OOD results: We show additional experimental results on Cora and CiteSeer datasets in Table 16 with a consistent observation. Notice that, all experimental results are average with 10 random seeds including (1, 3, 5, 7, 11, 13, 17, 19, 23, 29).
Table 15: MMD distance between train and validation, test sets on both i.i.d. and ood settings.
Dataset	Cora	CiteSeer	PubMed	Arxiv	Chameleon	Squirrel
IID valid	0.565	0.345	0.082	0.149	0.951	1.04
IID test	0.610	0.600	0.050	0.276	0.882	0.92
OOD valid	0.564	0.233	0.127	0.211	0.977	1.192
OOD test	0.597	0.598	0.442	0.420	0.854	0.92
Table 16: Additional performance with OOD data split on Cora and CiteSeer.
Dataset	Cora	CiteSeer
GCN (i.i.d)	87.53±0.35	77.30±1.26
GCN	51.05±0.58	43.74±0.88
MLP	51.09±0.74	45.26±0.62
GLNN	54.14±0.89	47.21±0.33
GCNII	55.33±0.41	46.47±0.59
GPRGNN	55.16±0.51	45.76±0.50
SRGNN	62.79±0.87	49.62±0.96
EERM	54.34±0.78	47.72±0.50
EERM(II)	70.91±1.89	51.12 ± 0.82
(c) IGB-tiny  $(h = 0.58)$
(d) Actor  $(h = 0.22)$
(e) twitch-gamers  $(h = 0.56)$
(f) Amazon-ratings  $(h = 0.38)$
Figure 14: Performance comparison between GCN and MLP-based models on more datasets. Each bar represents the accuracy gap on a specific test node subgroup exhibiting a local homophily ratio within the range specified on the x-axis.
H Additional results on performance comparison between GCN and MLP-based models
In Section 3, we conduct a performance comparison between GCN with two MLP-based models, i.e., MLP and GLNN, on test nodes with different structural patterns. In this section, we provide more elaboration on why using GLNN and conduct additional experiments on more datasets.
We first provide more elaborations on why using GLNN. The experiment is to examine the effectiveness of GCN on utilizing different structural patterns. Therefore, we compare GCN and MLP
architectures as GCN utilizes graph structure during inference while MLP cannot, serving as a structure-agnostic baseline. When GCN surpasses MLP,it indicates GNN benefits from structural patterns effectively, and vice visa. Notably, GLNN can be viewed as a better-trained MLP model. GLNN also utilizes the same MLP model architecture as vanilla MLP, the only difference between vanilla MLP and GLNN is that GLNN is trained in an advanced distillation manner while vanilla MLP is trained with cross-entropy loss. The reason why we utilize GLNN rather than only comparing GCN with vanilla MLP is that MLP meets optimization issue on training. Such an obstacle leads to a large performance gap (more than  $20\%$ ) between under-trained vanilla MLP and well-trained GCN. Consequently, a large performance gap induced by training difficulty hinders the potential for MLP architecture. Contrastly, GLNN enjoys a better training process leads to a more clear comparison between well-trained GNN and well-trained MLP(GLNN) architecture with a convincing conclusion
Additional experiments are illustrated in Fig 14. Cora and CiteSeer are two Planetoid datasets, showing similar properties as PubMed. Observations on Cora and CiteSeer are consistent with those in Section 3, further substantiating our conclusions. It should be noted that our paper primarily focuses on homophilic graphs and heterophilic graphs with a clear major structural pattern. Nonetheless, there exists more complicated real-world graphs with more diverse structural properties, as demonstrated in Figure 13. We conduct further investigations on these datasets, including IGB-tiny, Twitch-gamers, Amazon-ratings, and Actor. A comprehensive discussion on those datasets can be found in Appendix G.3. Experimental results are depicted in Figure 14. We make initial observations as follows: (1) IGB-tiny is a homophilic dataset exhibiting consistent observations in line with the main analysis in Section 3, where the MLP-based models outperform GCN on the heterophilic nodes and underperform on homophilic nodes. (2) Twitch-gamers and Amazon-ratings are two heterophilic datasets exhibiting consistent observations in line with the main analysis in Section 3, where the MLP-based models outperform GCN on the homophilic nodes while falling short on the heterophily nodes. (3) Actor is a heterophilic dataset where graph information is of limited utility. Empirical evidence shows that MLP-based models show an overall better performance than GCN. From a local perspective, the MLP-based models outperform GCN on the heterophilic nodes while underperforming on the homophily nodes. Those observations indicate our conclusion can be extended on most datasets with diverse structural patterns Nonetheless, it still remains a mystery on the Actor dataset. We leave a more comprehensive investigation especially on the Actor dataset as the future work.
I Additional results on performance comparison between GCN and deeper GNN models
In Section 4.1, we conduct a performance comparison between GCN with deeper GNN models, on test nodes with different structural patterns. In this section, we conduct additional experiments on more datasets, correspondingly. Cora and CiteSeer are two Planetoid datasets, showing similar properties as PubMed. Notably, deeper GNNs generally show better performance across different structural patterns, where the improvement still primarily lies in the minority groups, Observations on Cora and CiteSeer are consistent with those in Section 3, further substantiating our conclusions.
It should be noted that our paper primarily focuses on homophilic graphs and heterophilic graphs with a clear major structural pattern. Nonetheless, there exist more complicated real-world graphs with more diverse structural properties, as demonstrated in Figure 13. We conduct further investigations on these datasets, including IGB-tiny, Twitch-gamers, Amazon-ratings, and Actor. A comprehensive discussion on those datasets can be found in Appendix G.3. New experimental results are depicted in Figure 15. We observe that deeper GNNs generally outperform GCN on the heterophilic node while underperforming on the homophilic nodes on those datasets with diverse structural properties. We can conclude that the effectiveness of GNN on those datasets with no clear majority structural patterns primarily from the improvement from the heterophilic nodes. We leave a more comprehensive investigation as the future work.
J Additional Experiment on performance disparity across subgroups
In Section 3.4, we conduct experiments to empirically examine the effects of two-hop aggregated-feature distance and the homophily ratio difference on the test performance of different GNN models. Despite the empirical success on the second hop, we conduct additional experiment results focusing
(c) IGB-tiny  $(h = 0.57)$
(f) Amazon-ratings  $(h = 0.38)$
Figure 15: Performance comparison between GCN and deeper GNN models on more datasets. Each bar represents the accuracy gap on a specific test node subgroup exhibiting a local homophily ratio within the range specified on the x-axis.
on observations on higher-order homophily ratios, higher-order aggregated feature distance, and more datasets in this section. Notably, despite more complicated datasets, our theoretical analysis can still achieve empirical success.
(a) PubMed
(b) Ogbn-arxiv
(c) Chameleon
Figure 16: Test accuracy disparity across node subgroups by aggregated-feature distance and higher-order homophily ratio differences to training nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing differences to training set.
(d) Squirrel
(a) PubMed
Figure 17: Test accuracy disparity across node subgroups by higher-order aggregated-feature distance and homophily ratio differences to training nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing difference to training set.
(c) Chameleon
(b) Ogbn-arxiv
Figure 18: Test accuracy disparity across node subgroups by higher-order aggregated feature distance and higher-order homophily ratio differences to training nodes. Each figure corresponds to a dataset, and each bar cluster corresponds to a GNN model. Bars labeled 1 to 5 represent subgroups with increasing difference to training set.
J.1 Additional investigation on higher-order neighborhood
Our theory 1 in Section 3.3 focuses on the one-hop aggregation while it could be easily extended to the higher-order aggregation case in the real-world scenario. More experiments revolving on higher-order homophily ratio and aggregated features are shown in this subsection. In particular, the  $k$ -hop node homophily ratio can be calculated as:
$$
h _ {i} ^ {(k)} = \frac {\left| \left\{u \in \mathcal {N} _ {k} \left(v _ {i}\right) : y _ {u} = y _ {v} \right\} \right|}{\left| \mathcal {N} _ {k} \left(v _ {i}\right) \right|} \tag {52}
$$
where  $\mathcal{N}_k(v_i)$  denotes the  $k$ -hop neighbor node set of  $v_i$ . We then sort test nodes in terms of the disparity score and split them into 5 equal-size. Figures 16 and 17 illustrate results on higher-order (third hop) homophily ratio difference and higher-order aggregated feature distance (third hop), respectively while Figure 18 illustrates on the result on higher-order homophily ratio difference and higher-order aggregated feature distance.
A similar phenomenon can be found with the expected trend of test accuracy in terms of increasing differences on the aggregated feature distance and homophily ratio differences in different scenarios. Exceptional observation can be found in the following scenarios: (1) Squirrel dataset with higher-order aggregated feature. It may be because such higher-order feature aggregation leads to more distribution overlapping between different categories. (2) PubMed with both higher-order aggregated features and homophily ratio. The potential reason is that the PubMed dataset does not rely on higher-order information. Overall speaking, those results further indicate that for the generalization of our theory in the real-world scenario.
J.2 Additional results on more datasets