SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathrm {B E} = \mathbb {E} _ {\boldsymbol {x}} [ \mathbb {P} (z | \mathrm {C L} _ {\text {B a y e s}} (\boldsymbol {x}) \neq z) ] = \mathbb {E} _ {\boldsymbol {x}} [ 1 - \mathbb {P} (\mathrm {C L} _ {\text {B a y e s}} (\boldsymbol {x}) = z | \boldsymbol {x}) ]
$$
Specifically, the BE for CSBM-H can be written as
$$
\mathbf {B E} = \mathbb {P} (\boldsymbol {x} \in \mathcal {C} _ {0}) (1 - \mathbb {P} (\mathbf {C L} _ {\text {B a y e s}} (\boldsymbol {x}) = 0 | \boldsymbol {x} \in \mathcal {C} _ {0})) + \mathbb {P} (\boldsymbol {x} \in \mathcal {C} _ {1}) (1 - \mathbb {P} (\mathbf {C L} _ {\text {B a y e s}} (\boldsymbol {x}) = 1 | \boldsymbol {x} \in \mathcal {C} _ {1})). \tag {4}
$$
To estimate the above value, we compute Probabilistic Bayes Error (PBE) for CSBM-H as follows.
Probabilistic Bayes Error (PBE) The random variable in each dimension of  $\pmb{x}$  is independently normally distributed. As a result,  $Q(\pmb{x})$  defined in Theorem 1 follows a generalized  $\chi^2$  distribution [9][10](See the calculation in Appendix C). Specifically,
$$
\text {F o r} \boldsymbol {x} _ {i} \in \mathcal {C} _ {0}, Q (\boldsymbol {x} _ {i}) \sim \tilde {\chi} ^ {2} \left(w _ {0}, F _ {h}, \lambda_ {0}\right) + \xi ; \boldsymbol {x} _ {j} \in \mathcal {C} _ {1}, Q (\boldsymbol {x} _ {j}) \sim \tilde {\chi} ^ {2} \left(w _ {1}, F _ {h}, \lambda_ {1}\right) + \xi
$$
where  $w_0 = a\sigma_0^2, w_1 = a\sigma_1^2$ , the degree of freedom is  $F_h$ ,  $\lambda_0 = \left(\frac{\mu_0}{\sigma_0} + \frac{b}{2a\sigma_0}\right)^\top \left(\frac{\mu_0}{\sigma_0} + \frac{b}{2a\sigma_0}\right)$ ,  $\lambda_1 = \left(\frac{\mu_1}{\sigma_1} + \frac{b}{2a\sigma_1}\right)^\top \left(\frac{\mu_1}{\sigma_1} + \frac{b}{2a\sigma_1}\right)$  and  $\xi = c - \frac{b^\top b}{4a}$ . Then, by using the Cumulative Distribution Function (CDF) of  $\tilde{\chi}^2$ , we can calculate the predicted probabilities directly as,
$$
\mathbb {P} \left(\operatorname {C L} _ {\text {B a y e s}} (\boldsymbol {x}) = 0 \mid \boldsymbol {x} \in \mathcal {C} _ {0}\right) = 1 - \operatorname {C D F} _ {\tilde {\chi} ^ {2} \left(w _ {0}, F _ {h}, \lambda_ {0}\right)} (- \xi), \mathbb {P} \left(\operatorname {C L} _ {\text {B a y e s}} (\boldsymbol {x}) = 1 \mid \boldsymbol {x} \in \mathcal {C} _ {1}\right) = \operatorname {C D F} _ {\tilde {\chi} ^ {2} \left(w _ {1}, F _ {h}, \lambda_ {1}\right)} (- \xi).
$$
Suppose we have a balanced prior distribution  $\mathbb{P}(\pmb{x}\in \mathcal{C}_0) = \mathbb{P}(\pmb{x}\in \mathcal{C}_1) = 1 / 2$ . Then, PBE is,
$$
\frac {\mathrm {C D F} _ {\tilde {\chi} ^ {2} \left(w _ {0} , F _ {h} , \lambda_ {0}\right)} (- \xi) + \left(1 - \mathrm {C D F} _ {\tilde {\chi} ^ {2} \left(w _ {1} , F _ {h} , \lambda_ {1}\right)} (- \xi)\right)}{2}
$$
To investigate the impact of homophily on the ND for LP and HP filtered embeddings, we just need to replace  $\left(\pmb{\mu}_0,\sigma_0^2,\pmb{\mu}_1,\sigma_1^2\right)$  by  $\left(\tilde{\pmb{\mu}}_0,\tilde{\sigma}_0^2,\tilde{\pmb{\mu}}_1,\tilde{\sigma}_1^2\right)$  and  $\left(\tilde{\pmb{\mu}}_0^{\mathrm{HP}},(\tilde{\sigma}_0^{\mathrm{HP}})^2,\tilde{\pmb{\mu}}_1^{\mathrm{HP}},(\tilde{\sigma}_1^{\mathrm{HP}})^2\right)$  as equation 3
PBE can be numerically calculated and visualized to show the relationship between  $h$  and ND precisely. However, we do not have an analytic expression for PBE, which makes it less explainable and intuitive. To address this issue, we define another metric for ND in the following paragraphs.
Generalized Jeffreys Divergence The KL-divergence is a statistical measure of how a probability distribution  $P$  is different from another distribution  $Q$  [8]. It offers us a tool to define an explainable ND measure, generalized Jeffreys divergence.
Definition 2 (Generalized Jeffreys Divergence). For a random variable  $x$  which has either the distribution  $P(x)$  or the distribution  $Q(x)$ , the generalized Jeffreys divergence is defined as
$$
\underline {{D _ {G J} (P , Q)}} = \mathbb {P} (\boldsymbol {x} \sim P) \mathbb {E} _ {\boldsymbol {x} \sim P} \left[ \ln \frac {P (\boldsymbol {x})}{Q (\boldsymbol {x})} \right] + \mathbb {P} (\boldsymbol {x} \sim Q) \mathbb {E} _ {\boldsymbol {x} \sim Q} \left[ \ln \frac {Q (\boldsymbol {x})}{P (\boldsymbol {x})} \right]
$$
3The Bayes classifier for multiple categories  $(>2)$  can be computed by stacking multiple expectation terms using similar methods as in [12, 15]. We do not discuss the more complicated settings in this paper.
4Jeffreys divergence [25] is originally defined as  $D_{\mathrm{KL}}(P||Q) + D_{\mathrm{KL}}(Q||P)$
With  $\mathbb{P}(\pmb{x} \sim P) = \mathbb{P}(\pmb{x} \sim Q) = 1/2$ , the negative generalized Jeffreys divergence for the two-normal setting in CSBM-H can be computed by (See Appendix B for the calculation)
$$
D _ {\mathrm {N G J}} (\mathrm {C S B M - H}) = \underbrace {- d _ {X} ^ {2} \left(\frac {1}{4 \sigma_ {1} ^ {2}} + \frac {1}{4 \sigma_ {0} ^ {2}}\right)} _ {\text {N e g a t i v e N o r m a l i z e d D i s t a n c e}} \underbrace {- \frac {F _ {h}}{4} \left(\rho^ {2} + \frac {1}{\rho^ {2}} - 2\right)} _ {\text {N e g a t i v e V a r i a n c e R a t i o}} \tag {5}
$$
where  $d_X^2 = (\pmb{\mu}_0 - \pmb{\mu}_1)^\top (\pmb{\mu}_0 - \pmb{\mu}_1)$  is the squared Euclidean distance between centers;  $\rho = \frac{\sigma_0}{\sigma_1}$  and since we assume  $\sigma_0^2 < \sigma_1^2$ , we have  $0 < \rho < 1$ . For  $\pmb{h}$  and  $\pmb{h}^{\mathrm{HP}}$ , we have  $d_H^2 = (2h - 1)^2 d_X^2$ ,  $d_{\mathrm{HP}}^2 = 4(1 - h)^2 d_X^2$ . The smaller  $D_{\mathrm{NGJ}}$  the CSBM-H has, the more distinguishable the embeddings are.
From equation 5, we can see that  $D_{\mathrm{NGJ}}$  implies that ND relies on two terms, Expected Negative Normalized Distance (ENND) and the Negative Variance Ratio (NVR): 1. ENND depends on how large is the inter-class ND  $d_X^2$  compared with the normalization term  $\frac{1}{4\sigma_1^2} + \frac{1}{4\sigma_0^2}$ , which is determined by intra-class ND (variances  $\sigma_0, \sigma_1$ ); NVR depends on how different the two intra-class NDs are, i.e., when the intra-class ND of high-variation class is significantly larger than that of low-variation class ( $\rho$  is close to 0), NVR is small which means the nodes are more distinguishable and vice versa.
Now, we can investigate the impact of homophily on ND through the lens of PBE and  $D_{\mathrm{NGJ}}$ . Specifically, we set the standard CSBM-H as  $\pmb{\mu}_0 = [-1,0], \pmb{\mu}_1 = [0,1], \sigma_0^2 = 1, \sigma_1^2 = 2, d_0 = 5, d_1 = 5$ . And as shown in Figure 2 its PBE and  $D_{\mathrm{NGJ}}$  curves for LP filtered feature  $h$  are bell-shaped. This indicates that, contrary to the prevalent belief that heterophily has the most negative impact on ND, a medium level of homophily actually has a more detrimental effect on ND than extremely low levels of homophily. We refer to this phenomenon as the mid-homophily pitfall.
The PBE and  $D_{\mathrm{NGJ}}$  curves for  $\pmb{h}^{\mathrm{HP}}$  are monotonically increasing, which means that the high-pass filter works better in heterophily areas than in homophily areas. Moreover, it is observed that  $\pmb{x}$ ,  $\pmb{h}$ , and  $\pmb{h}^{\mathrm{HP}}$  will get the lowest PBE and  $D_{\mathrm{NGJ}}$  in different homophily intervals, which we refer to as the "FP regime (black)", "LP regime (green)", and "HP regime (red)" respectively. This indicates that LP filter works better at very low and very high homophily intervals (two ends), HP filter works better at low to medium homophily interval [8] the original (i.e., full-pass or FP filtered) features works better at medium to high homophily area.
Researchers have always been interested in exploring how node degree relates to the effect of homophily [38, 54]. In the upcoming subsection, besides node degree, we will also take a deeper look at the impact of class variances via the homophily-ND curves and the FP, LP and HP regimes.
(a) PBE
(b)  $D_{\mathrm{NGJ}}$
Figure 2: Visualization of CSBM-H  $(\pmb{\mu}_0 = [-1,0],\pmb{\mu}_1 = [0,1],\sigma_0^2 = 1,\sigma_1^2 = 2,d_0 = 5,d_1 = 5)$
(c) ENND
(d) NVR
3.4 Ablation Study on CSBM-H
Increase the Variance of High-variation Class  $(\sigma_0^2 = 1, \sigma_1^2 = 5)$  From Figure 3 it is observed that as the variance in  $\mathcal{C}_1$  increases and the variance between  $\mathcal{C}_0$  and  $\mathcal{C}_1$  becomes more imbalanced, the PBE and  $D_{\mathrm{NGJ}}$  of the three curves all go up which means the node embeddings become less distinguishable under HP, LP and FP filters. The significant shrinkage of the HP regimes and the expansion of the FP regime indicates that the original features are more robust to imbalanced variances
<sup>5</sup>We provide an open-ended discussion of imbalanced prior distributions in Appendix D.
See two more metrics, negative squared Wasserstein distance and Hellinger distance, in Appendix E.3
This is consistent with the empirical results found in [35] that the relationship between the prediction accuracy of GNN and homophily value is a U-shaped curve.
This verifies the conjecture made in [35] saying that high-pass filter cannot address all kinds of heterophily and only works well for certain heterophily cases.
(b)  $D_{\mathrm{NGJ}}$
(c) ENND
Figure 3: Comparison of CSBM-H with  $\sigma_0^2 = 1$ ,  $\sigma_1^2 = 5$ .
(c) ENND
(d) Negative Variance Ratio
Figure 4: Comparison of CSBM-H with  $\sigma_0^2 = 1.9$ ,  $\sigma_1^2 = 2$ .
especially in the low homophily area. From Figure 3(d), we can see that the main cause is that the NVR of the 3 curves all move down but the HP curve moves less in low homophily area than other 2 curves. This implies that the HP curve exhibits less sensitivity to  $\rho$  within the area of low homophily.
Increase the Variance of Low-variation Class  $(\sigma_0^2 = 1.9, \sigma_1^2 = 2)$  As shown in Figure4, when the variance in  $C_0$  increases and the variance between  $C_0$  and  $C_1$  becomes more balanced, PBE and  $D_{\mathrm{NGJ}}$  curves go up, which means the node embeddings become less distinguishable. The LP, HP and the FP regimes almost stays the same because the magnitude of NVR becomes too small that it almost has no effect to ND as shown in Figure4(d).
Interestingly, we found the change of variances cause less differences of the 3 regimes in ENND than that in NVR and HP filter is less sensitive to  $\rho$  changes in low homophily area than LP and FP filters. This insensitivity will have significant impact to the 3 regimes when  $\rho$  is close to 0 and have trivial effect when  $\rho$  is close to 1 because the magnitude of NVR is too small.
Increase the Node Degree of High-variation Class  $(d_0 = 5, d_1 = 25)$  From Figure 5 it can be observed that as the node degree of the high-variation class increases, the PBE and  $D_{\mathrm{NGJ}}$  curves of FP and HP filters almost stay the same while the curves of LP filters go down with a large margin. This leads to a substantial expansion of LP regime and shrinkage of FP and HP regime. This is mainly due to the decrease of ENND of LP filters and the decrease of its NVR in low homophily area also plays an important role.
Increase the Node Degree of Low-variation Class  $(d_0 = 25, d_1 = 5)$  From Figure 6 we have the similar observation as when we increase the node degree of high-variation class. The difference is that the expansion of LP regime and shrinkage of FP and HP regimes are not as significant as before.
From  $\tilde{\sigma}_0^2$ ,  $\tilde{\sigma}_1^2$  we can see that increasing node degree can help LP filter reduce variances of the aggregated features so that the ENND will decrease, especially for high-variation class while HP filter is less sensitive to the change of variances and node degree.
3.5 More General Theoretical Analysis
Besides the toy example, in this subsection, we aim to gain a deeper understanding of how LP and HP filters affect ND in a broader context beyond the two-normal settings. To be consistent with previous literature, we follow the assumptions outlined in [38], which are: 1. The features of node  $i$  are sampled from distribution  $\mathcal{F}_{z_i}$ , i.e.,  $\pmb{x}_i\sim \mathcal{F}_{z_i}$ , with mean  $\mu_{z_i}\in \mathbb{R}^{F_h}$ ; 2. Dimensions of  $\pmb{x}_i$  are independent to each other; 3. Each dimension in feature  $\pmb{x}_i$  is bounded, i.e.,  $a\leq x_{i,k}\leq b$ ; 4. For node  $i$ , the labels of its neighbors are independently sampled from neighborhood distribution
To verify this, we increase  $\sigma_0^2$  and  $\sigma_{1}^{2}$  proportionally. From Figure 10 in Appendix F relative sizes of the FP, LP, and HP areas remain similar.
Figure 5: Comparison of CSBM with different  $d_0 = 5$ ,  $d_1 = 25$  setups.
(a) PBE
Figure 6: Comparison of CSBM with different  $d_0 = 25$ ,  $d_1 = 5$  setups.
$\mathcal{D}_{z_i}$  and repeated for  $d_{i}$  times. We refer to a graph that follows the above assumptions as  $\mathcal{G} = \{\mathcal{V},\mathcal{E},\{\mathcal{F}_c,c\in \mathcal{C}\} ,\{\mathcal{D}_c,c\in \mathcal{C}\} \} ,\mathcal{C} = \{1,\ldots ,C\}$  and  $(b - a)^2$  reflects how variation the features are. The authors in [38] analyze the distance between the aggregated node embedding and its expectation, i.e.,  $\| h_i - \mathbb{E}(h_i)\| _2$  , which only considers the intra-class ND and has been shown to be inadequate for a comprehensive understanding of ND. Instead, we investigate how significant the intra-class embedding distance is smaller than the inter-class embedding distance in the following theorem, which is a better way to understand ND.
Theorem 2. Suppose a graph  $\mathcal{G} = \{\mathcal{V},\mathcal{E},\{\mathcal{F}_c,c\in \mathcal{C}\} ,\{\mathcal{D}_c,c\in \mathcal{C}\} \}$  meets all the above assumptions (1-4). For nodes  $i,j,v\in \mathcal{V}$ , suppose  $z_{i}\neq z_{j}$  and  $z_{i} = z_{v}$ , then for constants  $t_x,t_h,t_\mathrm{HP}$  that satisfy  $t_x\geq \sqrt{F_h} D_x(i,j)$ ,  $t_h\geq \sqrt{F_h} D_h(i,j)$ ,  $t_\mathrm{HP}\geq \sqrt{F_h} D_\mathrm{HP}(i,j)$  we have
$$
\mathbb {P} \left(\| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {j} \| _ {2} \geq \| \boldsymbol {x} _ {i} - \boldsymbol {x} _ {v} \| _ {2} + t _ {x}\right) \leq 2 F _ {h} \exp \left(- \frac {\left(D _ {x} (v , j) - \frac {t _ {x}}{\sqrt {F _ {h}}}\right) ^ {2}}{V _ {x} (v , j)}\right),
$$
$$
\mathbb {P} \left(\left\| \boldsymbol {h} _ {i} - \boldsymbol {h} _ {j} \right\| _ {2} \geq \left\| \boldsymbol {h} _ {i} - \boldsymbol {h} _ {v} \right\| _ {2} + t _ {h}\right) \leq 2 F _ {h} \exp \left(- \frac {\left(D _ {h} (v , j) - \frac {t _ {h}}{\sqrt {F _ {h}}}\right) ^ {2}}{V _ {h} (v , j)}\right), \tag {6}
$$
$$
\mathbb {P} \left(\left\| \boldsymbol {h} _ {i} ^ {\mathrm {H P}} - \boldsymbol {h} _ {j} ^ {\mathrm {H P}} \right\| _ {2} \geq \left\| \boldsymbol {h} _ {i} ^ {\mathrm {H P}} - \boldsymbol {h} _ {v} ^ {\mathrm {H P}} \right\| _ {2} + t _ {\mathrm {H P}}\right) \leq 2 F _ {h} \exp \left(- \frac {\left(D _ {\mathrm {H P}} (v , j) - \frac {t _ {\mathrm {H P}}}{\sqrt {F _ {h}}}\right) ^ {2}}{V _ {\mathrm {H P}} (v , j)}\right),
$$
where  $D_{x}(v,j) = \left\| \pmb{\mu}_{z_{v}} - \pmb{\mu}_{z_{j}}\right\|_{2}, V_{x}(v,j) = (b - a)^{2}, D_{h}(v,j) = \left\| \tilde{\pmb{\mu}}_{z_{v}} - \tilde{\pmb{\mu}}_{z_{j}}\right\|_{2}, V_{h}(v,j) = \left(\frac{1}{2d_{v}} +\frac{1}{2d_{j}}\right)(b - a)^{2},$
$$
D _ {\mathrm {H P}} (v, j) = \left\| \boldsymbol {\mu} _ {z _ {v}} - \tilde {\boldsymbol {\mu}} _ {z _ {v}} - \left(\boldsymbol {\mu} _ {z _ {j}} - \tilde {\boldsymbol {\mu}} _ {z _ {j}}\right) \right\| _ {2}, V _ {\mathrm {H P}} (v, j) = \left(1 + \frac {1}{2 d _ {v}} + \frac {1}{2 d _ {j}}\right) (b - a) ^ {2}, \tilde {\boldsymbol {\mu}} _ {z _ {v}} = \sum_ {u \in \mathcal {N} (v)} \mathbb {E} _ {\boldsymbol {x} _ {u} \sim \mathcal {F} _ {z _ {u}}} \left[ \frac {1}{d _ {v}} \boldsymbol {x} _ {u} \right].
$$
See the proof in Appendix G
We can see that, the probability upper bound mainly depends on a distance term (inter-class ND) and normalized variance term (intra-class ND). The normalized variance term of HP filter is less sensitive to the changes of node degree than that of LP filter because there is an additional 1 in the constant term. Moreover, we show that the distance term of HP filter actually depends on the relative center distance, which is a novel discovery. As shown in Figure[7] when homophily decreases, the aggregated centers will move away from the original centers, and the relative center distance (purple) will get larger which means the embedding distance of nodes from different classes will have larger probability to be big. This explains how HP filter work for some heterophily cases. Overall, in a more general setting with weaker assumptions, we can see that ND is also described by the intra- and inter-class ND terms together rather than intra-class ND only, which is consistent with CSBM-H.
4 Empirical Study of Node Distinguishability
Cornell	Wisconsin	Texas	Film	Chameleon	Squirrel	Cora	CiteSeer	PubMed
Baseline
Homophily
Metrics	Hedge	0.5669	0.4480	0.4106	0.3750	0.2795	0.2416	0.8100	0.7362	0.8024
Hnode	0.3855	0.1498	0.0968	0.2210	0.2470	0.2156	0.8252	0.7175	0.7924
Hclass	0.0468	0.0941	0.0013	0.0110	0.0620	0.0254	0.7657	0.6270	0.6641
Hagg	0.8032	0.7768	0.694	0.6822	0.61	0.3566	0.9904	0.9826	0.9432
HGE	0.31	0.34	0.35	0.16	0.0152	0.0157	0.17	0.19	0.27
Hadj	0.1889	0.0826	0.0258	0.1272	0.0663	0.0196	0.8178	0.7588	0.7431
L1	0.0169	0.1311	0.1923	0.0002	0.048	0.0015	0.5904	0.4508	0.4093
Classifier-based
Performance Metrics	KRNGP	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
GNB	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	1.00
SGC v.s. MLP-1	p-value	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	0.00
ACC SGC	70.98 ± 8.39	70.38 ± 2.85	83.28 ± 5.43	25.26 ± 1.18	64.86 ± 1.81	47.62 ± 1.27	85.12 ± 1.64	79.66 ± 0.75	85.5 ± 0.76
ACC MLP-1	93.77 ± 3.34	93.87 ± 3.33	93.77 ± 3.34	34.53 ± 1.48	45.01 ± 1.58	29.17 ± 1.46	74.3 ± 1.27	75.51 ± 1.35	86.23 ± 0.54
Diff Acc	-22.79	-23.49	-10.49	-9.27	19.85	18.45	10.82	4.15	-0.73
GCN v.s. MLP-2	p-value	0.00	0.00	0.00	0.00	1.00	1.00	1.00	1.00	0.00
ACC GCN	82.46 ± 3.11	75.5 ± 2.92	83.11 ± 3.2	35.51 ± 0.99	64.18 ± 2.62	44.76 ± 1.39	87.78 ± 0.96	81.39 ± 1.23	88.9 ± 0.32
ACC MLP-2	91.30 ± 0.70	93.87 ± 3.33	92.26 ± 0.71	38.58 ± 0.25	46.72 ± 0.46	31.28 ± 0.27	76.44 ± 0.30	76.25 ± 0.28	86.43 ± 0.13
Diff Acc	-8.84	-18.37	-9.15	-3.07	17.46	13.48	11.34	5.14	2.47
Table 1: P-values, homophily values and classifier-based performance metrics on 9 real-world benchmark datasets. Cells marked by grey are incorrect results for both SGC v.s. MLP-1 and GCN v.s. MLP-2 and cells marked by blue are incorrect for 1 of the 2 tests. We use 0.5 as the threshold value of the homophily metrics.
Besides theoretical analysis, in this section, we will conduct experiments to verify whether the effect of homophily on the performance of GNNs really relates to its effect on ND. If a strong relation can be verified, then it indicates that we can design new training-free ND-based performance metrics beyond homophily metrics, to evaluate the superiority and inferiority of G-aware models against its coupled G-agnostic models.
Figure 7: Demonstration of how HP filter captures the relative center distance.
4.1 Hypothesis Testing on Real-world Datasets
To test whether "intra-class embedding distance is smaller than the inter-class embedding distance" strongly relates to the superiority of G-aware models to their coupled G-agnostic models in practice, we conduct the following hypothesis testing [10].
Experimental Setup We first train two G-aware models GCN, SGC-1 and their coupled G-agnostic models MLP-2 and MLP-1 with fine-tuned hyperparameters provided by [35]. For each trained model, we calculate the pairwise Euclidean distance of the node embeddings in output layers. Next, we compute the proportion of nodes whose intra-class node distance is significantly smaller than inter-class node distance [1] e.g., we obtain Prop(GCN) for GCN. We use Prop to quantify ND and we train the models multiple times for samples to conduct the following hypothesis tests:
$\mathrm{H}_0:\mathrm{Prop}(\mathrm{G - }$  aware model)  $\geq$  Prop(G-agnostic model);  $\mathrm{H}_{1}:\mathrm{Prop}(\mathrm{G - }$  aware model)  $<  \mathrm{Prop}(\mathrm{G - }$ agnostic model)
Specifically, we compare GCN vs. MLP-2 and SGC-1 vs. MLP-1 on 9 widely used benchmark datasets with different homophily values for 100 times. In each time, we randomly split the data into training/validation/test sets with a ratio of  $60\% / 20\% / 20\%$ . For the 100 samples, we conduct  $T$ -test for the means of two independent samples of scores, and obtain the corresponding p-values. The test results and model performance comparisons are shown in Table  $\boxed{I}$  (See more experimental tests on state-of-the-art model in Appendix  $\boxed{H}$ ).
Authors in [33] also conduct hypothesis testing to find out when to use GNNs for node classification, but they test the differences between connected nodes and unconnected nodes instead of intra- and inter-class nodes.
<sup>11</sup>A node is considered as "significantly smaller" when the p-value for its intra-class node distance being smaller than inter-class node distance is smaller than 0.05. In other words, this node is considered as significantly distinguishable. This second statistical test is necessary to avoid noisy nodes. In practice, we noticed that the ratio of intra-class node distance to inter-class node distance is roughly 1 for lots of nodes. This is particularly evident when the labels are sparse and when we use sampling method. It will not only cause instability of the outputs, but also result in false results sometimes. Thus, we don't want to take account these "marginal nodes" into the comparison of Prop values and we found that using another hypothesis test would be helpful.
It is observed that, in most cases (except for GCN vs. MLP-2 on PubMed[12]), when  $\mathrm{H}_{1}$  significantly holds, G-aware models will underperform the coupled G-agnostic models and vice versa. This supports our claim that the performance of G-aware models is closely related to "intra-class vs. inter-class node embedding distances", no matter the homophily levels. It reminds us that the p-value can be a better performance metric for GNNs beyond homophily. Moreover, the p-value can provide a statistical threshold, such as  $p \leq 0.05$ . This property is not present in existing homophily metrics.
However, it is required to train and fine-tune the models to obtain the p-values, which make it less practical because of computational costs. To overcome this issue, in the next subsection, we propose a classifier-based performance metric that can provide p-values without training.
4.2 Towards A Better Metric Beyond Homophily: Classifier-based Performance Metric
A qualified classifier should not require iterative training. In this paper, we choose Gaussian Naïve Bayes (GNB)[20] and Kernel Regression (KR) with Neural Network Gaussian Process (NNGP)[30, 2, 16, 41] to capture the feature-based linear or non-linear information.
To get the p-value efficiently, we first randomly sample 500 labeled nodes from  $\mathcal{V}$  and splits them into  $60\% /40\%$  as "training" and "test" data. The original features  $X$  and aggregated features  $H$  of the sampled training and test nodes can be calculated and are then fed into a given classifier. The predicted results and prediction accuracy of the test nodes will be computed directly with feedforward method. We repeat this process for 100 times to get 100 samples of prediction accuracy for  $X$  and  $H$ . Then, for the given classifier, we compute the p-value of the following hypothesis testing,
$$
\mathrm {H} _ {0}: \operatorname {A c c} (\operatorname {C l a s s i e r} (H)) \geq \operatorname {A c c} (\operatorname {C l a s s i e r} (X)); \mathrm {H} _ {1}: \operatorname {A c c} (\operatorname {C l a s s i e r} (H)) <   \operatorname {A c c} (\operatorname {C l a s s i e r} (X))
$$