Title: What Is Missing For Graph Homophily? Disentangling Graph Homophily For Graph Neural Networks
Venue: NeurIPS 2024
Accepted: yes
Keywords: Graph Neural Networks

Graph Homophily
Abstract: Graph homophily refers to the phenomenon that connected nodes tend to share similar characteristics. Understanding this concept and its related metrics is crucial for designing effective Graph Neural Networks (GNNs). The most widely used homophily metrics, such as edge or node homophily, quantify such "similarity" as label consistency across the graph topology. These metrics are believed to be able to reflect the performance of GNNs, especially on node-level tasks. However, many recent studies have empirically demonstrated that the performance of GNNs does not always align with homophily metrics, and how homophily influences GNNs still remains unclear and controversial. Then, a crucial question arises: What is missing in our current understanding of homophily? To figure out the missing part, in this paper, we disentangle the graph homophily into three aspects: label, structural, and feature homophily, which are derived from the three basic elements of graph data. We argue that the synergy of the three homophily can provide a more comprehensive understanding of GNN performance. Our new proposed structural and feature homophily consider the neighborhood consistency and feature dependencies among nodes, addressing the previously overlooked structural and feature aspects in graph homophily. To investigate their synergy, we propose a Contextual Stochastic Block Model with three types of Homophily (CSBM-3H), where the topology and feature generation are controlled by the three metrics. Based on the theoretical analysis of CSBM-3H, we derive a new composite metric, named Tri-Hom, that considers all three aspects and overcomes the limitations of conventional homophily metrics. The theoretical conclusions and the effectiveness of Tri-Hom have been verified through synthetic experiments on CSBM-3H. In addition, we conduct experiments on $31$ real-world benchmark datasets and calculate the correlations between homophily metrics and model performance. Tri-Hom has significantly higher correlation values than $17$ existing metrics that only focus on a single homophily aspect, demonstrating its superiority and the importance of homophily synergy. Our code is available at https://github.com/zylMozart/Disentangle_GraphHom.

What Is Missing In Homophily? Disentangling Graph Homophily For Graph Neural Networks

Abstract

Graph homophily refers to the phenomenon that connected nodes tend to share similar characteristics. Understanding this concept and its related metrics is crucial for designing effective Graph Neural Networks (GNNs). The most widely used homophily metrics, such as edge or node homophily, quantify such "similarity" as label consistency across the graph topology. These metrics are believed to be able to reflect the performance of GNNs, especially on node-level tasks. However, many recent studies have empirically demonstrated that the performance of GNNs does not always align with homophily metrics, and how homophily influences GNNs still remains unclear and controversial. Then, a crucial question arises: What is missing in our current understanding of homophily? To figure out the missing part, in this paper, we disentangle the graph homophily into three aspects: label, structural, and feature homophily, which are derived from the three basic elements of graph data. Weargue that the synergy of the three homophily can provide a more comprehensive understanding of GNN performance. Our new proposed structural and feature homophily consider the neighborhood consistency and feature dependencies among nodes, addressing the previously overlooked structural and feature aspects in graph homophily. To investigate their synergy, we propose a Contextual Stochastic Block Model with three types of Homophily (CSBM-3H), where the topology and feature generation are controlled by the three metrics. Based on the theoretical analysis of CSBM-3H, we derive a new composite metric, named Tri-Hom, that considers all three aspects and overcomes the limitations of conventional homophily metrics. The theoretical conclusions and the effectiveness of Tri-Hom have been verified through synthetic experiments on CSBM-3H. In addition, we conduct experiments on 31 real-world benchmark datasets and calculate the correlations between homophily metrics and model performance. Tri-Hom has significantly higher correlation values than 17 existing metrics that only focus on a single homophily aspect, demonstrating its superiority and the importance of homophily synergy. Our code is available at https://github.com/zylMozart/Disentangle_GraphHom .

1 Introduction

Graph Neural Networks (GNNs) have been widely used in processing non-Euclidean data due to their superiority in extracting topological relations [10, 13, 26, 41, 35]. They have achieved great success on numerous real-world applications, e.g., recommendation [49, 58], bio-informatics [21, 20] and telecommunication [34]. It is found that their success, especially on node-level tasks, is closely related to the homophily assumption [47, 72, 38, 43, 36, 71], i.e., similar nodes tend to be connected [17]. On the other hand, when dissimilar nodes are more likely to be connected, which is known as the non-homophily/heterophily scenario, GNNs fail to capture the useful neighbor information and even underperform Multilayer perceptrons (MLPs) [37]. Several homophily metrics, such as edge homophily [72, 1] and node homophily [50] were proposed, which were believed to be able to recognize the difficult datasets [43] and measure the performance of GNNs [39].

However, recent studies [46, 39, 51, 40] show that the conventional homophily metrics [72, 1, 50] are insufficient to measure the performance of GNNs: Luan et al. [38] show that the homophily metrics cannot tell if GNNs work well under heterophily. Ma et al. [46] reveals that homophily is not a necessary assumption for effective GNNs and they propose to identify "good" and "bad" heterophily to explain why GNNs still work well under heterophily. Luan et al. [39] discovers a mid-homophily pitfall, showing the performance of GNNs reaches the worst in a medium level of homophily instead of the lowest. Then, a crucial question arises based on the above studies: What is missing in our current understanding of homophily?

In this paper, we fill the missing parts by investigating different perspectives of the "node similarity". Conventional homophily metrics quantify the "similarity" as an indicator function of whether connected nodes share the same label while ignoring the co-existence of three basic elements in graph data: label, structural, and node feature information. The ignorance of structural and feature information leads to insufficient understanding and unsatisfactory alignment between homophily metrics and GNN performance.

A complete understanding of graph homophily should include all the above three basic elements. To this end, we disentangle graph homophily into three corresponding aspects: label, structural, and feature homophily. Specifically, our new proposed structural homophily quantifies the "similarity" by considering the neighborhood structure consistency, and feature homophily measures the dependencies of node features across the topology. To investigate how their synergy affects GNN performance, we propose a Contextual Stochastic Block Model controlled with three types of Homophily (CSBM-3H). The node feature generation process in CSBM-3H breaks the i.i.d. assumption in previous studies [46, 39], which is closer to real-world scenarios [14, 59, 61]. With the three metrics, CSBM-3H enables a more comprehensive study on the impact of graph homophily than previous analysis [46, 39, 29, 60].

From the theoretical study of CSBM-3H, we derive a new composite metric named Tri-Hom to measure the synergy, which includes all three homophily aspects. Through CSBM-3H, our theoretical analysis and simulation results both show that the performance of GNNs is highly influenced by Tri-Hom. It can help explain how the three types of homophily influence GNN behavior individually or collectively. In addition, our theoretical findings can explain some interesting phenomena observed in previous literature, such as "good" or "bad" heterophily [46, 38] and the impact of feature shuffling on GNNs [29]. To verify the effectiveness of Tri-Hom, we conduct experiments on 31 real-world datasets. The results show that GNN performance is significantly better aligned with Tri-Hom than the other 17 existing metrics that focus only on a single homophily aspect. This implies that Tri-Hom can complete the absent parts in existing homophily metrics.

2 Preliminary

We denote G = ( V , E ) as an undirected graph, where V is the node set and E is the edge set. The graph has N nodes with C classes. The adjacency matrix of the graph is denoted as A ∈ R N × N . We use A uv = 1 or e uv ∈ E to denote the existence of an edge between node u and v , otherwise A uv = 0 or e uv / ∈ E . Node degree vector is denoted as D ∈ R N where D u is the degree of node u . Node label vector is denoted as Y ∈ R N and its one-hot encoding matrix is Z ∈ R N × C . The number of nodes in class c is denoted as N c = |{ u | Y u = c, u ∈ V}| . The neighbor set of node u is denoted as N u = { v | e uv ∈ E} . The features of all the nodes is denoted as X ∈ R N × M , where X v, : are the features of node v with M dimensions. We use I E ∈ R E × E and 1 E ∈ R E × E to denote identify matrix and all-ones matrix with size E , respectively.

Graph homophily metrics are used to measure the similarity between connected nodes. Edge [1, 72] and node homophily [50] are 2 most commonly used metrics and are defined as follows,

These metrics qualify the ratio of whether the labels of two connected nodes are the same in a graph. However, this definition of graph homophily only considers a label aspect and neglects structural and feature aspects, resulting in a partial understanding of graph homophily. Therefore, we propose to disentangle the graph homophily as label, structural, and feature homophily in the next section.

Graph-aware models M G and graph-agnostic models M ¬G refer to the models that either utilize structure information or do not, respectively. For example, baseline graph-aware models

M G , such as Graph Convolutional Network (GCN) [26], Graph Attention Network (GAT) [57] and GraphSage [16], encode both graph structure and node feature information in each layer; the corresponding graph-agnostic models M ¬G are the Multilayer Perceptrons (MLPs), which only encode node features [39].

Structural-agnostic features refer to the node features X that are conditionally independent of graph topology A given Y , i.e., ( X ⊥ ⊥ A | Y ) ; structural-aware features indicate ( X ̸ ⊥ ⊥ A | Y ) .

3 Disentangled Graph Homophily

In this section, we first introduce the definition of disentangled graph homophily from label, structural, and feature aspects to complete the missing part of the graph homophily. Then, in the next section, we will introduce how they collectively impact the performance of GNNs.

3.1 Label Homophily

Definition 1. Label homophily is defined as the consistency of node labels across the topology.

Label homophily is the most widely used conventional metric of graph homophily and it qualifies the similarity between connected nodes u and v using an indicator function ✶ ( Y u = Y v ) . Most of the conventional homophily metrics focus on label homophily, including edge homophily [1, 72], node homophily [50], class homophily [33], adjusted homophily [51], densityaware homophily [30], 2-hop neighbor class homophily [5], and neighbor homophily [15].

However, label homophily only focuses on the consistency of label information for connected nodes while neglecting structural and feature information, which are two indispensable components of graph data. Hence, it offers only a partial understanding of graph homophily, which cannot always align well with the performance of GNNs [46, 38, 39]. To capture the missing structural and feature information and better understand graph homophily, we give the definitions of structural and feature homophily in the following 2 subsections.

3.2 Structural homophily

For structural homophily, the "atom" information of a node is structural information instead of the label. It is meaningless to define the structural homophily using the consistency across the graph topology as in the label homophily because the structural information already contains the information from the graph topology. Therefore, we define the structural information as the consistency of structural information among the

(a)

h

L

= 0

.

1

(b)

h

L

= 0

.

5

(c)

h

L

= 0

.

9

(d)

h

S

= 0

.

1

(e)

h

S

= 0

.

5

(f)

h

S

= 0

.

9

(g)

h

F

=

-

0

.

7

(h)

h

F

= 0

.

7

(i)

h

F

= 0

.

9

nodes from the same classes 1 , which better disentangles itself from the label homophily.

Definition 2. Structural homophily is defined as the consistency of structural information of nodes within the same class. The structural homophily in a graph is defined as:

where h S,c is the class-wise structural homophily for class c , function S ( · ) measures structural information, σ denotes standard deviation of structural information, and σ max denotes the maximum value of σ .

In this paper, we quantify the structural information for node u through neighbor distribution (the class distribution of local neighbors) D N u = [ p u, 1 , p u, 2 , . . . , p u,c ] , where p u,k = |{ Y v = k | v ∈N u }| |N u | is the proportion of neighbors of node u that belong to class c . Ahigh structural homophily indicates that the graph-aware models leveraging structural encoding will have similar embeddings for intra-class nodes after aggregation, which are expected to outperform graph-agnostic models, irrespective of a low label homophily. There are also some homophily metrics that focus on the structural aspect in previous studies, including label informativeness [51], neighborhood similarity [46], and aggregation homophily [38], which is similar as the structural homophily defined there.

3.3 Feature Homophily

Previous feature-based graph homophily metrics, such as generalized edge homophily [23], local similarity [7], attribute homophily [67], and class-controlled feature homophily [29], mainly focus on the consistency of node features across the graph topology, which is similar as the definition of Dirichlet energy in graphs. However, these homophily metrics on feature consistency cannot fully disentangle itself from label homophily: Since the features of nodes in a graph are supposed to depend on their classes, when the graph shows a high/low label homophily, the connected nodes are more likely to share the same/different labels, resulting in a high/low feature similarity. Therefore, these feature-based homophily metrics are dependent on label homophily. Such dependency contains redundancy, which decreases the useful information inside feature-based homophily and impedes our understanding of the relationship between node features and GNN performance.

To disentangle the feature effect from label and graph structure, we define the feature homophily as the dependencies of node features across the graph topology, thereby dissociating it from label homophily and structural homophily. Inspired by graph diffusion [6] and interactive particle systems [55, 64], we have the structural-agnostic unobserved feature X (0) and the observed structural-aware feature X that satisfy the following relation

The detailed process of this relation is given in Appendix B. Here ω ∈ ( -1 ρ ( A ) , 1 ρ ( A ) ) is a parameter that controls the feature dependencies, where a positive, negative, or zero value corresponds to an attractive relation, repulsive relation, or independence of the nodes with their neighbors in graphs [55, 64]. The feature dependencies ( ω A ) t of t -order neighbors are introduced to structuralagnostic features X (0) . Finally, the state of all the nodes will converge to an equilibrium with structure-aware feature X . The ω in Eq. (3) is independent of the graph topology because no matter how the label homophily or structural homophily changes, ω will remain unaffected. To disentangle feature homophily from label homophily and structural homophily, we define the feature homophily based on ω as follows.

Definition 3. Feature homophily is defined as the degree of feature dependencies of nodes across the topology. For the linear case of the graph diffusion process with feature dependencies, the feature homophily for feature m satisfies

where ρ ( A ) is the spectral radius of A , X (0) ∼ p ( X | Y ) are the unseen structural-agnostic node features, and X ∼ p ( X | Y , A ) is the observed structural-aware node features. The feature homophily for the whole graph is the averaged feature homophily for all the features

Note that it is easy to control the feature homophily in the generation of synthetic graphs. However, since both h F,m and X : ,m (0) are unknown in Eq. (4), one more condition is required to estimate feature homophily in real-world datasets. To address this issue, we consider the case where the intra-class distances of X : ,m (0) are small. This case holds in lots of real-world scenarios [38, 39]

and we can utilize this property to estimate h F,m without solving X : ,m (0) . Specifically, the feature homophily h F,m for feature m can be estimated with the following optimization process

The estimation of feature homophily is invariant to the operations of feature shifts, scaling, or changing variance, where the proof is shown in Appendix C. This estimation process will be used in Section 5.2 for calculation.

Remark To better understand the definitions of three types of graph homophily, Figure 1 visualizes examples of graphs under varying h L , h S , and h F : 1) Label homophily. As label homophily h L increases, as shown in Figures 1(a), (b), and (c), nodes are more likely to connect with others that share the same label. Particularly, a high h L (Figure 1(c)) results in several clusters with distinct class boundaries, while a low h L causes nodes to more likely connect to nodes with different classes. 2) Structural homophily. As structural homophily h S increases, as shown in Figures 1(d), (e), and (f), the neighbor distributions of intra-class nodes become more consistent. Therefore, a high h S is expected to capture effective structural information with message aggregation. Interestingly, we also find that a higher h S makes a graph resemble planar graphs [3] and periodic graphs [9]. We hypothesize this phenomenon occurs because stable structural information leads to more regular and meaningful patterns, which would be interesting to explore the connection between h S and these geometric properties of graphs in the future. 3) Feature homophily. Figures 1(g), (h), and (i) illustrate different levels of feature homophily ( h F ) within the same graph topology. Figure 1 (h) demonstrates that under a medium positive h F , features of some boundary nodes exhibit characteristics of both neighboring classes. For instance, a node on the boundary of the red class and the blue class appears purple, a mixture of these classes. A higher h F (Figure 1 (i)) increases feature dependencies, particularly affecting more nodes closer to class boundaries. In social networks, a positive h F indicates that people's opinions are influenced by their friends, resulting in similar characteristics. Conversely, a negative h F causes nodes to become more dissimilar from their neighbors. As shown in Figure 1 (g), a negative h F creates a distinct boundary between classes. Additionally, for the intra-class nodes in Figure 1 (g), the node colors differ in shades from their neighbors. This occurs because node features become more dissimilar due to the "repulsive force" rather than the "attractive force" induced by a negative h F . For example, in online social media, people are likely to argue with those holding different opinions on certain topics. After such interactions, individuals may reinforce their original opinions, a phenomenon resulting from the "repulsive force" associated with a negative h F .

4 Impact of Disentangled Graph Homophily

To study the model performance in a graph, the Contextual Stochastic Block Model (CSBM) has been widely used to study the performance of GNNs with controlled graph topology and node features. Previous studies [51, 29, 39, 46] on graph homophily generally adopt a modified CSBM to control the label homophily through assigning nodes with different probabilities that connect to the nodes from other classes. Then the node features are sampled solely based on the classes. However, this graph modeling, which only considers label homophily, has two drawbacks: First, the probabilities of nodes from the same class connecting to the nodes with different classes are uniform, which lacks diversity. Second, the sampled node features are independent with their structures i.e., ( X ⊥ ⊥ A | Y ) , which is uncommon in real-world scenarios where interactions influence the attributes of connected nodes [61, 14, 59]. Therefore, we propose a Contextual Stochastic Block Model with three types of Homophily (CSBM-3H), a random graph generative model that integrates the three types of homophily (Section 4.1), where the newly proposed structural homophily h S and feature homophily h F can well address the aforementioned two drawbacks and fills the missing part of graph homophily. Then, Based on CSBM-3H, we theoretically study how the graph-agnostic and graph-aware models are affected by label, structural, and feature homophily metrics to explore their relationship and verify the effectiveness of proposed metrics (Section 4.2).

4.1 CSBM-3H

Graph Topology Generation Following the topology generation process in existing studies [39, 60], we assume all the nodes are class-balanced and share the same node degree d . We use node homophily h L to control the label consistency across the graph topology and h S to control the consistency of neighbor distribution D N of nodes within the same classes. Then, the neighbor distribution can be expressed as:

where D N ∈ R N × C is the neighbor distribution for all the nodes, S ∈ R C × C is a class-sampling matrix, and ϵ ∈ R C × C is a noise matrix. Each entry of ϵ is a noise of neighbor sampling that follows a Gaussian distribution N (0 , (1 -h S ) 2 c -1 ) . The class-sampling matrix S should be legal in practice [60] i.e., S u,v > 0 and ∑ v S u,v = 1 . Then an adjacency matrix A can be sampled from a neighborhood sampling matrix A p = Cd N D N Z T , where E [ A uv = 1] = ( A p ) uv for each pair of nodes u, v . In this way, we control the label homophily h L and structural homophily h S in a graph.

Node Feature Generation . For any node u in a graph, we first sample its structural-agnostic features X u (0) ∈ R F from a class-wised Gaussian distribution X u (0) ∼ N Y u ( µ Y u , Σ Y u ) with µ Y u ∈ R F and Σ Y u ∈ R F × F . We also assume each dimension of the feature vector is independent of each other, thereby Σ Y u ∈ R F × F is a diagonal matrix. Then the observed structural-aware features can be generated by the unseen structure-agnostic feature as described in Eq. (4).

4.2 Node distinguishability

Suppose we have the representations of node u as H u : = 1 d ∑ v ∈N u X v : for graph-aware models M G and H u : = X u : for graph-agnostic models M ¬G . Inspired by the principles of neural collapse [28, 27] and node distinguishability [39], we quantify the impact of the aforementioned homophily metrics on both the graph-aware models M G and graph-agnostic models M ¬G by measuring the ratio of intra-class node distance to inter-class distance. To ideally distinguish nodes from different classes, a smaller intra-class distance D intra ( H ) and larger inter-class distance D intra ( H ) is preferred because this will reduce boundary nodes and increase the margins among classes. The metric is defined as follows,

̸

A smaller J indicates better node embeddings for the model performance and vice versa, which has been proved in [39]. With the proposed CSBM-3H, we can analyze the impacts of h L , h S , and h F on M ¬G and M G by studying their relations with J , which will be derived in the following theorems.

Theorem 1. In CSBM-3H, the ratio of the expectation of intra-class distance to the expectation of inter-class distance of node representations for graph-agnostic models M ¬G and graph-aware models G is:

̸

From Theorem 1 we can see that, J N is a normalized distance term which is a constant given the distribution of structural-agnostic features and is irrelevant with graph information; J ¬G h and J G h are controlled by the three types of homophily, which can reflect the influence of graph homophily on model performance. We name J ¬G h and J G h as Tri-Hom for M ¬G and M G . To study the effect of Tri-Hom in more detail, we take the partial derivative of J G h with respect to h S , h F , and h L to show the analytical results of their influences 2 . (See calculation in Appendix G.5, G.6, and G.7.)

Theorem 2.1. The partial derivative of J G h with respect to label homophily h L satisfies,

From Theorem 2.1 we can see that, the worst performance of M G is reached when h L = 1 C , which corresponds to the scenario with the highest number of unpredictable boundary nodes among classes. This finding explains the misalignment between label homophily and GNN performance mentioned in previous studies [46, 38, 39].

2 We also calculate the partial derivative of J ¬G h and discuss the impact of three types of homophily on M ¬G in Appendix D.

Theorem 2.2. The partial derivative of J G h with respect to structural homophily h S satisfies,

From Theorem 2.2 we can see that, a larger h S consistently improves the performance of M G . This is intuitive for M G because more consistent intra-class neighbor distributions will lead to closer intra-class node representations after feature aggregation. This conclusion is also shown in Wang et al. [60], where the topological noise (which is inversely proportional to h S ) has a detrimental impact on node separability.

Theorem 2.3. The partial derivative of J G h with respect to feature homophily h F satisfies,

where 0 < h -L < h + L < 1 and -1 < ˆ h F < 1 . The expressions and detailed calculation of h -L , h + L , and ˆ h F are shown in Appendix G.7.

From Theorem 2.3 we can see that, when h L is high in a graph, nodes with the same labels tend to be connected, thereby a larger h F makes the intra-class nodes share more similar representations and positively affect M G ; when h L is low in a graph, nodes with the different labels tend to be connected, thereby a larger h F makes the inter-class nodes share more similar representations and negatively affect M G ; when h L is on a medium level i.e., h L ∈ ( h -L , h + L ) , an increase of h F will first improve and then reduces the performance of M G with the cut-off point at h F = ˆ h F . There the ˆ h F is influenced by h L , h S , C , and ρ ( A ) , and the h -L and h + L are influenced by h S , C , and ρ ( A ) .

Remark Apart from the new findings mentioned above, Tri-Hom can help explain other interesting but under-explored phenomena of the graph homophily in previous studies, e.g., 1) "good" or "bad" heterophily [46], which states that GNN can still perform well in some heterophily cases; and 2) feature shuffling [29], which states that shuffling the node features randomly within the same class can improves the performance of GNNs on node classification. Our explanations with Tri-Hom are: 1) The occurrence of "good" or "bad" heterophily is due to the fact that the model performance is influenced by a combination of h L , h F , and h S , instead of h L alone. When the h L is low, the graph-aware models can still achieve good performance with a high h S or a low h F ; 2) feature shuffling is due to the existence of the structural-aware features of nodes. When h F > 0 , nodes are positively dependent on their neighbors. In this case, the nodes at the class boundaries or class centers are the hardest or easiest ones to predict because of the feature dependencies. If we randomly shuffle the nodes inside their classes, the nodes at the class boundaries will be easier to be classified because their features are more likely to be replaced by the nodes from the center, which are more distinguishable. For the nodes close to the class centers, it will be compensated by their neighbors.

5 Experimental Results

In Section 5.1, we conduct experiments on synthetic data generated by CSBM-3H to verify the conclusions from Theorem 2.1, 2.2, and 2.3, demonstrate the synergy of label h L , structural h S , and feature homophily h F and test whether Tri-Hom J G h can reflect GNN performance. In Section 5.2, we evaluate the effectiveness of Tri-Hom on real-world benchmark datasets to test how well it can predict the model performance in real-world scenarios. In addition, we calculate the correlation between Tri-Hom value and model performance and compare them with the results of other 17 existing metrics. The results show that Tri-Hom has a significantly higher correlation with the model performance, demonstrating the necessity of filling the missing part by disentangling graph homophily from three aspects.

5.1 Experiments on Synthetic Datasets

To verify our theoretical results in a more general case, we measure the performance of GCN on synthetic datasets, where we can easily control h L , h S , and h F . Specifically,

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

G

h

0

.

0

0

.

5

1

.

0

M

G

, h

S

=

0

.

2

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

G

h

0

.

0

0

.

5

1

.

0

M

G

, h

S

=

0

.

4

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

G

h

0

.

0

0

.

5

1

.

0

M

G

, h

S

=

0

.

6

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

G

h

0

.

0

0

.

5

1

.

0

M

G

, h

S

=

0

.

8

(a) Numerical results of Tri-Hom

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

GCN

,

h

S

=

0

.

2

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

GCN

,

h

S

=

0

.

4

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

GCN

,

h

S

=

0

.

6

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

GCN

,

h

S

=

0

.

8

Synthetic Data Generation We generate the synthetic graphs using CSBM-3H with a given tuple ( h L , h S , h F ) , where h L ∈ { 0 , 0 . 1 , . . . , 0 . 9 , 1 } , h S ∈ { 0 , 0 . 1 , . . . , 0 . 9 , 1 } and h F ∈ {-0 . 8 , 0 . 6 , . . . , 0 . 6 , 0 . 8 } . For each ( h L , h S , h F ) , we generate 1000 nodes with three balanced classes and the node degrees are sampled from a uniform distribution [1 , 10] . For node features, we first sample the structural-agnostic features from class-specific Gaussian distributions, then we use Eq. (4) to propagate these features across the graph topology with feature homophily h F to generate the observed structural-aware features as the final node features. Then, we evaluate the node classification performance of GCN [26] on these synthetic graphs. To get a robust evaluation and mitigate the numerical instability, we generate 10 graphs with 10 random seeds for each ( h L , h S , h F ) and report the average and standard deviation of classification accuracy on validation sets. The detailed process of topology and feature generation is shown in Appendix E.1.1.

Numerical Results of Tri-Hom To verify whether Tri-Hom J G h can reflect the behavior of GCN, we calculate its numerical results with the same setting as the synthetic graphs and make a comparison. Specifically, we set C = 3 and ρ ( A ) = 10 to mitigate the influences of varying numbers of classes and spectral radius.

Comparison and Analysis The results are shown in Figure 2. For a better demonstration, we only show the results for h S = { 0 . 2 , 0 . 4 , 0 . 6 , 0 . 8 } and each subfigure is a slice of h S that visualizes the impact of h L and h F on GCN and Tri-Hom. From the comparison of Figure 2a and 2b, we have two main observations: 1) Overall, the impact of h L , h S , and h F on synthetic datasets aligns well with the numerical results of Tri-Hom J G h . The only difference is that h F seems to have less impact on GCN. We speculate that this is because the parameters in the graph filter in GCN are optimized during the training process, while these parameters are simplified as a fixed value in the theoretical analysis, leading to the difference 3 . 2) Our theoretical analysis of the impact of h L , h F and h S on Tri-Hom is consistent with GCN's behavior in Figure 2b: Theorem 2.1 shows the worst performance of M G is reached when h L = 1 C , corresponding to the ravine in Figure 2b for h L ; Theorem 2.2 shows an increase of h S consistently improves the performance of M G , corresponding to the overall increases of the accuracy in Figure 2b from the left to the right; Theorem 2.3 shows the influences of h F to M G is determined by the h L . Even if this is not obvious in Figure 2b, we show more detailed figures of the individual impact of h L , h S , and h F in Appendix E.6, confirming the impact of h F in Theorem 2.3.

J

5.2 Experiments on Real-world Datasets

In this subsection, we show the superiority of our proposed Tri-Hom over the existing metrics by studying the correlation between the estimated metric values and model performance on real-world graph data.

Experimental Settings To verify the effectiveness of our proposed Tri-Hom and compare with existing metrics, we train baseline models, MLP 4 , GCN [26], GraphSage [16], GAT [57] and estimate metrics, J ¬G h , J G h , h F , h S , and h L (we use h node in our experiments), on 31 real-world heterophilic and homophilic datasets. These datasets include Roman-Empire, Amazon-Ratings, Mineweeper, Tolokers , and Questions from [52]; Squirrel, Chameleon, Actor, Texas, Cornell, Wisconsin originally from [50, 53] and refined by [52]; Cora, PubMed , and CiteSeer from [69]; CoraFull, Amazon-Photo, Amazon-Computer, Coauthor-CS , and Coauther-Physics from [54]; Flickr from [70]; WikiCS from [48]; Blog-Catalog from [68]; Ogbn-Arxiv from [19]; Genius, Twitch-DE, Twitch-ENGB, Twitch-ES, Twitch-FR, Twitch-PTBR, Twitch-RU , and Twitch-TW from [32] 5 . Besides, we also calculate other graph homophily and performance metrics on the benchmark datasets, the metrics include label-based homophily: h edge [72], h node [50], h class [33], h adj [51], h den [30], h 2hop [5], h nei [15]; structuralbased homophily: LI [51], h NS [46], h agg [38]; feature-based homophily: h GE [23], h LS-cos [7], h LS-euc [7], h attr [67], h CF [29]; and classifier-based homophily metrics [39]: h KR, h GNB, h SVM on these datasets. The detailed definitions of these metrics are summarized in Appendix A and the details of all estimations are shown in Appendix E.1.2.

Correlations with Model Performance To find out which metric can better align with the performance of GNNs on graphs with different properties, in Table 1, we show Pearson correlation between all the metrics and model performance on the 31 real-world datasets 6 . For example, to show

how well h edge align with the performance of GCN, we measure the values of h edge (in Table 3) and GCN performance (in Table 4) on node classification tasks across 31 datasets then calculate their correlation. For each model, the homophily metrics with the best, second, and third highest correlation values are highlighted in red, blue and purple, respectively. To get more robust comparison results, we rank the metrics for each model and report the average rank in the last column.

Comparison and Analysis The results show that: 1) J G h achieves the highest correlation values with all models, much better than the other metrics that only consider a single aspect of graph data. This confirms the effectiveness of Tri-Hom in filling the missing part of graph homophily by taking all the three types of homophily h L , h S , h F into account, providing a comprehensive understanding of graph homophily; 2) J ¬G h , the Tri-Hom for M ¬G , also shows high correlation values. This indicates the existence of structural-aware features of nodes in graphs, which justifies our modeling of feature homophily in CSBM-3H. Compared with other variants of CSBM [46, 39], which assume node features are conditionally independent of graph topology given node labels, the setting of CSBM-3H is closer to real-world scenarios. This is one of the important reasons that the metrics derived from CSBM-3H are better than the existing metrics.

6 Conclusions

In this paper, we study the missing components of graph homophily by disentangling it from label, structural, and feature perspectives. Compared with previous homophily metrics, the combination of the three homophily metrics provides a unique and comprehensive understanding of graph homophily. Notably, our proposed feature homophily can measure the feature dependencies among nodes, fully disentangling itself from the label and structural homophily, which helps us analyze the disentangled impact. The theoretical study on CSBM-3H leads us to Tri-Hom, a combination of the three types of homophily. By investigating how each type influences Tri-Hom, we gain deeper insights into the effect of graph homophily on model performance and elucidate intriguing phenomena observed in previous studies [29, 60]. The synthetic experiments on CSBM-3H verify the theoretical results and the effectiveness of Tri-Hom. The high correlation with GNN behaviors on 31 real-world benchmark datasets confirms the superiority of Tri-Hom over 17 existing metrics.

7 Future Directions

In the future, it will be interesting to investigate the disentangled graph homophily in more general CSBM settings without some assumptions, e.g., uniform node degrees, balanced class, and linear feature dependencies. Additionally, the estimation of three types of homophily in unsupervised or weak-supervised scenarios would be important for label-scarcity cases. Since the theoretical results in the paper reveal a nuanced understanding of graph homophily, we briefly introduce future studies on model designs or applications as follows.

For the model designs, previous studies [38, 50, 42] of heterophily-oriented GNNs mostly focus on label homophily, neglecting structural homophily h S and feature homophily h F , that also influence GNN performance. For h S , the proposed structural information function S ( · ) could be replaced by any of the measurements regarding the task. Besides, the consistency of structural information could be different in the different classes, leading to the class-specific designs for each class. For h F , it will be interesting to propose new approaches to identify how node features are influenced by their neighbors from both the global and local perspectives and then design new methods to treat the graphs or nodes differently according to h F . Please refer to Appendix F.1 for more detailed explanations of the future directions on model designs

For the applications, such as social networks, recommendation systems, or urban computing, our proposed disentangled graph homophily could provide new insights. For example, in recommendation systems, the preferences of each user could be influenced by their neighbors. When people recognize or reject others' suggestions, it leads to a high h F or low h F , causing them more likely or less likely to buy specific items. Please refer to Appendix F.2 for more details on the applications of the disentangled graph homophily.

References

S. Abu-El-Haija, B. Perozzi, A. Kapoor, N. Alipourfard, K. Lerman, H. Harutyunyan, G. Ver Steeg, and A. Galstyan. Mixhop: Higher-order graph convolutional architectures via sparsified neighborhood mixing. In international conference on machine learning , pages 21-29. PMLR, 2019.

J. L. Ba, J. R. Kiros, and G. E. Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

M. Barthelemy. Morphogenesis of spatial networks . Springer, 2018.

D. J. Best and P. G. Gipps. Algorithm as 71: The upper tail probabilities of kendall's tau. Journal of the Royal Statistical Society. Series C (Applied Statistics) , 23(1):98-100, 1974.

A. Cavallo, C. Grohnfeldt, M. Russo, G. Lovisotto, and L. Vassio. 2-hop neighbor class similarity (2ncs): A graph structural metric indicative of graph neural network performance. arXiv preprint arXiv:2212.13202 , 2022.

B. Chamberlain, J. Rowbottom, M. I. Gorinova, M. Bronstein, S. Webb, and E. Rossi. Grand: Graph neural diffusion. In International Conference on Machine Learning , pages 1407-1418. PMLR, 2021.

Y. Chen, Y. Luo, J. Tang, L. Yang, S. Qiu, C. Wang, and X. Cao. Lsgnn: towards general graph neural network in node classification by local similarity. arXiv preprint arXiv:2305.04225 , 2023.

E. Chien, J. Peng, P. Li, and O. Milenkovic. Adaptive universal generalized pagerank graph neural network. arXiv preprint arXiv:2006.07988 , 2020.

E. Cohen and N. Megiddo. Recognizing properties of periodic graphs. In Applied geometry and discrete mathematics , pages 135-146. Citeseer, 1990.

M. Defferrard, X. Bresson, and P. Vandergheynst. Convolutional neural networks on graphs with fast localized spectral filtering. arXiv , abs/1606.09375, 2016.

B. Finkelshtein, X. Huang, M. Bronstein, and I. I. Ceylan. Cooperative graph neural networks. arXiv preprint arXiv:2310.01267 , 2023.

N. Gholinejad and M. H. Chehreghani. Heterophily-aware fair recommendation using graph convolutional networks. arXiv preprint arXiv:2402.03365 , 2024.

J. Gilmer, S. S. Schoenholz, P. F. Riley, O. Vinyals, and G. E. Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning-Volume 70 , pages 1263-1272. JMLR. org, 2017.

P. Goldsmith-Pinkham and G. W. Imbens. Social networks and the identification of peer effects. Journal of Business & Economic Statistics , 31(3):253-264, 2013.

S. Gong, J. Zhou, C. Xie, and Q. Xuan. Neighborhood homophily-based graph convolutional network. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , pages 3908-3912, 2023.

W. Hamilton, Z. Ying, and J. Leskovec. Inductive representation learning on large graphs. Advances in neural information processing systems , 30, 2017.

W. L. Hamilton. Graph representation learning . Morgan & Claypool Publishers, 2020.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770778, 2016.

W. Hu, M. Fey, M. Zitnik, Y. Dong, H. Ren, B. Liu, M. Catasta, and J. Leskovec. Open graph benchmark: Datasets for machine learning on graphs. Advances in neural information processing systems , 33:22118-22133, 2020.

C. Hua, Y. Liu, D. Zhang, O. Zhang, S. Luan, K. K. Yang, G. Wolf, D. Precup, and S. Zheng. Enzymeflow: Generating reaction-specific enzyme catalytic pockets through flow matching and co-evolutionary dynamics. arXiv preprint arXiv:2410.00327 , 2024.

C. Hua, S. Luan, M. Xu, Z. Ying, J. Fu, S. Ermon, and D. Precup. Mudiff: Unified diffusion for complete molecule generation. In Learning on Graphs Conference , pages 33-1. PMLR, 2024.

W. Jiang, X. Gao, G. Xu, T. Chen, and H. Yin. Challenging low homophily in social recommendation. In Proceedings of the ACM on Web Conference 2024 , pages 3476-3484, 2024.

D. Jin, R. Wang, M. Ge, D. He, X. Li, W. Lin, and W. Zhang. Raw-gnn: Random walk aggregation based graph neural network. arXiv preprint arXiv:2206.13953 , 2022.

K. Z. Khanam, G. Srivastava, and V. Mago. The homophily principle in social network analysis: A survey. Multimedia Tools and Applications , 82(6):8811-8854, 2023.

D. P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

T. N. Kipf and M. Welling. Semi-supervised classification with graph convolutional networks. arXiv preprint arXiv:1609.02907 , 2016.

V. Kothapalli. Neural collapse: A review on modelling principles and generalization. arXiv preprint arXiv:2206.04041 , 2022.

V. Kothapalli, T. Tirer, and J. Bruna. A neural collapse perspective on feature evolution in graph neural networks. Advances in Neural Information Processing Systems , 36, 2024.

S. Y. Lee, S. Kim, F. Bu, J. Yoo, J. Tang, and K. Shin. Feature distribution on graph topology mediates the effect of graph convolution: Homophily perspective. International Conference on Machine Learning , 2024.

S. Li, D. Kim, and Q. Wang. Restructuring graph for higher homophily via adaptive spectral clustering. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 37, pages 8622-8630, 2023.

X. Li, R. Zhu, Y. Cheng, C. Shan, S. Luo, D. Li, and W. Qian. Finding global homophily in graph neural networks when meeting heterophily. In International Conference on Machine Learning , pages 13242-13256. PMLR, 2022.

D. Lim, F. Hohne, X. Li, S. L. Huang, V. Gupta, O. Bhalerao, and S. N. Lim. Large scale learning on non-homophilous graphs: New benchmarks and strong simple methods. Advances in Neural Information Processing Systems , 34:20887-20902, 2021.

D. Lim, X. Li, F. Hohne, and S.-N. Lim. New benchmarks for learning on non-homophilous graphs. arXiv preprint arXiv:2104.01404 , 2021.

Q. Lu, S. Luan, and X.-W. Chang. Gcepnet: Graph convolution-enhanced expectation propagation for massive mimo detection. arXiv preprint arXiv:2404.14886 , 2024.

Q. Lu, J. Zhu, S. Luan, and X.-W. Chang. Flexible diffusion scopes with parameterized laplacian for heterophilic graph learning. arXiv preprint arXiv:2409.09888 , 2024.

S. Luan, C. Hua, Q. Lu, L. Ma, L. Wu, X. Wang, M. Xu, X.-W. Chang, D. Precup, R. Ying, et al. The heterophilic graph learning handbook: Benchmarks, models, theoretical analysis, applications and challenges. arXiv preprint arXiv:2407.09618 , 2024.

S. Luan, C. Hua, Q. Lu, J. Zhu, X.-W. Chang, and D. Precup. When do we need graph neural networks for node classification? In International Conference on Complex Networks and Their Applications , pages 37-48. Springer, 2023.

S. Luan, C. Hua, Q. Lu, J. Zhu, M. Zhao, S. Zhang, X.-W. Chang, and D. Precup. Revisiting heterophily for graph neural networks. Advances in neural information processing systems , 35:1362-1375, 2022.

S. Luan, C. Hua, M. Xu, Q. Lu, J. Zhu, X.-W. Chang, J. Fu, J. Leskovec, and D. Precup. When do graph neural networks help with node classification? investigating the homophily principle on node distinguishability. Advances in Neural Information Processing Systems , 36, 2024.

S. Luan, Q. Lu, C. Hua, X. Wang, J. Zhu, X.-W. Chang, G. Wolf, and J. Tang. Are heterophilyspecific gnns and homophily metrics really effective? evaluation pitfalls and new benchmarks. arXiv preprint arXiv:2409.05755 , 2024.

S. Luan, M. Zhao, X.-W. Chang, and D. Precup. Break the ceiling: Stronger multi-scale deep graph convolutional networks. Advances in neural information processing systems , 32, 2019.

S. Luan, M. Zhao, C. Hua, X.-W. Chang, and D. Precup. Complete the missing half: Augmenting aggregation filtering with diversification for graph convolutional networks. arXiv preprint arXiv:2008.08844 , 2020.

S. Luan, M. Zhao, C. Hua, X.-W. Chang, and D. Precup. Complete the missing half: Augmenting aggregation filtering with diversification for graph convolutional networks. In NeurIPS 2022 Workshop: New Frontiers in Graph Learning , 2022.

M. Luenendonk. Niche to discount: 12 major types of retail stores & retailers. In FounderJar , 2023.

Y. Luo, L. Shi, and X.-M. Wu. Classic gnns are strong baselines: Reassessing gnns for node classification. arXiv preprint arXiv:2406.08993 , 2024.

Y. Ma, X. Liu, N. Shah, and J. Tang. Is homophily a necessity for graph neural networks? arXiv preprint arXiv:2106.06134 , 2021.

M. McPherson, L. Smith-Lovin, and J. M. Cook. Birds of a feather: Homophily in social networks. Annual review of sociology , 27(1):415-444, 2001.

P. Mernyei and C. Cangea. Wiki-cs: A wikipedia-based benchmark for graph neural networks. arXiv preprint arXiv:2007.02901 , 2020.

R. K. Ong, W. Qiu, and A. W. Khong. Quad-tier entity fusion contrastive representation learning for knowledge aware recommendation system. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management , CIKM '23, page 1949-1959, New York, NY, USA, 2023. Association for Computing Machinery.

H. Pei, B. Wei, K. C.-C. Chang, Y. Lei, and B. Yang. Geom-gcn: Geometric graph convolutional networks. arXiv preprint arXiv:2002.05287 , 2020.

O. Platonov, D. Kuznedelev, A. Babenko, and L. Prokhorenkova. Characterizing graph datasets for node classification: Homophily-heterophily dichotomy and beyond. Advances in Neural Information Processing Systems , 36, 2024.

O. Platonov, D. Kuznedelev, M. Diskin, A. Babenko, and L. Prokhorenkova. A critical look at the evaluation of gnns under heterophily: Are we really making progress? arXiv preprint arXiv:2302.11640 , 2023.

B. Rozemberczki, C. Allen, and R. Sarkar. Multi-Scale Attributed Node Embedding. Journal of Complex Networks , 9(2), 2021.

O. Shchur, M. Mumme, A. Bojchevski, and S. Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint arXiv:1811.05868 , 2018.

D. Shi, A. Han, L. Lin, Y. Guo, Z. Wang, and J. Gao. Design your own universe: A physics-informed agnostic method for enhancing graph neural networks. arXiv preprint arXiv:2401.14580 , 2024.

S. Suresh, V. Budde, J. Neville, P. Li, and J. Ma. Breaking the limit of graph neural networks by improving the assortativity of graphs with local mixing patterns. In Proceedings of the 27th ACM SIGKDD Conference on Knowledge Discovery & Data Mining , pages 1541-1551, 2021.

P. Velickovi´ c, G. Cucurull, A. Casanova, A. Romero, P. Lio, and Y. Bengio. Graph attention networks. arXiv preprint arXiv:1710.10903 , 2017.

B. Wang, J. Chen, C. Li, S. Zhou, Q. Shi, Y. Gao, Y. Feng, C. Chen, and C. Wang. Distributionally robust graph-based recommendation system. In Proceedings of the ACM on Web Conference 2024 , pages 3777-3788, 2024.

G.-N. Wang, H. Gao, L. Chen, D. N. Mensah, and Y. Fu. Predicting positive and negative relationships in large social networks. PloS one , 10(6):e0129530, 2015.

J. Wang, Y. Guo, L. Yang, and Y. Wang. Understanding heterophily for graph neural networks. arXiv preprint arXiv:2401.09125 , 2024.

J. Wang, H. Xie, F. L. Wang, L.-K. Lee, and M. Wei. Jointly modeling intra-and inter-session dependencies with graph neural networks for session-based recommendations. Information Processing & Management , 60(2):103209, 2023.

K. Wang, G. Li, S. Wang, G. Zhang, K. Wang, Y. You, J. Fang, X. Peng, Y. Liang, and Y. Wang. The snowflake hypothesis: Training and powering gnn with one node one receptive field. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 3152-3163, 2024.

Y. Wang, S. Xiang, and C. Pan. Improving the homophily of heterophilic graphs for semisupervised node classification. In 2023 IEEE International Conference on Multimedia and Expo (ICME) , pages 1865-1870. IEEE, 2023.

Y. Wang, K. Yi, X. Liu, Y. G. Wang, and S. Jin. Acmp: Allen-cahn message passing for graph neural networks with particle phase transition. arXiv preprint arXiv:2206.05437 , 2022.

C. Xiao, J. Zhou, J. Huang, T. Xu, and H. Xiong. Spatial heterophily aware graph neural networks. In Proceedings of the 29th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2752-2763, 2023.

K. Xu, W. Hu, J. Leskovec, and S. Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:1810.00826 , 2018.

L. Yang, M. Li, L. Liu, C. Wang, X. Cao, Y. Guo, et al. Diverse message passing for attribute with heterophily. Advances in Neural Information Processing Systems , 34:4751-4763, 2021.

R. Yang, J. Shi, X. Xiao, Y. Yang, S. S. Bhowmick, and J. Liu. Pane: scalable and effective attributed network embedding. The VLDB Journal , 32(6):1237-1262, 2023.

Z. Yang, W. Cohen, and R. Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning , pages 40-48. PMLR, 2016.

H. Zeng, H. Zhou, A. Srivastava, R. Kannan, and V. Prasanna. Graphsaint: Graph sampling based inductive learning method. arXiv preprint arXiv:1907.04931 , 2019.

Y. Zheng, J. Xu, and L. Chen. Learn from heterophily: Heterophilous information-enhanced graph neural network. arXiv preprint arXiv:2403.17351 , 2024.

J. Zhu, Y. Yan, L. Zhao, M. Heimann, L. Akoglu, and D. Koutra. Beyond homophily in graph neural networks: Current limitations and effective designs. Advances in Neural Information Processing Systems , 33, 2020.

A Related Work on Homophily Measurements

In general, graph homophily metrics can be categorized as either statistic-based metrics or classifierbased metrics. The former can be further classified into the homophily on label, structural, or feature aspect. The definitions of these metrics are introduced as follows.

A.1 Homophily on Label Aspect

Edge homophily [72] measures the graph homophily at the edge level, which is defined as the fraction of edges in a graph that connects nodes with the same labels:

Node homophily [50] measures the graph homophily at the node level, where the homophily degree for each node is computed as the proportion of the neighbors sharing the same class. Then the node homophily for the whole graph is defined as the average homophily degree for all the nodes:

Class homophily [33] addresses class imbalance by treating all classes equally. This metric mitigates the sensitivity of edge homophily and node homophily to the number of classes and nodes in each class. The definition of class homophily is given by:

Adjusted homophily [51] considers the probability of an edge endpoint connecting to a node with a particular class and adjusts the edge homophily using node degrees, which is defined as:

where D c represents the total degree of class c , i.e., D k = ∣ ∣ { e uv | Y u = c or Y v = c, e uv ∈ E} ∣ ∣ .

Density-aware homophily [30] is introduced as an improvement over class homophily, which only captures relative edge proportions and disregards graph connectivity. This limitation results in inflated homophily scores for highly disconnected graphs [51]. The proposed density-aware homophily aims to provide a more accurate measurement of edge density and is defined as:

where ζ k is the edge density of the subgraph formed by intra-class edges of class k and ˆ ζ k is the maximum intra-class edge density of class k .

2-hop Neighbor Class Similarity [5]. Since the information of 1-hop neighbors might be less representative or even misleading [5], 2-hop Neighbor Class Similarity extends the concept of "neighbors" from 1-hop to 2-hop, which is defined as:

where N (2) u = { ⋃ v ∈N u N v }\{ u } represents the two-hop neighbors of node u .

Neighbor Homophily [15] is proposed to address the "good" and "bad" heterophily issue by considering the dominant neighbors. The homophily score for any given node u is based on the number

of nodes in which its class holds dominance among its k-hop neighbors. The definition of neighbor homophily in a graph is defined as:

where N ( k ) v represents the k-hop neighbors of node u .

A.2 Homophily on Structure Aspect

Label informativeness [51] measures the informativeness of a neighbor's label for a node's label using condition entropy:

where p ( c 1 , c 2 ) = ∑ ( u,v ) ∈E ✶ { Y u = c 1 ,Y v = c 2 } 2 |E| represents mutual distribution for a randomly sampled edge from class c 1 to class c 2 and ¯ p c = D c 2 |E| represents the distribution of the node degree for class c .

Neighborhood Similarity [46] measures the similarity of the neighbor distributions between two classes. A high similarity of intra-class neighbor distributions and a low similarity of inter-class neighbor distributions ensure the neighborhood patterns for nodes with different labels are distinguishable [46]. Therefore, the ratio of inter-class to intra-class neighborhood similarity could reflect the performance of GNNs 7

where N ( k ) v is the k-hop neighbors of node u .

̸

Aggregation homophily [38] measures the ratio of nodes in a graph that has a higher intra-class aggregation similarity than inter-class aggregation similarity. The aggregation similarity of node u and u is the multiplication of their neighbor distribution i.e., D N u D N v . The definition of aggregation homophily is given as:

̸

̸

A.3 Homophily on Feature Aspect

Generalized edge homophily [23] defines the feature homophily in graphs as the feature consistency across the graph topology:

The difference between this homophily with edge homophily is that generalized edge homophily replaced the indicator function of two connected nodes in edge homophily, i.e., ✶ { Y u = Y v } , to a similarity measurement of node features, i.e., sim ( X u , X v ) .

Local Similarity [7] measures feature homophily at the node level based on the hypothesis that nodes with similar features are likely to belong to the same class. The definition is given based on either cosine similarity

or Euclidean similarity.

Attribute homophily [67] considers the homophily with respect to each feature, which is defined as:

where h attr represents the attribute homophily for the whole graph and h attr,m represents the attribute homophily for feature m .

Class-controlled feature homophily [29] considers the interplay between graph topology and feature dependence through the disparity of nodes' expected distances to random nodes with their neighbors, which is defined as:

where X u | Y represents class-controlled features and d ( · ) denotes a distance function.

A.4 Classifier-based homophily

Classifier-based homophily [39] uses a classifier to capture the feature-based linear or non-linear information without iterative training. To determine when graph-aware models perform better than graph-agnostic models, a hypothesis test is conducted on the original feature X and the aggregated features H , as shown below:

H 0 : Prop ( G-aware model ) ≥ Prop ( G-agnostic model ); H 1 : Prop ( G-aware model ) < Prop ( G-agnostic model )

The resulting p-value from this hypothesis test can indicate whether the performance of H is superior to that of X . Three types of classifier, Gaussian Naive Bayes, Kernel Regression, and Support Vector Machine are used in [39], which correspond to the metrics h GNB , h KR , and h SV M in this paper.

B Structural-Aware Node Features

This section explores the impact of graph topology on node features. Unlike the data in Euclidean space, where samples are i.i.d. , the data sampled from a graph for each node are structural-aware i.e., ( X ̸ ⊥ ⊥ A | Y ) . To model the feature dependencies of nodes in graphs, we follow [55] and adopt a graph diffusion process [6], which is shown as follows

Here the structural-agnostic node features X (0) ∼ p ( X | Y ) are sampled from the distributions with respect to each class, while the structural-aware node features X ( t > 0) ∼ p ( X | Y , A ) are generated through the diffusion process. In ∂ X ( t ) ∂t , the first term describes how the dependencies are introduced with a feature dependency function F : ( A ) ↦→ R N × N and the second term X (0) preserves the node distinguishability. Then, based on the Eq. (28), we have

We can also interpret the process as an interactive particle system [55] where all node features collapse into an equilibrium eventually. The equilibrium requires Eq. (29) to have a closed form when t →∞ , implying the spectral radius of the adjacency matrix ρ ( F ( A )) < 1 . This constraint also implies that |F ( A ) | k ≥ |F ( A ) | k -1 for k > 0 , which aligns with the common relations in graphs that nodes are likely to have higher dependencies with their closer neighbors than with farther neighbors.

For simplicity and a better understanding of feature dependencies, we consider a linear case as in [6, 55] and use a parameter ω to control the feature dependencies i.e., F ( A ) = ω A with a range of ( -1 ρ ( A ) , 1 ρ ( A ) ) . Then we can represent the structural-aware features as

C Properties of Feature Homophily

We further investigate properties of the estimation of feature homophily under specific feature transformations, including shifts, scaling, and changes in variance. The problem is defined as follows: In a graph G = {V , E} with N nodes, there are node labels Y and node features X : ,m in dimension m . We can estimate the feature homophily h ∗ F,m for feature X : ,m in dimension m as:

Then we prove the estimation of feature homophily is invariant to the operations of feature shifts, scaling, or changing variance.

A. Shifts

Let's consider a shift of node features X : ,m by a constant vector C :

Then we have structural-agnostic features as

Then a new estimation of h ′ F,m under this feature shift can be expressed as

Since C is a constant vector, we have C u -C v = 0 . Next, we have

Therefore, we proved the estimation of feature homophily is invariant to the operation of feature shifts.

B. Scaling

Let's consider scaling node features X : ,m by a constant α :

Then we have structural-agnostic features as

Then a new estimation of h ′ F,m under this feature shift can be expressed as

Since arg min x ( · ) is invariant to the scaling, e.g. arg min x ( cf ( x )) = arg min x ( f ( x )) , we have

Therefore, we proved the estimation of feature homophily is invariant to the operation of feature scaling.

C. Variance Changing

Changing the variance of X : ,m can be seen as the combination of scaling and shifts. Assume node features follow a Gaussian distribution N ( µ, σ 2 ) , after the operation of changing the variance from σ 2 to βσ 2 , we have new node features as

where X ′ : ,m is calculated by deducing µ , multiplying √ β , and adding µ . We already show the estimation of feature homophily is invariant to the operations of feature shifts and scaling. Since the operation of variance changing is a combination of scaling and shifts, we can conclude that the estimation of feature homophily is invariant to the variance changing.

D Impact of Graph Homophily on Graph-agnostic Models

In addition to the impact of label homophily h L , structural homophily h S , and feature homophily h F on Graph-aware models as discussed in Section 4.2, we further discuss the impact on Graph-agnostic models through J ¬G h in this section. Specifically, we compute the partial derivative of J ¬G h with respect to h S , h F , and h L to reveal the analytical influences. (See calculation in Appendix G.2, G.3, and G.4.)

Theorem 3.1. The partial derivative of J ¬G h with respect to label homophily h L satisfies,

From Theorem 3.1 we can see that, under a positive h F i.e., the features of connected nodes become similar, the increase of h L makes the features of intra-class nodes more distinguishable, thereby improving the performance of M ¬G . Conversely, under a negative h F i.e., the features of connected nodes become dissimilar, the increase of h L makes the features of intra-class nodes more indistinguishable, resulting in a degradation of the performance of M ¬G .

Theorem 3.2. The partial derivative of J ¬G h with respect to structural homophily h S satisfies,

From Theorem 3.2 we can see that, since the existence of feature dependencies, a larger h S makes the structural-aware features more distinguishable among different classes, improving the performance of M ¬G .

Theorem 3.3. The partial derivative of J ¬G h with respect to feature homophily h F satisfies,

where 0 < h ∗ , -L < h ∗ , + L < 1 and -1 < h ∗ F < 1 .

From Theorem 3.3 we can see, how the impact of h F on M ¬G is determined by h L . This result is similar to the Theorem 2.3 in the case of M G . The increase of h F makes node features more similar to their neighbors. As a result, under a high h L , features of intra-class nodes become more similar, thereby improving the performance of M ¬G ; under a low h L , features of intra-class nodes becomes more similar, thereby reducing the performance of M ¬G .

E Experimental Details

E.1 Datasets

E.1.1 Synthetic Datasets

We show the detailed process of constructing CSBM-3H in Algorithm 1. First, graph topology is constructed with label homophily h L and structural homophily h S . Then, structural-aware node features are constructed with feature homophily h F . To investigate how three types of homophily influence the model performance, we generate graphs with h L ∈ [0 , 0 . 1 , . . . , 1] , h S ∈ [0 , 0 . 1 , . . . , 1] , and h F ∈ [ -0 . 8 , -0 . 6 , . . . , 0 . 8] . For each given h L , h S , and h F , we generate the graphs with 10 seeds from [0 , 1 , . . . , 9] to mitigate random deviations. Each graph contains 1000 nodes distributed across 3 classes, with node degrees sampled from a uniform distribution in the range

For the random graph generative models with homophily, current studies [51, 29, 39, 46] generally adopt a Contextual Stochastic Block Model with Homophily(CSBM-H) to control the label homophily h L through assigning nodes with different probabilities that connect to the nodes from other classes.

Algorithm 1 Stochastic Block Model controlled with 3 types of Homophily(CSBM-3H)

Input:

Label homophily

h

L

, structural homophily number of nodes

N

, number of classes

C

h

S

, and feature homophily

M

, random seed class-wised Gaussian mean

µ

M

D

,

ϕ

µ

= [

1

, . . . ,

,

µ

C

.

T

,

T

and Σ c ∈ R × Output: Adjacency matrix A , node features X and node labels Y of a synthetic graph.

M

for class

Initialize node features

X

c

= [

Σ

Y

1

, . . . ,

R

Σ

C

]

with

µ

c

N

. Set random seed

∈ R

ϕ

.

▷

∈

▷ One-hot encoding of node labels ▷ Introduce label homophily h L

▷

Construct neighbor distribution

▷ Introduce structural homophily h S

▷ Construct neighbor sampling matrix with node degrees ▷ Bound with [0 , 1] before sampling

▷ Binarization sampling and symmetrize adjacency matrix ▷ Sample Structural-agnostic node features

▷ Introduce feature homophily h F

Then the node features are sampled solely based on the classes. However, these random graph generative models have two drawbacks: First, the probabilities of nodes connecting to the nodes with different classes are uniform, which lacks diversity. Second, the sampled node features are independent with their structures i.e., ( X ⊥ ⊥ A | Y ) , which is uncommon in real-world scenarios where interactions influence the attributes of connected nodes [61, 14, 59]. Our proposed CSBM-3H well address these drawbacks by considering h S and h F , thereby providing a more comprehensive and realistic model.

E.1.2 Real-World Datasets

We conduct our experiments on 31 real-world datasets: Roman-Empire, Amazon-Ratings, Mineweeper, Tolokers, and Questions from [52]; Squirrel, Chameleon, Actor, Texas, Cornell, Wisconsin originally from [50] and refined by [52]; Cora, PubMed, and CiteSeer from [69]; CoraFull, AmazonPhoto, Amazon-Computer, Coauthor-CS, and Coauther-Physics from [54]; Flickr from [70]; WikiCS from [48]; Blog-Catalog from [68]; Ogbn-Arxiv from [19]; Genius, Twitch-DE, Twitch-ENGB, Twitch-ES, Twitch-FR, Twitch-PTBR, Twitch-RU, and Twitch-TW from [32]. These datasets contain both the homophilic and heterophilic graphs that come from citation networks, webpage networks, purchase networks, image description networks, coauthor networks, actor networks, and social networks. The diversity of these datasets enables us to evaluate the model performance in a general case. For these datasets, we show basic statistics in Table 2 and the graph homophily metrics or model performance metrics in Table 3.

E.2 Training Detail

For all the datasets, we randomly split the train, validation, and test set as 50%:25%:25% for 10 runs. We use the Adam optimizer [25] with a learning rate of 0.001. The maximum training epoch is set to 1000 with a patience of 40 for early stopping. To enhance performance, we incorporated

Y

]

, with

, covariance

Σ

X

∈

R

N

×

M

,

h

F

M

skip connections [18] and layer normalization [2] in each layer. All models are trained on a single NVIDIA RTX A5000 GPU with 24GB memory. For hyperparameter tuning, we perform a grid search on the validation set. The search space included the following hyperparameters:

Number of layers: {1, 2},

Hidden dimension: {64, 128, 256},

Dropout rate: {0.2, 0.4, 0.6, 0.8},

Weight decay: {1e-3, 1e-4, 1e-5}.

E.3 Node Classification Performance

Table 4 shows the averaged mean and stand deviation accuracy of node classification performance for MLP, GCN, GraphSage, and GAT across 10 runs. Notably, graph-aware models M G outperform graph-agnostic models M ¬G in most datasets. This phenomenon holds true for both homophilic datasets, such as Cora, Citeseer, and PubMed, and heterophilic datasets, such as Roman-empire, Chameleon-filtered, and Flickr. Consequently, relying solely on label homophily is insufficient to determine the performance of M G , which aligns with previous studies [46, 39]

E.4 Tri-Hom for Graph-agnostic Models

We show the numerical results of Tri-Hom J ¬G h for graph-agnostic models M ¬G in Figure 3a, where each subfigure is a slicer of h S that visualizes the influences of h L and h F on J G h . For the impact of h L , h F , and h S individually, we can get the same conclusions as in Theorem 3.1, 3.2, and 3.3. To validate our theoretical results of M ¬G in a more general case, we further show the impact of three types of homophily with MLP on synthetic datasets in Figure 3b. The results correlate well with our

numerical results, showing the effectiveness of J ¬G h in measuring the performance of M ¬G under the influences of three types of homophily.

E.5 Numerical Results of Tri-Hom with Three types of Homophily

We show how the label homophily h L , structural homophily h S , and feature homophily h F collectively influence the J ¬G h and J G h in Figure 4 in a more general case, where the x-axis, y-axis, and z-axis denotes h L , h F , and h N . Alighter color (close to yellow) indicates a larger J ¬G h or J G h , which also implies a better model performance. These numerical results align well with our theoretical results of the individual impact on J ¬G h or J G h with respect to h L , h S , or h F . Furthermore, by observing the gradients in Figure 4, we can know how these three homophily metrics influence the model performance collectively.

E.6 Influences of A Single Type of Homophily on Synthetic Datasets

To verify our theoretical results, we show how the performances of GCN and MLP are affected by three types of homophily individually in Figure 5, which shows the influences of label homophily h L , structural homophily h S , and feature homophily h F on the node classification accuracy of GCN and MLP on synthetic datasets. First, the top 3 subfigures show the influences of h L under different h F and h S . With the increase of h L , the accuracy of GCN first decreases and then increases with a pitfall on a medium-level of homophily, and the accuracy of MLP is dependent on the sign of h F , leading to

J

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

¬G

h

0

.

8

1

.

0

1

.

2

M

¬G

, h

S

=

0

.

2

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

¬G

h

0

.

8

1

.

0

1

.

2

M

¬G

, h

S

=

0

.

4

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

J

¬G

h

0

.

8

1

.

0

1

.

2

M

¬G

, h

S

=

0

.

6

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

¬G

h

0

.

8

1

.

0

1

.

2

M

¬G

, h

S

=

0

.

8

(a) Numerical results of Tri-Hom

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

MLP

,

h

S

=

0

.

2

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

MLP

,

h

S

=

0

.

4

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

MLP

,

h

S

=

0

.

6

h

F

-

1

.

0

-

0

.

5

0

.

0

0

.

5

1

.

0

h

L

0

.

00

0

.

25

0

.

50

0

.

75

1

.

00

0

.

6

0

.

8

1

.

0

MLP

,

h

S

=

0

.

8

0.0

0.2

0.4

0.6

0.8

1.0

hL

0.0

0.2

0.4

0.6

0.8

1.0

hS

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

hF

(a) Graph-agnostic Models

0.0

0.2

0.4

0.6

0.8

1.0

hL

0.0

0.2

0.4

0.6

0.8

1.0

hS

1.00

0.75

0.50

0.25

0.00

0.25

0.50

0.75

1.00

hF

(b) Graph-aware Models

the same results as in Theorem 2.1 and 3.1. Second, the medium 3 subfigures show the influences of h F . With the increases of h F , the accuracy of both of the GCN and MLP will decreases with a low h L , increases with a high h L , and increases first and then decreases with a medium h L . The result also aligns well with Theorem 2.3 and 3.3. Last, the bottom 3 subfigures show the influences of h S . Even if the influences under different h L or h F would slightly vary, an increase of h S generally improves the performance of both the GCN and MLP, aligning well with Theorem 2.2 and 3.2.

E.7 Visualization of Real-world Dataset with Three Types of Homophily

To better visualize the impact of h L , h S , and h F to graph-aware models M G on real-world datasets, we show the performance of GCN on all the datasets with these homophily metrics in Figure 6, where h L , h F , and h S are shown as the x-axis, y-axis, and the size of the scatter respectively. Generally, the correlation of three homophily metrics with the performance of GCN is similar to our theoretical results in Figure 2a. With the increase of h L , the performance of GCN decreases first and then

0.00

0.25

0.50

0.75

1.00

hL

0.5

0.6

0.7

0.8

0.9

1.0

hF

=-0.8,

hN

=0.9

GCN

MLP

0.00

0.25

0.50

0.75

1.00

hL

0.5

0.6

0.7

0.8

0.9

1.0

hF

=0.0,

hN

=0.9

GCN

MLP

0.00

0.25

0.50

0.75

1.00

hL

0.5

0.6

0.7

0.8

0.9

1.0

hF

=0.8,

hN

=0.9

GCN

MLP

0.5

0.0

0.5

hF

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.1,

hN

=0.9

GCN

MLP

0.5

0.0

0.5

hF

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.3,

hN

=0.9

GCN

MLP

0.5

0.0

0.5

hF

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.9,

hN

=0.9

GCN

MLP

0.00

0.25

0.50

0.75

1.00

hS

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.1,

hF

=-0.8

GCN

MLP

0.00

0.25

0.50

0.75

1.00

hS

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.9,

hF

=-0.8

GCN

MLP

0.00

0.25

0.50

0.75

1.00

hS

0.5

0.6

0.7

0.8

0.9

1.0

hL

=0.9,

hF

=0.8

GCN

MLP

increases, with a pitfall (as shown by datasets of Actor, Squirrel, and Chameleon). We also see that a higher h S leads to a better performance when we fix h F and h L (as shown by datasets of Question with Amazon-photo, and Cora with Coauthor-CS). As for the h F , since its influence is not so obvious compared with h L and h S as shown in Figure 5 and the scarcity of real-world datasets, it is hard to see its influence in real-world datasets.

E.8 Influences of Class-wised Structural Homophily on Real-world Datasets

As shown in Figure 7, we investigate more nuanced influences of structural homophily h S without the interference of h L and h F on real-world datasets with respect to each class. Specifically, we calculate node classification accuracy with respect to each class inside one dataset and show the alignment of the class-wised accuracy and class-wised structural homophily h S . The results show for some of the datasets such as Citeseer, Amazon-ratings, and Coauthor-physics, we can clearly observe the accuracy increases with h S while for datasets such as Corafull and Coauthor-CS, we can only see the general tendency of alignment with noises. We speculate that for Corafull and Coauthor-CS, the h S is not as uniform as other datasets for the nodes in one class, leading to the noises.

E.9 Correlation of Metrics with Performance Gap of GNNs and MLP

To investigate when GNNs are better than MLP, we show the differences in the performance between GNNs and MLP in Figure 5. The results indicate that, among the statistic-based homophily metrics,

0.2

0.4

0.6

0.8

hL

0.0

0.2

0.4

0.6

0.8

1.0

hF

Roman-Empire

Amazon-Ratings

Minesweeper

Tolokers

Questions

Chameleon

Actor

texas

Cornell

Wisconsin

CoraFull

CiteSeer

PubMed

Flickr

Amazon-Photo

Amazon-Computer

Coauthor-CS

Coauthor-Physics

WikiCS

Blog-Catalog

Ogbn-Arxiv

Genius

Twitch-DE

Twitch-ES

Twitch-FR

Twitch-PTBR

Twitch-TW

0.4

0.5

0.6

0.7

0.8

0.9

0.2

0.3

0.4

hS

0.4

0.6

0.8

CiteSeer

MLP

GCN

0.475

0.500

0.525

0.550

0.575

hS

0.0

0.2

0.4

0.6

Amazon-Ratings

MLP

GCN

0.65

0.70

0.75

0.80

0.85

hS

0.7

0.8

0.9

1.0

Amazon-Photo

MLP

GCN

0.6

0.7

0.8

hS

0.00

0.25

0.50

0.75

CoraFull

MLP

GCN

0.60

0.65

0.70

0.75

0.80

hS

0.7

0.8

0.9

1.0

Coauthor-CS

MLP

GCN

0.65

0.70

0.75

0.80

hS

0.90

0.92

0.94

0.96

0.98

Coauthor-Physics

MLP

GCN

the structural homophily h S shows a strong correlation with the differences in the performance between GNNs and MLP. This is because h S measure the consistency of the structural information of nodes within the same class, thereby reflecting the differences in the performance between GNNs and MLP. We also observe that the h KR has the highest correlation with differences, confirming the effectiveness of the classifier-based metrics in measuring the differences in the performance between GNNs and MLP [39].

E.10 Other Types of Correlation

In our experiments on real-world datasets, we explore the correlation between all the metrics and model performance. In addition to the widely used Pearson correlation, we employ Kendall's Tau rank correlation to assess these relationships. Let x and y be two observed variables with x , y ∈ R N , the Pearson correlation ρ can be calculated as

where ¯ x and ¯ y are the means of x and y respectively.

Kendall's Tau rank correlation [4] τ can be calculated as

where concordant pairs occur when the ranks of both variables agree, while discordant pairs occur when they disagree.

Pearson correlation measures linear correlation between two variables, which is widely used and easy to interpret. Kendall's Tau rank correlation measures the similarity ranking between two variables, which has no assumptions of the data distribution and are robust to outliers.

The correlation of all the metrics with model performance measured by Kendall's Tau rank correlation is shown in Table 6, which is similar as the results in Pearson correlation. Our Tri-Hom J G h still shows the highest correlation with GNNs performance compared with other types of metrics, which confirms the necessity of disentangling graph homophily.

E.11 Correlation of Homophily Metrics

To investigate the similarity of the information contained in graph homophily metrics and GNNs performance metrics, we show the Pearson correlation of these metrics on real-world datasets in Figure 8. We can see a high correlation among the homophily on label aspect (including h edge , h node , h class , h adj , h den , h 2 hop , and h nei ). These metrics measure the label consistency across the graph topology, sharing a similar characteristic. This also holds for the homophily on the structural aspect (including LI , h NC , h agg , and h S ), since these metrics measure how informative the neighbors are for the node labels. However, the Pearson correlation among the homophily on feature aspect ( h GE , h LS -cos , h LS -cos , h attr , h CF , and h F ) is low. We speculate that this is because different types of feature aspect ( h GE , h CF , and h F ) or different similarity measurements ( h LS -cos and h LS -cos ) could vary a lot for real-world datasets.

F Future Directions

Each of the disentangled graph homophily, including label homophily, structural homophily, or feature homophily, derives many interesting directions for future research. In this section, we introduce these potential future directions from the perspective of model design and applications.

F.1 Model Design

A. Label homophily

We discussed how label homophily influences GCN and MLP, providing both theoretical proof and empirical experiments. Our results show that GCN performs better than MLP in conditions of extremely low homophily (good heterophily [46]), but significantly worse than MLP in medium levels of homophily (mid-homophily pitfall [39]). This suggests that GCNs sometimes fail to extract effective topological information. To mitigate this weakness, it is preferable to add a residual connection to GNNs or introduce a learnable parameter that allows the model to balance graph-aware and graph-agnostic information. The necessity of residual connections has been verified in previous studies [66, 45, 52].

In addition to model design at the graph level, we can also consider a fine-grained approach at the node level. Our results indicate that GCN does not always outperform MLP, suggesting that different

h

edge

h

node

h

class

h

adj

h

den

h

2

hop

h

LI

h

NS

h

agg

h

S

h

GE

h

LS

-

cos

h

LS

-

euc

h

attr

h

CF

h

F

h

KR

h

GNB

h

SV M

J

¬G

h

J

G

h

h

edge

h

node

h

class

h

adj

h

den

h

2

hop

h

LI

h

NS

h

agg

h

S

h

GE

h

LS

-

cos

h

LS

-

euc

h

attr

h

CF

h

F

h

KR

h

GNB

h

SV M

J

¬G

h

J

G

h

1.0

0.99 0.68 0.77 0.58 0.81 0.71 0.79 0.47 0.18 0.28 0.37

0.4

-0.07-0.07 0.16 0.58 -0.05 0.12 0.08 0.72 0.81

0.99

1.0

0.69 0.77 0.57 0.79 0.69 0.79 0.47 0.16 0.36 0.33 0.36 -0.08-0.09 0.18 0.55

-0.0

0.14

0.1

0.71 0.83

0.68 0.69

1.0

0.96 0.33 0.43 0.36 0.82 0.61 0.73 0.42 0.17 0.19 -0.18-0.18 0.13 0.55 -0.19-0.09 0.15 0.73 0.79

0.77 0.77 0.96

1.0

0.51 0.43 0.35 0.87 0.66 0.63

0.4

0.2

0.23 -0.17-0.13 0.15

0.6

-0.16-0.03 0.01 0.73 0.77

0.58 0.57 0.33 0.51

1.0

0.11 0.02 0.38 0.18 -0.02 0.19

-0.0

0.01

-0.1

0.09 0.19

0.3

0.09 0.12 -0.34 0.27

0.3

0.81 0.79 0.43 0.43 0.11

1.0

0.97 0.51

0.3

0.01 0.01 0.46 0.48 0.16 -0.05 0.11 0.44 -0.05

0.1

0.36 0.61 0.69

0.71 0.69 0.36 0.35 0.02 0.97

1.0

0.38 0.26 0.03 -0.07 0.45 0.47 0.22

0.0

0.11 0.41 -0.03 0.12

0.4

0.55 0.62

0.79 0.79 0.82 0.87 0.38 0.51 0.38

1.0

0.67 0.46 0.55 0.36 0.37 -0.15-0.18-0.04 0.53 -0.06-0.06 0.08

0.7

0.78

0.47 0.47 0.61 0.66 0.18

0.3

0.26 0.67

1.0

0.52 0.43 0.37 0.43 0.02 -0.16 0.16 0.36 -0.14-0.08

0.2

0.47 0.59

0.18 0.16 0.73 0.63 -0.02 0.01 0.03 0.46 0.52

1.0

0.3

0.07 0.06 -0.17-0.01 0.15 0.36 -0.29 -0.2

0.3

0.42 0.46

0.28 0.36 0.42

0.4

0.19 0.01 -0.07 0.55 0.43

0.3

1.0

-0.08-0.05 -0.1 -0.14-0.03 0.07 0.21 0.04

0.2

0.21 0.52

0.37 0.33 0.17

0.2

-0.0

0.46 0.45 0.36 0.37 0.07 -0.08

1.0

0.99 -0.01 -0.0

0.15 0.62 -0.26-0.42 0.08 0.54 0.32

0.4

0.36 0.19 0.23 0.01 0.48 0.47 0.37 0.43 0.06 -0.05 0.99

1.0

-0.07-0.03 0.24 0.63 -0.28 -0.4

0.12 0.57 0.35

-0.07-0.08-0.18-0.17 -0.1

0.16 0.22 -0.15 0.02 -0.17 -0.1 -0.01-0.07

1.0

0.01 -0.56-0.29 0.11 0.15 0.11 -0.17-0.07

-0.07-0.09-0.18-0.13 0.09 -0.05

0.0

-0.18-0.16-0.01-0.14 -0.0 -0.03 0.01

1.0

0.26 0.35 0.35 -0.11 0.26 -0.07-0.19

0.16 0.18 0.13 0.15 0.19 0.11 0.11 -0.04 0.16 0.15 -0.03 0.15 0.24 -0.56 0.26

1.0

0.48 -0.26-0.29 0.21 0.34 0.24

0.58 0.55 0.55

0.6

0.3

0.44 0.41 0.53 0.36 0.36 0.07 0.62 0.63 -0.29 0.35 0.48

1.0

-0.18-0.36 0.13 0.86 0.56

-0.05 -0.0 -0.19-0.16 0.09 -0.05-0.03-0.06-0.14-0.29 0.21 -0.26-0.28 0.11 0.35 -0.26-0.18

1.0

0.41 0.01 -0.33-0.16

0.12 0.14 -0.09-0.03 0.12

0.1

0.12 -0.06-0.08 -0.2

0.04 -0.42 -0.4

0.15 -0.11-0.29-0.36 0.41

1.0

-0.15-0.26-0.04

0.08

0.1

0.15 0.01 -0.34 0.36

0.4

0.08

0.2

0.3

0.2

0.08 0.12 0.11 0.26 0.21 0.13 0.01 -0.15

1.0

0.18 0.38

0.72 0.71 0.73 0.73 0.27 0.61 0.55

0.7

0.47 0.42 0.21 0.54 0.57 -0.17-0.07 0.34 0.86 -0.33-0.26 0.18

1.0

0.81

0.81 0.83 0.79 0.77

0.3

0.69 0.62 0.78 0.59 0.46 0.52 0.32 0.35 -0.07-0.19 0.24 0.56 -0.16-0.04 0.38 0.81

1.0

models could be applied to nodes with varying levels of label homophily. This type of personalized design has been explored in current GNN research on heterophilic graphs [62, 11], yet it holds significant potential for improving overall model performance.

B. Structural homophily

As mentioned in Theorem 2.2, the performance of graph-aware models improves with an increase in structural homophily. This leads to a crucial question: how can we deal with graphs exhibiting varying levels of structural homophily? To address this issue, we can enhance current GNNs using message-passing calibration and graph rewiring.

For the message passing calibration, several methods, such as GPRGNN [8], FB-GNNs [43], and ACM-GNNs [38], propose adding an additional high-pass filter to capture local variations and details in the graph structure. When structural homophily is low, the high-pass filter captures the diversification of individual nodes. Along with a low-pass filter, these methods perform well on graphs with varying levels of homophily. However, the high-pass filters used in these methods cannot capture more complicated structural information. Since S ( · ) in structural homophily can be any measurement of structural homophily, this metric can evaluate more complex graph structures. In the

future, it is promising to design novel filters based on structural homophily to capture more intricate structural information and improve model performance.

For the graph rewiring, many methods (MVGCN [63], GloGNN [31], WRGNN [56]) propose deriving a new graph topology based on node features or embeddings. This operation improves the connectivity of nodes with similar semantic contexts, thereby enhancing model performance. However, this rewiring could connect nodes from different classes, which impedes GNN performance. To resolve this, we can measure class-wise structural homophily as shown in Eq. (2) and design adaptations for different classes, which will be particularly beneficial for class-imbalanced graphs, such as in bot detection and fraud detection. Furthermore, we can adapt the structural measurement function S ( · ) . Since most current rewiring methods do not evaluate the informativeness of their rewiring basis (node embeddings with structural information), the proposed structural homophily can serve as a metric to evaluate which types of rewiring basis to select. For example, Geom-GNN [50] uses structural node embeddings to construct new neighbors of nodes and empirically determines the best approach. Our structural homophily could identify the most effective embedding approach before training GNNs. Therefore, structural homophily provides a guideline for graph rewiring methods.

C. Feature homophily

The feature homophily proposed in this paper measures how node features are influenced by their neighbors. To our knowledge, only a few GNNs [55] consider these feature dependencies in their design. There is significant potential to explore how feature dependencies function in graphs. For instance, in social networks, people's opinions are affected by those around them. Identifying different user types while filtering out the noise introduced by their neighbors remains an open question. Both our theoretical results (Theorem 2.3) and empirical results (Figure 4) demonstrate the synergy between feature homophily and label homophily in enhancing model performance. Based on these findings, future work could focus on designing various graph filters to optimize the objective in Eq. (8) by considering both label and feature homophily. Furthermore, feature homophily can explain the presence of node features, making it worthwhile to investigate how much features are influenced by their neighbors, particularly in temporal graphs.

F.2 Applications

Current real-world applications of graph homophily only focus on label homophily, which cannot align well with GNNs performance as shown in this paper and other studies [46, 39]. To address this weakness, our findings provide a comprehensive view of 3 types of homophily, which could be applied in many real-world applications, such as social networks, recommendation, and urban computing.

A. Social Networks

In social networks, homophily is defined as the tendency for people to seek out or be drawn to others who are similar to themselves [24]. This definition primarily explains the consistency of certain characteristics of people within the network topology. Our proposed concepts of structural homophily and feature homophily offer additional insights into social networks.

Structural homophily refers to the similarity of the local neighbors of individuals of the same type, which can be used to analyze the friend circles of specific user groups. For instance, in fraud detection on social media, fraudsters often target older individuals who are more vulnerable to scams, resulting in a high level of structural homophily. Therefore, we can identify potential fraudsters based on their structural connections. However, structural information can vary and may not always be informative. If fraudsters randomly select users to contact, identifying them through their neighbors becomes challenging, leading to a low level of structural homophily. Future research could focus on measuring the level of structural homophily in social networks to better understand user behaviors.

Feature homophily, on the other hand, describes how individuals are influenced by their neighbors. Different types of networks exhibit varying levels of feature homophily. When people discuss similar events online, their opinions may be influenced by those they follow, leading to a higher similarity in features with their neighbors, indicating a high level of feature homophily. Conversely, during online arguments, individuals connect with those holding different opinions. After such interactions, they may reinforce their original viewpoints, resulting in greater dissimilarity with their neighbors, indicating a low level of feature homophily. Investigating user behavior through feature homophily

can reveal underlying intentions and improve model performance. Furthermore, feature homophily provides valuable insights into the extent to which users are influenced by their neighbors.

B. Recommendation

Previous studies [22, 12] on recommendation systems using graph homophily have primarily focused on label homophily, which may result in misalignment with model performance, similar to what occurs in purely homogeneous graphs. To address this issue, it would be beneficial to define structural homophily within recommendation systems. For instance, we can measure the structural information of users within the same community by assessing the consistency of the items they have purchased. This approach allows us to determine whether this topological information can effectively predict links between users and specific items.

C. Urban Computing

A recent study [65] proposes a method to measure spatial graph homophily in urban computing using a spatial diversity score with direction-aware and distance-aware partitions. However, this metric focuses solely on label homophily, leaving a significant opportunity to explore structural homophily. Structural homophily involves measuring the consistency of geographic information among similar types of locations. For example, bookstores are often surrounded by coffee shops, where customers can enjoy coffee while reading books [44], indicating a high level of structural homophily. Conversely, convenience stores might be located near a high-end fashion boutique, a fast-food restaurant, or an office building. Since the geographic information does not reliably indicate the presence of a convenience store, it exhibits a low level of structural homophily. Future research could explore structural homophily in urban graphs to analyze the behaviors of various urban objects, aiding in better city planning.

G Proof of Theorems

We first specify the ranges of three types of homophily and spectral radius 8 of an adjacency matrix, which are used for the approximation in partial derivative process:

G.1 Proof of Theorem 1

Theorem 1. In CSBM-3H, the ratio of the expectation of intra-class distance to the expectation of inter-class distance of node representations for graph-agnostic models M ¬G and graph-aware models M G is:

̸

Proof . We have the objective

̸

Then, we derive the objective for graph-agnostic models M ¬G and graph-aware models M G respectively as follows

G.1.1 Graph-agnostic Models

For the graph-agnostic models, we let H u = X u . The structural-agnostic feature m in class c is sampled from a Gaussian distribution N c,m ( µ c,m , σ 2 c,m ) . We use N ∈ R C × M to denote the distributions of structural-aware for all the classes with mean µ ∈ R C × M and standard deviation σ 2 ∈ R C × M in the matrix form.

For the structural-aware feature m of node u , we have 9

Next, to simplify the expression, we let p 1 = 1 -h L C -1 + ϵ and p 0 = h L -1 -h L C -1 ,

Then we have the feature distribution of node u where

When two nodes u and v share same labels, i.e., Y u = Y v , we have

Then the expectation of intra-class distance of X can be expressed as

When two nodes u and v share different labels, i.e., Y u = Y v , we have

̸

Then the expectation of inter-class distance of X can be expressed as

̸

̸

̸

Then, we come back to the objective by combining Eq. (54) and Eq. (56)

̸

̸

Then we take back p 0 , p 1 , and ω with h L , h S , and h F

So we have

Then, we can rewrite the objective as

where

̸

̸

̸

G.1.2 Graph-aware Models

For graph-aware models, we have H u = 1 D u ∑ v ∈N ( u ) X v , where X is the structural-aware features as in Section G.1.1. For the representation m in node u , we have

We take the result of ( I C -ω S ) -1 N : ,m as shown in Eq. (50). Then we have

Then we have the distribution of the representation of node u

where

When two nodes u and v share same labels, i.e., Y u = Y v , we have

Then the expectation of intra-class distance of H can be expressed as

When two nodes u and v share different labels, i.e., Y u = Y v , we have

̸

Then the expectation of inter-class distance of H can be expressed as

̸

̸

Then, we come back to the objective

̸

̸

̸

Let's consider the term

We notice the second term (1 -ω 2 ( C E [ p 2 1 ]+ p 2 0 )) (1 -ωp 0 ) 2 = J ¬G h , thereby we simplify the expression as

Then we take back p 0 , p 1 , and ω with h L , h S , and h F as in Eq. (58)

Then, we can rewrite the objective as

̸

G.2 Proof of Theorem 3.1

Theorem 3.1. The partial derivative of J ¬G h with respect to label homophily h L satisfies,

Proof. For the graph-agnostic models, we take the partial derivative of J ¬G h with respect to h L

Take the approximations of T h, UP and T h, DOWN are given in Eq. (87) back to ∂ J ¬G h ∂h L , we have

It is easy to see 2 C [ ρ ( A )( C -1)] > 0 , so the sign of ∂ J ¬G h ∂h L is solely depend on h F . Therefore, we have

where

G.3 Proof of Theorem 3.2

Theorem 3.2. The partial derivative of J ¬G h with respect to structural homophily h S satisfies,

Proof. We take the partial derivative of J ¬G h with respect to h S

From Eq. (46), we have

Therefore, we can approximate ∂ J ¬G h ∂h S that satisfies

G.4 Proof of Theorem 3.3

Theorem 3.3. The partial derivative of J ¬G h with respect to feature homophily h F satisfies, where 0 < h -L < h + L < 1 and -1 < ˆ h F < 1 .

Proof. We take the partial derivative of J ¬G h with respect to h F

where

Next, we approximate T h, UP and T h, DOWN with Eq. (46)

Then take back to Eq. (86)

where

As a result, the sign of ∂ J ¬G h ∂h F satisfies the following conditions:

Then we discuss the influence of h L to ∂ J ¬G h ∂h F . If ˆ h F ≥ 1 , then ∂ J ¬G h ∂h F > 0 holds ∀ h F ∈ ( -1 , 1) . In this case, we have

The solution of this condition for h L ∈ [0 , 1] is h L > h + L , where

Conversely, If ˆ h F ≤ -1 , then ∂ J ¬G h ∂h F < 0 holds ∀ h F ∈ ( -1 , 1) . In this case, we have

The solution of this condition for h L ∈ [0 , 1] is h L < h -L , where

It is easy to show that 0 < h -L < h + L < 1 . So far, we already know that ∂ J ¬G h ∂h F > 0 when h L > h + L and ∂ J ¬G h ∂h F < 0 when h L < h -L for all h F ∈ ( -1 , 1) . Then, for h -L < h L < h + L , the sign of ∂ J ¬G h ∂h F is dependent on h F as shown in Eq. (88). In conclusion, we have

G.5 Proof of Theorem 2.1

Theorem 2.1. The partial derivative of J G h with respect to label homophily h L satisfies,

Proof. For the graph-aware models, we approximate the J G h with Eq. (46)

Then we take the partial derivative with respect to h L

We can see that the first term monotonically increases with respect to h L and the second term is always positive for h L ∈ [0 , 1] . So we have

G.6 Proof of Theorem 2.2

Theorem 2.2. The partial derivative of J G h with respect to structural homophily h S satisfies,

Proof. We take the partial derivative of J G h with respect to h F

From Eq. (46), we have

Together with J ¬G h > 0 and ∂ J ¬G h ∂h S ≥ 0 , we can approximate ∂ J G h ∂h S as

G.7 Proof of Theorem 2.3

Theorem 2.3. The partial derivative of J G h with respect to feature homophily h F satisfies,

where 0 < h -L < h + L < 1 and -1 < ˆ h F < 1 . The expressions and detailed calculation of h -L , h + L , and ˆ h F are shown in Appendix G.4.

Proof. For the graph-aware models, we take the partial derivative of J G h with respect to h F

Since the term ( h L C -1 C -1 ) 2 ( C ( 1 -h L C -1 ) 2 + C (1 -h S ) 2 C -1 +( h L C -1 C -1 ) 2 ) ≥ 0 , the sign of ∂ J G h ∂h F is the determined by ∂ J ¬G h ∂h F except for h L = 1 C , which leads to the similar results as graph-aware models

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We accurately answered the claims and contributions in the abstract and introduction, including why we shouldn't discard label homophily and the superiority of disentangled homophily.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discussed the limitations in Section 6

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: We provided the necessary assumptions for our proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: We provide the reproducible code of our experiments, including node classification and homophily metrics.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: All the datasets used in this paper are public. Our provided codes well reproduce the main experimental results.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are shown in Appendix E.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the accuracy and standard deviation of the node classification performance with 10 runs with random splits. We report the p-value for the Pearson correlation.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [No]

Justification: We only provide computer resources used in our experiments. We didn't report the memory and time of execution because the main focus of this paper is to disentangle graph homophily instead of proposing a new method.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: All the data used in this paper is public and the research doesn't violate the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This paper focuses on improving homophily measurement, which is irrelevant to societal impacts.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: This paper focuses on disentangling graph homophily, which is more conceptual and has no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We provided the necessary references for all the data used in this paper.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: We didn't release new assets

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper doesn't involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper doesn't involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper analyzes graph homophily and distinguishes three types of homophily: label, structural, and feature homophily. Theoretical analysis is based on CSBM-3H (contextual stochastic block model with three types of homophily).  Based on this analysis, a new combined measure Tri-Hom is proposed. The relation between GNN performance and three types of homophily is empirically analyzed on CSBM-3H. Then, the agreement of Tri-Hom with GNN performance is analyzed using several real datasets. It is shown that the proposed homophily measure better agrees with the GNN performance than other existing measures.\n\nSTRENGTHS:\n1. The proposed model CSBM-3H is more flexible and models more complicated relations between graph structure, node features, and node labels than other typically used models. Such a model can be used as a synthetic benchmark in research papers.\n\n2. Theoretical analysis shows how different homophily aspects relate to the distinguishability of aggregated and non-aggregated node features.\n\nWEAKNESSES:\n1. There seems to be a terminological inconsistency in the text. In the abstract, it is written that \"graph homophily refers to the phenomenon that connected nodes tend to share similar characteristics.\" However, structural homophily introduced in Section 3.2 is about a different phenomenon: nodes from the same class tend to have similar neighbors (but they can be not connected). In some previous papers, e.g. in [31], these concepts are distinguished, and LI is not called a homophily measure.\n\n2. The particular form of structural homophily (2) is not motivated. In Section 3.2, it is written that there are several existing structural homophily measures, but they are not chosen. Why the proposed expression (2) is better?\n\n3. Similarly, there is not much motivation for the feature homophily proposed in (3). More motivation on why this diffusion process is supposed to reflect homophily would be helpful. In particular, (3) seems to assume a particular form of feature propagation. It is unclear how limiting this assumption is.\n\n4. Throughout the paper, the class sizes are assumed to be balanced. In line 192, it is written that this is done without loss of generality, but it is not clear why so. In particular, several papers previously discussed that node/edge homophily is not a suitable measure for class-imbalanced datasets. In other words, class balance plays a crucial role for homophily measures and thus considering only balanced classes is a significant limitation. One of the examples where class balance seems to be critical is Theorem 2.2: here the threshold 1/C holds only for balanced classes.\n\nSome typos:\n- Line 12: \"element\" -> \"elements\"\n- Line 61: \"i.i.d.assumption\" — missing space\n- Line 62: \"Compared previous\" -> \"Compared to the previous\"\n- Line 69: \"In addition, Our\" -> \"In addition, our\"\n- Line 82: It is written that $I_E$ is a matrix, in this case, it should be from $R^{E\\times E}$\n\nQUESTIONS:\n1. In Section 3.1, many label homophily measures are introduced. Which one is chosen by the authors?\n\n2. In equation (2), how exactly $\\sigma_{max}$ is computed?\n\n3. Could you give more details/motivation about the difference between label and structural homophily (in general)? Since both are expected to describe the relation between structure and labels.\n\n4. In equation (5), it would be useful to discuss whether the obtained values $h$ are comparable across features of different natures. In particular, how these values behave when we do certain feature transformations: shifts, scaling, or changing variance. These would give more intuition about the proposed form of feature homophily.\n\n5. How performance is measured in Table 1?\n\nLIMITATIONS:\nSome of the limitations are discussed in Section 6. I agree with the authors that societal impact is irrelevant for this type of work.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper combines three different metrics (label, structural, and feature) for homophily proposing a Contextual Stochastic Block Model (CSBM-3H) describing these three types of homophily. By doing so it can control the topology and feature generation based on these three homophily metrics. Furthermore, there exists an extensive theoretical analysis of CSBM-3H, also defining new composite metric, named Tri-Hom, that considers all three homophily aspects and overcomes the limitations of the previously proposed and conventional homophily metrics. Importantly, they validate the correlation of model performance (under node classification) based on experimental set-up including 31 real-world benchmark datasets, outperforming various baseline metrics.\n\nSTRENGTHS:\n1) The paper is interesting, and combining all three homophily metrics into one unified metric is novel.\n\n2) The theoretical analysis of the proposed method is robust. The paper introduces three theorems that establish connections to important aspects verified in the literature, thereby unifying multiple works under a single framework.\n\n3) The proposed Contextual Stochastic Block Model, which incorporates three types of homophily, is innovative and could be highly useful for benchmarking general models under varying types and levels of homophily.\n\n4) The experimental section is extensive, including 31 datasets, and yields promising results.\n\nWEAKNESSES:\n1) The paper validates performance only under the node classification task. It would be very interesting to also validate Tri-Hom for the task of link prediction.\n\n2) A minor weakness is that some results in Table 1 for certain baselines are quite close, with only marginal improvements. Could the authors comment on this phenomenon? Why might this be the case, and could it indicate that for some models or networks, having only one homophily metric would be sufficient?\n\nQUESTIONS:\nWould it be possible to visualize the generated SBM networks under different levels of homophily and observe the resulting block structures with respect to all different types of homophily?\n\nLIMITATIONS:\nThe authors should discuss limitations of their paper and proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper proposes a novel approach to understanding graph homophily by disentangling it into label, structural, and feature homophily. The introduction of the Tri-Hom metric combines these aspects to provide a more comprehensive measure of GNN performance. CSBM-3H is used to study the impact of these types of homophily. Extensive experimental results on synthetic and real-world datasets show the effectiveness of the findings.

STRENGTHS:
1. The paper takes a comprehensive thought on graph homophily and provides with a more detailed understanding of homophily.
2. The introduction of Tri-Hom metric as well as and the CSBM-3H model which is based on the metric is novel.
3. The paper includes both theoretical analysis and empirical validation through synthetic and real-world datasets.

WEAKNESSES:
The introduction of multiple new concepts and metrics might be overwhelming for readers who are not well-versed in graph theory and GNNs. Simplifying explanations or providing more intuitive examples could help.

QUESTIONS:
1. The definitions of structural and feature homophily are somewhat abstract. Could you provide more concrete examples or case studies to illustrate these concepts?
2. How to translate the finding to real-world applications?

LIMITATIONS:
Lack concrete examples for readers’ understanding.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2024-07-11

comments: SUMMARY:
This paper evaluates graph homophily from the perspectives of label, structure, and feature, which disentangle the dependencies of these three aspects. The theoretical analysis and experimental evaluations demonstrate the effectiveness of Tri-Hom.

STRENGTHS:
1. This paper is innovative and significant in evaluating graph properties from their central components, namely, label, feature, and structure.
2. This paper highlight the missing component on evaluating the graph homophily, marking it as novel compared to existing methods for designing graph learning methods and new evaluation metrics.
3. There have been extensive experiments conducted on synthetic and real-world datasets.

WEAKNESSES:
1. This paper lacks suggestions for designing models. New metrics are provided for evaluating graph homophily properties. However, as new models are continually proposed, it would be beneficial if the authors could provide some guidelines for designing models when tackling new datasets. 

2. The connections and distinctions between the proposed metrics and the existing metrics from the label, feature, and structure perspectives need further clarification.

QUESTIONS:
None

LIMITATIONS:
please check the weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-11