Is Homophily a Necessity for Graph Neural Networks?

1 INTRODUCTION Graph neural networks ( GNNs ) are a prominent approach for learning representations for graph structured data . Thanks to their great capacity in jointly leveraging attribute and graph structure information , they have been widely adopted to promote improvements for numerous graph-related learning tasks ( Kipf and Welling , 2016 ; Hamilton et al. , 2017 ; Ying et al. , 2018 ; Fan et al. , 2019 ; Zitnik et al. , 2018 ) , especially centered around node representation learning and semi-supervised node classification ( SSNC ) . GNNs learn node representations by a recursive neighborhood aggregation process , where each node aggregates and transforms features from its neighbors . The node representations can then be utilized for downstream node classification or regression tasks . Due to this neighborhood aggregation mechanism , several existing works posit that many GNNs implicitly assume strong homophily and homophily is critical for GNNs to achieve strong performance on SSNC ( Zhu et al. , 2020b ; a ; Chien et al. , 2021 ; Maurya et al. , 2021 ; Halcrow et al. , 2020 ; Lim et al. , 2021 ) . In general , homophily describes the phenomenon that nodes tend to connect with “ similar ” or “ alike ” others . Homophily is observed in a wide range of real-world graphs including friendship networks ( McPherson et al. , 2001 ) , political networks ( Gerber et al. , 2013 ; Newman , 2018 ) , citation networks ( Ciotti et al. , 2016 ) and more . Under the homophily assumption , through the aggregation process , a node ’ s representation is “ smoothed ” via its neighbors ’ representations , since each node is able to receive additional information from neighboring nodes , which are likely to share the same label . Several recent works ( Zhu et al. , 2020b ; a ) claim that GNNs are implicitly ( or explicitly ) designed with homophily in mind , are not suitable for graphs exhibiting heterophily , where connected nodes are prone to have different properties or labels , e.g dating networks or molecular networks ( Zhu et al. , 2020b ) . Such works accordingly design and modify new architectures and demonstrate outperformance over other GNN models on several heterophilous graphs . Present work . In our work , we empirically find that the graph convolutional network ( GCN ) Kipf and Welling ( 2016 ) , a fundamental , representative GNN model ( which we focus on in this work ) is actually able to outperform such heterophily-specific models on some heterophilous graphs after careful hyperparameter tuning . This motivates us to reconsider the popular notion in the literature that GNNs exhibit a homophilous inductive bias , and more specifically that strong homophily is crucial to strong GNN performance . Counter to this idea , we find that GCN model has the potential to work well for heterophilous graphs under suitable conditions . We demonstrate intuition with the following toy example : Consider the perfectly heterophilous graph ( with all inter-class edges ) shown in Figure 1 , where the color indicates the node label . Blue-labeled and orange-labeled nodes are associated with the scalar feature 0 and 1 , respectively . If we consider a single-layer GCN by performing an averaging feature aggregation over all neighboring nodes , it is clear that all blue nodes will have a representation of 1 , while the orange nodes will have that of 0 . Additional layers/aggregations will continue to alternate the features between the two types of nodes . Regardless of the number of layers , the two classes can still be perfectly separated . In this toy example , each blue ( orange ) node only connects orange ( blue ) nodes , and all blue ( orange ) nodes share similar neighborhood patterns in terms of their neighbors ’ label/feature distributions . Our work elucidates this intuition and extends it to a more general case : put simply , given a ( homophilous or heterophilous ) graph , GCN has the potential to achieve good performance if nodes with the same label share similar neighborhood patterns . We theoretically support this argument by investigating the learned node embeddings from the GCN model . We find that homophilous graphs always satisfy such assumptions , which explains why GCN typically works well for them . On other hand , there exist both “ good ” and “ bad ” heterophily , and GCNs can actually achieve strong performance for “ good ” heterophily settings while they usually fail on “ bad ” heterophily settings . Our work characterizes these settings , and provides a new perspective and solid step towards deeper understanding for heterophilous graphs . In short : Our contributions . ( 1 ) We reveal that strong homophily is not a necessary assumption for the GCN model . The GCN model can perform well over some heterophilous graphs under certain conditions . ( 2 ) We carefully characterize these conditions and provide theoretical understandings on how GCNs can achieve good SSNC performance under these conditions by investigating their embedding learning process . ( 3 ) We carefully investigate commonly used homophilous and heterophilous benchmarks and reason about GCN ’ s performs on them utilizing our theoretical understanding . 2 PRELIMINARIES . Let G = { V , E } denote a graph , where V and E are the sets of nodes and edges , respectively . The graph connection information can also be represented as an adjacency matrix A ∈ { 0 , 1 } |V|×|V| , where |V| is the number of nodes in the graph . The i , j-th element of the adjacency matrix A [ i , j ] is equal to 1 if and only if nodes i and j are adjacent to each other , otherwise A [ i , j ] = 0 . Each node i is associated with a l-dimensional vector of node features xi ∈ Rl ; the features for all nodes can be summarized as a matrix X ∈ R|V|×l . Furthermore , each node i is associated with a label yi ∈ C , where C denotes the set of labels . We also denote the set of nodes with a given label c ∈ C as Vc . We assume that labels are only given for a subset of nodes Vlabel ⊂ V . The goal of semi-supervised node classification ( SSNC ) is to learn a mapping f : V → C utilizing the graph G , the node features X and the labels for nodes in Vlabel . 2.1 HOMOPHILY IN GRAPHS . In this work , we focus on investigating performance in the context of graph homophily and heterophily properties . Homophily in graphs is typically defined based on similarity between connected node pairs , where two nodes are considered similar if they share the same node label . The homophily ratio is defined based on this intuition following Zhu et al . ( 2020b ) . Definition 1 ( Homophily ) . Given a graph G = { V , E } and node label vector y , the edge homophily ratio is defined as the fraction of edges that connect nodes with the same labels . Formally , we have : h ( G , { yi ; i ∈ V } ) = 1 |E| ∑ ( j , k ) ∈E 1 ( yj = yk ) , ( 1 ) where |E| is the number of edges in the graph and 1 ( · ) is the indicator function . A graph is typically considered to be highly homophilous when h ( · ) is large ( typically , 0.5 ≤ h ( · ) ≤ 1 ) , given suitable label context . On the other hand , a graph with a low edge homophily ratio is considered to be heterophilous . In future discourse , we write h ( · ) as h when discussing given a fixed graph and label context . 2.2 GRAPH NEURAL NETWORKS . Graph neural networks learn node representations by aggregating and transforming information over the graph structure . There are different designs and architectures for the aggregation and transformation , which leads to different graph neural network models ( Scarselli et al. , 2008 ; Kipf and Welling , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2017 ; Gilmer et al. , 2017 ; Zhou et al. , 2020 ) . One of the most popular and widely adopted GNN models is the graph convolutional network ( GCN ) . A single GCN operation takes the following form H′ = D−1AHW , where H and H′ denote the input and output features of layer , W ( k ) ∈ Rl×l is a parameter matrix to transform the features , and D is a diagonal matrix and D [ i , i ] = deg ( i ) with deg ( i ) denoting the degree of node i . From a local perspective for node i , the process can be written as a feature averaging process hi = 1 deg ( i ) ∑ j∈N ( i ) Wxj , where N ( i ) denotes the neighbors of node i . The neighborhood N ( i ) may contain the node i itself . Usually , when building GCN model upon GCN operations , nonlinear activation functions are added between consecutive GCN operations . 3 GRAPH CONVOLUTIONAL NETWORKS UNDER HETEROPHILY . Considerable prior literature posits that graph neural networks ( such as GCN ) work by assuming and exploiting homophily assumptions in the underlying graph ( Maurya et al. , 2021 ; Halcrow et al. , 2020 ; Wu et al. , 2018 ) . To this end , researchers have determined that such models are considered to be ill-suited for heterophilous graphs , where the homophily ratio is low ( Zhu et al. , 2020b ; a ; Chien et al. , 2021 ) . To deal with this limitation , researchers proposed several methods including H2GNN ( Zhu et al. , 2020b ) , CPGNN ( Zhu et al. , 2020a ) and GPRGNN ( Chien et al. , 2021 ) , which are explicitly designed to handle heterophilous graphs via architectural choices ( e.g . adding skipconnections , carefully choosing aggregators , etc . ) In this section , we revisit the claim that GCNs have fundamental homophily assumptions and are not suited for heterophilous graphs . To this end , we first observe empirically that the GCN model achieves fairly good performance on some of the commonly used heterophilous graphs ; specifically , we present SSNC performance on two commonly used heterophilous graph datasets , Chameleon and Squirrel in Table 1 ( see Appendix D for further details about the datasets and models ) . Both Chameleon and Squirrel are highly heterophilous ( h≈0.2 ) . We find that with some hyperparameter tuning , GCN can outperform alternative methods uniquely designed to operate on some certain heterophilous graphs . This observation suggests that GCN does not always “ underperform ” on heterophilous graphs , and it leads us to reconsider the prevalent assumption in literature . Hence , we next examine how GCNs learn representations , and how this information is used in downstream SSNC tasks . 3.1 WHEN DOES GCN LEARN SIMILAR EMBEDDINGS FOR NODES WITH THE SAME LABEL ? . GCN is considered to be unable to tackle heterophilous graphs due to its feature averaging process ( Zhu et al. , 2020b ; Chien et al. , 2021 ) . Namely , a node ’ s newly aggregated features are considered “ corrupted ” by those neighbors that do not share the same label , leading to the intuition that GCN embeddings are noisy and un-ideal for SSNC . However , we find that crucially , for some heterophilous graphs , the features of nodes with the same label are “ corrupted in the same way. ” Hence , the obtained embeddings still contain informative characteristics and thus facilitate SSNC . We next illustrate when GCN learns similar embeddings for nodes with the same label , beginning with a toy example and generalizing to more practical cases . a b Figure 2 : Two nodes share the same neighborhood distribution ; GCN learns equivalent embeddings for a and b. GCNs have been shown to be able to capture the local graph topological and structural information ( Xu et al. , 2019 ; Morris et al. , 2019 ) . Specifically , the aggregation step in the GCN model is able to capture and discriminate neighborhood distribution information , e.g . the mean of the neighborhood features ( Xu et al. , 2019 ) . Let us con- sider the two nodes a and b shown in Figure 2 , where we use color to indicate the label of each node . If we further assume that all nodes sharing the same label are associated with exactly the same features , then clearly , after 1-step aggregation , the GCN operation will output exactly the same embedding for nodes a and b . Accordingly , Xu et al . ( 2019 ) reasons that the GCN model lacks expressiveness due to its inability to differentiate the two nodes in the embedding space . However , in the SSNC task , mapping a and b to the same location in the embedding space is explicitly desirable . Intuitively , if all nodes with the same label are mapped to the same embedding and embeddings for different labels are distinct , SSNC is effortless ( Zhao et al. , 2020 ) . Such assumptions are hard to meet in practice . Thus , to consider a more practical scenario , we assume that both features and neighborhood patterns for nodes with a certain label are sampled from some fixed distributions . Under these conditions , same-label nodes may not share fixed embeddings , but we can aim to characterize their closeness . Intuitively , if the learned embeddings for same-label nodes are close and embeddings for other-label nodes are far , we expect strong SSNC performance to be good , given class separability ( low intra-class variance and high inter-class variance ) ( Fisher , 1936 ) . We prove that , for graphs meeting suitable conditions the distance between GCN-learned embeddings of any same-label node pair is bounded by a small quantity with high probability . Assumptions on Graphs . We consider a graph G , where each node i has features xi ∈ Rl and label yi . We assume that ( 1 ) The features of node i are sampled from feature distribution Fyi , i.e , xi ∼ Fyi , with µ ( Fyi ) denoting its mean ; ( 2 ) Dimensions of xi are independent to each other ; ( 3 ) The features in X are bounded by a positive scalar B , i.e , maxi , j |X [ i , j ] | ≤ B ; ( 4 ) For node i , its neighbor ’ s labels are independently sampled from neighbor distribution Dyi . The sampling is repeated for deg ( i ) times to sample the labels for deg ( i ) neighbors . We denote a graph following these assumptions ( 1 ) - ( 4 ) as G = { V , E , { Fc , c ∈ C } , { Dc , c ∈ C } } . Note that we use the subscripts in Fyi and Dyi to indicate that these two distributions are shared by all nodes with the same label as node i . Next , we analyze the embeddings obtained after a GCN operation . Following previous works ( Li et al. , 2018 ; Chen et al. , 2020 ; Baranwal et al. , 2021 ) , we drop the non-linearity in the analysis . Theorem 1 . Consider a graph G = { V , E , { Fc , c ∈ C } , { Dc , c ∈ C } } , which follows Assumptions ( 1 ) - ( 4 ) . For any node i ∈ V , the expectation of the pre-activation output of a single GCN operation is given by E [ hi ] = W ( Ec∼Dyi , x∼Fc [ x ] ) . ( 2 ) and for any t > 0 , the probability that the distance between the observation hi and its expectation is larger than t is bounded by P ( ‖hi − E [ hi ] ‖2 ≥ t ) ≤ 2 · l · exp ( − deg ( i ) t 2 2ρ2 ( W ) B2l ) , ( 3 ) where l denotes the feature dimensionality and ρ ( W ) denotes the largest singular value of W. The detailed proof can be found in Appendix A. Theorem 1 demonstrates two key ideas . First , in expectation , all nodes with the same label have the same embedding ( Eq . ( 2 ) ) . Second , the distance between the output embedding of a node and its expectation is small with a high probability . Specifically , this probability is related to the node degree and higher degree nodes have higher probability to be close to the expectation . Together , these results show that the GCN model is able to map nodes with the same label to an area centered around the expectation in the embedding space under given assumptions . Then , the downstream classifier in the GCN model is able to assign these nodes to the same class with high probability . To ensure that the classifier achieves strong performance , the centers ( or the expectations ) of different classes must be distant from each other ; if we assume that µ ( Fyi ) are distinct from each other ( as is common ) , then the neighbor distributions { Dc , c ∈ C } must be distinguishable to ensure good SSNC performance . Based on these understandings and discussions , we have the following key ( informal ) observations on GCN ’ s performance for graphs with homophily and heterophily . Observation 1 ( GCN under Homophily ) . In homophilous graphs , the neighborhood distribution of nodes with the same label ( w.l.o.g c ) can be approximately regarded as a highly skewed discrete Dc , with most of the mass concentrated on the category c. Thus , different labels clearly have distinct distributions . Hence , the GCN model typically in SSNC on such graph , with high degree nodes benefiting more , which is consistent with previous work ( Tang et al. , 2020b ) . Observation 2 ( GCN under Heterophily ) . In heterophilous graphs , if the neighborhood distribution of nodes with the same label ( w.l.o.g . c ) is ( approximately ) sampled from a fixed distribution Dc , and different labels have distinguishable distributions , then GCN can excel at SSNC , especially when node degrees are large . Otherwise , GCNs may fail for heterophilous graphs . Notably , our findings illustrate that disruptions of certain conditions inhibit GCN performance on heterophilous graphs , but heterophily is not a sufficient condition for poor GCN performance . GCNs are able to achieve reasonable performance for both homophilous and heterophilous graphs if they follow certain assumptions as discussed in the two observations . In Section 3.2 , we theoretically demonstrate these observations for graphs sampled from the Contextual Stochastic Block Model ( CSBM ) ( Deshpande et al. , 2018 ) with two classes , whose distinguishablilty of neighborhood distributions can be explicitly characterized . Furthermore , in Section 3.3 , we empirically demonstrate these observations on graphs with multiple classes . We note that although our derivations are for GCN , a similar line of analysis can be used for more general message-passing neural networks .

Summary: The paper revisited the performance of GCN on graph with heterophily and provide negative evidence that heterophily does not always result in the poor performance of GCN, which contradicts with the assumptions/observations of many previous papers. They demonstrated that the GCN embeddings are still label-distinguishable on a special type of graphs with assumptions that the nodes with same labels have the same node feature distribution as well as the same neighborhood label distributions. They theoretically analyzed the case of CSBM model with two classes, and also empirically investigated on synthetic/real-world graphs with multiple classes. The conclusion is that GCNs can achieve good performance on heterophilous graphs under certain conditions.