Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks

1 INTRODUCTION . In recent years , GCNs ( Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Veličković et al. , 2017 ) have been widely used in applications ranging from social science ( Li & Goldwasser , 2019 ) and biology ( Yan et al. , 2019 ) to program understanding ( Allamanis et al. , 2018 ; Shi et al. , 2019 ) . A typical GCN architecture ( Gilmer et al. , 2017 ) for the node classification task can be decomposed into two main components : propagation/aggregation , and combination . Messages are first exchanged between neighboring nodes , then aggregated . Afterwards , the messages are combined with the self-representations ( a.k.a. , the current node representations ) to update the node representations . Though GCNs are effective , they have some key limitations . The first limitation is the “ oversmoothing '' problem ( Li et al. , 2018 ) : the performance of GCNs degrade when stacking many layers . Recent work has found that oversmoothing could be caused by GCNs exponentially losing expressive power in the node classification task ( Oono & Suzuki , 2019 ) and that the node representations converge to a stationary state which is decided by the degree of the nodes and the input features ( Chen et al. , 2020 ; Wang et al. , 2019 ; Rong et al. , 2019 ; Rossi et al. , 2020 ) . These works focus on the analysis of the steady state in the limit of infinite layers , but they do not explore the dynamics on how oversmoothing is triggered , or which nodes tend to cause it . Other works like ( Xu et al. , 2018 ; Wang et al. , 2019 ; Chen et al. , 2020 ) focus on heuristic-based model designs to alleviate the oversmoothing problem . To fill this gap , we seek to theoretically analyze the dynamics around oversmoothing . In our empirical analysis , by dividing the nodes in real graphs into several degree groups and tracking their average accuracy while increasing the layers of GCN ( Kipf & Welling , 2016 ) , we observe two phases ( Fig . 1 ) : the initial stage when higher-degree nodes have higher accuracy and the developing stage when the accuracy of higher-degree nodes decreases more sharply ( cf . § 5.4 for details ) . As we will show theoretically and empirically , the low-degree nodes are the ones that trigger oversmoothing . Though Chen et al . ( 2020 ) provide some explanations for the developing stage by analyzing the convergence rate , their theory fails to characterize the initial stage . In fact , we find that the analysis of initial stage provides valuable insights about oversmoothing . The second limitation of GCNs is their poor performance on heterophilous graphs ( Pei et al. , 2019 ; Lim et al. , 2021 ) , which—unlike homophilous graphs—have many neighboring nodes that belong to different classes ( Newman , 2002 ) . For instance , in protein networks , amino acids of different types tend to form links ( Zhu et al. , 2020 ) , and in transaction networks , fraudsters are more likely to connect to accomplices than to other fraudsters ( Pandit et al. , 2007 ) . Most GCNs ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ) fail to effectively capture heterophily , so emerging works have proposed some effective designs ( Pei et al. , 2019 ; Zhu et al. , 2020 ) , including signed messages ( positive and negative interactions between nodes ) proposed by concur- rent works ( Chien et al. , 2021 ; Bo et al. , 2021 ) . Though these works justify signed messages from a spectral perspective , they focus on asymptotic results , requiring an infinite number of filters . Our work does not make this assumption , and our theory works in more practical scenarios ( § 3.4 ) . These two limitations have mostly been studied independently , but recent work on oversmoothing ( Chen et al. , 2020 ) was shown empirically to address heterophily , and vice versa ( Chien et al. , 2021 ) . Motivated by this empirical observation , we set out to understand the theoretical connection between the oversmoothing and heterophily problems by studying the change in node representations during message passing . Specifically , we make the following contributions : • Theory : We introduce the first theory that explains the connections between heterophily and oversmoothing which opens up the possibility of jointly studying the two problems . • Insights : We find that under conditions , the relative degree of a node ( compared to its neighbors ) and its heterophily level affect the misclassification rate of node labels in the oversmoothing and heterophily problems . We further prove that signed messages can mitigate their negative impact . • Improved model & empirical analysis : Based on these insights , we design a generalized GCN model , GGCN , that allows negative interactions between nodes and compensates for the effect of low-degree nodes through a learned rescaling scheme . Our empirical results show that our model is robust to oversmoothing , achieves state-of-the-art performance on datasets with high levels of heterophily , and achieves competitive performance on homophilous datasets . 2 PRELIMINARIES . We first provide the notations & definitions that we use in the paper , and a brief background on GCNs . Notation . We denote an unweighted and self-loop-free graph as G ( V , E ) and its adjacency matrix as A . We represent the degree of node vi ∈ V by di , and the degree matrix—which is a diagonal matrix whose elements are node degrees—by D. Let Ni be the set of nodes directly connected to vi , i.e. , its neighbors . I is the identity matrix . We denote the node representations at l-th layer as F ( l ) , and the i-th row of F is f ( l ) i , which is the representation of node vi . The input node features are given by F ( 0 ) . The weight matrix and bias vector at the l-th layer are denoted as W ( l ) and b ( l ) , respectively . Homophily and Heterophily . Given a set of node labels/classes , homophily captures the tendency of a node to have the same class as its neighbors . Specifically , the homophily of node vi is defined as hi ≡ E ( |N si | |Ni| ) , where N si is the set of neighboring nodes with the same label as vi , | · | is the cardinality operator , and the expectation is taken over the randomness of the node labels . High homophily corresponds to low heterophily , and vice versa , so we use these terms interchangeably . Supervised Node Classification Task . We focus on node classification : given a subset of labeled nodes ( from a label set L ) , the goal is to learn a mapping F : f ( 0 ) i 7→ yi between each node vi and its ground truth label yi ∈ L. We use ŷi to represent the predicted label of vi and define the misclassification rate of a fixed learnt model over a node set V as : E ( | { vi|ŷi 6=yi , vi∈V } | |V| ) . The expectation is taken over the randomness of graph structures and input node features . Graph Convolutional Neural Networks . In node classification tasks , an L-layer GCN contains two components ( Gilmer et al. , 2017 ) : ( 1 ) neighborhood propagation and aggrega- tion : f̂ ( l ) i = AGGREGATE ( f ( l ) j , vj ∈ Ni ) , and ( 2 ) combination : f ( l+1 ) i = COMBINE ( f̂ ( l ) i , f ( l ) i ) , where AGGREGATE and COMBINE are learnable functions . The loss is given by LCE=CrossEntropy ( Softmax ( f ( L ) i W ( L ) + b ( L ) ) , yi ) . The vanilla GCN suggests a renormalization trick on the adjacency A to prevent gradient explosion ( Kipf & Welling , 2016 ) . The ( l + 1 ) -th output is given by : F ( l+1 ) = σ ( ÃF ( l ) W ( l ) ) , where Ã = D̃−1/2 ( I + A ) D̃−1/2 , D̃ is the degree matrix of I + A , and σ is ReLU . When the non-linearities in the vanilla GCN are removed , it reduces to a linear model called SGC ( Wu et al. , 2019 ) , which has competitive performance and is widely used in theoretical analyses ( Oono & Suzuki , 2019 ; Chen et al. , 2020 ) . For SGC , the l-th layer representations are given by : F ( l ) = ÃlF ( 0 ) and the last layer is a logistic-regression layer : ŷi = Softmax ( F ( L ) W ( L ) + b ( L ) ) . We note that only one weight matrix W ( L ) is learned ; it is equivalent to the products of all weight matrices in a linear GCN . Find more related works in § 6 . 3 THEORETICAL ANALYSIS . In this section , we formalize the connection between the oversmoothing and heterophily problems in two steps : ( 1 ) We show how the movements of node representations relate to the misclassification rate of node labels ; and ( 2 ) we identify the factors that affect the movements of representations . We use the SGC model to analyze binary node classification with nodes in class 1 denoted as set V1 and nodes in class 2 denoted as set V2 . We show empirically ( § 5 ) that the insights obtained in this section are effective for other non-linear models in multi-class classification . 3.1 ASSUMPTIONS . Next , we use “ i.d. ” to represent random variables/vectors that follow the same marginal distribution and their joint probability density function ( PDF ) is a permutation-invariant function p ( x1 , . . . , xn ) = p ( P ( x1 , . . . , xn ) ) , where P ( · ) means permutation . ( 1 ) Random graph : Node degrees { di } are i.d . random variables . { ( · ) i } represents a set with i = 1 , . . . , |V| . ( 2 ) Inputs : ( 2.1 ) Node labels { yi } are i.d . Bernoulli random variables given by the ratio ρ : ρ ≡ P ( yi=1 ) P ( yi=2 ) , ∀i . ( 2.2 ) Initial input node features { f ( 0 ) i } are i.d . random vectors given by ( PDF ) f ( x ) , which is expressed as : f ( x ) = { f1 ( x ) , when yi = 1. f2 ( x ) , when yi = 2 . E ( f ( 0 ) i |yi ) = { µ , when yi = 1 . −ρµ , when yi = 2. so E ( f ( 0 ) i ) = 0 . ( 3 ) Independence : { di } are independent of { yi } and { f ( 0 ) i } . 3.2 MISCLASSIFICATION RATE & MOVEMENTS OF NODE REPRESENTATIONS . The following lemmas illustrate why the movements of node representations are a good indicator of SGC ’ s performance . We view the changes of node representations across the layers as their movements ; i.e. , f ( l+1 ) i is moved from f ( l ) i , ∀i . The misclassification rate of an L-layer SGC is determined by { f ( L ) i } , which is the input to the last regression layer , and can be studied through the decision boundary of f ( L ) i . Lemma 1 In SGC , the decision boundaries w.r.t . { f ( L ) i } in multi-class classification are linear hyperplanes . In particular , the decision boundary is a single hyperplane for binary classification . Proof . We provide the proof in App . A.1 . The decision boundary of an ( L + 1 ) -layer SGC can be viewed as a perturbation to the decision boundary of an L-layer SGC due to the movements from { f ( L ) i } to { f ( L+1 ) i } . Due to Lemma 1 , we can use the hyperplane xw + b = 0 to represent the decision boundary of { f ( L ) i } , where w is a unit vector , b is a scalar , and the { f ( L ) i } are classified to class 1 if f ( L ) i w + b > 0 . Suppose when w = w∗ and b = b∗ , we find an optimal hyperplane xw∗ + b∗ = 0 that achieves the lowest total misclassification rate ( § 2 ) for { f ( L ) i } . Due to the movements of node representations , the new optimal hyperplane is decided by : xw∗′ + b∗′ = 0 . In Lemma 2 , we show that , under constraints , moving towards the original decision boundary with a non-zero step results in a non-decreasing total misclassification rate under the new decision boundary . Lemma 2 Moving representations { f ( L ) i } from class 1 by adding −tw∗ T , t > 0 , s.t . w∗Tw∗′ > 0 , the new total misclassification rate is no less than the misclassification rate before the movements . Proof . We provide the proof in App . A.2 . Note that the special case where the representations of the two classes swap positions ( e.g , bipartite graphs ) violates the condition w∗Tw∗′ > 0 and leads to different conclusions . We refer to the condition w∗Tw∗′ > 0 as `` non-swapping condition '' and throughout the paper , we analyze SGC under that . Lemma 2 shows that , under the `` non-swapping condition '' , if { f ( l ) i } move towards the original decision boundary ( or the other class ) , SGC tends to perform worse .

Summary: The paper studies two problems plaguing graph neural networks (GCNs/GNNs): 1) oversmoothing of GCNs/GNNs; and 2) GCNs/GNNs often do not yield good performance on heterophilous networks. The authors aim to develop a theory to explain these issues simultaneously, where a few lemmas and theorems were developed based on certain simplifying assumptions. Based on these theoretical results, the authors develop a generalized GCN (GGCN) with negative message passing. Evaluation is conducted and compared with the state-of-the-art.