Two Sides of the Same Coin: Heterophily and Oversmoothing in Graph Convolutional Neural Networks

1 INTRODUCTION . In recent years , GCNs ( Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Veličković et al. , 2017 ) have been widely used in applications ranging from social science ( Li & Goldwasser , 2019 ) and biology ( Yan et al. , 2019 ) to program understanding ( Allamanis et al. , 2018 ; Shi et al. , 2019 ) . A typical GCN architecture ( Gilmer et al. , 2017 ) for the node classification task can be decomposed into two main components : propagation/aggregation , and combination . Messages are first exchanged between neighboring nodes , then aggregated . Afterwards , the messages are combined with the self-representations ( a.k.a. , the current node representations ) to update the node representations . Though GCNs are effective , they have some key limitations . The first limitation is the “ oversmoothing '' problem ( Li et al. , 2018 ) : the performance of GCNs degrade when stacking many layers . Recent work has found that oversmoothing could be caused by GCNs exponentially losing expressive power in the node classification task ( Oono & Suzuki , 2019 ) and that the node representations converge to a stationary state which is decided by the degree of the nodes and the input features ( Chen et al. , 2020 ; Wang et al. , 2019 ; Rong et al. , 2019 ; Rossi et al. , 2020 ) . These works focus on the analysis of the steady state in the limit of infinite layers , but they do not explore the dynamics on how oversmoothing is triggered , or which nodes tend to cause it . Other works like ( Xu et al. , 2018 ; Wang et al. , 2019 ; Chen et al. , 2020 ) focus on heuristic-based model designs to alleviate the oversmoothing problem . To fill this gap , we seek to theoretically analyze the dynamics around oversmoothing . In our empirical analysis , by dividing the nodes in real graphs into several degree groups and tracking their average accuracy while increasing the layers of GCN ( Kipf & Welling , 2016 ) , we observe two phases ( Fig . 1 ) : the initial stage when higher-degree nodes have higher accuracy and the developing stage when the accuracy of higher-degree nodes decreases more sharply ( cf . § 5.4 for details ) . As we will show theoretically and empirically , the low-degree nodes are the ones that trigger oversmoothing . Though Chen et al . ( 2020 ) provide some explanations for the developing stage by analyzing the convergence rate , their theory fails to characterize the initial stage . In fact , we find that the analysis of initial stage provides valuable insights about oversmoothing . The second limitation of GCNs is their poor performance on heterophilous graphs ( Pei et al. , 2019 ; Lim et al. , 2021 ) , which—unlike homophilous graphs—have many neighboring nodes that belong to different classes ( Newman , 2002 ) . For instance , in protein networks , amino acids of different types tend to form links ( Zhu et al. , 2020 ) , and in transaction networks , fraudsters are more likely to connect to accomplices than to other fraudsters ( Pandit et al. , 2007 ) . Most GCNs ( Kipf & Welling , 2016 ; Veličković et al. , 2017 ) fail to effectively capture heterophily , so emerging works have proposed some effective designs ( Pei et al. , 2019 ; Zhu et al. , 2020 ) , including signed messages ( positive and negative interactions between nodes ) proposed by concur- rent works ( Chien et al. , 2021 ; Bo et al. , 2021 ) . Though these works justify signed messages from a spectral perspective , they focus on asymptotic results , requiring an infinite number of filters . Our work does not make this assumption , and our theory works in more practical scenarios ( § 3.4 ) . These two limitations have mostly been studied independently , but recent work on oversmoothing ( Chen et al. , 2020 ) was shown empirically to address heterophily , and vice versa ( Chien et al. , 2021 ) . Motivated by this empirical observation , we set out to understand the theoretical connection between the oversmoothing and heterophily problems by studying the change in node representations during message passing . Specifically , we make the following contributions : • Theory : We introduce the first theory that explains the connections between heterophily and oversmoothing which opens up the possibility of jointly studying the two problems . • Insights : We find that under conditions , the relative degree of a node ( compared to its neighbors ) and its heterophily level affect the misclassification rate of node labels in the oversmoothing and heterophily problems . We further prove that signed messages can mitigate their negative impact . • Improved model & empirical analysis : Based on these insights , we design a generalized GCN model , GGCN , that allows negative interactions between nodes and compensates for the effect of low-degree nodes through a learned rescaling scheme . Our empirical results show that our model is robust to oversmoothing , achieves state-of-the-art performance on datasets with high levels of heterophily , and achieves competitive performance on homophilous datasets . 2 PRELIMINARIES . We first provide the notations & definitions that we use in the paper , and a brief background on GCNs . Notation . We denote an unweighted and self-loop-free graph as G ( V , E ) and its adjacency matrix as A . We represent the degree of node vi ∈ V by di , and the degree matrix—which is a diagonal matrix whose elements are node degrees—by D. Let Ni be the set of nodes directly connected to vi , i.e. , its neighbors . I is the identity matrix . We denote the node representations at l-th layer as F ( l ) , and the i-th row of F is f ( l ) i , which is the representation of node vi . The input node features are given by F ( 0 ) . The weight matrix and bias vector at the l-th layer are denoted as W ( l ) and b ( l ) , respectively . Homophily and Heterophily . Given a set of node labels/classes , homophily captures the tendency of a node to have the same class as its neighbors . Specifically , the homophily of node vi is defined as hi ≡ E ( |N si | |Ni| ) , where N si is the set of neighboring nodes with the same label as vi , | · | is the cardinality operator , and the expectation is taken over the randomness of the node labels . High homophily corresponds to low heterophily , and vice versa , so we use these terms interchangeably . Supervised Node Classification Task . We focus on node classification : given a subset of labeled nodes ( from a label set L ) , the goal is to learn a mapping F : f ( 0 ) i 7→ yi between each node vi and its ground truth label yi ∈ L. We use ŷi to represent the predicted label of vi and define the misclassification rate of a fixed learnt model over a node set V as : E ( | { vi|ŷi 6=yi , vi∈V } | |V| ) . The expectation is taken over the randomness of graph structures and input node features . Graph Convolutional Neural Networks . In node classification tasks , an L-layer GCN contains two components ( Gilmer et al. , 2017 ) : ( 1 ) neighborhood propagation and aggrega- tion : f̂ ( l ) i = AGGREGATE ( f ( l ) j , vj ∈ Ni ) , and ( 2 ) combination : f ( l+1 ) i = COMBINE ( f̂ ( l ) i , f ( l ) i ) , where AGGREGATE and COMBINE are learnable functions . The loss is given by LCE=CrossEntropy ( Softmax ( f ( L ) i W ( L ) + b ( L ) ) , yi ) . The vanilla GCN suggests a renormalization trick on the adjacency A to prevent gradient explosion ( Kipf & Welling , 2016 ) . The ( l + 1 ) -th output is given by : F ( l+1 ) = σ ( ÃF ( l ) W ( l ) ) , where Ã = D̃−1/2 ( I + A ) D̃−1/2 , D̃ is the degree matrix of I + A , and σ is ReLU . When the non-linearities in the vanilla GCN are removed , it reduces to a linear model called SGC ( Wu et al. , 2019 ) , which has competitive performance and is widely used in theoretical analyses ( Oono & Suzuki , 2019 ; Chen et al. , 2020 ) . For SGC , the l-th layer representations are given by : F ( l ) = ÃlF ( 0 ) and the last layer is a logistic-regression layer : ŷi = Softmax ( F ( L ) W ( L ) + b ( L ) ) . We note that only one weight matrix W ( L ) is learned ; it is equivalent to the products of all weight matrices in a linear GCN . Find more related works in § 6 . 3 THEORETICAL ANALYSIS . In this section , we formalize the connection between the oversmoothing and heterophily problems in two steps : ( 1 ) We show how the movements of node representations relate to the misclassification rate of node labels ; and ( 2 ) we identify the factors that affect the movements of representations . We use the SGC model to analyze binary node classification with nodes in class 1 denoted as set V1 and nodes in class 2 denoted as set V2 . We show empirically ( § 5 ) that the insights obtained in this section are effective for other non-linear models in multi-class classification . 3.1 ASSUMPTIONS . Next , we use “ i.d. ” to represent random variables/vectors that follow the same marginal distribution and their joint probability density function ( PDF ) is a permutation-invariant function p ( x1 , . . . , xn ) = p ( P ( x1 , . . . , xn ) ) , where P ( · ) means permutation . ( 1 ) Random graph : Node degrees { di } are i.d . random variables . { ( · ) i } represents a set with i = 1 , . . . , |V| . ( 2 ) Inputs : ( 2.1 ) Node labels { yi } are i.d . Bernoulli random variables given by the ratio ρ : ρ ≡ P ( yi=1 ) P ( yi=2 ) , ∀i . ( 2.2 ) Initial input node features { f ( 0 ) i } are i.d . random vectors given by ( PDF ) f ( x ) , which is expressed as : f ( x ) = { f1 ( x ) , when yi = 1. f2 ( x ) , when yi = 2 . E ( f ( 0 ) i |yi ) = { µ , when yi = 1 . −ρµ , when yi = 2. so E ( f ( 0 ) i ) = 0 . ( 3 ) Independence : { di } are independent of { yi } and { f ( 0 ) i } . 3.2 MISCLASSIFICATION RATE & MOVEMENTS OF NODE REPRESENTATIONS . The following lemmas illustrate why the movements of node representations are a good indicator of SGC ’ s performance . We view the changes of node representations across the layers as their movements ; i.e. , f ( l+1 ) i is moved from f ( l ) i , ∀i . The misclassification rate of an L-layer SGC is determined by { f ( L ) i } , which is the input to the last regression layer , and can be studied through the decision boundary of f ( L ) i . Lemma 1 In SGC , the decision boundaries w.r.t . { f ( L ) i } in multi-class classification are linear hyperplanes . In particular , the decision boundary is a single hyperplane for binary classification . Proof . We provide the proof in App . A.1 . The decision boundary of an ( L + 1 ) -layer SGC can be viewed as a perturbation to the decision boundary of an L-layer SGC due to the movements from { f ( L ) i } to { f ( L+1 ) i } . Due to Lemma 1 , we can use the hyperplane xw + b = 0 to represent the decision boundary of { f ( L ) i } , where w is a unit vector , b is a scalar , and the { f ( L ) i } are classified to class 1 if f ( L ) i w + b > 0 . Suppose when w = w∗ and b = b∗ , we find an optimal hyperplane xw∗ + b∗ = 0 that achieves the lowest total misclassification rate ( § 2 ) for { f ( L ) i } . Due to the movements of node representations , the new optimal hyperplane is decided by : xw∗′ + b∗′ = 0 . In Lemma 2 , we show that , under constraints , moving towards the original decision boundary with a non-zero step results in a non-decreasing total misclassification rate under the new decision boundary . Lemma 2 Moving representations { f ( L ) i } from class 1 by adding −tw∗ T , t > 0 , s.t . w∗Tw∗′ > 0 , the new total misclassification rate is no less than the misclassification rate before the movements . Proof . We provide the proof in App . A.2 . Note that the special case where the representations of the two classes swap positions ( e.g , bipartite graphs ) violates the condition w∗Tw∗′ > 0 and leads to different conclusions . We refer to the condition w∗Tw∗′ > 0 as `` non-swapping condition '' and throughout the paper , we analyze SGC under that . Lemma 2 shows that , under the `` non-swapping condition '' , if { f ( l ) i } move towards the original decision boundary ( or the other class ) , SGC tends to perform worse .

Summary: Two issues confronting GCNs, heterophily and over-smoothing, share a common cause. This article asserts that the relative degree and homophily level account for node representation movement and further affect the likelihood of misclassification.   The contributions of this paper are as follows:  1) it attempts to establish a connection between heterophily and over-smoothing by identifying common causes;  2) it establishes that the relative degree and homophily level have an effect on the misclassification rate; and  3) it also designs and presents a model that allows for negative interactions between nodes, which is shown to be robust to over-smoothing and capable of achieving state-of-the-art performance on heterophily datasets.