Adaptive Universal Generalized PageRank Graph Neural Network

1 INTRODUCTION . Graph-centered machine learning has received significant interest in recent years due to the ubiquity of graph-structured data and its importance in solving numerous real-world problems such as semisupervised node classification and graph classification ( Zhu , 2005 ; Shervashidze et al. , 2011 ; Lü & Zhou , 2011 ) . Usually , the data at hand contains two sources of information : Node features and graph topology . As an example , in social networks , nodes represent users that have different combinations of interests and properties captured by their corresponding feature vectors ; edges on the other hand document observable friendship and collaboration relations that may or may not depend on the node features . Hence , learning methods that are able to simultaneously and adaptively exploit node features and the graph topology are highly desirable as they make use of their latent connections and thereby improve learning on graphs . Graph neural networks ( GNN ) leverage their representational power to provide state-of-the-art performance when addressing the above described application domains . Many GNNs use message ∗equal contribution 1https : //github.com/jianhao2016/GPRGNN passing ( Gilmer et al. , 2017 ; Battaglia et al. , 2018 ) to manipulate node features and graph topology . They are constructed by stacking ( graph ) neural network layers which essentially propagate and transform node features over the given graph topology . Different types of layers have been proposed and used in practice , including graph convolutional layers ( GCN ) ( Bruna et al. , 2014 ; Kipf & Welling , 2017 ) , graph attention layers ( GAT ) ( Velickovic et al. , 2018 ) and many others ( Hamilton et al. , 2017 ; Wijesinghe & Wang , 2019 ; Zeng et al. , 2020 ; Abu-El-Haija et al. , 2019 ) . However , most of the existing GNN architectures have two fundamental weaknesses which restrict their learning ability on general graph-structured data . First , most of them seem to be tailor-made to work on homophilic ( associative ) graphs . The homophily principle ( McPherson et al. , 2001 ) in the context of node classification asserts that nodes from the same class tend to form edges . Homophily is also a common assumption in graph clustering ( Von Luxburg , 2007 ; Tsourakakis , 2015 ; Dau & Milenkovic , 2017 ) and in many GNNs design ( Klicpera et al. , 2018 ) . Methods developed for homophilic graphs are nonuniversal in so far that they fail to properly solve learning problems on heterophilic ( disassortative ) graphs ( Pei et al. , 2019 ; Bojchevski et al. , 2019 ; 2020 ) . In heterophilic graphs , nodes with distinct labels are more likely to link together ( For example , many people tend to preferentially connect with people of the opposite sex in dating graphs , different classes of amino acids are more likely to connect within many protein structures ( Zhu et al. , 2020 ) etc ) . GNNs model the homophily principle by aggregating node features within graph neighborhoods . For this purpose , they use different mechanisms such as averaging in each network layer . Neighborhood aggregation is problematic and significantly more difficult for heterophilic graphs ( Jia & Benson , 2020 ) . Second , most of the existing GNNs fail to be “ deep enough ” . Although in principle an arbitrary number of layers may be stacked , practical models are usually shallow ( including 2-4 layers ) as these architectures are known to achieve better empirical performance than deep networks . A widely accepted explanation for the performance degradation of GNNs with increasing depth is feature-oversmoothing , which may be intuitively explained as follows . The process of GNN feature propagating represents a form of random walks on “ feature graphs , ” and under proper conditions , such random walks converge with exponential rate to their stationary points . This essentially levels the expressive power of the features and renders them nondiscriminative . This intuitive reasoning was first described for linear settings in Li et al . ( 2018 ) and has been recently studied in Oono & Suzuki ( 2020 ) for a setting involving nonlinear rectifiers . We address these two described weaknesses by combining GNNs with Generalized PageRank techniques ( GPR ) within a new model termed GPR-GNN . The GPR-GNN architecture is designed to first learn the hidden features and then to propagate them via GPR techniques . The focal component of the network is the GPR procedure that associates each step of feature propagation with a learnable weight . The weights depend on the contributions of different steps during the information propagation procedure , and they can be both positive and negative . This departures from common nonnegativity assumptions ( Klicpera et al. , 2018 ) allows for the signs of the weights to adapt to the homophily/heterophily structure of the underlying graphs . The amplitudes of the weights trade-off the degree of smoothing of node features and the aggregation power of topological features . These traits do not change with the choice of the initialization procedure and elucidate the process used to combine node features and the graph structure so as to achieve ( near ) -optimal predictions . In summary , the GPR-GNN method can simultaneously learn the node label patterns of disparate classes of graphs and prevent feature over-smoothing . The excellent performance of GPR-GNN is demonstrated empirically , on real world datasets , and further supported through a number of theoretical findings . In the latter setting , we show that the GPR procedure relates to general polynomial graph filtering , which can naturally deal with both high and low frequency parts of the graph signals . In contrast , recent GNN models that utilize Personalized PageRanks ( PPR ) with fixed weights ( Wu et al. , 2019 ; Klicpera et al. , 2018 ; 2019 ) inevitably act as low-pass filters . Thus , they fail to learn the labels of heterophilic graphs . We also establish that GPR-GNN can provably mitigate the feature-over-smoothing issue in an adaptive manner even after large-step propagation ( i.e. , after a large number of propagation steps ) . Hence , the method is able to make use of informative large-step propagation . To test the performance of GPR-GNN on homophilic and heterophilic node label patterns and determine the trade-off between node and topological feature exploration , we first describe the recently proposed contextual stochastic block model ( cSBM ) ( Deshpande et al. , 2018 ) . The cSBM allows for smoothly controlling the “ informativeness ratio ” between node features and graph topology , where the graph can vary from being highly homophilic to highly heterophilic . We show that GPRGNN outperforms all other baseline methods for the task of semi-supervised node classification on the cSBM consistently from strong homophily to strong heterophily . We then proceed to show that GPR-GNN offers state-of-the-art performance on node-classification benchmark real-world datasets which contain both homophilic and heterophilic graphs . Due to the space limit , we put all proofs , formal theorem statements , and the conclusion section in the Supplement . 2 PRELIMINARIES . Let G = ( V , E ) be an undirected graph with nodes V and edges E. Let n denote the number of nodes , assumed to belong to one of C ≥ 2 classes . The nodes are associated with the node feature matrix X ∈ Rn×f , where f denotes the number of features per node . Throughout the paper , we use Xi : to indicate the ith row and X : j to indicate the jth column of the matrix X , respectively . The symbol δij is reserved for the Kronecker delta function . The graph G is described by the adjacency matrix A , while Ã stands for the adjacency matrix for a graph with added self-loops . We let D̃ be the diagonal degree matrix of Ã and Ãsym = D̃−1/2ÃD̃−1/2 denote the symmetric normalized adjacency matrix with self-loops . 3 GPR-GNNS : MOTIVATION AND CONTRIBUTIONS . Generalized PageRanks . Generalized PageRank ( GPR ) methods were first used in the context of unsupervised graph clustering where they showed significant performance improvements over Personalized PageRank ( Kloumann et al. , 2017 ; Li et al. , 2019 ) . The operational principles of GPRs can be succinctly described as follows . Given a seed node s ∈ V in some cluster of the graph , a one-dimensional feature vector H ( 0 ) ∈ Rn×1 is initialized according to H ( 0 ) v : = δvs . The GPR score is defined as ∑∞ k=0 γkÃ k symH ( 0 ) = ∑∞ k=0 γkH ( k ) , where the parameters γk ∈ R , k = 0 , 1 , 2 , . . . , are referred to as the GPR weights . Clustering of the graph is performed locally by thresholding the GPR score . Certain PangRank methods , such as Personalized PageRank or heat-kernel PageRank ( Chung , 2007 ) , are associated with specific choices of GPR weights ( Li et al. , 2019 ) . For an excellent in-depth discussion of PageRank methods , the interested reader is referred to ( Gleich , 2015 ) . The work in Li et al . ( 2019 ) recently introduced and theoretically analyzed a special form of GPR termed Inverse PR ( IPR ) and showed that long random walk paths are more beneficial for clustering then previously assumed , provided that the GPR weights are properly selected ( Note that IPR was developed for homophilic graphs and optimal GPR weights for heterophilic graphs are not currently known ) . Equivalence of the GPR method and polynomial graph filtering . If we truncate the infinite sum in the definition of GPR at some natural number K , ∑K k=0 γkÃ k sym corresponds to a polynomial graph filter of orderK . Thus , learning the optimal GPR weights is equivalent to learning the optimal polynomial graph filter . Note that one can approximate any graph filter using a polynomial graph filter ( Shuman et al. , 2013 ) and hence the GPR method is able to deal with a large range of different node label patterns . Also , increasing K allows one to better approximate the underlying optimal graph filter . This once again shows that large-step propagation is beneficial . Universality with respect to node label patterns : Homophily versus heterophily . In their recent work , Pei et al . ( 2019 ) proposed an index to measure the level of homophily of nodes in a graph H ( G ) = 1|V | ∑ v∈V Number of neighbors of v ∈ V that have the same label as v Number of neighbors of v . Note that H ( G ) → 1 corresponds to strong homophily while H ( G ) → 0 indicates strong heterophily . Figures 1 ( b ) and ( c ) plot the GPR weights learnt by our GPR-GNN method on a homophilic ( Cora ) and heterophilic ( Texas ) dataset . The learnt GPR weights from Cora match the behavior of IPR ( Li et al. , 2019 ) , which verifies that large-step propagation is indeed of great importance for homophilic graphs . The GPR weights learnt from Texas behave significantly differently from all known PR variants , taking a number of negative values . These differences in weight patterns are observed under random initialization , demonstrating that the weights are actually learned by the network and not forced by specific initialization . Furthermore , the large difference in the GPR weights for these two graph models illustrates the learning power of GPR-GNN and their universal adaptability . The over-smoothing problem . One of the key components in most GNN models is the graph convolutional layer , described by H ( k ) GCN = ReLU ( ÃsymH ( k−1 ) GCN W ( k ) ) , P̂GCN = softmax ( ÃsymH ( K−1 ) GCN W ( k ) ) , where H ( 0 ) GCN = X and W ( k ) represents the trainable weight matrix for the kth layer . The key issue that limits stacking multiple layers is the over-smoothing phenomenon : If one were to remove ReLU in the above expression , limk→∞ ÃksymH ( 0 ) = H ( ∞ ) , where each row of H ( ∞ ) only depends on the degree of the corresponding node , provided that the graph is irreducible and aperiodic . This shows that the model looses discriminative information provided by the node features as the number of layers increases . Mitigating graph heterophily and over-smoothing issues with the GPR-GNN model . GPRGNN first extracts hidden state features for each node and then uses GPR to propagate them . The GPR-GNN process can be mathematically described as : P̂ = softmax ( Z ) , Z = K∑ k=0 γkH ( k ) , H ( k ) = ÃsymH ( k−1 ) , H ( 0 ) i : = fθ ( Xi : ) , ( 1 ) where fθ ( . ) represents a neural network with parameter set { θ } that generates the hidden state features H ( 0 ) . The GPR weights γk are trained together with { θ } in an end-to-end fashion . The GPRGNN model is easy to interpret : As already pointed out , GPR-GNN has the ability to adaptively control the contribution of each propagation step and adjust it to the node label pattern . Examining the learnt GPR weights also helps with elucidating the properties of the topological information of a graph ( i.e. , determining the optimal polynomial graph filter ) , as illustrated in Figure 1 ( b ) and ( c ) . Placing GPR-GNNs in the context of related prior work . Among the methods that differ from repeated stacking of GCN layers , APPNP ( Klicpera et al. , 2018 ) represents one of the state-of-theart GNNs that is related to our GPR-GNN approach . It can be easily seen that APPNP as well as SGC ( Wu et al. , 2019 ) are special cases of our model since APPNP fixes γk = α ( 1−α ) k , γK = ( 1− α ) K , while SGC removes all nonlinearities with γk = δkK , respectively . These two weight choices correspond to Personalized PageRank ( PPR ) ( Jeh & Widom , 2003 ) , which is known to be suboptimal compared to the IPR framework when applied to homophilic node classification ( Li et al. , 2019 ) . Fixing the GPR weights makes the model unable to adaptively learn the optimal propagation rules which is of crucial importance : As we will show in Section 4 , the fixed PPR weights corresponds to low-pass graph filters which makes them inadequate for learning on heterophilic graphs . The recent work ( Klicpera et al. , 2018 ) showed that fixed PPR weights ( APPNP ) can also provably resolve the over-smoothing problem . However , the way APPNP prevents over-smoothing is independent on the node label information . In contrast , the escape of GPR-GNN from over-smoothing is guided by the node label information ( Theorem 4.2 ) . A detailed discussion of this phenomena along with illustrative examples is delegated to the Supplement . Among the GCN-like models , JK-Net ( Xu et al. , 2018 ) exhibits some similarities with GPR-GNN . It also aggregates the outputs of different GCN layers to arrive at the final output . On the other hand , the GCN-Cheby method ( Defferrard et al. , 2016 ; Kipf & Welling , 2017 ) is related to polynomial graph filtering , where each convolutional layer propagates multiple steps and the graph filter is related to Chebyshev polynomials . In both cases , the depth of the models is limited in practice ( Klicpera et al. , 2018 ) and they are not easy to interpret as our GPR-GNN method . Some prior work also emphasizes adaptively learning the importance of different steps ( Abu-El-Haija et al. , 2018 ; Berberidis et al. , 2018 ) . Nevertheless , none of the above works is applicable for semisupervised learning with GNNs and considers heterophilic graphs .

Summary: The paper proposes a new GNN architecture based on Generalized PageRank to handle two weakness in some existing GNNs: the difficulty of neighborhood aggregation on heterophilic graphs, and the oversmoothing problem when stacking GNN layers. The proposed GPR-GNN can be viewed as an extension of the Personalized PageRank-based GNNs, such as APPNP and SGC, which also aimed to handle oversmoothing problem. Pros:  The paper highlights and clearly explains why optimizing the proposed layerwise GPR weights in GNN could tackle the two challenges. Although the final GPR-GNN architecture is simple, the ideas behind are significant. The paper is technically sound. The paper clearly analyzes the differences and relationships between related prior works and the proposed approach. In experiment, datasets are sufficient, including both homophilic and heterophilic graphs (both synthetic datasets and 10 real benchmarks are sufficient) which proves that GPR-GNN can achieve state-of-the-art performances especially on heterophilic graphs.