Predict then Propagate: Graph Neural Networks meet Personalized PageRank

1 INTRODUCTION . Graphs are ubiquitous in the real world and its description through scientific models . They are used to study the spread of information , to optimize delivery , to recommend new books , to suggest friends , or to find a party ’ s potential voters . Deep learning approaches have achieved great success on many important graph problems such as link prediction ( Grover & Leskovec , 2016 ; Bojchevski et al. , 2018 ) , graph classification ( Duvenaud et al. , 2015 ; Niepert et al. , 2016 ; Gilmer et al. , 2017 ) and semi-supervised node classification ( Yang et al. , 2016 ; Kipf & Welling , 2017 ) . There are many approaches for leveraging deep learning algorithms on graphs . Node embedding methods use random walks or matrix factorization to directly train individual node embeddings , often without using node features and usually in an unsupervised manner , i.e . without leveraging node classes ( Perozzi et al. , 2014 ; Tang et al. , 2015 ; Nandanwar & Murty , 2016 ; Grover & Leskovec , 2016 ; Qiu et al. , 2018 ) . Many other approaches use both graph structure and node features in a supervised setting . Examples for these include spectral graph convolutional neural networks ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ) , message passing ( or neighbor aggregation ) algorithms ( Kearnes et al. , 2016 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Pham et al. , 2017 ; Monti et al. , 2017 ; Gilmer et al. , 2017 ) , and neighbor aggregation via recurrent neural networks ( Scarselli et al. , 2009 ; Li et al. , 2016 ; Dai et al. , 2018 ) . Among these categories , the class of message passing algorithms has garnered particular attention recently due to its flexibility and good performance . Several works have been aimed at improving the basic neighborhood aggregation scheme by using attention mechanisms ( Kearnes et al. , 2016 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) , random walks ( Abu-El-Haija et al. , 2018a ; Ying et al. , 2018 ; Li et al. , 2018 ) , edge features ( Kearnes et al. , 2016 ; Gilmer et al. , 2017 ; Schlichtkrull et al. , 2018 ) and making it more scalable on large graphs ( Chen et al. , 2018 ; Ying et al. , 2018 ) . However , all of these methods only use the information of a very limited neighborhood for each node . A larger neighborhood would be desirable to provide the model with more information , especially for nodes in the periphery or in a sparsely labelled setting . Increasing the size of the neighborhood used by these algorithms , i.e . their range , is not trivial since neighborhood aggregation in this scheme is essentially a type of Laplacian smoothing and too 1https : //www.kdd.in.tum.de/ppnp many layers lead to oversmoothing ( Li et al. , 2018 ) . Xu et al . ( 2018 ) highlighted the same problem by establishing a relationship between the message passing algorithm termed Graph Convolutional Network ( GCN ) by Kipf & Welling ( 2017 ) and a random walk . Using this relationship we see that GCN converges to this random walk ’ s limit distribution as the number of layers increases . The limit distribution is a property of the graph as a whole and does not take the random walk ’ s starting ( root ) node into account . As such it is unsuited to describe the root node ’ s neighborhood . Hence , GCN ’ s performance necessarily deteriorates for a high number of layers ( or aggregation/propagation steps ) . To solve this issue , in this paper , we first highlight the inherent connection between the limit distribution and PageRank ( Page et al. , 1998 ) . We then propose an algorithm that utilizes a propagation scheme derived from personalized PageRank instead . This algorithm adds a chance of teleporting back to the root node , which ensures that the PageRank score encodes the local neighborhood for every root node ( Page et al. , 1998 ) . The teleport probability allows us to balance the needs of preserving locality ( i.e . staying close to the root node to avoid oversmoothing ) and leveraging the information from a large neighborhood . We show that this propagation scheme permits the use of far more ( in fact , infinitely many ) propagation steps without leading to oversmoothing . Moreover , while propagation and classification are inherently intertwined in message passing , our proposed algorithm separates the neural network from the propagation scheme . This allows us to achieve a much higher range without changing the neural network , whereas in the message passing scheme every additional propagation step would require an additional layer . It also permits the independent development of the propagation algorithm and the neural network generating predictions from node features . That is , we can combine any state-of-the-art prediction method with our propagation scheme . We even found that adding our propagation scheme during inference significantly improves the accuracy of networks that were trained without using any graph information . Our model achieves state-of-the-art results while requiring fewer parameters and less training time compared to most competing models , with a computational complexity that is linear in the number of edges . We show these results in the most thorough study ( including significance testing ) of message passing models using graphs with text-based features that has been done so far . 2 GRAPH CONVOLUTIONAL NETWORKS AND THEIR LIMITED RANGE . We first introduce our notation and explain the problem our model solves . Let G = ( V , E ) be a graph with nodes V and edges E. Let n denote the number of nodes and m the number of edges . The nodes are described by the feature matrix X ∈ Rn×f , with the number of features f per node , and the class ( or label ) matrix Y ∈ Rn×c , with the number of classes c. The graphG is described by the adjacency matrixA ∈ Rn×n . Ã = A+ In denotes the adjacency matrix with added self-loops . One simple and widely used message passing algorithm for semi-supervised classification is the Graph Convolutional Network ( GCN ) . In the case of two message passing layers its equation is ZGCN = softmax ( ˆ̃A ReLU ( ˆ̃AXW0 ) W1 ) , ( 1 ) where Z ∈ Rn×c are the predicted node labels , ˆ̃A = D̃ −1/2 ÃD̃ −1/2 is the symmetrically normalized adjacency matrix with self-loops , with the diagonal degree matrix D̃ij = ∑ k Ãikδij , and W0 andW1 are trainable weight matrices ( Kipf & Welling , 2017 ) . With two GCN-layers , only neighbors in the two-hop neighborhood are considered . There are essentially two reasons why a message passing algorithm like GCN can not be trivially expanded to use a larger neighborhood . First , aggregation by averaging causes oversmoothing if too many layers are used . It , therefore , loses its focus on the local neighborhood ( Li et al. , 2018 ) . Second , most common aggregation schemes use learnable weight matrices in each layer . Therefore , using a larger neighborhood necessarily increases the depth and number of learnable parameters of the neural network ( the second aspect can be circumvented by using weight sharing , which is typically not the case , though ) . However , the required neighborhood size and neural network depth are two completely orthogonal aspects . This fixed relationship is a strong limitation and leads to bad compromises . We will start by concentrating on the first issue . Xu et al . ( 2018 ) have shown that for a klayer GCN the influence score of node x on y , I ( x , y ) = ∑ i ∑ j ∂Zyi ∂Xxj , is proportional in expectation to a slightly modified k-step random walk distribution starting at the root node x , Prw ’ ( x → y , k ) . Hence , the information of node x spreads to node y in a random walk-like manner . If we take the limit k → ∞ and the graph is irreducible and aperiodic , this random walk probability distribution Prw ’ ( x → y , k ) converges to the limit ( or stationary ) distribution Plim ( → y ) . This distribution can be obtained by solving the equation πlim = ˆ̃Aπlim . Obviously , the result only depends on the graph as a whole and is independent of the random walk ’ s starting ( root ) node x . This global property is therefore unsuitable for describing the root node ’ s neighborhood . 3 PERSONALIZED PROPAGATION OF NEURAL PREDICTIONS . From message passing to personalized PageRank . We can solve the problem of lost focus by recognizing the connection between the limit distribution and PageRank ( Page et al. , 1998 ) . The only differences between these two are the added self-loops and the adjacency matrix normalization in ˆ̃A . Original PageRank is calculated via πpr = Arwπpr , with Arw = AD−1 . Having made this connection we can now consider using a variant of PageRank that takes the root node into account – personalized PageRank ( Page et al. , 1998 ) . We define the root node x via the teleport vector ix , which is a one-hot indicator vector . Our adaptation of personalized PageRank can be obtained for node x using the recurrent equation πppr ( ix ) = ( 1−α ) ˆ̃Aπppr ( ix ) +αix , with the teleport ( or restart ) probability α ∈ ( 0 , 1 ] . By solving this equation , we obtain πppr ( ix ) = α ( In − ( 1− α ) ˆ̃A ) −1 ix . ( 2 ) Introducing the teleport vector ix allows us to preserve the node ’ s local neighborhood even in the limit distribution . In this model the influence score of root node x on node y , I ( x , y ) , is proportional to the y-th element of our personalized PageRank πppr ( ix ) . This value is different for every root node . How fast it decreases as we move away from the root node can be adjusted via α . By substituting the indicator vector ix with the unit matrix In we obtain our fully personalized PageRank matrix Πppr = α ( In − ( 1 − α ) ˆ̃A ) −1 , whose element ( yx ) specifies the influence score of node x on node y , I ( x , y ) ∝ Π ( yx ) ppr . Note that due to symmetry Π ( yx ) ppr = Π ( xy ) ppr , i.e . the influence of x on y is equal to the influence of y on x . This inverse always exists since 11−α > 1 and therefore can not be an eigenvalue of ˆ̃A ( see Appendix A ) . Personalized propagation of neural predictions ( PPNP ) . To utilize the above influence scores for semi-supervised classification we generate predictions for each node based on its own features and then propagate them via our fully personalized PageRank scheme to generate the final predictions . This is the foundation of personalized propagation of neural predictions . PPNP ’ s model equation is ZPPNP = softmax ( α ( In − ( 1− α ) ˆ̃A ) −1 H ) , Hi , : = fθ ( Xi , : ) , ( 3 ) where X is the feature matrix and fθ a neural network with parameter set θ generating the predictions H ∈ Rn×c . Note that fθ operates on each node ’ s features independently , allowing for parallelization . Furthermore , one could substitute ˆ̃A with any propagation matrix , such asArw . As a consequence , PPNP separates the neural network used for generating predictions from the propagation scheme . This separation additionally solves the second issue mentioned above : the depth of the neural network is now fully independent of the propagation algorithm . As we saw when connecting GCN to PageRank , personalized PageRank can effectively use even infinitely many neighborhood aggregation layers , which is clearly not possible in the classical message passing framework . Furthermore , the separation gives us the flexibility to use any method for generating predictions , e.g . deep convolutional neural networks for graphs of images . While generating predictions and propagating them happen consecutively during inference , it is important to note that the model is trained end-to-end . That is , the gradient flows through the propagation scheme during backpropagation ( implicitly considering infinitely many neighborhood aggregation layers ) . Adding these propagation effects significantly improves the model ’ s accuracy . Efficiency analysis . Directly calculating the fully personalized PageRank matrix Πppr , is computationally inefficient and results in a dense Rn×n matrix . Using this matrix would lead to a computational complexity and memory requirement of O ( n2 ) for training and inference . To solve this issue , reconsider the equation Z = α ( In − ( 1 − α ) ˆ̃A ) −1H . Instead of viewing this equation as a combination of a dense fully personalized PageRank matrix with the prediction matrix , we can also view it as a variant of topic-sensitive PageRank , with each class corresponding to one topic ( Haveliwala , 2002 ) . In this view every column of H defines an ( unnormalized ) distribution over nodes that acts as a teleport set . Hence , we can approximate PPNP via an approximate computation of topic-sensitive PageRank . Approximate personalized propagation of neural predictions ( APPNP ) . More precisely , APPNP achieves linear computational complexity by approximating topic-sensitive PageRank via power iteration . While PageRank ’ s power iteration is connected to the regular random walk , the power iteration of topic-sensitive PageRank is related to a random walk with restarts . Each power iteration ( random walk/propagation ) step of our topic-sensitive PageRank variant is , thus , calculated via Z ( 0 ) = H = fθ ( X ) , Z ( k+1 ) = ( 1− α ) ˆ̃AZ ( k ) + αH , Z ( K ) = softmax ( ( 1− α ) ˆ̃AZ ( K−1 ) + αH ) , ( 4 ) where the prediction matrix H acts as both the starting vector and the teleport set , K defines the number of power iteration steps and k ∈ [ 0 , K−2 ] . Note that this method retains the graph ’ s sparsity and never constructs an Rn×n matrix . The convergence of this iterative scheme can be shown by investigating the resulting series ( see Appendix B ) . Note that the propagation scheme of this model does not require any additional parameters to train – as opposed to models like GCN , which typically require more parameters for each additional propagation layer . We can therefore propagate very far with very few parameters . Our experiments show that this ability is indeed very beneficial ( see Section 6 ) . A similar model expressed in the message passing framework would therefore not be able to achieve the same level of performance . The reformulation of PPNP via fixed-point iterations illustrates a connection to the original graph neural network ( GNN ) model ( Scarselli et al. , 2009 ) . While the latter uses a learned fixed-point iteration , our approach uses a predetermined iteration ( adapted personalized PageRank ) and applies a learned feature transformation before propagation . In both PPNP and APPNP , the size of the neighborhood influencing each node can be adjusted via the teleport probability α . The freedom to choose α allows us to adjust the model for different types of networks , since varying graph types require the consideration of different neighborhood sizes , as shown in Section 6 and described by Grover & Leskovec ( 2016 ) and Abu-El-Haija et al . ( 2018b ) .

Summary: This paper proposes a GCN variant that addresses a limitation of the original model, where embedding is propagated in only a few hops. The architectural difference may be explained in the following: GCN interleaves the individual node feature transformation and the single-hop propagation, whereas the proposed architecture first transforms the node features, followed by a propagation with an (in)finite number of hops. The propagation in the proposed method follows personalized PageRank, where in addition to following direct links, there is a nonzero probably jumping to a target node.