Are Graph Convolutional Networks Fully Exploiting the Graph Structure?

Graph Convolutional Networks ( GCNs ) represent the state-of-the-art for many graph related tasks . At every layer , GCNs rely on the graph structure to define an aggregation strategy where each node updates its representation by combining information from its neighbours . A known limitation of GCNs is their inability to infer long-range dependencies . In fact , as the number of layers increases , information gets smoothed and node embeddings become indistinguishable , negatively affecting performance . In this paper we formalize four levels of injection of graph structural information , and use them to analyze the importance of long-range dependencies . We then propose a novel regularization technique based on random walks with restart , called RWRReg , which encourages the network to encode long-range information into node embeddings . RWRReg does not require additional operations at inference time , is model-agnostic , and is further supported by our theoretical analysis connecting it to the Weisfeiler-Leman algorithm . Our experimental analysis , on both transductive and inductive tasks , shows that the lack of long-range structural information greatly affects the performance of state-ofthe-art models , and that the long-range information exploited by RWRReg leads to an average accuracy improvement of more than 5 % on all considered tasks . 1 INTRODUCTION . Graphs are a ubiquitous data representation of many real world phenomena , with applications ranging from social networks , to chemistry , biology , and recommendation systems ( Zhou et al. , 2018 ) . Graph Neural Networks ( GNNs ) are the generalization of deep learning for graph structured data , and have received a huge amount of attention from the research community . One class of GNN models , the Graph Convolutional Network ( GCN ) , has demonstrated to be extremely effective and is the current state-of-the-art for tasks such as graph classification , node classification , and link prediction . GCNs adopt a message passing mechanism where at each layer every node in the graph receives a message ( e.g . a feature vector ) from its 1-hop neighbours . The massages are then aggregated with a permutation invariant function ( e.g . by mean or sum ) and are used to update the node ’ s representation vector with a learnable , possibly non-linear , transformation . The final node embedding vectors are used to make predictions , and the whole process is trained end-to-end . Empirically , the best results are obtained when the message passing procedure is repeated 2 or 3 times , as a higher number of layers leads to over-smoothing ( Li et al. , 2018 ; Xu et al. , 2018b ) . Thus , GCNs are only leveraging the graph structure in the form of the 2-hop or 3-hop neighbourhood of each node . A direct consequence of this phenomenon is that GCNs are not capable of extracting and exploiting long-range dependencies between nodes . Random walks with restart ( Page et al. , 1998 ) have proven to be very effective at quantifying how closely related two nodes are ( Tong et al. , 2006 ) , regardless of their distance in the graph . In fact random walks with restart can capture the global structure of a graph , and have been used for many tasks including ranking , link prediction , and community detection ( Jin et al. , 2019 ) . On the other hand , random walks with restart do not consider node features , which are instead heavily exploited by GCNs . Combining GCNs and random walks with restart could then provide a powerful method to fully exploit the information contained in a graph . In this work we are not interested in defining new state-of-the-art results , or proposing novel GNN models . We focus on studying the impact of long-range dependencies , and identifying a first strategy , which can easily be applied to any existing model , to incorporate this information . Our Contribution . In more detail , we assess whether the injection of information on the graph structure that can not be captured by 2 or 3-hop neighbourhoods has a significant impact on the performance of several state-of-the-art GCN models . In this regard , our contributions are fourfold . Firstly , we propose and formalize four different levels of structural information injection . Secondly , we propose a novel and practical regularization strategy , Random Walk with Restart Regularization ( RWRReg ) , to inject structural information using random walks with restart , allowing GCNs to leverage long-range dependencies . RWRReg does not require additional operations at inference time , maintains the permutation-invariance of GCN models , and leads to an average 5 % increase in accuracy on both node classification , and graph classification . Thirdly , we prove a theoretical result linking random walks with restart and the Weisfeiler-Leman algorithm , providing a theoretical foundation for their use in GCNs . Fourthly , we test how the injection of structural information can impact the performance of 6 different GCN models on node classification , graph classification , and on the task of triangle counting . Results show that current state-of-the-art models lack the ability to extract long-range information , and this is severely affecting their performance . 2 INJECTING LONG-RANGE INFORMATION IN GCNS . To test if GCNs are missing on important information that is encoded in the structure of a graph , we inject additional structural information into existing GCN models , and test how the performance of these models changes in several graph related tasks . Intuitively , based on a model ’ s performance when injected with different levels of structural information , we can understand how much information is not captured by GCNs , and if this additional knowledge can improve performance on the considered tasks . In the rest of this section we present the notation used throughout the paper , the four levels of structural information injection that we consider , and an analytical result proving the effectiveness of using information from random walks with restart . 2.1 PRELIMINARIES . We use uppercase bold letters for matrices ( M ) , and lowercase bold letters for vectors ( v ) . We use plain letters with subscript indices to refer to a specific element of a matrix ( Mi , j ) , or of a vector ( vi ) . We refer to the vector containing the i-th row of a matrix with the subscript “ i , : ” ( Mi , : ) , while we refer to the i-th column with the subscript “ : , i ” ( M : ,i ) . For a graph G = ( V , E ) , where V = { 1 , .. , n } is the set of nodes and E ⊆ V × V is the set of edges , the input is given by a tuple ( X , A ) . X is an n × d matrix where the i-th row contains the d-dimensional feature vector of the i-th node , and A is the n × n adjacency matrix . For the sake of clarity we restrict our presentation to undirected graphs , but similar concepts can be applied to directed graphs . 2.2 STRUCTURAL INFORMATION INJECTION . We consider four different levels of structural information injection , briefly described below . We remark that not all the injection strategies presented in this section are made for practical use , as the scope of these strategies is to help us understand the importance of missing structural information . In particular , in Section 4 we study the impact of the different types of structural information injection , and hence quantify the information that is not exploited by current GCN models . We then discuss scalability and practicality aspects in Section 5 . Adjacency Matrix . We concatenate each node ’ s adjacency matrix row to its feature vector . This explicitly empowers the GCN model with the connectivity of each node , and allows for higher level structural reasoning when considering a neighbourhood ( the model will have access to the connectivity of the whole neighbourhood when aggregating messages from neighbouring nodes ) . Random Walk with Restart ( RWR ) Matrix . We perform random walks with restart ( RWR ) ( Page et al. , 1998 ) from each node v , thus obtaining a n-dimensional vector ( for each node ) that gives a score of how much v is “ related ” to each other node in the graph . We concatenate this vector of RWR features to each node ’ s feature vector . The choice of RWR is motivated by their capability to capture the relevance between two nodes in a graph ( Tong et al. , 2006 ) , and by the possibility to modulate the exploration of long-range dependencies by changing the restart probability . Intuitively , if a RWR starting at node v is very likely to visit a node u ( e.g . there are multiple paths that connect the two ) , then there will be a high score in the RWR vector for v at position u . This gives the GCN model higher level information about the structure of the graph that goes beyond the 1- hop neighbourhood of each node , and , again , it allows for high level reasoning on neighbourhood connectivity . RWR Regularization . We define a novel regularization term that pushes nodes with mutually high RWR scores to have embeddings that are close to each other ( independently of how far they are in the graph ) . This regularization term encourages the message passing procedure defined by GCNs , that acts on neighbouring nodes , to produce embeddings where pairs of nodes with high RWR score have similar representations . Therefore , the model is encouraged to extract global information , from local communications . The final embeddings are then a combination of local information and longrange information provided by RWR . Let S be the n × n matrix with the RWR scores . We define the RWRReg ( Random Walk with Restart Regularization ) loss as follows : LRWRReg = ∑ i , j∈V Si , j ||Hi , : −Hj , :||2 where H is a matrix of size n × d containing d-dimensional node embeddings that are in between graph convolution layers ( see Appendix A for the exact point in which H is considered for each model ) . With this approach , the loss function used to train the model becomes : L = Loriginal + λLRWRReg , where Loriginal is the original loss function for each model , and λ is a balancing term . RWR Matrix + RWR Regularization . We combine the previous two types of structural knowledge injection . The intuition is that it should be easier to enforce the RWRReg by having the additional long-range information provided by the RWR features . We expect this type of information injection to have the highest impact on performance of the models on downstream tasks . 2.3 RELATIONSHIP BETWEEN THE 1-WEISFEILER-LEMAN ALGORITHM AND RWRS . In this section we provide analytical evidence that the information from RWR significantly empowers GCNs . In particular , we prove an interesting connection between the 1-Weisfeiler-Leman ( 1-WL ) algorithm and RWR . The 1-WL algorithm for graph isomorphism testing uses an iterative coloring , or relabeling , scheme , in which all nodes are initially assigned the same label ( e.g. , the value 1 ) . It then iteratively refines the color of each node by aggregating the multiset of colors in its neighborhood . The final feature representation of a graph is the histogram of resulting node colors . ( For a more detailed description of the 1-WL algorithm we refer the reader to Shervashidze et al . ( 2011 ) . ) It is known that there are non-isomorphic graphs that are not distinguishable by the 1-WL algorithm , and that n iterations are enough to distinguish two graphs of n vertices which are distinguishable by the 1-WL algorithm . There is a well known connection ( Kipf & Welling , 2017 ; Xu et al. , 2018a ) between 1-WL and aggregation-based GCNs , which can be seen as a differentiable approximation of the algorithm . In particular , graphs that can be distinguished in k iterations by the 1-WL algorithm , can be distinguished by certain GCNs in k message passing iterations ( Morris et al. , 2019 ) . Here , we prove that graphs that are distinguishable by 1-WL in k iterations have different feature representations extracted by RWR of length k. Given a graphG = ( V , E ) , we define its k-step RWR representation as the set of vectors rv = [ rv , u1 , . . . , rv , un ] , v ∈ V , where each entry rv , u describes the probability that a RWR of length k starting in v ends in u ∈ V . Proposition 1 . Let G1 = ( V1 , E1 ) and G2 = ( V2 , E2 ) be two non-isomorphic graphs for which the 1-WL algorithm terminates with the correct answer after k iterations and starting from the labelling of all 1 ’ s . Then the k-step RWR representations of G1 and G2 are different . The proof can be found in Appendix B . Given that k iterations of the 1-WL algorithm require GCNs of depth k to be performed , but in practice GCNs are limited to depth 2 or 3 , the result above shows that RWR can empower GCNs with relevant information that is discarded in practice . Recent work ( Micali & Zhu , 2016 ) has shown that anonymous random walks ( i.e. , random walks where the global identities of nodes are not known ) of fixed length starting at node u are sufficient to reconstruct the local neighborhood within a fixed distance of a node u ( Micali & Zhu , 2016 ) . Subsequently , anonymous random walks have been introduced in the context of learning graph representations ( Ivanov & Burnaev , 2018 ) . Such results are complementary to ours , since they assume access to the distribution of entire walks of a given length , while our RWR representation only stores information on the probability of ending in a given node . In addition , such works do not provide a connection between RWR and 1-WL .

Summary: This paper studies GCNs when long-range dependencies have been added to the model as a regularizer. The regularizer proposed in this work is based on a random walk with restart (RWR) approach as RWR encourages the model to consider long-range dependencies. This paper shows that infusing the long-range dependencies using RWR regularizer improves the performance of some models for node classification and graph classification.