Geometric Random Walk Graph Neural Networks via Implicit Layers

1 INTRODUCTION . Recent years have witnessed an enormous growth in the amount of data represented as graphs . Indeed , graphs emerge naturally in several domains , including social networks , bioinformatics , and neuroscience , just to name a few . Besides the increase in the amount of graph-structured data , there is also a growing interest in applying machine learning techniques to data modeled as graphs . Among others , the graph classification and graph regression tasks have attracted a great deal of attention in the past years . These tasks have served as the fundamental building block within applications that deal with problems ranging from drug design Kearnes et al . ( 2016 ) to session-based recommendation Wu et al . ( 2019 ) . Graph Neural Networks ( GNNs ) provide a powerful tool for machine learning on graphs , So far , the field of GNNs has been largely dominated by message passing architectures . Indeed , most of them share the same basic idea , and can be reformulated into a single common framework , socalled message passing neural networks ( MPNNs ) Gilmer et al . ( 2017 ) . These models employ a message passing procedure to aggregate local information of vertices . For graph-related tasks , MPNNs usually apply some permutation invariant readout function to the vertex representations to produce a representation for the entire graph . The family of MPNNs has been heavily studied in the past few years , and there are now available very expressive models which have achieved state-ofthe-art results in several tasks Xu et al . ( 2019 ) ; Morris et al . ( 2019 ) . Although the family of MPNNs is perhaps the most successful story in the field of graph representation learning , there exist models that follow different design paradigms and do not fall into this family . An example of such a model is the recently proposed Random Walk Graph Neural Network ( RWNN ) Nikolentzos & Vazirgiannis ( 2020 ) . This model contains a number of trainable “ hidden graphs ” , and it compares the input graphs against these graphs using a random walk kernel which counts the number of common walks in two graphs . The emerging kernel values are fed into a fully-connected neural network which acts as the classifier or regressor . The employed random walk kernel is differentiable , and thus RWNN is end- to-end trainable . However , this kernel considers only random walks of a small length . Such local patterns may fail to capture the overall large-scale shape of the graphs , while several interesting properties of graphs depend on the graph ’ s global structure . Furthermore , increasing the length of the walks has a direct impact on the model ’ s computational complexity . In this paper , we propose a novel approach to tackle these challenges . Specifically , we propose a new architecture , called Geometric Random Walk Graph Neural Network ( GRWNN ) , that generalizes the RWNN model such that it can count common walks of infinite length in two graphs . The model contains a number of trainable “ hidden graphs ” , and it compares the input graphs against these graphs using the geometric random walk kernel . Thus , instead of walks of small length , the proposed model considers walks of infinite length . To compute the kernel , GRWNN uses a fixedpoint iteration approach . The kernel values are then passed on to a fully-connected neural network which produces the output . The proposed neural network is end-to-end trainable since we can directly differentiate through the fixed-point equations via implicit differentation , which leads to a very efficient implementation in terms of memory requirements . Hence , we can still update the “ hidden graphs ” during training with backpropagation . We compare the performance of the proposed model to state-of-the-art graph kernels and recently-proposed neural architectures on several graph classification datasets . Results show that in most cases , the GRWNN model matches or outperforms competing methods . Our main contributions are summarized as follows : • We propose a novel neural network model , Geometric Random Walk Graph Neural Network , which employs the geometric random walk kernel to produce graph representations . The model counts common walks of infinite length in the input graph and a set of randomly initialized “ hidden graphs ” . • We employ an efficient scheme to compute the random walk graph kernel using fixed-point iterations . We show that we can directly differentiate through the fixed-point equations via implicit differentation , which leads to an efficient implementation . • We evaluate the model ’ s performance on several standard graph classification datasets and show that it achieves results similar and in some cases superior to those obtained by recent GNNs and graph kernels . 2 RELATED WORK . Graph kernels have a long history in the field of graph representation learning Kriege et al . ( 2020 ) . A graph kernel is a kernel function between graphs , i. e. , a symmetric positive semidefinite function defined on the space of graphs . These methods generate implicitly ( or explicitly ) graph representations and enable the application of kernel methods such as the SVM classifier to graphs . Most graph kernels are instances of the R-convolution framework Haussler ( 1999 ) , and they compare substructures extracted from the graphs to each other . Such substructures include shortest paths Borgwardt & Kriegel ( 2005 ) , random walks Gärtner et al . ( 2003 ) ; Kashima et al . ( 2003 ) , small subgraphs Shervashidze et al . ( 2009 ) , and others . Our work is related to random walk kernels , i. e. , kernels that compare random walks to each other . The first such kernels were proposed by Gärtner et al . Gärtner et al . ( 2003 ) and by Kashima et al . Kashima et al . ( 2003 ) . The work of Kashima et al . was later refined by Mahé et al . Mahé et al . ( 2004 ) . Vishwanathan et al . Vishwanathan et al . ( 2010 ) and Kang et al . Kang et al . ( 2012 ) proposed new algorithms for efficiently computing random walk kernels . These algorithms improve the time complexity of kernel computation . Sugiyama and Borgwardt studied the problem of halting ( i. e. , longer walks are downweighted so much that the kernel value is completely dominated by the comparison of walks of length 1 ) that occurs in random walk kernels , and showed that its extent depends on properties of the graphs being compared Sugiyama & Borgwardt ( 2015 ) . Zhang et al . defined a different kernel which does not compare random walks to each other , but instead , compares the return probabilities of random walks Zhang et al . ( 2018b ) . Finally , Kalofolias et al . proposed a variant of the random walk kernel where structurally dissimilar vertices are not just down-weighed , but are not allowed to be visited during the simultaneous walk Kalofolias et al . ( 2021 ) . Although the first GNNs were proposed several years ago Sperduti & Starita ( 1997 ) ; Scarselli et al . ( 2009 ) ; Micheli ( 2009 ) , until recently , these models had attracted limited attention . In recent years , with the rise of deep learning , a lot of models started to emerge Bruna et al . ( 2014 ) ; Li et al . ( 2015 ) ; Duvenaud et al . ( 2015 ) ; Atwood & Towsley ( 2016 ) ; Defferrard et al . ( 2016 ) ; Lei et al . ( 2017 ) . Most models update the representation of each vertex by aggregating the feature vectors of its neighbors . This update procedure can be viewed as a form of message passing algorithm and thus , these models are known as message passing neural networks ( MPNNs ) Gilmer et al . ( 2017 ) . To compute a feature vector for the entire graph , MPNNs apply some permutation invariant readout function to all the vertices of the graph . The family of MPNNs has been heavily studied in the past few years , and there are now available several sophisticated models which can produce expressive graph representations Xu et al . ( 2019 ) ; Morris et al . ( 2019 ) ; Dehmamy et al . ( 2019 ) ; Morris et al . ( 2020 ) . Despite the general recent focus on MPNNs , some works have proposed architectures that are not variants of this family of models Niepert et al . ( 2016 ) ; Maron et al . ( 2019b ; a ) ; Nikolentzos & Vazirgiannis ( 2020 ) . The work closest to ours is the one reported in Nikolentzos & Vazirgiannis ( 2020 ) which presents the Random Walk Graph Neural Network ( RWNN ) model . In fact , in this paper , we generalize the RWNN model to compare random walks of infinite length in two graphs . Recently , another method that uses random walks to extract features which are then processed by a standard convolutional neural network was proposed Toenshoff et al . ( 2021 ) . However , the proposed approach decouples data representation from learning since random walks are sampled in a preprocessing stage . Our work is also related to implicit models which have been applied successfully to many problems de Avila Belbute-Peres et al . ( 2018 ) ; Chen et al . ( 2018 ) ; Amos et al . ( 2018 ) ; Bai et al . ( 2019 ) . The outputs of these models are determined implicitly by a solution of some underlying sub-problem . Implicit models have also been defined in the context of graph representation learning . For instance , Gu et al . proposed IGNN , a model that seeks the fixed-point of some equation which is equivalent to running an infinite number of message passing iterations Gu et al . ( 2020 ) . Thus , the final representation potentially contains information from all neighbors in the graph capturing long-range dependencies . Gallicchio and Micheli proposed a similar model which generates graph representations based on the fixed point of a recursive/dynamical system , but is actually only partially trained Gallicchio & Micheli ( 2020 ) . In contrast to these approaches whose objective is to apply a large ( or infinite ) number of message passing layers implicitly , in our setting , we employ a fixed-point iteration approach to compute the random walk kernel and then we directly differentiate through the fixed point equations via implicit differentation . 3 PRELIMINARIES . 3.1 NOTATION . Let [ n ] = { 1 , . . . , n } ⊂ N for n ≥ 1 . Let G = ( V , E ) be an undirected graph , where V is the vertex set and E is the edge set . We will denote by n the number of vertices and by m the number of edges . The adjacency matrix A ∈ Rn×n of a graph G is a symmetric ( typically sparse ) matrix used to encode edge information in the graph . The element of the ith row and jth column is equal to the weight of the edge between vertices vi and vj if such an edge exists , and 0 otherwise . The degree d ( v ) of a vertex v is equal to the sum of the weights of the edges that are adjacent to the vertex . For vertex-attributed graphs , every vertex in the graph is associated with a feature vector . We use X ∈ Rn×d to denote the vertex features where d is the feature dimensionality . The feature of a given vertex vi corresponds to the ith row of X . The direct ( tensor ) productG× = ( V× , E× ) of two graphsG = ( V , E ) andG′ = ( V ′ , E′ ) is defined as follows : V× = { ( v , v′ ) ∈ V × V ′ } E× = { ( ( v , v′ ) , ( u , u′ ) ) ∈ V× × V× | ( v , u ) ∈ E , and ( v′ , u′ ) ∈ E′ } We denote by A× the adjacency matrix ofG× , and denote by ∆× and d̄× the maximum and average of the vertex degrees of G× , respectively . Thus , d̄× = 1/n ∑ v∈V× d ( v ) . A walk in a graph is a sequence of vertices such that consecutive vertices are linked by an edge . Performing a random walk on the direct product G× of two graphs G and G′ is equivalent to performing a simultaneous random walk on the two graphs G and G′ . We use ⊗ to represent the Kronecker product , and use to represent elementwise multiplication between two matrices or vectors of the same dimension . For a p × q matrix V , vec ( V ) ∈ Rpq represents the vectorized form of V , obtained by stacking its columns . Let also vec−1 denote the inverse vectorization operator which transforms a vector into a matrix , i. e. , for a pq vector v , V = vec−1 ( v ) where V ∈ Rp×q ( see the appendix for the exact definition of the vec and vec−1 operators ) .

Summary: The paper discusses random walk graph kernels for graph neural network feature generation in other works. They work by generating features for graphs by comparing random walks present in each graph compared to a set of trainable hidden graphs. It is stated in this paper that an advantage of random walk kernels is that they are end to end differentiable, but current implementations have the weakness of only being efficient for walks of very short length. They introduce a way to efficiently extend walks to ‘infinite length’, theoretically making features generated by the kernel more informative. They compare their results with top GNNs and with other graph kernel approaches such as the mentioned random walk kernels on graph classification datasets. Out of 10 datasets they achieve best performance on 2 (1 of the 2 is tied for best), second best on 3, and third best on 1, It should be noted that many placements are well within the margin of error.