Graph Kernel Neural Networks

1 INTRODUCTION . In recent years , graph neural networks ( GNNs ) have gained increasing traction in the machine learning community . Graphs have long been used as a powerful abstraction for a wide variety of realworld data where structure plays a key role , from collaborations ( Lima et al. , 2014 ; Kipf & Welling , 2017 ) to biological ( Gilmer et al. , 2017 ; Ye et al. , 2015 ) and physical ( Shlomi et al. , 2020 ) data , to mention a few . Before the advent of GNNs , graph kernels provided a principled way to deal with graph data in the traditional machine learning setting ( Shervashidze et al. , 2011 ; Bai et al. , 2015 ; Minello et al. , 2019 ) . However , with the attention of machine learning researchers steadily shifting away from hand-crafted features toward end-to-end models where both the features and the model are learned together , neural networks have quickly overtaken kernels as the framework of choice to deal with graph data , leading to multiple popular architectures such as ( Kipf & Welling , 2017 ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ) The main obstacle that both traditional and deep learning methods have to overcome when dealing with graph data is also the source of interest for using graphs as data representations . The richness of graphs means that there is no obvious way to embed them into a vector space , a necessary step when the learning method expects a vector input . One reason is the lack of a canonical ordering of the nodes in a graph , requiring either permutation-invariant operations or an alignment to a reference structure . Moreover , even if the order or correspondence can be established , the dimension of the embedding space may vary , as a result of structural modifications , i.e. , changes in the number of nodes and edges . In traditional machine learning , kernel methods , and particularly graph kernels , provide an elegant way to sidestep this issue by replacing explicit vector representations of the data points with a positive semi-definite matrix of their inner products . Thus , any algorithm that can be formulated in terms of scalar products between input vectors , can be applied to a set of data ( such as graphs ) on which a kernel is defined . GNNs apply a form of generalised convolution operation to graphs , which can be seen as a message passing strategy where the node features are propagated over the graph to capture node interactions . In this paper we propose a neural architecture that bridges the two worlds of graph kernels and GNNs . The main idea is an analogy between the traditional convolution operator , which can be seen as performing an inner product between an input matrix and a filter , and graph kernels , which compute an inner product of graphs . As such , graph kernels provide the natural tool to generalise the concept of convolution to the graph domain , which we term graph kernel convolution ( GKC ) . Given a graph kernel of choice , in each GKC layer the input graph is compared against a series of structural masks ( analogous to the convolutional masks in CNNs ) , which effectively represent learnable sub-graphs . These in turn can offer better intrepretability , in the form of insights into what structural patterns in the input graphs are related to the corresponding output . Our main contributions are : 1 . Our model is fully structural , unlike existing approaches that require embedding the input graph into a larger , relaxed space . While the latter allows one to seek more complex and arbitrary decision boundaries , it also has the potential of overfitting the problem and creating more local optima , as evidenced by the need to use dropout strategies to avoid overfitting in Graph Convolutional Networks ( GCNs ) ( Rong et al. , 2019 ; Cong et al. , 2021 ) . 2 . Our architecture allows to plug-in any number and type of graph kernel ( not just a single differentiable kernel as in Nikolentzos & Vazirgiannis ( 2020 ) ) . 3 . As an added benefit , our model provides some interpretability in terms of the structural masks that are learned during the training process , similarly to convolutional masks in traditional CNNs . 4 . Finally , we analyse the expressive power of our model and we show that it is greater than that of standard message-passing GNNs and the equivalent Weisfeiler–Lehman ( WL ) graph isomorphism test . The fundamental goal of our model is to reconstruct the structural information needed for classification and is thus particularly suited for problems where the structure plays a pivotal role . These are often graph classification problems , where structure provides the most relevant information for classification , while node/edge features take a secondary role . The situation is usually reversed for node classification problems , where the distribution of features across the immediate neighbours of a node holds most of the information needed to classify the node itself . For these reasons in this paper we focus on graph classification problems . However we stress that our model can be indeed extended to tackle node-level tasks . The remainder of this paper is organised as follows . In Section 2 , we review the related work . In Section 3 , we present our model , where graph kernels are used to redefine convolution on graph data . In Section 4 , we investigate the hyper-parameters of our architecture through an extensive ablation study , provide some insight on the interpretability of the structural masks , and evaluate our architecture on standard graph classification benchmarks . Finally , Section 5 concludes the paper . 2 RELATED WORK . The majority of graph kernels belongs to one of two main categories : 1 ) bag-of-structures and 2 ) information propagation kernels . Bag-of-structures kernels compute the similarity between a pair of input graphs by first decomposing them into simpler sub-structures and then counting the number of isomorphic sub-structures between the two input graphs . Depending on the type of sub-structure considered , one can build a multitude of different kernels , e.g. , sub-trees ( Ramon & Gärtner , 2003 ) , shortest paths ( Borgwardt & Kriegel , 2005 ) , and graphlets ( Shervashidze et al. , 2009 ) . Information propagation kernels , on the other hand , include methods where pairs of input graphs are compared based on how information diffuses on them . Examples include random walk kernels ( Kashima et al. , 2003 ; Bai & Hancock , 2013 ) , quantum walk kernels ( Bai et al. , 2015 ; Rossi et al. , 2015 ) , and kernels based on iterative label refinements ( Shervashidze et al. , 2011 ) . While some kernels work only on undirected and unattributed graphs , other kernels are designed to handle attributes as well , either discrete- or continuous-valued ( Shervashidze et al. , 2011 ; Da San Martino et al. , 2017 ) . For a detailed review and historical perspective on graph kernels , we refer the reader to the recent survey of Kriege et al . ( 2020 ) . In recent years , with the advent of deep learning and the renewed interest in neural architectures , the focus of graph-based machine learning researchers has quickly moved to extending deep learning approaches to deal with graph data . Fundamentally , the principle underpinning most GNNs is that of exploiting the structure of the graph to propagate the node feature information iteratively . One of the first papers to propose the idea of GNNs is that of Scarselli et al . ( 2008 ) , where an information diffusion mechanism is used to learn the nodes ’ latent representations by exchanging neighbourhood information . Depending on the form this diffusion takes , Bronstein et al . ( 2021 ) distinguish between convolutional ( Kipf & Welling , 2017 ; Atwood & Towsley , 2016 ; Levie et al. , 2018 ) , atten- tional Veličković et al . ( 2018 ) , or message passing ( Gilmer et al. , 2017 ) types of GNNs , with the latter being the most general and formally equivalent to the Weisfeiler-Lehman graph isomorphism test under some technical conditions ( Xu et al. , 2018 ; Morris et al. , 2019 ) . More recently , Eliasof et al . ( 2021 ) proposed a reinterpretation of graph convolution in terms of partial differential equations on graphs , noting that different problems can benefit from different networks dynamics , hence suggesting the need to look beyond diffusion . For a comprehensive survey of GNNs we refer the reader to Wu et al . ( 2020 ) . In this work , we argue that a natural extension of the convolution operation to the graph domain , and thus an ideal candidate to build graph CNNs , already exists in the form of graph kernels . A number of works in the literature have investigated potential synergies between GNNs and kernels . Lei et al . ( 2017 ) , introduce a class of deep recurrent neural architectures show that this lies in the reproducing kernel Hilbert space ( RKHS ) of graph kernels . Nikolentzos et al . ( 2018 ) compute continuous embeddings of graphs using kernels and plug them into a neural network . Xu et al . ( 2018 ) ; Morris et al . ( 2019 ) show that GNNs have the same expressiveness as the WL graph kernel ( Shervashidze et al. , 2011 ) and propose a new generalised architecture with increased expressive power . Du et al . ( 2019 ) take a somewhat different approach and exploit neural networks to introduce a new graph kernel kernel . This in turn is shown to be equivalent to an infinitely-wide GNN initialized with random weights and trained with gradient descent . Chen et al . ( 2020 ) propose a graph neural architecture where each layer enumerates local sub-structures around each node and then maps them to a RKHS via a Gaussian kernel mapping . In the context of graph compression , Bouritsas et al . ( 2021 ) learn how to best decompose the graph into small substructures that are also learned . The closest work to ours we are aware of is that of Nikolentzos & Vazirgiannis ( 2020 ) , who propose a GNN where the first layer consists of a series of hidden graphs that are compared against the input graph using a random walk kernel . However , due to their optimisation strategy , their model only works with a single differentiable kernel , whereas our model allows us to tap into the expressive power of any type and number of kernels . Moreover , in the architecture of Nikolentzos & Vazirgiannis ( 2020 ) the learned structural masks are assumed to be complete weighted graphs , whereas we allow for graphs with arbitrary structure . 3 GRAPH NEURAL NETWORKS FROM GRAPH KERNELS . 3.1 PRELIMINARIES . Kernel methods are a class of algorithms that are capable of learning when presented with a particular pairwise similarity measure on the input data , known as a kernel . Consider a set X and a positive semi-definite kernel function K : X × X → R such that there exists a map φ : X → H into a Hilbert space H and K ( x , y ) = φ ( x ) > φ ( y ) for all x , y ∈ X . Crucially , X can represent any set of data on which a kernel can be defined , from Rd to a finite set of graphs . Hence , the field of machine learning is ripe with examples of graph kernels ( see Section 2 ) , which are nothing but positive semi-definite pairwise similarity measures on graphs . These can be either implicit ( only K is computed ) or explicit ( φ is also computed ) . Weisfeiler-Lehman test . The WL kernel ( Shervashidze et al. , 2011 ) is one of the most powerful and commonly used graph kernels and it is based on the 1-dimensional Weisfeiler-Lehman ( 1-WL ) graph isomorphism test . The idea underpinning the test is that of partitioning the node set by iteratively propagating the node labels between adjacent nodes . With each iteration , the set of labels accumulated at each node of the graph is then mapped to a new label through a hash function . This procedure is repeated until the cardinality of the set of labels stops growing . Two graphs can then be compared in terms of their label sets at convergence , with two graphs being isomorphic only if their label sets coincide . Learning on graphs . Let G = ( V , X , E ) be an undirected graph with |V | nodes and |E| edges , where each node v is associated to a label x ( v ) belonging to a dictionary D. A common goal in graph machine learning problems is to produce a vector representation of G that is aware of both the node labels and the structural information of G. Generalizing the convolution operator to graphs , GNNs learn the parameters Θ of a function h that embeds the graph into a vector Z by performing message passing ( Gilmer et al. , 2017 ) in the 1-hop neighborhood N 1G ( v ) of each node v ∈ V , i.e. , z ( v ) = ∑ u∈N 1G ( v ) hΘ ( x ( v ) , x ( u ) ) . ( 1 ) The convolution is usually repeated for L layers , and the final vectorial representation Z is obtained by applying a node-wise permutation invariant aggregation operator on the node features z ( v ) .

Summary: The authors propose a method that is based on graph kernels to perform graph convolution, which add to the interpretability of GNNs. A framework for learning graph kernels is introduced and studied on multiple datasets. The results show improvement in some datasets.