Graph Kernel Neural Networks

1 INTRODUCTION . In recent years , graph neural networks ( GNNs ) have gained increasing traction in the machine learning community . Graphs have long been used as a powerful abstraction for a wide variety of realworld data where structure plays a key role , from collaborations ( Lima et al. , 2014 ; Kipf & Welling , 2017 ) to biological ( Gilmer et al. , 2017 ; Ye et al. , 2015 ) and physical ( Shlomi et al. , 2020 ) data , to mention a few . Before the advent of GNNs , graph kernels provided a principled way to deal with graph data in the traditional machine learning setting ( Shervashidze et al. , 2011 ; Bai et al. , 2015 ; Minello et al. , 2019 ) . However , with the attention of machine learning researchers steadily shifting away from hand-crafted features toward end-to-end models where both the features and the model are learned together , neural networks have quickly overtaken kernels as the framework of choice to deal with graph data , leading to multiple popular architectures such as ( Kipf & Welling , 2017 ; Gilmer et al. , 2017 ; Veličković et al. , 2018 ) The main obstacle that both traditional and deep learning methods have to overcome when dealing with graph data is also the source of interest for using graphs as data representations . The richness of graphs means that there is no obvious way to embed them into a vector space , a necessary step when the learning method expects a vector input . One reason is the lack of a canonical ordering of the nodes in a graph , requiring either permutation-invariant operations or an alignment to a reference structure . Moreover , even if the order or correspondence can be established , the dimension of the embedding space may vary , as a result of structural modifications , i.e. , changes in the number of nodes and edges . In traditional machine learning , kernel methods , and particularly graph kernels , provide an elegant way to sidestep this issue by replacing explicit vector representations of the data points with a positive semi-definite matrix of their inner products . Thus , any algorithm that can be formulated in terms of scalar products between input vectors , can be applied to a set of data ( such as graphs ) on which a kernel is defined . GNNs apply a form of generalised convolution operation to graphs , which can be seen as a message passing strategy where the node features are propagated over the graph to capture node interactions . In this paper we propose a neural architecture that bridges the two worlds of graph kernels and GNNs . The main idea is an analogy between the traditional convolution operator , which can be seen as performing an inner product between an input matrix and a filter , and graph kernels , which compute an inner product of graphs . As such , graph kernels provide the natural tool to generalise the concept of convolution to the graph domain , which we term graph kernel convolution ( GKC ) . Given a graph kernel of choice , in each GKC layer the input graph is compared against a series of structural masks ( analogous to the convolutional masks in CNNs ) , which effectively represent learnable sub-graphs . These in turn can offer better intrepretability , in the form of insights into what structural patterns in the input graphs are related to the corresponding output . Our main contributions are : 1 . Our model is fully structural , unlike existing approaches that require embedding the input graph into a larger , relaxed space . While the latter allows one to seek more complex and arbitrary decision boundaries , it also has the potential of overfitting the problem and creating more local optima , as evidenced by the need to use dropout strategies to avoid overfitting in Graph Convolutional Networks ( GCNs ) ( Rong et al. , 2019 ; Cong et al. , 2021 ) . 2 . Our architecture allows to plug-in any number and type of graph kernel ( not just a single differentiable kernel as in Nikolentzos & Vazirgiannis ( 2020 ) ) . 3 . As an added benefit , our model provides some interpretability in terms of the structural masks that are learned during the training process , similarly to convolutional masks in traditional CNNs . 4 . Finally , we analyse the expressive power of our model and we show that it is greater than that of standard message-passing GNNs and the equivalent Weisfeiler–Lehman ( WL ) graph isomorphism test . The fundamental goal of our model is to reconstruct the structural information needed for classification and is thus particularly suited for problems where the structure plays a pivotal role . These are often graph classification problems , where structure provides the most relevant information for classification , while node/edge features take a secondary role . The situation is usually reversed for node classification problems , where the distribution of features across the immediate neighbours of a node holds most of the information needed to classify the node itself . For these reasons in this paper we focus on graph classification problems . However we stress that our model can be indeed extended to tackle node-level tasks . The remainder of this paper is organised as follows . In Section 2 , we review the related work . In Section 3 , we present our model , where graph kernels are used to redefine convolution on graph data . In Section 4 , we investigate the hyper-parameters of our architecture through an extensive ablation study , provide some insight on the interpretability of the structural masks , and evaluate our architecture on standard graph classification benchmarks . Finally , Section 5 concludes the paper . 2 RELATED WORK . The majority of graph kernels belongs to one of two main categories : 1 ) bag-of-structures and 2 ) information propagation kernels . Bag-of-structures kernels compute the similarity between a pair of input graphs by first decomposing them into simpler sub-structures and then counting the number of isomorphic sub-structures between the two input graphs . Depending on the type of sub-structure considered , one can build a multitude of different kernels , e.g. , sub-trees ( Ramon & Gärtner , 2003 ) , shortest paths ( Borgwardt & Kriegel , 2005 ) , and graphlets ( Shervashidze et al. , 2009 ) . Information propagation kernels , on the other hand , include methods where pairs of input graphs are compared based on how information diffuses on them . Examples include random walk kernels ( Kashima et al. , 2003 ; Bai & Hancock , 2013 ) , quantum walk kernels ( Bai et al. , 2015 ; Rossi et al. , 2015 ) , and kernels based on iterative label refinements ( Shervashidze et al. , 2011 ) . While some kernels work only on undirected and unattributed graphs , other kernels are designed to handle attributes as well , either discrete- or continuous-valued ( Shervashidze et al. , 2011 ; Da San Martino et al. , 2017 ) . For a detailed review and historical perspective on graph kernels , we refer the reader to the recent survey of Kriege et al . ( 2020 ) . In recent years , with the advent of deep learning and the renewed interest in neural architectures , the focus of graph-based machine learning researchers has quickly moved to extending deep learning approaches to deal with graph data . Fundamentally , the principle underpinning most GNNs is that of exploiting the structure of the graph to propagate the node feature information iteratively . One of the first papers to propose the idea of GNNs is that of Scarselli et al . ( 2008 ) , where an information diffusion mechanism is used to learn the nodes ’ latent representations by exchanging neighbourhood information . Depending on the form this diffusion takes , Bronstein et al . ( 2021 ) distinguish between convolutional ( Kipf & Welling , 2017 ; Atwood & Towsley , 2016 ; Levie et al. , 2018 ) , atten- tional Veličković et al . ( 2018 ) , or message passing ( Gilmer et al. , 2017 ) types of GNNs , with the latter being the most general and formally equivalent to the Weisfeiler-Lehman graph isomorphism test under some technical conditions ( Xu et al. , 2018 ; Morris et al. , 2019 ) . More recently , Eliasof et al . ( 2021 ) proposed a reinterpretation of graph convolution in terms of partial differential equations on graphs , noting that different problems can benefit from different networks dynamics , hence suggesting the need to look beyond diffusion . For a comprehensive survey of GNNs we refer the reader to Wu et al . ( 2020 ) . In this work , we argue that a natural extension of the convolution operation to the graph domain , and thus an ideal candidate to build graph CNNs , already exists in the form of graph kernels . A number of works in the literature have investigated potential synergies between GNNs and kernels . Lei et al . ( 2017 ) , introduce a class of deep recurrent neural architectures show that this lies in the reproducing kernel Hilbert space ( RKHS ) of graph kernels . Nikolentzos et al . ( 2018 ) compute continuous embeddings of graphs using kernels and plug them into a neural network . Xu et al . ( 2018 ) ; Morris et al . ( 2019 ) show that GNNs have the same expressiveness as the WL graph kernel ( Shervashidze et al. , 2011 ) and propose a new generalised architecture with increased expressive power . Du et al . ( 2019 ) take a somewhat different approach and exploit neural networks to introduce a new graph kernel kernel . This in turn is shown to be equivalent to an infinitely-wide GNN initialized with random weights and trained with gradient descent . Chen et al . ( 2020 ) propose a graph neural architecture where each layer enumerates local sub-structures around each node and then maps them to a RKHS via a Gaussian kernel mapping . In the context of graph compression , Bouritsas et al . ( 2021 ) learn how to best decompose the graph into small substructures that are also learned . The closest work to ours we are aware of is that of Nikolentzos & Vazirgiannis ( 2020 ) , who propose a GNN where the first layer consists of a series of hidden graphs that are compared against the input graph using a random walk kernel . However , due to their optimisation strategy , their model only works with a single differentiable kernel , whereas our model allows us to tap into the expressive power of any type and number of kernels . Moreover , in the architecture of Nikolentzos & Vazirgiannis ( 2020 ) the learned structural masks are assumed to be complete weighted graphs , whereas we allow for graphs with arbitrary structure . 3 GRAPH NEURAL NETWORKS FROM GRAPH KERNELS . 3.1 PRELIMINARIES . Kernel methods are a class of algorithms that are capable of learning when presented with a particular pairwise similarity measure on the input data , known as a kernel . Consider a set X and a positive semi-definite kernel function K : X × X → R such that there exists a map φ : X → H into a Hilbert space H and K ( x , y ) = φ ( x ) > φ ( y ) for all x , y ∈ X . Crucially , X can represent any set of data on which a kernel can be defined , from Rd to a finite set of graphs . Hence , the field of machine learning is ripe with examples of graph kernels ( see Section 2 ) , which are nothing but positive semi-definite pairwise similarity measures on graphs . These can be either implicit ( only K is computed ) or explicit ( φ is also computed ) . Weisfeiler-Lehman test . The WL kernel ( Shervashidze et al. , 2011 ) is one of the most powerful and commonly used graph kernels and it is based on the 1-dimensional Weisfeiler-Lehman ( 1-WL ) graph isomorphism test . The idea underpinning the test is that of partitioning the node set by iteratively propagating the node labels between adjacent nodes . With each iteration , the set of labels accumulated at each node of the graph is then mapped to a new label through a hash function . This procedure is repeated until the cardinality of the set of labels stops growing . Two graphs can then be compared in terms of their label sets at convergence , with two graphs being isomorphic only if their label sets coincide . Learning on graphs . Let G = ( V , X , E ) be an undirected graph with |V | nodes and |E| edges , where each node v is associated to a label x ( v ) belonging to a dictionary D. A common goal in graph machine learning problems is to produce a vector representation of G that is aware of both the node labels and the structural information of G. Generalizing the convolution operator to graphs , GNNs learn the parameters Θ of a function h that embeds the graph into a vector Z by performing message passing ( Gilmer et al. , 2017 ) in the 1-hop neighborhood N 1G ( v ) of each node v ∈ V , i.e. , z ( v ) = ∑ u∈N 1G ( v ) hΘ ( x ( v ) , x ( u ) ) . ( 1 ) The convolution is usually repeated for L layers , and the final vectorial representation Z is obtained by applying a node-wise permutation invariant aggregation operator on the node features z ( v ) .

Summary: In this paper, the authors propose a model that uses graph kernels to extend the convolution operator to the graph domain. Given a graph, the model extracts a subgraph centered around each node of the graph and generates a representation for that node by comparing its subgraph against a number of structural masks using graph kernels. Since graph kernels are not in general differentiable functions, the authors use a discrete randomized descent strategy to optimize the structural masks. The authors also add a Jensen-Shannon divergence loss term to force the masks provide dissimilar responses from each other. The proposed model was evaluated on standard graph classification datasets where it was found to be competitive with the baselines.