Graph Structural Aggregation for Explainable Learning

1 INTRODUCTION . Convolution neural networks ( LeCun et al. , 1995 ) have proven to be very efficient at learning meaningful patterns for many articificial intelligence tasks . They convey the ability to learn hierarchical information in data with Euclidean grid-like structures such as images and text . Convolutional Neural Networks ( CNNs ) have rapidly become state-of-the art methods in the fields of computer vision ( Russakovsky et al. , 2015 ) and natural language processing ( Devlin et al. , 2018 ) . However in many scientific fields , studied data have an underlying graph or manifold structure such as communication networks ( whether social or technical ) or knowledge graphs . Recently there have been many attempts to extend convolutions to those non-Euclidean structured data ( Hammond et al. , 2011 ; Kipf & Welling , 2016 ; Defferrard et al. , 2016 ) . In these new approaches , the authors propose to compute node embeddings in a semi-supervised fashion in order to perform node classification . Those node embeddings can also be used for link prediction by computing distances between each node of the graph ( Hammond et al. , 2011 ; Kipf & Welling , 2016 ) . Graph classification is studied in many fields . Whether for predicting the chemical activity of a molecule or to cluster authors from different scientific domains based on their ego-networks ( Freeman , 1982 ) . However when trying to generalize neural network approaches to the task of graph classification there are several aspects that differ widely from image classification . When trying to perform graph classification , we can deal with graphs of different sizes . To compare them we first need to obtain a graph representation that is independant of the size of the graph . Moreover , for a fixed graph , nodes are not ordered . The graph representation obtained with neural network algorithms must be independant of the order of nodes and thus be invariant by node permutation . Aggregation functions are functions that operate on node embeddings to produce a graph representation . When tackling a graph classification task , the aggregation function used is usually just a mean or a max of node embeddings as illustrated in figure 1b . But when working with graphs of large sizes , the mean over all nodes does not allow us to extract significant patterns with a good discriminating power . In order to identify patterns in graphs , some methods try to identify structural roles for nodes . Donnat et al . ( 2018 ) define structural role discovery as the process of identifying nodes which have topologically similar network neighborhoods while residing in potentially distant areas of the network as illustrated in figure 1a . Those structural roles represent local patterns in graphs . Identifying them and comparing them among graphs could improve the discriminative power of graph embeddings obtained with graph neural networks . In this work , we build an aggregation process based on the identification of structural roles , called StructAgg . The main contributions of this work are summarized bellow : 1 . Learned aggregation process . A differentiable aggregation process that learns how to aggregate node embeddings in order to produce a graph representation for a graph classification task . 2 . Identification of structural roles . Based on the definition of structural roles from Donnat et al . ( 2018 ) , our algorithm learns structural roles during the aggregation process . This is innovative because most algorithms that learn structural roles in graphs are not based on graph neural networks . 3 . Explainability of selected features for a graph classification task . The identification of structural roles enables us to understand and explain what features are selected during training . Graph neural networks often lack explainability and there are only few works that tackle this issue . One contribution of this work is the explainability of the approach . We show how our end-to-end model provides interpretability to a graph classification task based on graph neural networks . 4 . Experimental results . Our method achieves state-of-the-art results on benchmark datasets . We compare it with kernel methods and state-of-the-art message passing algorithms that use pooling layers as aggregation processes . 2 RELATED WORK . The identification of nodes that have similar structural roles is usually done by an explicit featurization of nodes or by algorithms that rely on random walks to explore nodes ’ neighborhoods . A well known algorithm in this line of research is RolX ( Gilpin et al. , 2013 ; Henderson et al. , 2012 ) , a matrix factorization that focuses on computing a soft assignment matrix based on a listing of topological properties set as inputs for nodes . Similarly struct2vec builds a multilayered graph based on topological metrics on nodes and then generates random walks to capture structural information . In another line of research , many works rely on graphlets to capture nodes ’ topological properties and identify nodes with similar neighborhoods ( Rossi et al. , 2017 ; Lee et al. , 2018 ; Ahmed et al. , 2018 ) . In their work , Donnat et al . ( 2018 ) compute node embeddings from wavelets in graphs to caracterize nodes ’ neighborhood at different scales . In this work , we introduce an aggregation process based on the identification of structural roles in graphs that is computed in an end-to-end trainable fashion . We build a hierarchical representation of nodes by using neural network models in graphs to propagate nodes ’ features at different hops . Recently there has been a rich line of research , inspired by deep models in images , that aims at redefining neural networks in graphs and in particular convolutional neural networks ( Defferrard et al. , 2016 ; Kipf & Welling , 2016 ; Veličković et al. , 2017 ; Hamilton et al. , 2017 ; Bronstein et al. , 2017 ; Bruna et al. , 2013 ; Scarselli et al. , 2009 ) . Those convolutions can be viewed as message passing algorithms that are composed of two phases . A message passing phase that runs for T steps is first defined in terms of message functions and vertex update functions . A readout phase then computes a feature vector for the whole graph using some readout function . In this work we will see how to define a readout phase that is learnable and that is representative of meaningful patterns in graphs . 3 PROPOSED METHOD . In this section we introduce the structural aggregation layer ( StructAgg ) . We show how we identify structural classes for nodes in graphs ; how those classes are used in order to develop an aggregation layer ; and how this layer allows us to compare significant structural patterns in graphs for a supervised classification task . 3.1 NOTATIONS . Let G = ( V , E , X ) be a graph , where V is the set of nodes of G , E the set of edges and X ∈ Rn×f the feature matrix of G ’ s nodes where f is the dimensionality of node features . Let n = |V | be the number of nodes of G and e = |E| the number of edges of G. Let A be the adjacency matrix of graph G and D be its degree diagonal matrix . Let vi and vj be the ith and jth nodes of G , we have : Aij = { 1 if ( vi , vj ) ∈ E 0 otherwise , Dii = ∑ j Aij Let S = { G1 , ... , Gd } be a set of d graphs and { y1 , ... , yd } be the labels associated with these graphs . 3.2 HIERARCHICAL STRUCTURAL EMBEDDING . Graph neural networks . We build our work upon graph neural networks ( GNNs ) . Several architectures of graph neural networks have been proposed by Defferrard et al . ( 2016 ) ; Kipf & Welling ( 2016 ) ; Veličković et al . ( 2017 ) or Bruna & Li ( 2017 ) . Those graph neural network models are all based on propagation mechanisms of node features that follow a general neural message passing architecture ( Ying et al. , 2018 ; Gilmer et al. , 2017 ) : X ( l+1 ) =MP ( A , X ( l ) ; W ( l ) ) ( 1 ) where X ( l ) ∈ Rn×fl are the node embeddings computed after l steps of the GNN , X ( 0 ) = X , and MP is the message propagation function , which depends on the adjacency matrix . W ( l ) is a trainable weight matrix that depends on layer l. Let fl be the dimension of the node vectors after l steps of the GNN , f0 = f . The aggregation process that we introduce next can be used with any neural message passing algorithm that follows the propagation rule 1 . In all the following of our work we denote by MP the algorithm . For the experiments , we consider Graph Convolutional Network ( GCN ) defined by ( Kipf & Welling , 2016 ) . This model is based on an approximation of convolutions on graphs defined by ( Defferrard et al. , 2016 ) and that use spectral decompositions of the Laplacian . The popularity of this model comes from its computational efficiency and the state-of-the-art results obtained on benchmark datasets . This layer propagates node features to 1-hop neighbors . Its propagation rule is the following : X ( l+1 ) =MP ( A , X ( l ) ; W ( l ) ) = GCN ( A , X ( l ) ) = ρ ( D̃−1/2ÃD̃−1/2X ( l ) W ( l ) ) ( 2 ) Where ρ is a non-linear function ( a ReLU in our case ) , Ã = A + In is the adjacency matrix with added self-loops and D̃ii = ∑ j Ãij . This propagation process allows us to obtain a node representation representing its l-hop neighborhood after l layers of GCN . We build a hierarchical representation for nodes by concatenating their embeddings after each step of GCN . The final representation Xstructi of a node i is given by : Xstructi = Ln l=1 X ( l ) i ( 3 ) Where L is the total number of GCN layers applied . Identifying structural classes . Embedding nodes with MP creates embeddings that are close for nodes that are structurally equivalent . Some use a handcrafted node embedding based on propagation processes with wavelets in graphs to identify structural clusters based on hierarchical representation of nodes ( Donnat et al. , 2018 ) . By analogy , we learn hierarchical node embeddings and an aggregation layer that identifies structural roles for node in graphs . Those structural roles are consistent along graphs of a dataset which allows us to bring interpretability to our graph classification task . Node features Xstruct contain the information of their L-hop neighborhood decomposed into L vectors each representing their l-hop neighborhood for l varying between 1 and L. We will show next that nodes that have the same L-hop neighborhood are embedded into the same vector . To identify structural roles , we thus project each node embedding on a matrix p ∈ Rfstruct×c where c is the number of structural classes and fstruct = L∑ l=1 fl is the dimensionality of Xstructi for each node i . We obtain a soft assignment matrix : C = softmax ( Xstructp ) ∈ Rn×c ( 4 ) Where the softmax function is applied in a row-wise fashion . This way , Cij represents the probability that node i belongs to cluster j . Definition 1 . Let i and j be two nodes of a graphG = ( V , E , X ) . LetNl ( i ) = { i′ ∈ N |d ( i , i′ ) ≤ l } be the l-hop neighborhood of i , which means all the nodes that are at distance lower of equal to l of i , d being the shortest-path distance . Let XNl ( i ) be the feature matrix of the l-hop neighborhood of u . Let Gi , l be the subgraph of G composed of the l-hop neighborhood of i . We say that i and j are l-structurally equivalent if there exists an isomorphism ψ from Nl ( j ) to Nl ( i ) such that the two following conditions are verified : • Gi , l = ψ ( Gj , l ) • ∀j′ ∈ Nl ( j ) , Xψ ( j′ ) = Xj′ Theorem 1 . Two nodes i and j that are L-structurally equivalent have the same final embedding , Xstructi = Xstructj .

Summary: This paper proposed the StructAgg, an aggregation algorithm in convolutional graph neural network that learns the structural roles for nodes in the graph embedding. In this algorithm, a structural representation of node is constructed through concatenation of latest p layers of node presentation in graph neural network, which consists of information from the p-hop neighborhood. The paper is good quality and its demonstration is clear.  The idea behind is original. However, learning embeddings through concatenation of multiple layers of neural representation is not completely new. The contribution of this paper to the community is not ground-breaking as well.  From the experiment results, it is hard to stress its significance as in most of times this method does not beat the state-of-the-art algorithms.