GRAPH ANALYSIS AND GRAPH POOLING IN THE SPATIAL DOMAIN

1 INTRODUCTION . Many of the modern data are naturally represented by graphs ( Bronstein et al. , 2017 ; Cook & Holder , 2006 ; Dadaneh & Qian , 2016 ; Qi et al. , 2017b ) and it is an important research problem to design neural network architectures which can work with graphs . The adjacency matrix of a graph exhibits the local connectivity of the nodes . Thus , it is straightforward to extend the local feature aggregation used in the Convolutional Neural Networks ( CNNs ) to the Graph Neural Networks ( GNNs ) ( Atwood & Towsley , 2016 ; Bruna et al. , 2014 ; Fey et al. , 2018 ; Gilmer et al. , 2017 ; Niepert et al. , 2016 ; Simonovsky & Komodakis , 2017 ; Zhang et al. , 2018 ) . However , when the nodes of the graphs are not labeled/attributed or the labels/attributes of the nodes do not carry information about the differences between the nodes or any information about their locations in the graph , the GNN can fail to extract discriminative features . The GNN maps nodes whose corresponding local structures are similar to same/close feature vectors in the continuous feature space . Although mapping nodes with similar local structures to similar feature vectors might be desirable in some applications ( Henderson et al. , 2012 ) , this feature can make the GNN unable to distinguish graphs whose difference is not in their local structures . In addition , when the nodes are not labeled/attributed , the spatial graph convolution only propagates information about the degree of the nodes which might not lead to extracting informative features about the topological structure of the graph . Another limitation of the current GNN architectures is that they are mostly unable to do the hierarchical feature learning employed in the CNNs ( He et al. , 2016 ; Krizhevsky et al. , 2012 ) . The main reason is that graphs lack the tensor representation and it is difficult to measure how accurate a subset of nodes represent the topological structure of the given graph . Summary of Contributions . In this paper , we focus on the graph classification problem . The main contributions of this paper can be summarized as follows . • A shortcoming of the GNNs is discussed and it is shown that the existing GNNs can fail to learn to perform even simple graph analysis tasks . It is shown that the proposed approach which leverages a spatial representation of the graph effectively addresses the shortcoming of the GNN . Several new experiments are presented to demonstrate the shortcoming of the GNNs and they show that providing the geometrical representation of the graph to the neural network substantially improves the capability of the GNN in inferring the structure of the graph . • The geometrical representation of the graph is leveraged to design a novel graph pooling method . The proposed approach simplifies the graph down sampling problem into a column/row sampling problem . The proposed approach samples a subset of the nodes such that they preserve the structure of the graph . It is shown that the proposed approach achieves competitive or better performance in comparison with the existing methods . Notation . Given a vector x , ‖x‖p denotes its ` p Euclidean norm , and x ( i ) denotes its ith element . Given a matrix X , xi denotes the ith row of X . A graph with n nodes is represented by two matrices A ∈ Rn×n and F ∈ Rn×df , where A is the adjacency matrix , F is the matrix of node labels/attributes , and df is the dimension of the attributes/labels of the nodes . The operation A⇐ B means that the content of A is set equal to the content of B . If I is a set of indices , XI is the matrix of the rows of X whose indexes are in I. Vector y = softmax ( x ) is defined as y ( i ) = exp ( x ( i ) ) ∑ k exp ( x ( k ) ) . The local structure corresponding to a node is the structure of the graph in the close neighbourhood of the node . 2 RELATED WORK . This paper mainly focuses on the graph classification problem . In the proposed approach , the geometrical representation of the graph provided by a graph embedding algorithm is utilized to make the neural network aware of the topological structure of the graph . In this section , some of the related research works in GNN and graph embedding are briefly reviewed . Graph Embedding : A graph embedding method aims at finding a continuous embedding vector for each node of the graph such that the topological structure of the graph is encoded in the spatial distribution of the embedding vectors . The nodes which are close on the graph or they share similar structural role are mapped to nearby points in the embedding space and vice versa . In general , the graph embedding methods fall into three broad categories : matrix factorization based methods ( Ahmed et al. , 2013 ; Belkin & Niyogi , 2002 ; Cao et al. , 2015 ; Ou et al. , 2016 ; Roweis & Saul , 2000 ) , random-walk based approaches ( Grover & Leskovec , 2016 ; Perozzi et al. , 2014 ) , and deep learning based methods ( Wang et al. , 2016 ) . In this paper , we use the DeepWalk graph embedding method ( Perozzi et al. , 2014 ) which is a random walked based method . Graph Neural Networks : In recent years , there has been a surge of interest in developing deep network architectures which can work with graphs ( Bronstein et al. , 2017 ; Bruna et al. , 2014 ; Duvenaud et al. , 2015 ; Fout et al. , 2017 ; Gilmer et al. , 2017 ; Hamilton et al. , 2017 ; Kipf et al. , 2018 ; Kipf & Welling , 2017 ; Li et al. , 2016 ; Niepert et al. , 2016 ; Simonovsky & Komodakis , 2017 ; Tixier et al. , 2019 ) . Local connectivity , weight sharing , and shift invariance of the convolution layer in the CNNs have led to remarkable achievements in computer vision and natural language processing ( Goodfellow et al. , 2016 ) . Accordingly , there are remarkable works focusing on the design of graph convolution layers encoding the traditional properties . Most of the existing graph convolution layers can be loosely divided into two main subsets : the spatial convolution layers and the spectral convolution layers . The spectral methods are based on the generalization of spectral filtering in graph signal processing ( Bruna et al. , 2014 ; Defferrard et al. , 2016 ; Henaff et al. , 2015 ; Levie et al. , 2017 ) . The major drawback of the spectral based methods is the non-generalizability to data residing over multiple graphs which is due to the dependency on the basis of the graph Laplacian . In contrast to spectral graph convolution , the spatial graph convolution performs the convolution directly in the nodal domain and can be generalized across graphs ( Nguyen et al. , 2018 ; Niepert et al. , 2016 ; Schlichtkrull et al. , 2018 ; Simonovsky & Komodakis , 2017 ; Veličković et al. , 2017 ; Verma & Zhang , 2018 ; Zhang et al. , 2018 ) . If the sum function is used to aggregate the local feature vectors , a simple spatial convolution layer can be written as X⇐ A f ( XΦ1 ) + f ( XΦ2 ) , ( 1 ) where f ( · ) is the element-wise non-linear function . Some papers use a normalized version of ( 1 ) via multiplying ( 1 ) with the inverse of the degree matrix ( Ying et al. , 2018 ) . The weight matrix Φ2 transforms the feature vector of the given node and Φ1 transforms the feature vectors of the neighbouring nodes . A drawback of the spatial convolution is that the aggregation function might not be an injective function , i.e. , two nodes with different local structures can be mapped to the same feature vector by the convolution layer . The authors of ( Xu et al. , 2019 ) showed that with a minor modification , the sum aggregation function can become an injective function . However , when the nodes are not labeled/attributed or the labels/attributes do not inform the GNN about the location of the nodes , the GNN is not able to distinguish graphs whose local structures are similar but their global structures are different . The approach proposed in our paper not only addresses the problem raised in ( Xu et al. , 2019 ) , it also helps the GNN to distinguish the similar local structures in different locations of the graph . In order to obtain a global representation of the graph , the local feature vectors obtained by the convolution layers should be aggregated into a final feature vector . The element-wise max/mean functions are used widely to aggregate all the local feature vectors . Inspired by the hierarchical feature extraction in the CNNs , tools have been developed to perform nonlocal feature aggregation ( Duvenaud et al. , 2015 ; Luzhnica et al. , 2019 ) . In ( Zhang et al. , 2018 ) , the nodes are ranked and the ranking is used to build a sequence of nodes using a subset of the nodes . Subsequently , a 1- dimensional CNN is applied to the sequence of the nodes to perform non-local feature aggregation . However , the way that ( Zhang et al. , 2018 ) builds the sequence of the nodes is not data driven . In addition , a 1-dimensional array might not have the capacity to preserve the topological structure of the graph . In ( Ying et al. , 2018 ) , a soft graph pooling method was proposed which learns a set of cluster centers that are used to down-sample the extracted local feature vectors . The graph pooling methods proposed in ( Gao & Ji , 2019 ; Lee et al. , 2019 ) are similar to the soft graph pooling method presented in ( Ying et al. , 2018 ) but they learn one cluster center which is used to rank the nodes and they sample the nodes which are closer to the learned cluster center . In ( Defferrard et al. , 2016 ; Fey et al. , 2018 ) , the graph clustering algorithms were used to perform graph down-sizing . 3 THE SHORTCOMING OF THE GNNS . Suppose the nodes of the given graphs are not labeled/attributed or assume that the labels/attributes do not contain any information about the role/location of the nodes in the global structure of the graph . Therefore , if two different nodes are not labeled or they are labeled similarly , they appear the same to the GNN . Accordingly , if the local structures corresponding to two nodes are similar , the convolution layer of the GNN maps them to the same feature vector in the continuous feature space . This means that if the local structures of two different graphs are similar , the GNN can not extract discriminative features to distinguish them . In the followings , we study an example to clarify this feature of the GNN . Illustrative example : Suppose we have a dataset of clustered unlabeled graphs ( nodes and edges are not labeled/attributed ) and assume that the clusters are not topologically different ( a common generator created all the clusters ) . Every node is connected to small a set of the other nodes in its corresponding cluster . One class of the graphs consist of two clusters and the other class of graphs consist of three clusters . Therefore , the task is to infer if a graph is made of two clusters or three clusters . Consider the simplest case in which there is no connection between the clusters . Assume that we use a typical GNN which is composed of multiple spatial convolution layers and a global element-wise max/mean pooling layer . Suppose that a given graph belongs to the first class , i.e. , it consists of two clusters . Define v1 and v2 as the aggregation of the local feature vectors corresponding to the first and the second clusters , respectively . The global feature vector of this graph is equal to the element-wise mean/max of v1 and v2 . Clearly , v2 can be indistinguishable from v1 since the clusters are generated using the same generator and the GNN is not aware of the location of the nodes . Therefore , the feature vector of the whole graph can also be indistinguishable from v1 . The same argument is also true for a graph with three clusters . Accordingly , the representation obtained by the GNN is unable to distinguish this two classes of graphs . If one trains a GNN on this task , the test accuracy is around 50 % which is not better than random guess . The main reason is that the local structures corresponding to all the nodes in a graph are similar . In addition , the local structures in a graph with two clusters is not different from the local structures in a graph with three clusters . Since the GNN is not aware of the location of the nodes , all the nodes are mapped to similar feature vectors in the feature space . Accordingly , the spatial distribution of the local feature vectors of the two classes of the graphs are indistinguishable . 1

Summary: In this work the authors point out an issue related to graph neural networks. Specifically, if two nodes, that may be far apart in the graph, may be represented as (almost) the same vector. This is simply because when no features/labels are associated with nodes, and the local structure around those two nodes is very similar then the local aggregation of information will result in a similar representation.  Therefore the authors introduce an embedding first of the graph in the Euclidean space using DeepWalk and then use this embedding in combination with the design of a CNN. The authors propose a pooling method that outperforms several state-of-the-art pooling techniques on real data. Overall, the empirical results are supportive of the fact that the proposed method can help improve the performance of GNNs.