DEEP GEOMETRICAL GRAPH CLASSIFICATION

Most of the existing Graph Neural Networks ( GNNs ) are the mere extension of the Convolutional Neural Networks ( CNNs ) to graphs . Generally , they consist of several steps of message passing between the nodes followed by a global indiscriminate feature pooling function . In many data-sets , however , the nodes are unlabeled or their labels provide no information about the similarity between the nodes and the locations of the nodes in the graph . Accordingly , message passing may not propagate helpful information throughout the graph . We show that this conventional approach can fail to learn to perform even simple graph classification tasks . We alleviate this serious shortcoming of the GNNs by making them a two step method . In the first of the proposed approach , a graph embedding algorithm is utilized to obtain a continuous feature vector for each node of the graph . The embedding algorithm represents the graph as a point-cloud in the embedding space . In the second step , the GNN is applied to the point-cloud representation of the graph provided by the embedding method . The GNN learns to perform the given task by inferring the topological structure of the graph encoded in the spatial distribution of the embedded vectors . In addition , we extend the proposed approach to the graph clustering problem and an architecture for graph clustering is presented . Moreover , the spatial representation of the graph is utilized to design a new graph pooling algorithm . We turn the problem of graph down-sampling into a column sampling problem , i.e. , the sampling algorithm selects a subset of the nodes whose feature vectors preserve the spatial distribution of all the feature vectors . We apply the proposed approach to several popular benchmark data-sets and it is shown that the proposed geometrical approach strongly improves the state-of-the-art result for several data-sets . For instance , for the PTC data-set , the state-of-the-art result is improved for more than 22 % . 1 INTRODUCTION . Many of the modern data/signals are naturally represented by graphs ( Bronstein et al. , 2017 ; Dadaneh & Qian , 2016 ; Cook & Holder , 2006 ; Qi et al. , 2017b ) and it is of great interest to design data analysis algorithms which can work directly with graphs . Due to the remarkable success of deep learning based methods in many machine learning applications , it is an attractive research problem to design new neural network architectures which can make the deep networks able to work with graphs . The adjacency matrix of a graph exhibits the local connectivity of the nodes . Thus , it is straightforward to extend the local feature aggregation in the Convolutional Neural Networks ( CNNs ) to the Graph Neural Networks ( GNNs ) ( Simonovsky & Komodakis , 2017 ; Zhang et al. , 2018 ; Atwood & Towsley , 2016 ; Niepert et al. , 2016 ; Bruna et al. , 2013 ; Fey et al. , 2018 ) . The local feature aggregation in the graphs is equivalent to message passing between the nodes ( Zhang et al. , 2018 ; Gilmer et al. , 2017 ) . However , if the graph is not labeled ( the nodes/edges do not have feature vectors ) or the feature vectors of the nodes do not carry information about the similarities between the nodes or any information about their structural role , the message passing may not propagate informative features throughout the graph . Thus , the GNN can fail to infer the topological structure of the graph . Another limitation of the current GNN architectures is that they are mostly unable to do the hierarchical feature learning employed in the CNNs ( Krizhevsky et al. , 2012 ; He et al. , 2016 ) . The main reason is that graphs lack the tensor representation and it is hard to measure how accurate a subset of nodes represent the topological structure of the given graph . In this paper , we address these two shortcomings of the GNNs . In the proposed approach , we provide a spatial representation of the graph to the GNN . The spatial representation makes the network aware of the similarity/difference between the nodes and also the location of the nodes in the graph . Accordingly , the local feature aggregation can propagate informative messages throughout the graph . In addition , we propose a down-sampling method to perform graph pooling . The proposed node sampling approach measures the similarities between the nodes in the spatial domain and merges the closest pairs of nodes . The main contributions of the proposed approach can be summarized as follows : • It is shown that the existing GNNs lack the important embedding step . The proposed approach enhances the capability of the GNNs in inferring the topological structure of the graphs . • A new graph pooling method is proposed which can be implemented in any GNN . The proposed pooling method preserves the spatial representation of the graph by merging the closest pairs of nodes in the spatial domain . • To the best of our knowledge , the proposed approach advances the state-of-the-art results for 5 established benchmark data-sets . For instance , the improvement in accuracy with the PTC data-set is more than 22 % . • An architecture for graph clustering is presented . In contrary to the conventional GNNs which can not be trained to cluster unlabeled graphs , the proposed approach utilizes the geometrical representation of the graph to infer the structure of the graph . 1.1 NOTATION . We use bold-face upper-case letters to denote matrices and bold-face lower-case letters to denote vectors . Given a vector x , ‖x‖p denotes its ` p Euclidean norm , and x ( i ) denotes its ith element . Given a setA , A [ i ] denotes the ith element of the set . Given a matrix X , xi denotes the ith row of X , xi denotes the ith column of X , and X ( i , j ) = xi ( j ) . A graph with n nodes is represented by two matrices A ∈ Rn×n and X ∈ Rn×d1 , where A is the adjacency matrix , X is the matrix of feature vectors of the nodes , and d1 is the dimension of the attributes of the nodes . The matrix of feature vectors at the input of the lth layer of the network is denoted by Xl ∈ Rnl×dl , where nl is the size of the graph at the output of the ( l− 1 ) th layer of the network and dl is the length of the feature vectors at the output of the ( l − 1 ) th layer of the network ( X0 = X and n0 = n ) . The set Ii contains the indices of the neighbouring nodes of the ith node . We also include i in Ii . The operation A ⇐ B means that the content of A is set equal to the content of B . The function byc returns the closest integer which is smaller than or equal to the real number y . 2 RELATED WORK . The proposed approach translates the graph analysis problem into a point-cloud ( Qi et al. , 2017a ) analysis problem . In the proposed method , the deep network is given the geometrical representation of the graph which is obtained by a graph embedding algorithm . Therefore , in this section we review some of the related research works in graph classification and graph embedding . Graph Embedding : A graph embedding method aims at finding a continuous feature vector for each node of the graph such that the topological structure of the graph is encoded in the spatial distribution of the feature vectors . The nodes which are close on the graph or they share similar structural role are mapped to nearby points by the embedding method . The graph embedding methods such as DeepWlak ( Perozzi et al. , 2014 ) and Node2Vec ( Grover & Leskovec , 2016 ) generalize the recent advances in word embedding to graphs . For instance , DeepWalk uses the local information obtained from the random walks to learn the nodes representations by treating the random walks as sentences . We refer the reader to ( Ivanov & Burnaev , 2018 ; Perozzi et al. , 2014 ; Grover & Leskovec , 2016 ) and the references therein for a more comprehensive review of graph embedding . Kernel Based Graph Classification Methods : Graph kernels are graph analysis tools which are able to measure the similarity between two graphs . They make the kernel machines such as SVM able to work directly on graphs . The primary idea , proposed in ( Haussler , 1999 ) , is to decompose a graph into small substructures and to compute the similarity between two graph by adding up the pairwise similarities between these components . The main difference between the graph kernels is in their choice of the substructures . The substructures include walks ( Vishwanathan et al. , 2010 ) , subgraphs ( Kriege & Mutzel , 2012 ) , paths ( Borgwardt & Kriegel , 2005 ) , and sub-trees ( Shervashidze et al. , 2011 ) . Despite the success of graph kernels , they have two major drawbacks . First , the graph kernels are computationally expensive . These methods need to fill out the kernel matrix by computing the similarity between every two graphs in the training data-set . Thus , the complexity of training a kernel based method scales with the square of the size of the training data . In addition , computing each element of the kernel matrix can be computationally expensive . For instance , the computation complexity of the shortest path graph kernel scales with the square of the number of nodes . This high computation complexity makes the algorithm inapplicable to large graphs . In Section 7.2 , we compare the performance of the proposed method with some of the graph kernels . Although the computation complexity of the proposed approach can be linear with the number of nodes and the complexity of the training of the proposed method is linear with the size of the training data , it strongly outperforms them on most of the data-sets . The second problem with graph kernels is that the features that they use for classification is independent from the data-set . In contrary to the deep learning based methods , the extracted features are not data driven . Graph Neural Networks : In recent years , there has been a surge of interest in developing deep network architectures which can work with graphs . The main trend is to adopt the structure of the convolutional networks . In contrary to images , we do not have any meaningful order of the neighbouring nodes in graphs . Thus , we can not assign a specific weight to each neighbouring node of a given node . Accordingly , the feature vector of a given node is uniformly aggregated with the feature vectors of its neighbouring nodes . Specifically , the matrix of feature vectors X is updated as X⇐ f ( AXW ) , where W is the weights matrix assigned to all the nodes and f ( · ) is the elementwise non-linear function . If this operation is repeated k-times , a given node receives ” messages ” from all the nodes with distance up to k steps away ( Gilmer et al. , 2017 ) . In addition to the lack of tensor representation in graphs , it is not straightforward to perform pooling on graphs . Thus , the standard approach in the GNNs is to globally aggregate all the feature vectors of the nodes using the element-wise mean function or the element-wise max function . In order to avoid the aggregation over the whole graph and to ” keep more vertex information ” , the recent architecture presented in ( Zhang et al. , 2018 ) builds a sequence of a set of sampled nodes and applies a one dimensional CNN to the sequence of the nodes . However , it is evident that a one dimensional space do not have the capacity to represent the structure of a graph . For instance , we lose all the spatial information in an image if we sort the pixels ( or their feature vectors ) in a 1-dimensional array . In ( Ying et al. , 2018 ) , a soft graph pooling method was proposed . The pooling method proposed in ( Ying et al. , 2018 ) maps the nodes with similar adjacency vectors to the same cluster in the pooled graph . However , large graphs are mostly sparsely conneted . Thus , the adjacency vectors of two nodes which lie in the same cluster can be orthogonal . Therefore , the pooling method presented in ( Ying et al. , 2018 ) might not be applicable to large graphs . In addition , the soft graph pooling method does not preserve the sparsity of the adjacency matrix . In ( Defferrard et al. , 2016 ; Fey et al. , 2018 ) , the graph clustering algorithms were combined with the GNN to perform the graph downsizing . In this paper , we do not assume that the given graph is clustered and the graph pooling is performed dynamically with respect to the distribution of the learned feature vectors .

Summary: The authors argue that graph neural networks based on the message passing frameworks are not able to infer the topological structure of graphs. Therefore, they propose to use the node embedding features from DeepWalk as (additional) input for the graph convolution. Moreover, a graph pooling operator is proposed, which clusters node pairs in a greedy fashion based on the l2-distances between feature vectors. The proposed methods are evaluated on seven common benchmark datasets and achieve better or comparable results to existing methods. Moreover, the method is evaluated using synthetic toy examples, showing that the proposed extensions help to infer topological structures.