IsoNN: Isomorphic Neural Network for Graph Representation Learning and Classification

1 INTRODUCTION . The graph structure is attracting increasing interests because of its great representation power on various types of data . Researchers have done many analyses based on different types of graphs , such as social networks , brain networks and biological networks . In this paper , we will focus on the binary graph classification problem , which has extensive applications in the real world . For example , one may wish to identify the social community categories according to the users ’ social interactions ( Gao et al. , 2017 ) , distinguish the brain states of patients via their brain networks ( Wang et al. , 2017 ) , and classify the functions of proteins in a biological interaction network ( Hamilton et al. , 2017 ) . To address the graph classification task , many approaches have been proposed . One way to estimate the usefulness of subgraph features is feature evaluation criteria based on both labeled and unlabeled graphs ( Kong & Yu , 2010 ) . Some other works also proposed to design a pattern exploration approach based on pattern co-occurrence and build the classification model ( Jin et al. , 2009 ) or develop a boosting algorithm ( Wu et al. , 2014 ) . However , such works based on BFS or DFS can not avoid computing a large quantity of possible subgraphs , which causes high computational complexity though the explicit subgraphs are maintained . Recently , deep learning models are also widely used to solve the graph-oriented problems . Although some deep models like MPNN ( Gilmer et al. , 2017 ) and GCN ( Kipf & Welling , 2016 ) learn implicit structural features , the explict structural information can not be maintained for further research . Besides , most existing works on graph classification use the aggregation of the node features in graphs as the graph representation ( Xu et al. , 2018 ; Hamilton et al. , 2017 ) , but simply doing aggregation on the whole graph can not capture the substructure precisely . While there are other models can capture the subgraphs , they often need more complex computation and mechanism ( Wang et al. , 2017 ; Narayanan et al. , 2017 ) or need additonal node labels to find the subgraph strcuture ( Gaüzere et al. , 2012 ; Shervashidze et al. , 2011 ) . However , we should notice that when we deal with the graph-structured data , different node-orders will result in very different adjacency matrix representations for most existing deep models which take the adjacency matrices as input if there is no other information on graph . Therefore , compared with the original graph , matrix naturally poses a redundant constraint on the graph node-order . Such a node-order is usually unnecessary and manually defined . The different graph matrix representations brought by the node-order differences may render the learning performance of the existing models to be extremely erratic and not robust . Formally , we summarize the encountered challenges in the graph classification problem as follows : • Explicit useful subgraph extraction . The existing works have proposed many discriminative models to discover useful subgraphs for graph classification , and most of them require manual efforts . Nevertheless , how to select the contributing subgraphs automatically without any additional manual involvement is a challenging problem . • Graph representation learning . Representing graphs in the vector space is an important task since it facilitates the storage , parallelism and the usage of machine learning models for the graph data . Extensive works have been done on node representations ( Grover & Leskovec , 2016 ; Lin et al. , 2015 ; Lai et al. , 2017 ; Hamilton et al. , 2017 ) , whereas learning the representation of the whole graph with clear interpretability is still an open problem requiring more explorations . • Node-order elimination for subgraphs . Nodes in graphs are orderless , whereas the matrix representations of graphs cast an unnecessary order on nodes , which also renders the features extracted with the existing learning models , e.g. , CNN , to be useless for the graphs . For subgraphs , this problem also exists . Thus , how to break such a node-order constraint for subgraphs is challenging . • Efficient matching for large subgraphs . To break the node-order , we will try all possible node permutations to find the best permutation for a subgraph . Clearly , trying all possible permutaions is a combinatorial explosion problem , which is extremly time-comsuming for finding large subgraph templates . The problem shows that how to accelerate the proposed model for large subgraphs also needs to be solved . In this paper , we propose a novel model , namely Isomorphic Neural Network ( ISONN ) and its variants , to address the aforementioned challenges in the graph representation learning and classification problem . ISONN is composed of two components : the graph isomorphic feature extraction component and the classification component , aiming at learning isomorphic features and classifying graph instances , respectively . In the graph isomorphic feature extraction component , ISONN automatically learns a group of subgraph templates of useful patterns from the input graph . ISONN makes use of a set of permutation matrices , which act as the node isomorphism mappings between the templates and the input graph . With the potential isomorphic features learned by all the permutation matrices and the templates , ISONN adopts one min-pooling layer to find the best node permutation for each template and one softmax layer to normalize and fuse all subgraph features learned by different kernels , respectively . Such features learned by different kernels will be fused together and fed as the input for the classification component . ISONN further adopts three fully-connected layers as the classification component to project the graph instances to their labels . Moreover , to accelerate the proposed model when dealing with large subgraphs , we also propose two variants of ISONN to gurantee the efficiency . 2 RELATED WORK . Our work relates to subgraph mining , graph neural networks , network embedding as well as graph classification . We will discuss them briefly in the followings . Subgraph Mining and Graph Kernel Methods : Mining subgraph features from graph data has been studied for many years . The aim is to extract useful subgraph features from a set of graphs by adopting some specific criteria . One classic unsupervised method ( i.e. , without label information ) is gSpan ( Yan & Han , 2002 ) , which builds a lexicographic order among graphs and map each graph to a unique minimum DFS code as its canonical label . GRAMI ( Elseidy et al. , 2014 ) only stores templates of frequent subgraphs and treat the frequency evaluation as a constraint satisfaction problem to find the minimal set . For the supervised model ( i.e. , with label information ) , CORK utilizes labels to guide the feature selection , where the features are generated by gSpan ( Thoma et al. , 2009 ) . Due to the mature development of the sub-graph mining field , subgraph mining methods have also been adopted in life sciences ( Mrzic et al. , 2018 ) . Moreover , several parallel computing based methods ( Qiao et al. , 2018 ; Hill et al. , 2012 ; Lin et al. , 2014 ) have proposed to reduce the time cost . On the other hand , graph kernel methods are also applied to discover the subgraph structures ( Kashima et al. , 2003 ; Vishwanathan et al. , 2010 ; Gaüzere et al. , 2012 ; Shervashidze et al. , 2011 ) . Among them , most existing works focus on the graph with node labels and the kernels methods only computes the similarity between pairwise graphs . Yet , in this paper , we are handling the graph without node labels . Moreover , we can not only compute the similarity between pairwise graphs but also learn subgraph templates , which can be further analyzed . Graph Neural Network and Network Embedding : Graph Neural Networks ( Monti et al. , 2017 ; Atwood & Towsley , 2016 ; Masci et al. , 2015 ; Kipf & Welling , 2016 ; Battaglia et al. , 2018 ) have been studied in recent years because of the prosperity of deep learning . Traditional deep models can not be directly applied to graphs due to the special data structure . The general graph neural model MoNet ( Monti et al. , 2017 ) employs CNN architectures on non-Euclidean domains such as graphs and manifold . The GCN proposed in ( Kipf & Welling , 2016 ) utilizes the normalized adjacency matrix to learn the node features for node classification ; ( Bai et al. , 2018 ) proposes the multiscale convolutional model for pairwise graph similarity with a set matching based graph similarity computation . However , these existing works based on graph neural networks all fail to investigate the node-orderless property of the graph data and to maintain the explicit structural information . Another important topic related to this paper is network embedding ( Bordes et al. , 2013 ; Lin et al. , 2015 ; Lai et al. , 2017 ; Abu-El-Haija et al. , 2018 ; Hamilton et al. , 2017 ) , which aims at learning the feature representation of each individual node in a network based on either the network structure or attribute information . Distinct from these network embedding works , the graph representation learning problem studied in this paper treats each graph as an individual instance and focuses on learning the representation of the whole graph instead . Graph Classification : Graph classification is an important problem with many practical applications . Data like social networks , chemical compounds , brain networks can be represented as graphs naturally and they can have applications such as community detection ( Zhang et al. , 2018 ) , anti-cancer activity identification ( Kong et al. , 2013 ; Kong & Yu , 2010 ) and Alzheimer ’ s patients diagnosis ( Tong et al. , 2017 ; 2015 ) respectively . Traditionally , researchers mine the subgraphs by DFS or BFS ( Saigo et al. , 2009 ; Kong et al. , 2013 ) , and use them as the features . With the rapid development of deep learning ( DL ) , many works are done based on DL methods . GAM builds the model by RNN with self-attention mechanism ( Lee et al. , 2018 ) . DCNN extend CNN to general graph-structured data by introducing a ‘ diffusion-convolution ’ operation ( Atwood & Towsley , 2016 ) . 3 TERMINOLOGY AND PROBLEM DEFINITION . In this section , we will define the notations and the terminologies used in this paper and give the formulation for the graph classification problem . 3.1 NOTATIONS . In the following sections , we will use lower case letters like x to denote scalars , lower case bold letters ( e.g . x ) to represent vectors , bold-face capital letters ( e.g . X ) to show the matrices . For tensors or sets , capital calligraphic letters are used to denote them . We use xi to represent the i-th element in x . Given a matrix X , we use X ( i , j ) to express the element in i-th row and j-th column . For i-th row vector and j-th column vector , we use X ( i , : ) and X ( : , j ) to denote respectively . Moreover , notations x > and X > denote the transpose of vector x and matrix X respectively . Besides , the F -norm of matrix X can be represented as ‖X‖F = ( ∑ i , j |Xi , j |2 ) 1 2 .

Summary: This paper proposes a new neural network architecture for dealing with graphs dealing with the lack of order of the nodes. The first step called the graph isomorphic layer compute features invariant to the order of nodes by extracting sub-graphs and cosidering all possible permutation of these subgraphs. There is no training involved here as no parameter is learned. Indeed the only learning part is in the so-called classification component which is a (standard) fully connected layer. In my opinion, any classification algorithm could be used on the features extracted from the graphs.