Neural Pooling for Graph Neural Networks

Tasks such as graph classification , require graph pooling to learn graph-level representations from constituent node representations . In this work , we propose two novel methods using fully connected neural network layers for graph pooling , namely Neural Pooling Method 1 and 2 . Our proposed methods have the ability to handle variable number of nodes in different graphs , and are also invariant to the isomorphic structures of graphs . In addition , compared to existing graph pooling methods , our proposed methods are able to capture information from all nodes , collect second-order statistics , and leverage the ability of neural networks to learn relationships among node representations , making them more powerful . We perform experiments on graph classification tasks in the bio-informatics and social network domains to determine the effectiveness of our proposed methods . Experimental results show that our methods lead to an absolute increase of upto 1.2 % in classification accuracy over previous works and a general decrease in standard deviation across multiple runs indicating greater reliability . Experimental results also indicate that this improvement in performance is consistent across several datasets . 1 INTRODUCTION . Over the past several years , there is a growing number of applications where data is generated from non-Euclidean domains and is represented as graphs with complex relationships and interdependency between entities . Deep learning generalised from grid-like data to the graph domain has led to the development of the remarkably successful Graph Neural Networks ( GNNs ) ( Fan et al. , 2019 ; Gao et al. , 2019 ; Ma et al. , 2019a ; Wang et al. , 2019b ) and its numerous variants such the Graph Convolutional Network ( GCN ) ( Kipf & Welling , 2017 ) , GraphSAGE ( Hamilton et al. , 2017 ) , graph attention network ( GAT ) ( Veličković et al. , 2018 ) , jumping knowledge network ( JK ) ( Xu et al. , 2018 ) , and graph isomorphism networks ( GINs ) ( Xu et al. , 2019 ) , etc . Pooling is a common operation in deep learning on grid-like data , such as images . Pooling layers provide an approach to down sampling feature maps by summarizing the presence of features in patches of the feature map . It reduces dimensionality and also provides local translational invariance . In the case of graph data , pooling is used to obtain a representation of a graph using its constituent node representations . However , it is challenging to develop graph pooling methods due to the some special properties of graph data such as the variable number of nodes in different graphs and the isomorphic structures of graphs . Firstly , the number of nodes varies in different graphs , while the graph representations are usually required to have the same fixed size to fit into other downstream machine learning models where they are used for tasks such as classification . Therefore , graph pooling should be capable of handling the variable number of node representations as inputs and producing fixed-sized graph representations . Secondly , unlike images and texts where we can order pixels and words according to the spatial structural information , there is no inherent ordering relationship among nodes in graphs . Therefore , isomorphic graphs should have the same graph representation , and hence , graph pooling should give the same output by taking node representations in any order as inputs . Our main contributions in this work are two novel graph pooling methods , Neural Pooling Method 1 and 2 . These new pooling methods allow us to do the following , i ) produce the same dimensional graph representation for graphs with variable number of nodes , ii ) remain invariant to the isomorphic structures of graphs , iii ) collect second- order statistics , iv ) leverage trainable parameters in the form of fully connected neural networks to learn relationships between underlying node representations to generate high quality graph representations which are then used for graph classification tasks . Experiments are performed on four benchmark bio-informatics datasets and five popular social network datasets to demonstrate the effectiveness and superiority of our proposed graph pooling methods . Experimental results show that our methods lead to an improvement in classification accuracy over existing methods and are also more reliable as compared to previous works . 2 RELATED WORK . 2.1 GRAPH NEURAL NETWORKS . A graph can be represented by its adjacency matrix and node features . Formally , for a graph G consisting of n nodes , its topology information can be represented by an adjacency matrix A ∈ { 0 , 1 } n × n and the node features can be represented as X ∈ Rn × d , assuming each node has a d-dimensional feature vector . GNNs learn feature representations for different nodes using these matrices ( Gilmer et al. , 2017 ) . Several approaches are proposed to investigate deep GNNs , and they generally follow a neighborhood information aggregation scheme ( Gilmer et al. , 2017 ; Xu et al. , 2019 ; Kipf & Welling , 2017 ; Hamilton et al. , 2017 ; Veličković et al. , 2018 ) . In each step , the representation of a node is updated by aggregating the representations of its neighbors . Graph Convolutional Networks ( GCNs ) are popular variants of GNNs and inspired by the first order graph Laplacian methods ( Kipf & Welling , 2017 ) . Graph pooling is used to connect embedded graphs outputted by GNN layers with classifiers for graph classification . Given a graph , GNN layers produce node representations , where each node is embedded as a vector . Graph pooling is applied after GNN layers to process node representations into a single feature vector as the graph representation . A classifier takes the graph representation and performs graph classification . 2.2 GRAPH POOLING . Early studies employ simple methods such as averaging and summation as graph pooling ( Xu et al. , 2019 ; Duvenaud et al. , 2015 ; Defferrard et al. , 2016 ) . However , averaging and summation do not capture the feature correlation information , curtailing the overall model performance ( Zhang et al. , 2018 ) . Other studies have proposed advanced graph pooling methods , including DIFFPOOL ( Ying et al. , 2018 ) , SORT-POOL ( Zhang et al. , 2018 ) , TOPKPOOL ( Gao & Ji , 2019 ) , SAGPOOL ( Lee et al. , 2019 ) , and EIGENPOOL ( Ma et al. , 2019b ) , and achieve great performance on multiple benchmark datasets . EIGENPOOL involves the computation of eigenvectors , which is slow and expensive . DIFFPOOL ( Ying et al. , 2018 ) treats the graph pooling as a node clustering problem . A cluster of nodes from the original graph are merged to form a new node in the new graph . DIFFPOOL ( Ying et al. , 2018 ) proposes to perform the graph convolution operation on node features to obtain node clustering assignment matrix . Intuitively , the class assignment of a given node should depend on the class assignments of other neighbouring nodes . However , DIFFPOOL does not explicitly consider high-order structural relationships , which we that are important for graph pooling . SORTPOOL ( Zhang et al. , 2018 ) , TOPKPOOL ( Gao & Ji , 2019 ) , and SAGPOOL ( Lee et al. , 2019 ) learn to select important nodes from the original graph and use these nodes to build a new graph . They share the similar idea to learn a sorting vector based on node representations , which indicates the importance of different nodes . Then only the top k important nodes are selected to form a new graph while the other nodes are ignored . However , the ignored nodes may contain important features and this information is lost during pooling . It is worth noting that all the graph pooling methods mentioned till now only collect first-order statistics ( Boureau et al. , 2010 ) . A recent study has proposed second order graph pooling methods SOPoolbimap and SOPoolattention ( Wang & Ji , 2020 ) . In this work , we propose two novel methods using fully connected neural network layers for graph pooling , namely Neural Pooling Method 1 and 2 . Compared to existing graph pooling methods , our proposed methods are able to capture information from all nodes , collect second-order statistics , and leverage the ability of neural networks to learn relationships among node representations , making them more powerful . 3 METHODOLOGY . 3.1 PROPERTIES OF GRAPH POOLING . Consider a graph G = ( A , X ) represented by its adjacency matrix A ∈ { 0 , 1 } n × n and node feature matrix X ∈ Rn × d , where n is the number of nodes in G and d is the dimension of node features . The node features may come from node labels or node degrees . Graph neural networks are known to be powerful in learning good node representation matrix H from A and X : H = [ h1 , h2 , ......... , hn ] T = GNN ( A , X ) ∈ Rn × f ( 1 ) where rows of H , hi ∈ Rf , i = 1 , 2 , ... , n , are representations of n nodes , and f is the dimension of the node representation obtained from the GNN and depends on the architecture of the GNN . The task that we focus on in this work is to obtain a graph representation vector hG from H , which is then fed into a classifier to perform graph classification : hG = g ( [ A ] , H ) ∈ Rc ( 2 ) where g ( · ) is the graph pooling function and c is the dimension of hG . Here , [ A ] means that the information from A can be optionally used in graph pooling . For simplicity , we omit it in the following discussion . Note that the function g ( · ) must satisfy two requirements to serve as graph pooling . First , g ( · ) should be able to take H with variable number of rows as the inputs and produce fixed-sized outputs . Specifically , different graphs may have different number of nodes , which means that n is a variable . On the other hand , c , which is the dimension of the graph representation hG is supposed to be fixed to fit into the classifier . Second , g ( · ) should output the same hG when the order of rows of H changes . This permutation invariance property is necessary to handle isomorphic graphs . To be concrete , if two graph G1 = ( A1 , X1 ) and G2 = ( A2 , X2 ) are isomorphic , GNNs will output the same multi set of node representations . That is , there exists a permutation matrix P ∈ { 0 , 1 } n × n such that H1 = PH2 , for H1 = GNN ( A1 , X1 ) and H2 = GNN ( A2 , X2 ) . However , the graph representation computed by g ( · ) should be the same , i.e. , g ( H1 ) = g ( H2 ) if H1 = PH2 . 3.2 NEURAL POOLING METHOD 1 . Our first proposed method is called Neural Pooling Method 1 . Consider a node representation matrix H obtained following Equation 1 in Section 3.1 . H is passed through a Fully Connected Neural Network Layer ( FCL1 ) to obtain H ′ as : H ′ = FCL1 ( H ) ∈ Rn × f ′ where f ′ < f ( 4 ) After this H ′ is again passed through a second Fully Connected Neural Network Layer ( FCL2 ) to obtain Q as : Q = FCL2 ( H ′ ) ∈ Rn × 1 ( 5 ) Finally the graph representation hG is obtained as : hG = H ′TQ ∈ Rf ′ × 1 ( 6 ) where H ′T denotes the transpose of H ′ . Neural Pooling Method 1 always outputs an f ′-dimensional graph representation for H ∈ Rn × f , regardless of the value of n. It is also invariant to permutation so that it outputs the same graph representation , even when the order of rows of H changes . Intuition : The FCL1 performs the role of reducing the dimensionality of the input node representations . The trainable parameters of this FCL1 can be thought of as learning a mapping from the f to the f ′-dimensional space . The FCL2 reduces the f ′-dimensional node representations to a 1 dimensional representation , Q. H ′ ∈ Rn × f ′ can be viewed as H ′ = [ l1 , l2 , . . . , lf ′ ] , where lj ∈ Rn , j=1 , 2 , ... , f ′ . The vector lj encodes the spatial distribution of the j-th feature in the graph . Based on this view , H ′TQ is able to capture the topology information and Q can be thought of as roughly encoding the position of nodes by learning the weights according to which the j-th feature is aggregated across the nodes . Neural Pooling Method 1 hence , leverages the ability of neural networks to learn the topological structure as well as correlation among the node representations in H . It captures the essential features and connections between underlying data . It also reduces the dimensionality of H , and results in an accurate representation of the input graph .

Summary: This paper proposes two fully-connected layers based neural graph pooling methods for graph neural networks, named Neural Pooling Method 1 and Neural Pooling Method 2. The first method uses a first FC to reduce the feature dimension and then FC2 to compute the weights to do weighted-average over features for different nodes. The second method uses two FC to reduce the dimension and then compute second-order statistics by Flatten(H^{\top}H). Experimental results on four datasets (PTC, PROTEINS, IMDB-BINARY, IMDB-MULTI) of two tasks (bioinformatics, social networks) show that the proposed graph pooling method can improve the performance by 0.5%-1.2% accuracy while decreasing the std.