MxPool: Multiplex Pooling for Hierarchical Graph Representation Learning

1 INTRODUCTION . Graphs are known to have complicated structures and have myriad of real world applications . Recently , great efforts have been put on utilizing deep learning methods for graph data analysis . Many newly proposed graph learning approaches are inspired by Convolutional Neural Networks ( CNNs ) ( LeCun & Bengio , 1998 ) , which have been greatly successful in learning two-dimensional image data ( grid structure ) . The convolution and pooling layers in CNNs have been redefined to process graph data . Multitude of different Graph Convolutional Networks ( GCNs ) ( Shuman et al. , 2013 ) have been proposed , which can learn node level representations by aggregating feature information from neighbors ( spatial-based approaches ) ( Hamilton et al. , 2017 ) or by introducing filters from the perspective of graph signal processing ( spectral-based approaches ) ( Bengio & LeCun , 2014 ) . On the other hand , similar to the original pooling layer which comes with CNNs , graph pooling module ( Defferrard et al. , 2016 ; Zhang et al. , 2018 ) could easily reduce the variance and computation complexity by down-sampling from original feature data , which is of vital importance , particularly for graph level classification tasks . Recently , hierarchical pooling methods that can learn hierarchical representations of graphs have been proposed ( Ying et al. , 2018 ; Gao & Ji , 2019 ; Lee et al. , 2019 ) and shows state-of-the-art performance for graph classification tasks . However , two properties of graph data have imposed significant challenges on existing graph learning techniques . 1 ) Diversity : each graph has a variable size of unordered nodes and has diverse node/edge types . 2 ) Complexity : graphs have not only node/edge features but also complex topological features . These two properties can bring troubles in both of the graph convolution operation and the graph pooling operation . For example , when performing node-representation learning tasks ( by graph convolution operation ) , it is enough to use small output embedding size for simple and small graphs , as shown in Figure 1 ( a ) , since large embedding size could result in overfitting problem . By contrast , it is necessary to set large output embedding sizes for complex and large graphs to learn complex graph structure properties , as shown Figure in 1 ( b ) . This creates a contradiction when processing a set of irregular graphs . For another example , when coarsening graphs ( by graph pooling operation ) , if more attention is put on the graph structure , we may obtain a coarsened graph as shown in Figure 1 ( c ) . If more attention is put on the node features , we may obtain another coarsened graph as shown in Figure 1 ( d ) . This creates another contradiction when processing graphs with not only node features but also topological features . The diversity property and the complexity property of graph data motivate us to use multiplex GNN structure to learn graph features in a diverse way . On the other hand , as known , a common solution for augmenting the traditional CNN convolution layers is to use multiple convolution kernels in order to learn multiple local features . The success of CNNs on image data also inspires us to concurrently use multiple graph convolution networks and multiple graph pooling networks to learn graph representations . In this paper , we propose MxPool in hierarchical graph representation learning for graph classification tasks1 . MxPool comprises multiple graph convolution networks to learn node-level representations and also comprises multiple graph pooling networks to coarsen the graph . The node-level representations resulted from multiple convolution networks and the coarsened graphs resulted from multiple pooling networks are merged in a learnable way , respectively . The merged node representations and the merged coarsened graph are then used to generate a new coarsened graph , which is used in the next layer . This multiplex structure can adapt to graphs with different sizes and can extract useful information from different perspectives . We conduct extensive experiments on numerous graph classification benchmarks and show that our MxPool has marked superiority over other state-of-the-art graph representation learning methods . For example , MxPool achieves 92.1 % accuracy on the D & D dataset while the second best method DiffPool only achieves 80.64 % accuracy . 2 RELATED WORK . In this section , we review the recent literature on GNNs , graph convolution variants , and graph pooling variants . Graph Neural Networks Inspired by Traditional Deep Learning Techniques . GNNs have recently drawn considerable attention due to their superiority in a wide variety of graph related tasks , including node classification ( Kipf & Welling. , 2017 ) , link prediction ( Schlichtkrull et al. , 2018 ) , and graph classification ( Dai et al. , 2016 ) . Many of these GNN models are inspired by traditional learning techniques . Inspired by the huge success of convolutional networks in the computer vision domain , a large number of Graph Convolutional Networks ( GCNs ) have emerged . Besides convolution operation , pooling operation , as another key component in CNNs , has also inspired research communities to propose graph pooling operations . There are also GNN optimizations originating from other learning approaches . Inspired by Recurrent Neural Networks ( RNNs ) , You et al . ( 2018a ) apply Graph RNN to the graph generation problem . DGNN ( Ma et al. , 2018 ) proposes using LSTM ( Hochreiter & Schmidhuber , 1997 ) to learn node representations in dynamic graphs . Inspired by the attention mechanism ( Vaswani et al. , 2017 ) Graph Attention Networks ( GATs ) ( Velickovic et al. , 1Our code is available at https : //github.com/JucatL/MxPool/ . 2017 ) introduce attentions into GCNs by differentiating the influence of neighbors . Graph AutoEncoders ( GAEs ) ( Wang et al. , 2016 ) origin from the autoencoder mechanism widely used for unsupervised learning and are suitable to learn node representations for graphs . GCPN ( You et al. , 2018b ) utilizes Reinforcement Learning ( RL ) for goal-directed molecular graph generation . Graph Convolution . Graph convolution operations fall into two categories , spectral-based approaches and spatial-based approaches . Bengio & LeCun ( 2014 ) first introduce convolution for graph data from spectral domain using the graph Laplacian matrix L. Besides , there exist numerous spectral-based graph convolution methods , such as ChebNet ( Defferrard et al. , 2016 ) , 1stChebNet ( Kipf & Welling , 2017 ) , and AGCN ( Li et al. , 2018 ) . In contrast , spatial-based convolution methods define graph convolution based on a node ’ s spatial relations . It takes the aggregation of a node representation and its neighbors ’ representations to obtain a new representation for this node . In order to explore the depth and breadth of a node ’ s receptive field , multiple graph convolution layer are stacked together , so that the features of two or more hops away neighbors can be learned . For example , GGNNs ( Li et al. , 2015 ) , MPNN ( Gilmer et al. , 2017 ) , GraphSage ( Hamilton et al. , 2017 ) , PATCHY-SAN ( Niepert et al. , 2016 ) , and DCNN ( Atwood & Towsley , 2016 ) all fall into the spatial-based category . Graph Pooling . Graph pooling operation is of vital importance for graph classification tasks ( Zhang et al. , 2018 ) . It coarsens a graph into sub-graphs ( Defferrard et al. , 2016 ; Ying et al. , 2018 ) or to sum/average over the node representations ( Duvenaud et al. , 2015 ; Gilmer et al. , 2017 ) , which can obtain a compact representation on graph level . The graph coarsening approaches obtain hierarchical graph representations either by using deterministic pooling methods or by using learned pooling methods . The deterministic pooling methods ( Defferrard et al. , 2016 ; Simonovsky & Komodakis , 2017 ) utilizes graph clustering algorithms to obtain next level coarsened graph that is going to be processed by GNNs , following a two-stage approach . On the other hand , the learned pooling methods ( Ying et al. , 2018 ; Lee et al. , 2019 ; Diehl , 2019 ; Gao & Ji , 2019 ) seek to learn the hierarchical structure , which have shown to outperform deterministic pooling methods . DiffPool ( Ying et al. , 2018 ) was the first to propose learned graph pooling . It learns a soft cluster assignment matrix in layer l which contains the probability values of nodes being assigned to clusters . A cluster in layer l will be reduced to a node in layer l + 1 . A GNN with input node features and adjacency matrix is used to generate the soft assignment matrix , based on which we can learn the cluster embeddings ( i.e. , node features in the next layer ) and the coarsened adjacency matrix denoting the connectivity strength between each pair of the clusters . Besides DiffPool , numerous graph pooling methods have emerged recently , including gPool ( Gao & Ji , 2019 ) , SAGPool ( Lee et al. , 2019 ) , EigenPooling ( Ma et al. , 2019 ) , and Relational Pooling ( Murphy et al. , 2019 ) . This paper will focus on the learned pooling . 3 PROPOSED METHOD . In this section , we propose MxPool to learn graph representations such that graph level classification can be applied . Before going to the details , we first introduce some notations and the problem setting . Problem Setting A graph can be represented as G = { A , F } , where A ∈ Rn×n denotes the adjacency matrix ( n is the number of nodes contained in G ) , and F ∈ RN×d denotes the node feature matrix ( d is the dimension of features ) . In the graph classification setting , given a set of graphs and each being associated with a label , we aim to train a model that takes an unseen graph as input and predicts its corresponding label . To make the prediction , it is important to extract useful information from multiple perspectives including both graph structure and node features . 3.1 OVERVIEW . MxPool is a multi-layer hierarchical GNN model . At each layer , MxPool consists of convolution operation and pooling operation . The convolution operation aims to learn node-level representations , while the pooling operation aims to learn a coarsened graph . The new coarsened graph can then be used as input to next layer . This process can be repeated several times , generating a multi-layer GNN model to learn hierarchical graph representations . The convolution operation and pooling operation are both important for graph representation learning . To simplify the illustration , we choose GCN ( Kipf & Welling. , 2017 ) as the convolution layer and DiffPool ( Ying et al. , 2018 ) as the pooling layer ( which is a differentiable pooling method ) , but it can be extended to use other convolution variant and pooling variant as well . Different from other hierarchical GNNs , MxPool launches multiple GCNs to learn node-level representations and also launches multiple pooling networks to coarsen the graph . The node-level representations resulted from multiple GCNs are then merged in a learnable way , and the coarsened graphs resulted from multiple pooling networks are also merged . The merged node embeddings and the merged coarsened graph are used to generate a new coarsened graph with a new set of node features . This multiplex structure can help extract useful information from different perspectives ( e.g. , graph structure perspective and node feature perspective ) and can adapt to graphs with different sizes . We provide an illustrative example as shown in Figure 3 . The procedure of the GCN ( Kipf & Welling. , 2017 ) is to “ horizontally ” learn node representations , as it can only “ pass message ” between nodes through edges . The procedure of differentiable pooling ( Ying et al. , 2018 ) is to “ vertically ” summarize the node features into the higher level graph representation . The procedure of multiplexing is to “ diversely ” learn node representations or graph representations from different perspectives . The procedure of merging is to “ synthetically ” learn the diverse results and put more attention to one or more perspectives . Since the convolution , pooling , and merging operations are all differentiable , we can define an end-to-end differentiable graph representation learning framework in a hierarchical manner .

Summary: This paper introduces a new hierarchical graph representation learning method for graph classification. It builds upon the diffpool method. Specifically, the authors propose the multiplex convolution and the multiplex pooling operation. The multiplex convolution learns multiple graph convolution operations (potentially with different parameters) and merges them using a neural network. Similarly, the multiplex pooling learns multiple assignment matrices utilizing diffpool and merges them using a neural network. The effectiveness of the proposed model is demonstrated with experiments in several standard benchmark datasets for graph classification.