Capsule Graph Neural Network

1 INTRODUCTION . GNN is a general type of deep-learning architectures that can be directly applied to structured data . These architectures are mainly generalized from other well-established deep-learning models like CNN ( Krizhevsky et al. , 2012 ) and RNN ( Mikolov et al. , 2010 ) . In this paper , we mainly focus on Convolution-based Graph Neural Networks which attract increasing interest recently . Convolution operation can be embedded into Graph Neural Networks from spectral or spatial perspective . Bruna et al . ( 2013 ) defines the convolution operation in the Fourier domain which needs to calculate the eigendecomposition of the graph Laplacian . This method is computationally expensive and the filters they defined are non-spatially localized . Later , Henaff et al . ( 2015 ) introduces Chebyshev expansion of the graph Laplacian to avoid computing eigenvectors and Kipf & Welling ( 2017 ) proposes to do convolution within 1-step neighbor nodes to reduce the complexity . From the spatial perspective , Hamilton et al . ( 2017 ) and Zhang et al . ( 2018 ) propose to define a node receptive-field and do convolution within this field during which the information of each node as well as their neighbor nodes is gathered and new representation of each node is generated through an activation function . Both of these two perspectives perform well in node representation learning and a number of variants ( Velikovi et al. , 2018 ) are developed based on the convolution idea and some of them have proven to achieve SOTA in various tasks . The success of GNN in node representation learning has inspired many deep-learning-based approaches to leverage on node embeddings extracted from GNN to generate graph embeddings for graph-based applications . However , during this procedure , the learned representation of each node will be considered as multiple individual scalar features instead of one vector . For example , Zhang et al . ( 2018 ) applies element-wise max-pooling to nodes embeddings when generating graph embeddings , Verma & Zhang ( 2018 ) generates graph embeddings by computing the element-wise covariance of all nodes . These operations indicate that the authors capture node features in the form of scalar when they generate graph embeddings which may not suffice to preserve the node/graph properties efficiently . To build high-quality graph embeddings , it is important to not only detect the presence of different structures around each node but also preserve their detailed properties such as position , direction , connection , etc . However , encoding these properties information in the form of scalar means activating elements in a vector one-by-one which is exponentially less efficient than encoding them with distributed representations . This has been identified discussed in Sabour et al . ( 2017 ) . Inspired by CapsNet , we propose to extend scalar to vector during the procedure of applying GNN to graph representation learning . Compared with scalar-based neural network , vector-based neural network preserves the information of node/graph properties more efficiently . The technique for extracting features in the form of vectors is proposed in Hinton et al . ( 2011 ) and improved in Sabour et al . ( 2017 ) and Hinton et al . ( 2018 ) . This technique is mainly devised for image processing . In their work , the extracted vector is referred to as capsule ( a group of neurons in neural network ) , so we follow the same notation in our work . Introducing capsules allows us to use routing mechanism to generate high-level features which we believe is a more efficient way for features encoding . Compared with max-pooling in CNN in which all information will be dropped except for the most active one , routing preserves all the information from low-level capsules and routes them to the closest high-level capsules . Besides , this allows to model each graph with multiple embeddings and each embedding reflects different properties of the graph . This is more representative than only one embedding used in other scalar-based approaches . In this paper , we propose Capsule Graph Neural Network ( CapsGNN ) , a novel deep learning architecture , which is inspired by CapsNet and uses node features extracted from GNN to generate high-quality graph embeddings . In this architecture , each graph is represented as multiple embeddings and each embedding reflects the graph properties from different aspects . More specifically , basic node features are extracted in the form of capsules through GNN and routing mechanism is applied to generate high-level graph capsules as well as class capsules . In the procedure of generating graph capsules , an Attention Module can be applied to tackle graphs in various sizes . It also assigns different weights to each capsule of each node so that this model focuses on critical parts of the graph . We validate the performance of generated graph embeddings on classification task over 5 biological datasets and 5 social datasets . CapsGNN achieves SOTA performance on 6 out of 10 benchmark datasets and comparable results on the rest . T-SNE ( Maaten & Hinton , 2008 ) is used to visualize the learned graph embeddings and the results show that different graph capsules indeed capture different information of the graphs . 2 BACKGROUND . Here , we provide a brief introduction to Graph Convolutional Networks ( GCNs ) ( Kipf & Welling , 2017 ) , routing mechanism in CapsNet and Attention mechanism which is used in CapsGNN . 2.1 GRAPH . By definition , a weighted directed graph can be represented by G = ( V , X , A ) where V = { v1 , v2 , ... vN } is the set of nodes and A ∈ { 0 , 1 } N×N is the adjacency matrix . If there is an edge from vi to vj , then Aij = 1 otherwise Aij = 0 . X ∈ RN×d represents the features of each node . d is the number of feature channels and N is the number of nodes . 2.2 GRAPH CONVOLULTIONAL NETWORK . GCN , a widely used GNN architecture , is chosen as one of the key building blocks in our work . At each layer of the GCN , the convolution operation is applied to each node as well as its neighbors and the new representation of each node is computed through an activation function . This procedure can be written as : Zl+1 = f ( TZlW l ) ( 1 ) where Zl ∈ RN×d represents nodes features at the layer l , d represents the number of feature channels and Z0 = X , W l ∈ Rd×d′ is a trainable weights matrix which serves as a channel filter , f is a nonlinear activation function , T ∈ RN×N is the information transform matrix and it is usually calculated from the adjacency matrix A for guiding the information flowing between nodes . A complete GNN usually stacks L layers to generate final nodes embeddings ZL . In the architecture proposed by Kipf & Welling ( 2017 ) , at the lth layer of GCN , the extracted features of each node actually take all its adjacent nodes within l steps into consideration . So l can be considered as the size of the node receptive-field at this layer . This special property inspired us to use nodes features extracted from different layers to generate the graph capsules . 2.3 CAPSULE NEURAL NETWORK . The concept of capsules is invented by Hinton ’ s team ( Hinton et al. , 2011 ) and used recently in Sabour et al . ( 2017 ) and Hinton et al . ( 2018 ) . CapsNet is designed for image features extraction and it is developed based on CNN . However , unlike traditional CNN in which the presence of feature is represented with scalar value in feature maps , the features in CapsNet are represented with capsules ( vectors ) . In Sabour et al . ( 2017 ) , the direction of capsules reflects the detailed properties of the features and the length of capsules reflects the probability of the presence of different features . The transmission of information between layers follows Dynamic Routing mechanism . The specific procedure of Dynamic Routing can be found in Appendix A for the completeness . Inspired by CapsNet , the capsule mechanism is adopted and fused with GNN in our proposed CapsGNN to generate graph capsules and class capsules on the basis of node capsules which are extracted from GNN . Dynamic Routing is applied to update weights between capsules from one layer to the next layer so that the properties captured by node capsules can be propagated to suitable graph capsules . Thus , each graph is modeled as multiple graph capsules , and then modeled as multiple class capsules . Different graph capsules reflect the properties of the graph from different aspects . 2.4 ATTENTION MECHANISM . Attention mechanism is widely applied in image ( Zheng et al. , 2017 ) and natural language processing domain ( Gehring et al. , 2016 ) where it is used to find the relevant parts of the input data to the task target . The main procedure of Attention mechanism is : 1 ) defining an attention measure which is used to measure the relevance of each part of the input data to the task target . 2 ) normalizing the generated attention value . 3 ) scaling each part with the normalized attention value . In CapsGNN , we apply Attention mechanism for two purposes : 1 ) scaling each node capsule so that the graph capsules that are generated from different graphs are still comparable even though these graphs are vastly different in sizes . 2 ) guiding the model to focus on more relevant parts of graphs . 3 CAPSULE GRAPH NEURAL NETWORK . In this section , we outline CapsGNN and show how it is used to generate high-quality graph capsules which then can be applied to graph classification task . Figure 1 shows a simplified version of CapsGNN . It consists of three key blocks : 1 ) Basic node capsules extraction block : GNN is applied to extract local vertices features with different receptive-field and then primary node capsules are built in this block . 2 ) High level graph capsules extraction block : Attention Module and Dynamic Routing are fused to generate multiple capsules for graphs . 3 ) Graph classification block : Dynamic Routing is applied again to generate class capsules for graph classification . The details of each block is explained in the following . 3.1 BASIC NODE CAPSULES . Firstly , the basic node features are extracted with GNN . Node degrees can be used as node attributes if nodes do not have attributes . We use the architecture improved by Kipf & Welling ( 2017 ) ( GCN ) as the node features extractor . The difference is that we extract multi-scale node features from different layers and the extracted features are represented in the form of capsules . The procedure can be written as : Zl+1j = f ( ∑ i D̃− 1 2 ÃD̃− 1 2ZliW l ij ) ( 2 ) where W lij ∈ Rd×d ′ is the trainable weights matrix . It serves as the channel filters from the ith channel at the lth layer to the jth channel at the ( l + 1 ) th layer . Here , we choose f ( · ) = tanh ( · ) as the activation function . Zl+1 ∈ RN×d′ , Z0 = X , Ã = A + I 1 and D̃ = ∑ j Ãij . To preserve features of sub-components with different sizes , we use nodes features extracted from all GNN layers to generate high-level capsules .

Summary: The authors provide an architecture that applies recent advances in the field of capsule networks in the graph neural network domain. First, hierarchical node level capsules are extracted using GCN layers. Second, after weighting each capsule by the output of a proposed attention module, graph level capsules are computed by performing a global dynamic routing. These graph level capsules are used for training a capsule classifier using a margin loss and a reconstruction loss.