Title: CAPSULE GRAPH NEURAL NETWORK

Abstract: The high-quality node embeddings learned from the Graph Neural Networks (GNNs) have been applied to a wide range of node-based applications and some of them have achieved state-of-the-art (SOTA) performance. However, when applying node embeddings learned from GNNs to generate graph embeddings, the scalar node representations may not suffice to preserve the node/graph properties efficiently, resulting in sub-optimal graph embeddings. Inspired by the Capsule Neural Network (CapsNet) (Sabour et al., 2017) , we propose the Capsule Graph Neural Network (CapsGNN), which adopts the concept of capsules to address the weakness in existing GNN-based graph embeddings algorithms. By extracting node features in the form of capsules, routing mechanism can be utilized to capture important information at the graph level. As a result, our model generates multiple embeddings for each graph to capture graph properties from different aspects. The attention module incorporated in CapsGNN is used to tackle graphs with various sizes which also enables the model to focus on critical parts of the graphs. Our extensive evaluations with 10 graph-structured datasets demonstrate that CapsGNN has a powerful mechanism that operates to capture macroscopic properties of the whole graph by data-driven. It outperforms other SOTA techniques on several graph classification tasks, by virtue of the new instrument.

1. INTRODUCTION:

GNN is a general type of deep-learning architectures that can be directly applied to structured data. These architectures are mainly generalized from other well-established deep-learning models like CNN (Krizhevsky et al., 2012) and RNN (Mikolov et al., 2010) . In this paper, we mainly focus on Convolution-based Graph Neural Networks which attract increasing interest recently. Convolution operation can be embedded into Graph Neural Networks from spectral or spatial perspective. Bruna et al. (2013) defines the convolution operation in the Fourier domain which needs to calculate the eigendecomposition of the graph Laplacian. This method is computationally expensive and the filters they defined are non-spatially localized. Later, Henaff et al. (2015) introduces Chebyshev expansion of the graph Laplacian to avoid computing eigenvectors and Kipf & Welling (2017) proposes to do convolution within 1-step neighbor nodes to reduce the complexity. From the spatial perspective, Hamilton et al. (2017) and Zhang et al. (2018) propose to define a node receptive-field and do convolution within this field during which the information of each node as well as their neighbor nodes is gathered and new representation of each node is generated through an activation function. Both of these two perspectives perform well in node representation learning and a number of variants (Velikovi et al., 2018) are developed based on the convolution idea and some of them have proven to achieve SOTA in various tasks.

The success of GNN in node representation learning has inspired many deep-learning-based approaches to leverage on node embeddings extracted from GNN to generate graph embeddings for graph-based applications. However, during this procedure, the learned representation of each node will be considered as multiple individual scalar features instead of one vector. For example, Zhang et al. (2018) applies element-wise max-pooling to nodes embeddings when generating graph embeddings, Verma & Zhang (2018) generates graph embeddings by computing the element-wise covariance of all nodes. These operations indicate that the authors capture node features in the form Published as a conference paper at ICLR 2019 of scalar when they generate graph embeddings which may not suffice to preserve the node/graph properties efficiently.

To build high-quality graph embeddings, it is important to not only detect the presence of different structures around each node but also preserve their detailed properties such as position, direction, connection, etc. However, encoding these properties information in the form of scalar means activating elements in a vector one-by-one which is exponentially less efficient than encoding them with distributed representations. This has been identified discussed in Sabour et al. (2017) . Inspired by CapsNet, we propose to extend scalar to vector during the procedure of applying GNN to graph representation learning. Compared with scalar-based neural network, vector-based neural network preserves the information of node/graph properties more efficiently. The technique for extracting features in the form of vectors is proposed in Hinton et al. (2011) and improved in Sabour et al. (2017) and Hinton et al. (2018) . This technique is mainly devised for image processing. In their work, the extracted vector is referred to as capsule (a group of neurons in neural network), so we follow the same notation in our work. Introducing capsules allows us to use routing mechanism to generate high-level features which we believe is a more efficient way for features encoding. Compared with max-pooling in CNN in which all information will be dropped except for the most active one, routing preserves all the information from low-level capsules and routes them to the closest high-level capsules. Besides, this allows to model each graph with multiple embeddings and each embedding reflects different properties of the graph. This is more representative than only one embedding used in other scalar-based approaches.

In this paper, we propose Capsule Graph Neural Network (CapsGNN), a novel deep learning architecture, which is inspired by CapsNet and uses node features extracted from GNN to generate high-quality graph embeddings. In this architecture, each graph is represented as multiple embeddings and each embedding reflects the graph properties from different aspects. More specifically, basic node features are extracted in the form of capsules through GNN and routing mechanism is applied to generate high-level graph capsules as well as class capsules. In the procedure of generating graph capsules, an Attention Module can be applied to tackle graphs in various sizes. It also assigns different weights to each capsule of each node so that this model focuses on critical parts of the graph. We validate the performance of generated graph embeddings on classification task over 5 biological datasets and 5 social datasets. CapsGNN achieves SOTA performance on 6 out of 10 benchmark datasets and comparable results on the rest. T-SNE (Maaten & Hinton, 2008) is used to visualize the learned graph embeddings and the results show that different graph capsules indeed capture different information of the graphs.

2. BACKGROUND:

Here, we provide a brief introduction to Graph Convolutional Networks (GCNs) (Kipf & Welling, 2017) , routing mechanism in CapsNet and Attention mechanism which is used in CapsGNN.

2.1. GRAPH:

By definition, a weighted directed graph can be represented by G = (V, X, A) where V = {v 1 , v 2 , ...v N } is the set of nodes and A ∈ {0, 1} N ×N is the adjacency matrix. If there is an edge from v i to v j , then A ij = 1 otherwise A ij = 0. X ∈ R N ×d represents the features of each node. d is the number of feature channels and N is the number of nodes.

2.2. GRAPH CONVOLULTIONAL NETWORK:

GCN, a widely used GNN architecture, is chosen as one of the key building blocks in our work. At each layer of the GCN, the convolution operation is applied to each node as well as its neighbors and the new representation of each node is computed through an activation function. This procedure can be written as:

Z l+1 = f (T Z l W l ) (1)

where Z l ∈ R N ×d represents nodes features at the layer l, d represents the number of feature channels and Z 0 = X, W l ∈ R d×d is a trainable weights matrix which serves as a channel filter, f is a nonlinear activation function, T ∈ R N ×N is the information transform matrix and it is usually calculated from the adjacency matrix A for guiding the information flowing between nodes.

A complete GNN usually stacks L layers to generate final nodes embeddings Z L . In the architecture proposed by Kipf & Welling (2017) , at the lth layer of GCN, the extracted features of each node actually take all its adjacent nodes within l steps into consideration. So l can be considered as the size of the node receptive-field at this layer. This special property inspired us to use nodes features extracted from different layers to generate the graph capsules.

2.3. CAPSULE NEURAL NETWORK:

The concept of capsules is invented by Hinton's team (Hinton et al., 2011) and used recently in Sabour et al. (2017) and Hinton et al. (2018) . CapsNet is designed for image features extraction and it is developed based on CNN. However, unlike traditional CNN in which the presence of feature is represented with scalar value in feature maps, the features in CapsNet are represented with capsules (vectors). In Sabour et al. (2017) , the direction of capsules reflects the detailed properties of the features and the length of capsules reflects the probability of the presence of different features. The transmission of information between layers follows Dynamic Routing mechanism. The specific procedure of Dynamic Routing can be found in Appendix A for the completeness.

Inspired by CapsNet, the capsule mechanism is adopted and fused with GNN in our proposed Caps-GNN to generate graph capsules and class capsules on the basis of node capsules which are extracted from GNN. Dynamic Routing is applied to update weights between capsules from one layer to the next layer so that the properties captured by node capsules can be propagated to suitable graph capsules. Thus, each graph is modeled as multiple graph capsules, and then modeled as multiple class capsules. Different graph capsules reflect the properties of the graph from different aspects.

2.4. ATTENTION MECHANISM:

Attention mechanism is widely applied in image (Zheng et al., 2017) and natural language processing domain (Gehring et al., 2016) where it is used to find the relevant parts of the input data to the task target. The main procedure of Attention mechanism is: 1) defining an attention measure which is used to measure the relevance of each part of the input data to the task target. 2) normalizing the generated attention value. 3) scaling each part with the normalized attention value.

In CapsGNN, we apply Attention mechanism for two purposes: 1) scaling each node capsule so that the graph capsules that are generated from different graphs are still comparable even though these graphs are vastly different in sizes. 2) guiding the model to focus on more relevant parts of graphs.

3. CAPSULE GRAPH NEURAL NETWORK:

In this section, we outline CapsGNN and show how it is used to generate high-quality graph capsules which then can be applied to graph classification task. Figure 1 where W l ij ∈ R d×d is the trainable weights matrix. It serves as the channel filters from the ith channel at the lth layer to the jth channel at the (l + 1)th layer. Here, we choose f (•) = tanh(•) as the activation function. Z l+1 ∈ R N ×d , Z 0 = X, Ã = A + I1 and D = j Ãij . To preserve features of sub-components with different sizes, we use nodes features extracted from all GNN layers to generate high-level capsules.

Z l+1 j = f ( i D-1 2 Ã D-1 2 Z l i W l ij ) (2)

3.2. HIGH-LEVEL GRAPH CAPSULES:

After getting local node capsules, global routing mechanism is applied to generate graph capsules. The input of this block contains N sets of node capsules, each set is S n = {s 11 , .., s 1C1 , ..., s LC L }, s lc ∈ R d , where C l is the number of channels at the lth layer of GNN, d is the dimension of each capsule. The output of this block is a set of graph capsules H ∈ R P ×d . Each of the capsules reflects the properties of the graph from different aspects. The length of these capsules reflects the probability of the presence of these properties and the angle reflects the details of the graph properties. Before generating graph capsules with node capsules, an Attention Module is introduced to scale node capsules.

Attention Module. In CapsGNN, primary capsules are extracted based on each node which means the number of primary capsules depends on the size of input graphs. In this case, if the routing mechanism is directly applied, the value of the generated high-level capsules will highly depend on the number of primary capsules (graph size) which is not the ideal case. Hence, an Attention Module is introduced to combat this issue.

The attention measure we choose is a two-layer fully connected neural network F attn (•). The number of input units of F attn (•) is d × C all where C all = l C l and the number of output units equals to C all . We apply node-based normalization to generate attention value in each channel and then scale the original node capsules. The details of Attention Module is shown in Figure 2 and the procedure can be written as:

EQUATION

where sn ∈ R 1×C all d is obtained by concatenating all capsules of the node n. s (n,i) ∈ R 1×d represents the ith capsule of the node n and F attn ( sn ) ∈ R 1×C all is the generated attention value. In this way, the generated graph capsules can be independent to the size of graphs and the architecture will focus on more important parts of the input graph. After Attention Module, coordinate addition module can be used to preserve the position information of each node during the procedure of generating node capsule votes. Here, we introduce coordinate addition as an additional module and more details can be found in Appendix C.

The procedure of generating multiple graph capsules is summarized as follows:

1) Scale primary capsules: Apply Attention Module to scale primary capsules. The results of this module should be S ∈ R N ×C all ×d .

2) Calculate votes: When calculating votes, capsules of different nodes from the same channel share the transform matrix. The result of this step is a set of votes V ∈ R N ×C all ×P ×d where C all denotes the number of channels. P denotes the defined number of graph capsules.

3) Dynamic Routing Mechanism: High-level graph capsules are computed with the procedure introduced in Section 2.3 based on votes produced in previous steps.

3.3. CLASSIFICATION:

This block is designed for graph classification using the graph capsules.

Classification Loss. Dynamic Routing is applied again over graph capsules to generate final class capsules C ∈ R K×d , where K is the number of graph classes. Here, we use margin loss function proposed in Sabour et al. (2017) to calculate the classification loss and it is computed as:

Loss c = k {T k max(0, m + -c k ) 2 + λ(1 -T k ) max(0, c k -m -) 2 } (4)

where m + = 0.9, m -= 0.1 and T k = 1 iff the input graph belongs to class k. λ is used to stop initial learning from reducing the length of all class capsules especially when K is large.

Reconstruction Loss. Following Sabour et al. (2017) , we use reconstruction loss as regularization method. Here, all class capsules are masked except the correct one and it is decoded with two fullyconnected layer to reconstruct the input information. The information we reconstruct here is the histogram of input nodes. The procedure can be written as:

EQUATION

where m i represents the number of nodes with the attribute i appear in the input graph, d i is the corresponding decoded value. M P i = 1 iff input graph contains nodes with attribute i. Equation 5 is used to prevent reducing reconstruction loss from setting all decoded value as 0 especially when most of the elements of the ground truth are 0.

The architecture details presented in section 3 describe the key design idea of CapsGNN which is based on the fusing of GNN and CapsNet. We also present a general comparison between CapsGNN with existing approaches in Appendix D.

4. EXPERIMENTS:

We verify the performance of the graph embeddings extracted from CapsGNN against a number of SOTA approaches and some classical approaches on classification task with 10 benchmark datasets.

Besides, we conduct experimental study to assess the impact of capsules in efficiency of encoding features of graphs. We also conduct brief analysis on the generated graph/class capsules. The experimental results and analysis is shown in the following. In addition to the analysis of the whole framework, we also provide a comparison experiment to evaluate the contribution of each module of CapsGNN with classification task. More details can be found in Appendix F.

4.1. GRAPH CLASSIFICATION:

The goal of graph classification is to predict the classes these graphs belong to by analyzing the structure and nodes labels information of graphs. More specifically, given a set of labeled graphs D = {(G 1 , y 1 ), (G 2 , y 2 ), . . . } where y i ∈ Y is the label of each graph G i . The objective of graph classification is to find a mapping f such that f : G → Y.

4.1.1. BASELINES METHODS:

We compare CapsGNN with both kernel-based and deep-learning-based algorithms. The details are given as follows:

Kernel-based Methods: Three kernel-based algorithms, namely the Weisfeiler-Lehman subtree kernel (WL) (Shervashidze et al., 2011) , the graphlet count kernel(GK) (Shervashidze et al., 2009) , and the Random Walk (RW) (Vishwanathan et al., 2010) . Typically, kernel-based algorithms first decompose graphs into sub-components based on the kernel definition, then build graph embeddings in a feature-based manner. Lastly, some machine learning algorithms (i.e., SVM) are applied to perform graph classification.

Deep-Learning-based Methods: Three types of deep-learning-based algorithms are selected:

1) Graph2vec (Narayanan et al., 2017) , Deep Graph Kernel (DGK) (Yanardag & Vishwanathan, 2015) and AWE (Ivanov & Burnaev, 2018) . Graph2vec, DGK and AWE require extracting substructures in advance while Graph2vec and AWE learn the representations of graphs in the manner of Doc2vec (Le & Mikolov, 2014) , DGK applies Word2vec (Mikolov et al., 2013) to learn the similarity between each pair of sub-structures which will be used to build the graph kernel. Then kernel-based machine learning methods (i.e., SVM) are applied to perform graph classification. These three algorithms as well as kernel-based methods are all sub-components based and they all require two stages to do graph classification. So although Graph2vec, DGK and AWE apply learning approaches to learn the embeddings, we still consider them and other kernel-based algorithms as the same type in our experiments and we mainly compare our proposed architecture with the other remained methods which are all end-to-end and totally data-driven architectures.

2) PATCHY-SAN (PSCN) (Niepert et al., 2016) . This method first sorts all nodes, then defines a receptive-field size for each node. These receptive-field are then filled with sorted neighbor nodes.

Lastly, 1-D CNN is applied to perform graph classification.

3) GCAPS-CNN (Verma & Zhang, 2018) , Dynamic Edge CNN (ECC) (Simonovsky & Komodakis, 2017) and Deep Graph CNN (DGCNN) (Zhang et al., 2018) . These methods are all GNN-based algorithms. GCAPS-CNN first extract FGSD (Verma & Zhang, 2017) features for nodes that do not have attributes and then generate capsules for each node with higher-order statistical moment value of its neighbor nodes. At the last layer, they calculate covariance between all nodes to generate graph embeddings. ECC extracts node features on the condition of edge labels in GNN and then apply multi-scale pyramid structure to coarsen the graph. It uses average pooling at the last layer to generate graph embeddings. DGCNN generates nodes embeddings through a multi-layer GNN and combine features extracted from all layers. Then they order the nodes based on the embeddings extracted from the last layer which is followed by 1-D CNN.

4.1.2. EXPERIMENTAL SET-UP:

Five biological graph datasets: MUTAG, ENZYMES, NCI1, PROTEINS, D&D and five social network datasets: COLLAB, IMDB-B, IMDB-M, RE-M5K, RE-M12K (Yanardag & Vishwanathan, 2015) are used for our experimental study. Details of these datasets can be found in Appendix B.

We applied 10-fold cross validation to evaluate the performance objectively. CapsGNN achieves the SOTA performance on social datasets. More specifically, we are able to improve the classification accuracy by a margin of 2.78% and 5.30% on RE-M5K and RE-M12K respectively. This demonstrates that learning features in the form of capsules and modeling a graph to multiple embeddings is beneficial to capture macroscopic properties of graphs which are more important in classifying social networks. These results also consistent with the property of CapsNet, as it focuses more on extracting important information from children capsules by voting. However, applying routing to the whole graph leads to preserve all the information at a graph level and this property is not suitable to give prominence to individual fine structures which might be more important to biological datasets analysis. This results in less robust of CapsGNN on biological datasets. Despite this, the performance of CapsGNN in graph classification task still demonstrates its capability of graph representation especially its high potential of large graph dataset analysis. In this experiment, the number of layers of GNN is set as L = 3, the number of channels at each layer is all set as C l = 2. We construct different CapsGNNs by adjusting the dimension of nodes (d n ) and graphs (d g ) capsules and the number of graph capsules (P ). The size of FC in scalar-based architectures is adjusted based on the size of CapsGNNs so that they have comparable number of trainable weights. Other hyper-parameters are the same as Appendix E. The details of the tested architectures are shown in Table 3 . Besides, NCI1 dataset, which has more than 4000 graphs, is used for the test. The accuracy of NCI1 on various architectures can be found in Figure 3 . In Table 3 and Figure 3 , the setting of different architectures is represented as d n -d g -P . Here, we choose the simplest setting (2-4-2) as an example: 2-4-2 means that the dimension of nodes capsules is d n = 2, the dimension of graph and class capsules is d g = 4 and the number of graph capsules P equals to 2. Besides, we set the dimension of FC of its corresponding scalar-based architecture as 12 so that they have comparable number of trainable weights. In this case, each graph is modeled as 2 4-dimensional graph embeddings in the CapsGNN or 1 12-dimensional graph embedding in its corresponding scalar-based architecture. Both architectures are sub-optimal to represent the whole dataset while CapsGNN can still reach higher accuracy compared with the scalar-based architecture.

As we can see from Figure 3 , the test accuracy of Caps-based architectures (CapsGNN) is higher than the corresponding scalar-based architectures in all settings. For the training accuracy, when the dimension of FC is slightly higher than the dimension of graph embeddings in CapsGNN, CapsGNN can still reach higher accuracy which indicates that CapsGNN is more powerful in representing the whole dataset. When we keep increasing the number of graph capsules in CapsGNN and enlarging the dimension of FC in scalar-based architectures, the difference between the dimension of graph embeddings and the size of FC becomes larger, their training accuracy will be closer. It is noted that the training accuracy of scalar-based architectures is slightly higher than the CapsGNNs when the dimension of FC is about 20% larger than the dimension of graph capsules. In this experiment, we use extremely simple architectures on purpose to simulate the situation where we need to model complex datasets with relatively simple architectures. Since each pair of CapsGNN and its corresponding scalar-based architecture have similar structure and comparable number of trainable weights, the higher training accuracy and testing accuracy of CapsGNN demonstrate its efficiency in feature encoding and its strong capability of generalization.

4.3. GRAPH CAPSULES ANALYSIS:

CapsGNN leverages on capsules idea to get multiple embeddings for each graph so that complex information underlying graphs can be captured more effectively. To explore the properties of the extracted graph/class capsules, we plot the graph distribution based on capsules extracted from different channels with t-SNE. Due to space constrain, we only take REDDIT-M12K as an example. We choose to depict the distribution of graphs which are generated from 3 categories, namely atheism , IAmA and mildlyinteresting with capsules extracted from the 1st, 2nd, 11th, 14th channel of graph capsules. As we can see from Table 4 , different channels of capsules represent different aspects of graph properties. atheism and IAmA can be discriminated obviously with capsules extracted from the 11th and the 14th channels while they are hard to be separated with capsules extracted from the 1st and the 2nd channels. However, atheism and mildlyinteresting can be discriminated with the capsules extracted from the 1st and the 2nd channels while they are mixed in the 11th and the 14th channels which is opposite to the case of atheism and IAmA. This phenomenon can also be observed in other multi-class datasets. It is still hard to figure out the specific aspects these capsules focus on. However, compared with scalar-based neural networks, modeling an object with multiple embeddings makes it possible to explore the meaning of each channel which may lead the model to learn more interpretable embeddings in the future. As we can see from Table 5 , different class capsules focus on different classification-related graph properties. For example, the capsules that represent athesism (first column) can well discriminate athesism (red) from the other two types of graphs while IAmA (green) and mildlyinteresting (blue) are mixed in this channel. The similar phenomenon can also be found in other class capsules. Besides, when we concatenate the capsules of these three classes together, three types of graphs can be well discriminated with the concatenated capsules which also directly reflect the classification performance. This property is quite different from standard scalar-based architectures where each graph is modeled with only one graph embedding2 . By introducing the concept of capsules, the graph and class capsules can not only preserve classification-related properties of each graph (reflected with the length of class capsules) but also other properties information (reflected with the angle of class capsules). The generated class capsules can also be useful in other follow-up work and we leave this to be explored in the future.

5. CONCLUSION:

We have proposed CapsGNN, a novel framework that fuses capsules theory into GNN for more efficient graph representation learning. Inspired by CapsNet, the concepts of capsules are introduced in this architecture to extract features in the form of vectors on the basis of nodes features extracted from GNN. As a result, one graph is represented as multiple embeddings and each embedding captures different aspects of the graph properties. The generated graph and class capsules can preserve not only the classification-related information but also other information with respect to graph properties which might be useful in the follow-up work and we leave this to be explored in the future. We believe this is a novel, efficient and powerful data-driven method to represent high-dimensional data such as graphs. Our model has successfully achieved better or comparable performance when compared with other SOTA algorithms on 6 out of 10 graph classification tasks especially on social datasets. Compared with similar scalar-based architectures, CapsGNN is more efficient in encoding features and this would be very beneficial for processing large datasets.

A SPECIFIC PROCEDURE OF ROUTING:

The specific procedure of routing is shown in Algorithm 1.

Algorithm 1 Dynamic routing mechanism returns parent capsules H given children capsules S, a set of trainable transform matrices W and the number of iterations t.

1: procedure DYNAMIC ROUTING(t, S, W)

2:

for all children capsule i :

v j|i = s T i W ij 3:

for all children capsule i to all parent capsule j: r ij ← 0 4:

for t iterations do 5:

for all children capsule i: ri ← sof tmax(r i )

6:

for all parent capsule j:

h j ← i rij v ij 7:

for all parent capsule j: hj ← squash(h j )

8:

for all children capsule i to all parent capsule j:

r ij ← r ij + hT j v ij 9:

end for 10:

return hj 11: end procedure

B DETAILS OF EXPERIMENTAL DATASETS:

The details of benchmark datasets we use in our experiment is shown in Table 6 . 

C COORDINATE ADDITION:

After Attention Module, coordinate addition can be used to preserve the position information of each node during the procedure of generating node capsule votes. The details of Coordinate Addition module can be found in Figure 4 . This module is not necessary in some datasets. Here, we propose this module as a selective optimization. When the GNN goes deeper, the extracted nodes features contain more specific position information of each node. Inspired by Zhang et al. (2018) where the node embeddings learned from the last layer of GNN are taken to order all nodes, we also take the capsules extracted from the last layer of GNN as the position indicators of corresponding nodes by concatenating it with each capsule of the node. The procedure of calculating votes with node position indicators can be written as:

EQUATION

where v (n,i)j ∈ R 1×(dn+dp) represents the node capsule vote from the ith channel of the nth node to the jth channel of graph capsules. W n ij ∈ R d×dn and W p j ∈ R d×dp are the transform matrices. s (n,i) is the same as introduced in Section 3.2 and represents concatenate operation. (2018) (GNN-based graph representation learning architectures), CapsGNN represents node features in the form of capsules. This is helpful to preserve the properties information contained in nodes more efficiently when generating graph embeddings. Besides, each graph is modeled as multiple embeddings in CapsGNN instead of only one embedding used in other approaches. This allows us to capture information of graphs from different aspects. The second difference is that, in these approaches, each part of the graph is given equal importance. However, the attention mechanism used in CapsGNN allows it to assign various weights to different nodes. This leads the model to focus on critical parts of input graphs. Lastly, different from Atwood & Towsley (2016) and Simonovsky & Komodakis (2017) , CapsGNN and Zhang et al. (2018) uses node features extracted from multiple layers of GNN so that different size of receptive-fields are applied to preserve more information.

2) GCAPS-CNN proposed by Verma & Zhang (2018) also introduced capsule-related concept into graph representation learning. However, they generate capsules in a feature-based manner instead of learning capsules as distributed embeddings. More specifically, when they extend a scalar feature to a capsule for the node n, P higher-order statistical moment value is calculated based on its neighbor nodes and these P value is concatenated to a P -dimensional capsule. Between layers, GCAPS-CNN performs dimension reduction to compress capsules back to scalar features, which defeats the purpose of having capsules in the first place. CapsGNN learns each dimension of capsules in a data-driven manner and apply routing mechanism between layers to preserve the learned meaning of each capsule. This also allows us to preserve multiple properties information contained in nodes more efficiently especially when generating graph embeddings.

3) Compared with CapsNet proposed by Sabour et al. (2017) which works well in image processing domain, CapsGNN needs to handle more complex situations when handling graphs. In image processing domain, the size of the input images can be standardized by resizing the images. However, it is not possible to simply resize the graphs. So, we introduced an additional Attention Module to tackle graphs that are vastly different in sizes and preserve important parts of graphs. We also propose to use features extracted from all layers of GNN since it is hard to define a suitable receptive-field size for graphs. Furthermore, compared with the architecture of CapsNet, CapsGNN has one additional graph capsules layer which is used to learn multiple graph embeddings and these embeddings reflect different aspects of graph properties which is valuable in future research. To the best of our knowledge, we are the first one to model a graph as multiple embeddings in the form of distributed capsules and we believe this approach of learning representations has a high potential for other complex data analysis which is not limited to graphs. Besides, CapsGNN has different explanation of linear transformation. In CapsNet, by applying a linear trainable transformation to pose vectors, the spatial relationship between object parts and the whole object can be well modeled. However, by applying linear trainable transformation, CapsGNN is simply computing the prediction vectors from nodes-level representations to graph-level representations. This transform matrix is not trying to model the change of viewpoint or capture viewpoint invariant knowledge but to model the relationship between the properties of nodes and the properties of the whole graph.

E EXPERIMENTAL SETTING:

The same architecture settings are used in CapsGNN for all datasets to show its robust performance. For the node capsules extraction, the GCN has 5 layers (L = 5), the number of channels at each layer is set as the same which is 2 (C l = 2). The number of graph capsules is fixed as 16 (P = 16). The dimension of all capsules are set as 8 (d = 8). The number of units in the hidden layer of Attention Module is set as 1 16 of the number of input units. The number of iterations in routing is set as 3. During training stage, we simultaneously reduce Loss c and Loss r and we scale Loss r with 0.1 so that the model focuses on classification task. λ is set as 0.5 and 1.0 for multi-class classification and binary classification respectively. As for the node attributes in different datasets, considering that REDDIT-M5K and REDDIT-M12K are large-scale datasets with widely distributed nodes degree, we set the attributes of all nodes in these two datasets as the same which means we consider the initial node embeddings for all the nodes as the same to avoid over-fitting. For the remained relatively small datasets, both of the node degree and other node attributes are sent to CapsGNN as node features to speed up the training and we apply dropout(dropput rate is 0.3) to the input node features to improve the learning performance. The settings are summarized in the Table 7 : 

F CONTRIBUTION OF EACH MODULE:

In addition to evaluate the performance of the whole architecture on the classification task, we also provide detailed study to quantify the contributions made by each module in CapsGNN on the

:

I represents identity matrix.

More details of graph distribution can be found at https://sites.google.com/view/ capsgnn/home where we provide more figures generated from COLLAB and REDDIT-MULTI-12K. We also plot the graph distribution based on graph embeddings learned from DGCNN as a reference and the figures can be found at https://sites.google.com/view/capsgnn/embeddings-visualization/ collab/class-capsules.

Figures and Tables:

Figure 1: Framework of CapsGNN. At first, GNN is used to extract node embeddings and form primary capsules. Attention module is used to scale node embeddings which is followed by Dynamic Routing to generate graph capsules. At the last stage, Dynamic Routing is applied again to perform graph classification.

Figure 2: The structure of Attention Module. We first flatten primary capsules and apply two layer fully-connected neural network to generate attention value for each capsule. Node-based normalization (normalize each row here) is applied to generate final attention value. Scaled capsules are calculated by multiplying the normalized value with primary capsules.

Figure 3: Comparison of efficiency in feature representation. The horizontal axis represents the setting of tested architectures. The vertical axis represents classification accuracy on NCI1.

Figure 4: The structure of Coordinate Addition Module. We take the capsules extracted from the final layer of GNN as the position indicators of corresponding nodes by concatenating it with each capsule of the node. The node capsules votes generated in this way contain more position information.

Each time we use 1 training fold as validation fold to adjust hyper-parameters, 8 training fold to train the architecture and the remained 1 testing fold to test the performance. We stop training when the performance on the validation fold reaches to the highest. Then we use the accuracy on the test fold as our test result. The final result is the average of these 10 test accuracy. By default, we use the results reported in the original work for baseline comparison. However, in cases where the results are not available, we use the best testing results reported inVerma & Zhang (2018),Zhang et al. (2018) andIvanov & Burnaev (2018). More details about experimental setting can be found in Appendix E. Experiment Result of Biological Dataset

Table content: <table><tr><td>Algorithm</td><td>MUTAG</td><td>NCI1</td><td>PROTEINS</td><td>D&amp;D</td><td>ENZYMES</td></tr><tr><td>WL</td><td colspan="5">82.05±0.36 82.19±0.18 74.68±0.49 79.78±0.36 52.22±1.26</td></tr><tr><td>GK</td><td colspan="5">81.58±2.11 62.49±0.27 71.67±0.55 78.45±0.26 32.70±1.20</td></tr><tr><td>RW</td><td colspan="2">79.17±2.07 &gt;3days</td><td colspan="2">74.22±0.42 &gt;3days</td><td>24.16±1.64</td></tr><tr><td>Graph2vec</td><td colspan="4">83.15±9.25 73.22±1.81 73.30±2.05 -</td><td>-</td></tr><tr><td>AWE</td><td colspan="2">87.87±9.76 -</td><td>-</td><td colspan="2">71.51±4.02 35.77±5.93</td></tr><tr><td>DGK</td><td colspan="5">87.44±2.72 80.31±0.46 75.68±0.54 73.50±1.01 53.43±0.91</td></tr><tr><td>PSCN</td><td colspan="5">88.95±4.37 76.34±1.68 75.00±2.51 76.27±2.64 -</td></tr><tr><td>DGCNN</td><td colspan="5">85.83±1.66 74.44±0.47 75.54±0.94 79.37±0.94 51.00±7.29</td></tr><tr><td>ECC</td><td>76.11</td><td>76.82</td><td>-</td><td>72.54</td><td>45.67</td></tr><tr><td colspan="2">GCAPS-CNN -</td><td colspan="4">82.72±2.38 76.40±4.17 77.62±4.99 61.83±5.39</td></tr><tr><td>CapsGNN</td><td colspan="5">86.67±6.88 78.35±1.55 76.28±3.63 75.38±4.17 54.67±5.67</td></tr></table>

Experiment Result of Social DatasetThe main objective of this experiment is to examine the efficiency of capsules in encoding graph features. More efficient in feature encoding here means representing more information with the similar number of neurons. We construct a scalar-based neural network for each CapsGNN and then compare the CapsGNN with its related scalar-based architecture by comparing their training and testing accuracy on graph classification task to demonstrate the efficiency in feature representation. More specifically,these scalar-based architectures are designed by replacing the graph capsules block and the class capsules block in CapsGNN with fully-connected layers (FC). In this case, the only difference between each pair of CapsGNN and its corresponding scalar-based architecture is that CapsGNN represents features with vectors and uses routing to propagate information between layers while the scalar-based architecture encodes features with scalar values.

Table content: <table><tr><td>Algorithm</td><td>COLLAB</td><td>IMDB-B</td><td>IMDB-M</td><td>RE-M5K</td><td>RE-M12K</td></tr><tr><td>WL</td><td colspan="5">79.02±1.77 73.40±4.63 49.33±4.75 49.44±2.36 38.18±1.30</td></tr><tr><td>GK</td><td colspan="5">72.84±0.28 65.87±0.98 43.89±0.38 41.01±0.17 31.82±0.08</td></tr><tr><td>DGK</td><td colspan="5">73.09±0.25 66.96±0.56 44.55±0.52 41.27±0.18 32.22±0.10</td></tr><tr><td>AWE</td><td colspan="5">73.93±1.94 74.45±5.83 51.54±3.61 50.46±1.91 39.20±2.09</td></tr><tr><td>PSCN</td><td colspan="5">72.60±2.15 71.00±2.29 45.23±2.84 49.10±0.70 41.32±0.42</td></tr><tr><td>DGCNN</td><td colspan="5">73.76±0.49 70.03±0.86 47.83±0.85 48.70±4.54 -</td></tr><tr><td colspan="6">GCAPS-CNN 77.71±2.51 71.69±3.40 48.50±4.10 50.10±1.72 -</td></tr><tr><td>CapsGNN</td><td colspan="5">79.62±0.91 73.10±4.83 50.27±2.65 52.88±1.48 46.62±1.90</td></tr><tr><td colspan="2">4.2 EFFICIENCY OF CAPSULES</td><td/><td/><td/><td/></tr></table>

Details of Tested Architectures in Efficiency Evaluation Experiment

Table content: <table><tr><td/><td colspan="8">2-4-2 2-4-4 2-4-8 2-4-16 4-4-2 4-4-4 4-4-8 4-4-16</td></tr><tr><td>No. trainable Para.(Caps)</td><td>208</td><td>367</td><td>688</td><td>1328</td><td>448</td><td>704</td><td colspan="2">1216 2240</td></tr><tr><td colspan="2">No. trainable Para.(Scalar) 216</td><td>370</td><td>692</td><td>1336</td><td>452</td><td>712</td><td colspan="2">1232 2246</td></tr><tr><td>Dim Node Feat.(Both)</td><td>2</td><td>2</td><td>2</td><td>2</td><td>4</td><td>4</td><td>4</td><td>4</td></tr><tr><td>Dim Graph emb.(Caps)</td><td colspan="8">2 × 4 4 × 4 8 × 4 16 × 4 2 × 4 4 × 4 8 × 4 16 × 4</td></tr><tr><td>Dim FC.(Scalar)</td><td>12</td><td>23</td><td>48</td><td>92</td><td>10</td><td>20</td><td>40</td><td>79</td></tr></table>

Visualization of Graph Capsules

Table content: <table><tr><td>Subreddit</td><td>Channel 1</td><td>Channel 2</td><td>Channel 11</td><td>Channel 14</td></tr><tr><td>atheism</td><td/><td/><td/><td/></tr><tr><td>&amp;</td><td/><td/><td/><td/></tr><tr><td>IAmA</td><td/><td/><td/><td/></tr><tr><td>atheism</td><td/><td/><td/><td/></tr><tr><td>&amp;</td><td/><td/><td/><td/></tr><tr><td>mildlyinteresting</td><td/><td/><td/><td/></tr></table>

Visualization of Class Capsules

Table content: <table><tr><td>Subreddit</td><td>atheism</td><td>IAmA</td><td>mildlyinteresting Concatenate</td></tr><tr><td>atheism(g)</td><td/><td/><td/></tr><tr><td>&amp;</td><td/><td/><td/></tr><tr><td>IAmA(r)</td><td/><td/><td/></tr><tr><td>&amp;</td><td/><td/><td/></tr><tr><td>mildlyinteresting(b)</td><td/><td/><td/></tr></table>

Dataset Description

Table content: <table><tr><td>Dataset</td><td colspan="6">Source Graphs Classes Nodes Avg. Edges Avg. Nodes Labels</td></tr><tr><td>MUTAG</td><td>Bio</td><td>188</td><td>2</td><td>17.93</td><td>19.79</td><td>7</td></tr><tr><td>ENZYMES</td><td>Bio</td><td>600</td><td>6</td><td>32.46</td><td>63.14</td><td>6</td></tr><tr><td>NCI1</td><td>Bio</td><td>4110</td><td>2</td><td>29.87</td><td>32.30</td><td>23</td></tr><tr><td>PROTEINS</td><td>Bio</td><td>1113</td><td>2</td><td>39.06</td><td>72.81</td><td>4</td></tr><tr><td>D&amp; D</td><td>Bio</td><td>1178</td><td>2</td><td>284.31</td><td>715.65</td><td>82</td></tr><tr><td>COLLAB</td><td>Social</td><td>5000</td><td>3</td><td>74.49</td><td>4914.99</td><td>-</td></tr><tr><td>IMDB-B</td><td>Social</td><td>1000</td><td>2</td><td>19.77</td><td>193.06</td><td>-</td></tr><tr><td>IMDB-M</td><td>Social</td><td>1500</td><td>3</td><td>13</td><td>131.87</td><td>-</td></tr><tr><td>REDDIT-M5K</td><td>Social</td><td>4999</td><td>5</td><td>508.5</td><td>1189.74</td><td>-</td></tr><tr><td colspan="2">REDDIT-M12K Social</td><td>11929</td><td>11</td><td>391.4</td><td>456.89</td><td>-</td></tr></table>

Experimental Setting for Graph Classification

Table content: <table><tr><td>Hyper-parameter description</td><td colspan="2">Notation Value</td></tr><tr><td>Number of GCN layers</td><td>L</td><td>5</td></tr><tr><td>Number of channels at each layer</td><td>C l</td><td>2</td></tr><tr><td>Number of graph capsules</td><td>P</td><td>16</td></tr><tr><td>Dimension of all capsules</td><td>d</td><td>8</td></tr><tr><td>Number of iterations in routing</td><td>t</td><td>3</td></tr><tr><td>Value used to scale Loss r</td><td>r</td><td>0.1</td></tr><tr><td>Value used to balance the loss from the positive and negative output</td><td>λ</td><td>0.5 for multi-class classification 1.0 for binary classification</td></tr></table>

Peer review (iclr2019):
 The paper fuses Capsule Networks with Graph Neural Networks. The idea seems technically correct and is well-written. With 13 pages the paper seems really long. Moreover, the experimental part seems to be too short. So, the theoretical and experimental part is not well-balanced. Minor concerns/ notes to the authors: 1. Page 1: The abbreviation GNN is used before it is defined. 2. Page 2: I guess there is a mistake in your indices. Capital N == n or? 3. Page 4: What is .mathbf{I}? I guess you mean the identity matrix. 4. Page 4: Could you define/describe C_all? 5. Page 5: Can you describe how you perform the coordinate addition or add a reference? 6. Page 6: The idea to use reconstruction as regularization method is not new. May you can add a respective reference? 7. Page 8: The abbreviations in your result tables are confusing. They are not aligned with the text. For example, what is Caps-CNN for a model? My major concern is about your experimental evaluation. Under a first look the result tables looking great. But that’s due to fact, that you marked the two best values in bold type. To be more precise, the method WL is in the most cases better than your proposed method. This makes me wondering if there is a real improvement by your method. It would be easier to decide if you would present the training/inference times and the number of parameters. By having that, I could relate your results regarding an accuracy-complexity tradeoff. Moreover, your t-SNE and attention visualizations are not convincing. As you may know, the output of a t-SNE strongly dependents on the chosen hyper-parameters like the perplexity, etc. You not mentioned the setting of these values. Additionally, it is hard to decide if your embeddings are good or not because you are not presenting a baseline or referencing a respective work. You are complaining that this is due to the space restrictions. But you have unlimited capacity in the appendix. So please provide some clarifying plots. Finally, I’m also not convinced that your attention mechanism works as expected. It’s again due to missing baseline results and/or a reference. If it’s not possible to add one of them, you could perform an easy experiment where you freeze your fully-connected layers of the attention module to fixed values (maybe such that it performs just an averaging) and repeat your experiments. In case your attention module works as expected you should observe a real change in terms of accuracy and in your visualizations too. You could also think about to publish your code or present further results/plots in a separate blog. Update: According to the revised version which addresses a lot of my concerns, I vote for marginally above acceptance threshold.