Graph Warp Module: an Auxiliary Module for Boosting the Power of Graph Neural Networks in Molecular Graph Analysis
Authors: Katsuhiko Ishiguro
Shin-ichi Maeda
Masanori Koyama
Venue:  (2020)
Keywords: Graph Neural Networks
molecular graph analysis
supernode
auxiliary module

Abstract: Graph Neural Network (GNN) is a popular architecture for the analysis of chemical molecules, and it has numerous applications in material and medicinal science.
Current lines of GNNs developed for molecular analysis, however, do not fit well on the training set, and their performance does not scale well with the complexity of the network. 
In this paper, we propose an auxiliary module to be attached to a GNN that can boost the representation power of the model without hindering the original GNN architecture. 
Our auxiliary module can improve the representation power and the generalization ability of a wide variety of GNNs, including those that are used commonly in biochemical applications. 

Under review as a conference paper at ICLR 2020
GRAPH WARP MODULE: AN AUXILIARY MODULE FOR BOOSTING THE POWER OF GRAPH NEURAL NET- WORKS IN MOLECULAR GRAPH ANALYSIS
Anonymous authors Paper under double-blind review
ABSTRACT
Graph Neural Network (GNN) is a popular architecture for the analysis of chemical molecules, and it has numerous applications in material and medicinal science. Current lines of GNNs developed for molecular analysis, however, do not fit well on the training set, and their performance does not scale well with the complexity of the network. In this paper, we propose an auxiliary module to be attached to a GNN that can boost the representation power of the model without hindering the original GNN architecture. Our auxiliary module can improve the representation power and the generalization ability a wide variety of GNNs, including those that used commonly in biochemical applications.
INTRODUCTION
Recently, Graph Neural Network (GNN) is popular choice of model in the analysis of molecular datasets in medicinal and material science. Many molecular datasets consist of molecular graphs with feature vectors associated
each atom, and numerous methods based on GNN has been proposed to date just for learning the features of chemical molecules (Wu et al. 2018 Duvenaud al. 2015 Kearnes et al. 2016 Li et al., 2017 Gilmer et al., 2017 Shang et al. 2018). such as those pertaining to electrical conductivity and toxicity. One problem in the application of GNN to molecular datasets the difficulty in reducing the training loss. Unlike in the applications of Deep Neural Networks (DNNs) to image datasets, the training loss of GNN not decrease consistently with the number of layers nor number of nodes per layers (cf. Fig. 4 in the appendix, thin dashed lines), and this seems to happen to numerous GNN architectures that used applications today (Duvenaud et al. 2013 Li et al. 2016 Kipf & Welling 2017; Xu et al. 2019 Busbridge al., 2018). Unfortunately. ly, many strong techniques developed for deep CNNs such as ResNet (He et al. 20166) cannot be applied naively to GNN, because the tasks for GNNs are oftentimes fundamentally different in nature from that of standard DNN. For example, each graph data to be passed to the network can differ in size, and it is also often desired that GNN is equivariant (invariant under the reordering of vertices) in general. To the best of the authors' knowledge, there have not been many studies done to date that directly addressed the problem of training loss.
molecular dataset does
In this study, we propose graph warp module (GWM), a supernode (Li et 2017 Gilmer et al. 2017 Battaglia et al. 2018) based auxiliary module that can be attached to generie GNNs of various types to improve its representation power. The I/O of the auxiliary module is defined independently from the GNN to which it is attached, and the users can install the GWM just by adding small segment of code.
Our GWM consists of three major components. The first component is viriual supernode (Li et al. 2017, Gilmer et al.) 2017), which communicates with all nodes in the graph and promotes the remote message passing. The second and third components are attention unit (Vaswani al.) [017]] Veličković et al.) 2018) and gating units (Cho et al. 2014). These adaptive weighting functions the module help to adjust the flow of messages and deliver message of appropriate strength to each node in the graph.
Our GWM can consistently improve the performance of various types of GNN various types of dataset. In Fig. [eee show the effect of GWM on the performance of four types of GNNs with the same embedding dimension and the same number of layers four molecular graph datasets. As

---

Under review as a conference paper at ICLR 2020
Reduction Ratio of Test Loss
1.00
Train loss reduction and test loss re- duction achieved by GWM on various model- dataset pair. The shape of each point presents the dataset used, and the color of each point presents the GNN model used. The horizon- tal axis denotes the ratio of reduced training loss, and the vertical axis denotes that of the test loss. That the x-coordinate of a point is positive implies that the attachment of GWM improved the train performance for the cor- responding model-dataset pair. That the y- coordinate of point is positive implies that GWM improves the test-performance.
Figure 1:
0.6
0.4_
GNN
Dataset
NPP
HIV
GGNN
Tox21
0.2.
RSGON
Lipo
GIN
0.0
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Reduction Ratio of Train Loss
can see in the figure, the attachment of GWM reduces both train loss and test loss for all but three model-dataset pairs. The GWM provides not only more representation powers (less train loss), but also better generalization performances (less test loss) for various GNNs.
As we will show in section 4.4, we can further improve the positive effect of GWM using hyperpa- rameter optimization softwares such as Optuna (Akiba et al. 2019).
Our contributions are as follows:
1. We introduce GWM, an auxiliary module that can help improve the representation power of the GNNs that are designed for the analysis of small graphs.
2. We show that the attachment GWM can improve both the representation power and the generalization ability of various GNN models on many popular molecular graph datasets.
2 RELATED WORK
2.1 VIRTUAL SUPERNODE
common challenge in the application of GNNs to a graphical dataset is the difficulty in prop- agating the information across remote parts of graphs. Previously proposed solutions include sub-sampling (Hamilton et al., 2017) and pooling neighbor nodes (Ying et al. 2018). However, these clustering approaches are not too effective on the analysis of small graphs, in which every node can have a strong influence on the graph label.
In this study, we use supernode (Gilmer al. 2017 Li al. 2017 Pham et al. 2017) to promote the global propagation of the information in molecular graphs. By adding a supernode to a graph, allow any pair of nodes in the graph to communicate through the supernode in one hop. Battaglia et al. (Battaglia et al. 2018) discusses a framework of GNNs that generalizes the supernode- augmented GNNs. One advantage of the supernode- basedd approach is that we can modify the network architecture while keeping the original GNN model intact. However, naive addition of supernode to graph can potentially lead inadvertent over-smoothing of information propagation (c.f. (Li et al. 2018)). ). In our study, we therefore make the supernode module by combining it with gated message passing mechanism. This auxiliary module enables us to regulate the amount and type of information that is propagated through the feature space of the supernode.
2.2 MESSAGE PASSING AND ATTENTION/ MECHANISM IN GNN
The supernode our GWM transmits information using the mechanism of message passing neural network (MPNN) GGimmer et al. 2017), which is defined recursively follows by composing multiple layers of the form:
h4 Ft ((he1,555 N(1) U {i)})
where j.j are indices of nodes in a graph. hy, is the feature vector of the node i at the 9th layer, (i) is the neighborhood of the node i, and Fr is appropriate choice of function that updates the
2

---

Under review as a conference paper at ICLR 2020
INPUT
Before Altaching the Graph Warp Module
Node feature veciors @ Layer 1-1
Node feelure veciors @Layer
Origiall Graph data
(molocular graph)
After Adttaching the Graph Warp Module
Feature voctors original graph structure
Node feature @Layer
Nade feature @Layerli 1-1
Transmitter Unit
Warp
+ virtual "supernode" connecting all nodes
Superode feature vector @ BLayer 1-1
Graph Warp Module
Supernode feature vector @Layer!
The overview of the proposed Graph Warp Module (GWM). A GWM consists of supernode, transmitter unit, and a warp gate unit. A GWM can be added the original GNN as an auxiliary module. At each layer, the supernode and the main network communicate through the transmitter and the warp gate.
Figure 2:
feature vectors of the previous layer. That is, MPNNs work by passing the information of each node to its neighbors in a recursive manner. Various methods are proposed for the choice f and for the method of pooling the information of the neighbors of each node (Schlichtkrull al. 2017 Kipf &
Welling 2017 Li et al. 2016 Bruna & Szlam 2014 Duvenaud et al. 2015)).
Attention mechanism is mechanism that helps the network regulate the importance of each node/edge
in message passing (cf. (Wang et al. 2018) ). A Relational GCN (Schlichtkrull et al. 2017) assumes that the aggregation weights of he 1.j is fixed a priori in all Fe. With such architectures, however, one cannot regulate the higher order correlation amongst the outputs from each node. Graph Attention Networks (GATs) (Veličković et al. (018)) introduce a self-attention mechanism (Vaswani al.) 2017), which is equipped with a trainable set of weights that controls the importance of edges for each node. The relational graph attention network (RGAT) (Busbridge et al. 2018) also builds upon GAT and constructed multiple types of attentions derived from relation-type- wise intermediate node representations. Finally, a GRU (Cho et al. 2014) a gating mechanism originally introduced for recurrent neural networks. Gated Graph sequence Neural Networks (GGNN) (Li et al.) 2016) are the first to apply GRUs to the GNNs, and their method aims to introduce a recurrence re relation between successive layers. Although not in the form of super-node, very recent paper (anonymous) 2019) uses a mechanism that combines GGNN and self-attention to capture both local and global relationships of a graph structure of program source codes. Our GWM equipped with both multi- relational attention mechanisms and GRUs to grant the module greater flexibility for the transmission of message between supernode and the bulk nodes.
3 GRAPH WARP MODULE
Our Graph Warp Module (GWM) is made of three building blocks: (1) a supernode, (2) transmitter unit, and (3) a warp gate unit (Fig. 2 In GWM-attached GNN, information is propagated across the graph through communication between the supernode and the original (main) GNN at each layer. Messages from the supernode and the main GNN are transmitted to the warp gate through the transmitter unit, and the results of the communication are passed back to the module/main network through the warp gate units. In this section, we describe the Graph Warp Module in detail, and present the motivation of design.
3.1 PREMISE: VANILLA GNN AND ITS I/O
Before describing our GWM, we need to present the I/O notation for the family of GNNs we consider, and explain how they will be used when GWM is attached to GNN. We denote arbitrary
3

---

Under review as a conference paper at ICLR 2020
graph with the edge set E and the node set V as G = (V, E). We will label the nodes in V as i = L.2,...[V), and represent each edge as pair of nodes in V. The adjacency matrix A R a matrix whose (i. j)th entry is the weight assigned to the edge between the node and the node Each instance of data passed the GNN is set of input feature vectors. We denote an input feature vector associated with node as 24. The type of GNN that we consider computes the output recursively by applying composition of smooth functions Fy to I,S. With the understanding that Tj = jo let he. = -1,,,(he-1..3 E V) E Rd be the vector of features to be assigned to the ith node by the éth layer of the GNN. When the GNN is operating on its own without the attachment of GWM, the GNN updates a feature vector using he, -
E V) E R Finally, the GNN reports some form of the aggregation of ( E V} as the final Readout output.
When a GWM is attached to the GNN, the main(bulk) GNN is requested to report V) E Rd as the message from the e - 1th main layer to the module, where it is treated as an element in the intermodule hyperspace and mixed with the transmission from the supernode. The GWM will return the mixed message he back to the 9th layer of the main GNN. At the same time, the GWM requests transmission message from the main GNN the éth supernode. The module will mix the transmission and the message from the l - 1th supernode and return the mixed message De to the fth supernode. The final output is produced by aggregating {KL E V} and gL.
1,j E
3.2 SUPERNODE
A supernode is special node that is connected to nodes in the original graph to promote global information propagation across the network (Fig. 2 A supernode is to be prepared for each 7th layer of the main GNN, and we associate feature vector ge to the supernode the (th layer. At each layer, the transmitter requests the following from the supernode: (1) a message 9 (gt) for the E + 1th layer and (2) a transmission to the main network, where 9e is an appropriate choice of smooth function.
Because a supernode is a superficial variable, we must initialize manually. For instance, we can use some form of aggregation of the global graph features (e.g. a number of nodes or edges, graph diameter, girth, cycle number, min, max, histogram, or an average of input node features,.... A detailed example of the aggregated feature is presented in the appendix.
3.3 TRANSMITTER UNIT
The transmitter unit handles the communications between the main GNN module and the GWM (Fig. 3.. The transmitter module responsible for translating the messages from the recipient into a form that can be mixed in the intermodule hyperspace. We will use multiple types of messages and thus use a separate attention mechanism for each type of message. Before transmitting messages from the main GNN to the supernode, the transmitter uses the K -head attention mechanism to aggregate messages of each type. We enumerate the components included therein:
main super aggregated message of head k from the main GNN to the supernode at layer é, t
1 maii--super, transmission from the main GNN to the supernode, derived from m e
main-ssuper
super-amail transmission from the supernode to the main at layer É.
98
The transmissions are to be constructed from the following set of equations. For a vector we use Umn E [((m=n) to denote the concatenation of the vectors Um, Um+1,,.. E IRd. Throughout, we use capital letters to denote the trainable coefficients.
h
main ssuper
tanh (We
(Weme m.1.k
main-ssuper
E R
m
main main-ssuper
Σ Cn..Ueehhee TR
(3)
aiiik = softmax (h7 Ac E (0.1).
(4)
where oggii, denotes an attention weight of the ith node at the kth head (type) and the Ith layer.
The transmission from the supernode to the main is simply given by:
9t main tanh (Fege-1) E DR.
(5)
4

---

Under review as a conference paper at ICLR 2020
Node vestors @Laayr
Intra-module of node festure vectors
(Main GNN)
Mixed messages vectors
Node festure vectors @laayrr
h1-1
h
Warp Gale Unit gated-sum mixing
GRU
h
Tranemitter Unit: Inter-modure message
super-a main
1
g
Transmitter Unit: inter-module message
main - supee
Warp Gate Unit: galld sum maxing
GRU
gi
gi
81 1
8/
Maxed messages for supemode feature vector
Supemode feeture vector @Layyrr!
Supernode featuge vector @Layer
Intra-module update supernode fealure vector
Figure 3: Details of the GWM computations.
There is no analogue of m for the supernode because we are not considering a set of messages of different types to be transmitted from the supernode.
3.4
WARP GATE
The warp gate is responsible for merging the transmitted messages and for passing the results the supernode and the main network through self recurrent units. The gate uses warp gate coefficients to control the mixing-rate of the messages. The components of the warp Gate are:
h inputs to the GRU unit at 9th layer that transmits the message to the main GNN.
c:: inputs to the GRU unit at 9th layer that transmits the message to the supernode.
he the message expressed in the intermodule hyperspace.
E V) e RR from the 1th layer of the main network,
ge: the message 9:-1 (gt-1) DR from the 1th supernode, where & is an appropriate smooth function with outputs in the intermodule hyperspace.
Zg,5 tensor of warp gate coefficients for the transmission from the supernode to the main GNN.
(8). 26. tensor of warp gate coefficients for the transmission from the main GNN to the supernode.
The module then mixes the transmissions and the messages from the previous layer by applying the following gated interpolations:
h4. (1-2e,) O h + 21, 9t
super mmiin mainn
E RR
(6)
9 2 ④ 1
ssuper
+ (1 (9)
3 ge DDD
=e. a HHhe, + Gegt
super mainn
28 (S)
H 115
(S) main ssuper
G Dt
where a is nonlinear function whose range lies in [0, 1]. Finally. the warp gate returns the mixed messages to the main GNN and the supernode through gated recurrent unit (GRU) s:
bg GRU (he-1,5, E ERD ge GRU (ge-1.92) E ER
As for the structure of GRU, used the original design introduced by CCoo al. 2014).
As we will show with ablation studies (Sec. 4.4), every component of GWM essential making the module work. The Attention coefficients (Eqs. 31) are important because the amount and the type of information that must be transmitted to remote nodes may differ for different nodes. The Gating coefficients ( are important because want regulate the transmission from each
5

---

Under review as a conference paper at ICLR 2020
node in the graph to the supernode and vice versa. We use different recurring units (Eq for the transmission from the module to the supernode and the transmission from the module to the main network for each layer because the amount of the information that must be reinforced may differ for the main network and the supernode at each layer.
3.4.1 COMPUTATIONAL COMPLEXITY
Let V be the vertex set, K be the number of attention heads, and D be the dimension of the node embedding. Then the additional computational cost incurred by the attachment of GWM is at most O(VV D²). As for the actual computation time, GWM attached module consumes approximately double the time of the original unattached version.
EXPERIMENTS
In this section, we present our experimental results on multiple molecular graph datasets, testing the efficacy of the GWM for graph regression tasks and graph classification tasks.
4.1 DATASETS
We used four datasets collected in MoleculeNet (Wu et al.) 2018). These datasets are described in the SMILES string format, which admits the graph representations we described above. For details, please see (Wu et al. 2018))
For the graph regression tasks, we used the QM9 dataset and the Lipophilicity (LIPO) dataset. QM9 is dataset with numerical labels, containing about 133K drug-like molecules with 12 important chemical-enen -neereale,nergetic, electronic, and thermodynamic properties, such as HOMO, LUMO, and electron gaps. The LIPO dataset is another numeric-valued dataset, containing the solubility values of roughly 4K drug molecules. Each instance of data in these datasets is pair of a molecular graph and a numerical value(s): the 12 chemical properties in the QM9 dataset, and the solubility in the LIPO dataset. For both datasets, the task is to predict the numerical value(s) from the molecular graph. We evaluated the performance of the models using mean absolute errors (MAEs). We report the averaged MAE over 12 sub-tasks (properties) for QM9.
For the graph classification tasks, we used the Tox21 and the HIV datasets. The Tox21 dataset contains about 8K pairs of molecular graph and 12 dimensional binary vector that represent the experimental outcomes of toxicity measurements on 12 different targets. The HIV dataset contains roughly 42K pairs of molecular graph and binary label that represent the medicinal effect of the molecule. For these datasets, the task is to predict the binary label(s) from the molecular graph. For these tasks, we use ROC-AUC values measure of performance. We report the averaged ROC-AUC over 12 sub-tasks (targets) for Tox21.
Throughout, we used the train/validationntess data splits of the "scaffold" type, which is considered by (Ruddigkeit et al., 20122] Ramakrishnan et al. 2014) as the difficult type for test predictions. Please find the appendix for details.
4.2 CHOICES OF THE MAIN GNN MODELS AND IMPLEMENTATIONS
We test GWMs on various GNN models. Neural Fingerprints (NFP) (Duvenaud et al. 2015) and Weavenet (Kearnes et al. 2016) are relatively classical baselines. A Gated Graph Neural Network (GGNN) (Li et al. 2016) is a strong GRU-based GNN. Renormalized Spectral Graph Convolutional Network (RSGCN
Kipf& & Welling, 2017). a popular GNN model approximating a CNN for graphs (Defferrard et al., 2016). The relational graph attention network (RGAT) (Busbridge et al. 2018) uses multiple attention mechanisms for a set of edge types. Graph Isomorphism Network (GIN) Xu al. 2019) employs multi layer perceptrons within each layer for richer transformations.
We implement all models in Chainer (Tokui et al. 2015). In the readout layer, we first aggregate all information from the main nodes in the same way as in the original paper, concatenated the result with the features from the supernode, and passed the concatenated tensor to fully connected layer.
Referred RSGCN in Chainer-Chemistry package, hut often simply referred "GCN" several papers.

---

Under review as a conference paper at ICLR 2020
Model Name (emulating)
Attention
Gatings
GRUs
Simple supernode (Li et al. 2017: Pham et al. 2017) NoGate GWM Gilmer et al. 2017
(supernode only)
Proposed GWM
Table 1: Supernode- models validated in the Experiment 4.4. Note that the "simple supernode" ablation model and the model used in (Pham et al., 2017) allow bi-directional message passings between original nodes and the supernode while the one used in (Li et al. 2017) only allows directional messages from the original nodes to the supernode.
For evaluation, used softmax cross entropy for classification tasks and mean squared error for regression tasks. We use fixed set of hyperparameters throughout the study.
All models trained with Adam (Kingma & Ba 2015). We report the results of the model snapshots of the epoch for which the best validation score was achieved. For implementation details including readouts and hyperparameters, please read the appendix.
4.3 TRAINING AND TEST LOss REDUCTION
We explain the details of the experiment that produced the result presented in Fig. For this experiment, we reported the average F of the loss reduction ratio r = 2 -- ge(+) for both training loss [All
and test loss over 10 runs. L denotes the loss of the vanilla model, and (e(+) denote the loss of the GWM-installed model. Ptrain denotes reduction ratio of the training loss, and Ftest denotes reduction ratio of the test loss.
Fig. is the scatter plot of the (Ftrain, Fteat), with the dotted slope representing Ttrain Ftest. For all GNNs and datasets, we set -= 3 and D = 50. As we can see in the plot, Ftrain were negative for the two GNNs in the HIV dataset (blue and brown circles). Ftrain for GGNN on QM9 (blue cross) was a very small negative value). For the other 13 (model-dataset) pairs, the attachment of GWM consistently reduces the training loss( i.e. improves the fit the training graph datasets. ) Remarkably, 15 out of 16 pairs had positive FrestS: the GWM improved generalization performances in most cases. It is worthy of note that Ptrain and Frest are positively correlated in this scatter plot. We were able to obtain similar results for all other choices of hyperparameters results with different hyperparameter values, see the appendix. This result implics that our GWM has the general effect of improving the generalization performance by augmenting the representation power of the main GNN.
tested. For the
4.4 EFFECT OF THE GWM ON THE REPRESENTATION POWER OF MODEL SPACE
In the second experiment, we studied the effect of GWM on the representation power of GNN models. To compare the GWM-augment GNNs with their vanilla GNN counterparts on fair grounds, we used Bayesian optimization to optimize the number of layers and the dimension of feature vectors for each -dael-d pair we tested. Hyperparameters are optimized via the Optuna (Akiba et al. 2019) library.
We conducted a set of ablation studies to investigate the effect of (1) attention mechanism, (2) Gating mechanism, and (3) the Recurrent unit. We used two ablation models. Simple supernode model is supernode without attention, gatings, and GRU functions. This ablation model can be considered variant of (Pham et al., 2017)'s supernode model in which the supernode and the bulk networks communicate with each other in bidirectional manner. This model can be also considered as an extended version of supemode proposed in (Li al. 2017).
NoGate GWM is a GWM without gatings, and lacks GRU for the main GNN.
Table I summarizes the details of ablation studies. For the detailed formulations of the two ablation models, please see the appendix.
Table 2 and Table 3 respectively present the MAEs on the regression tasks and the ROC-AUCs for the classification tasks, averaged over 10 random runs. In the tables, bold faces indicate the improvements from the vanilla GNN and asterisks indicate the best model among supernode models
7

---

Under review as a conference paper at ICLR 2020
Dataset
GNN model
NFP
Weave
RGAT
GGNN
RSGCN
GIN
# Improved
vanilla GNN
.677
1.19
.753
.582
.844
LIPO
Simple Superood
.693
1.01
.740
.604
.775
.819
4/6
+NoGate GWM
.675
.721
.688
.576
.787
.847
5/6
Proposed GWM
.672*
.688*
.65**
.569*
.752
.784
6/6
vanilla GNN
6.16
6,38
8.96
4.92
14.0
QM9
+ +Simplc Supemode
7.68
5.51
9.00
5.41
14.6
11.5
3/6
+ NooGate GWM
6.84
5,40*
9.21
5.52
12.5
12.9
3/6
Proposed GWM
6.64*
5.90
8.39
4.88*
11.9
11.8
5/6
Table 2: MAEs on the LIPO dataset and QM9 dataset. Smaller values are better. Scores on QM9 are the average MAEs over 12 sub-tasks. The score of select models are presented the appendix.
Dataset
GNN model
NFP
Weave
RGAT
GGNN
RSGCN
GIN
Improved
vanilla GNN
.724
.670
.707
.746
.746
.729
HIV
Simple supernode
.707
.676
.704
.764*
.728
.729
2/6
+NoGate GWM
.714
.680
.726
.744
.742
.739
3/6
Proposed GWM
.731
.681
.748
.762
.758
.755*
6/6
vanilla GNN
.763
.710
.764
.757
.760
.740
Tox21
SSimple supernode
.770
.750
.787
.790
.770
.763
6/6
+ NooGate GWM
.775
.764
.786
.792*
.759
.766
5/6
Proposed GWM
.769
.767
.787
.785
.766
.768*
6/6
Table 3: ROC-AUCs on the HIV dataset and Tox21 dataset. Larger values are better, Scores on Tox21 are the average MAES over 12 sub-tasks. Scores of select models are presented in the appendix.
for each (dataset, GNN) pair. Full tables with standard deviations are presented in the appendix. As we can see in the tables, the proposed GWM improves the generalization performances for 23 out of 24 (model-datasee pairs. These results suggest that the proposed (full) GWM can improve GNNs' performances irrespective of the choice of GNN models and the dataset.
A few words of caution are in order here. Two ablation models did not improve the generalization of GNNs for the QM9 and the HIV datasets (see the column "# Improved"). This suggests an appropriate combination of attentions, gatings, and GRUs is essential in making the supernode effective for the analysis of molecular graph datasets.
We shall emphasize that the goal of this study is not to find the specific network architecture that achieves the states of the art performance for selected dataset Instead, the goal of our work is propose attachable module that improves the representation power and generalization performance irrespective of the choice of GNN architecture. As we can see the presented result, the attachment of GWM improves the result in most cases; of the results of our GWM attached model for Tox21 is actually SOTA (0.787, achieved by GWM attached RGAT).
5 CONCLUSION
For generic DNN, numerous effective installable modules have been proposed for the improvement of the model (e.g. (Srivastava et al., 2014], Joffe & Szegedy 2015) Miyato et al.) 2018 He et al.) 2016)). The proposed GWM is the first of its kind to be installed to generic GNN as an auxiliary module. Experimental results show that the GWM can generally improve the representation power as well as the generalization performance the molecular graph datasets. We would like to emphasize that the choice of the internal structure of GWM is not limited to the ones we described in this study, and that there are possibly numerous ways construct GWM-like module. For example, there no provable justification for the of linear transformation in the transmissions or bilinear form in the attention coefficients a. Effective choices of supernode features are also open further research. Our study can possibly open an entirely new avenue for the architectural study of GNNs.
GNN, irrespective of the choice of GNN architecture and
For example, Yang al. datasct.
2019 and Li
2017 achieve the SotA AUCss (0.776) of GNNs on the HIV
8

---

Under review as a conference paper at ICLR 2020
REFERENCES
Takuya Akiba, Shotaro Sano, Toshihiko Yanase. Takeru Ohta, and Masanori Koyama. Optuna: A Next-generation Hyperparameter Optimization Framework. In Proceedings of the ACM SIGKDD International Conference on Knowledge Discovery and Data Mining (KDD), 2019.
anonymous. Global Relational Models of Source Code.
OpenReview, pp.
B
2019.
Peter W. Battaglia, Jessica B. Hamrick, Victor Bapst, Alvaro Sanchez-gonzan gonzalez, Vinicius Zambaldi, Mateusz Malinowski, Andrea Tacehetti, David Raposo, Adam Santoro, Ryan Faulkner, Caglar Gulcehre, Francis Song, Andrew Ballard, Justin Gilmer, George Dahl, Ashish Vaswani, Kelsey Allen, Charles Nash, Victoria Langston, Chris Dyer, Nicolas Heess, Daan Wierstra, Pushmeet Kohli, Matt Botvinick, Oriol Vinyals, Yujia Li, and Razvan Pascanu. Relational inductive biases. deep learning and graph networks. arXiv, pp. 1806.01261v3 [cs.LG], 2018.
Joan Bruna and Arthur Szlam. Spectral Networks and Deep Locally Connected Networks on Graphs. In Proceedings of the 2nd International Conference on Learning Representations (ICLR), 2014.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Hammerla. Relational Graph Attention Networks. OpenReview, 2018.
Kyunghyun Cho, Bart Van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Holger Schwenk, and Yoshua Bengio. Learning phrase representations using rmn encoder-decoder for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Methods in Neural Language Processing (EMNLP), pp. 1734, 2014.
Michaël Defferrard, Xavier Bresson, and Pierre Vandergheynst. Convolutional Neural Networks on Graphs with Fast Localized Spectral Filtering. In Advances Neural Information Processing Systems 29 (Proceedings of NIPS), 2016.
David Duvenaud, Dougal Maclaurin, Jorge aguilera-Iparraguirre Rafael Gómez-Boobarelli, Tim- othy Hirzel, Alán Aspuru-Guzik, and Ryan P. Adams. Convolutional Networks on Graphs for Learning Molecular Fingerprints. Advances in Neural Information Processing Systems 28 (Proceedings of NIPS), pp. 2224-2232, 2015. URL http::/axxiv.rrgg wwtp:/iwww.r//b///55g33
09292
Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol Vinyals, and George E. Dahl. Neural Message Passing for Quantum Chemistry. In Proceedings of the 34th International Conference on Machine Learning (ICML), pp. 1263-1272, 2017. h httpvv.xiiv..////
01212
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances in Neural Information Processing Systems 30 (Proceedings NIPS), 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual I Learning for Image Recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1-9, 2016. URL http: http://aaxvvvii. www.aaawww..gga////a.aaa..
03385
Sergey loffe and Christian Szegedy. Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift. In Proceedings of the 32nd International Conference on Machine Learning (ICML), pp. 448-456, 2015.
Steven Kearnes, Kevin McCloskey, Mare Berndl, Vijay Pande, and Patrick Riley. Molecular graph convolutions: moving beyond fingerprints. Journal of Computer-Aided Molecular Design, 30(8): 595-608, 2016.
Diederik P. Kingma and Jimmy Lei Ba. Adam: a Method for Stochastic Optimization. In Proceedings of the 3rd International Conference on Learning Representations (ICLR), 2015.
Thomas N. Kipf and Max Welling. Semi-supervised Classification with Graph Convolutional Net- works. In Proceedings of the 5th International Conference on Learning Representations (ICLR), 2017.

---

Under review as a conference paper at ICLR 2020
Junying Li, Deng Cai, and Xiaofei He. Learning Graph-Level Representation for Drug Discovery. arXiv. pp. 1709.03741v2 [cs,LG], 2017.
Qimai Li, Zhichao Han, and Xiao-ming Wu. Deeper Insights into Graph Convolutional Networks for Semi-supervised Learning. In Proceedings of the 32nd AAAI Conference Artificial Intelligence (AAAI-18), 2018.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated Graph Sequence Neural Net- works. In Proceedings of the 4th International Conference on Learning Representations (ICLR), 2016.
Minh-Thang Luong, Hieu Pham, and Christopher D. Manning. Effective Approaches to Attention- based Neural Machine Translation. In Proceedings of the 2015 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2015.
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral Normalization for Generative Adversarial Networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
Trang Pham, Truyen Tran, Hoa Dam, and Svetha Venkatesh. Graph Classification via Deep Learning with Virtual Nodes. arXiv, pp.
2017.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp, and O. Anatole von Lilienfeld. Quantum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1: 1:140022, 2014.
Lars Ruddigkeit, Ruud van Deursen, Lorenz C. Blum, and Reymond Jean-Louis. Enumeration of 166 billion organic small molecules in the chemical universe database gdb-17. Journal of chemical information and modeling, 52(1 1) 28864 2012.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling Relational Data with Graph Convolutional Networks. arXiv, pp. 1703.06103v4 [stat.ML.], 2017.
Chao Shang. Qinqing Liu, Ko-shin Chen, Jiangwen Sun, Jin Lu, Jinfeng Yi, and Jinbo Bi. Edge Attention-based Multi-Relational Graph Convolutional Networks. arXiv, pp. 1802.04944v1 [stat.MI.J, 2018.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, Ilya Sutskever, and Ruslan Salakhutdinov. Dropout: A Simple Way of Prevent Neural Networks from Overfitting. Journal of Machine Learning Research, 15:1929-1958, 2014.
Seiya Tokui, Kenta Oono, Shohei Hido, and Justin Clayton. Chainer: a Next- Open Source Framework for Deep Learning. In Proceedings of Workshop Machine Learning Systems (LearningSys) in The Twenty-ninth Annual Conference on Neural Information Processing Systems (NIPS), 2015. URL URL cha iner.org//
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention Is All You Need. In Advances in Neural Information Processing Systems 30 (Proceedings of NIPS), 2017.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In Proceedings of the 6th International Conference on Learning Representations (ICLR), 2018.
Xiaolong Wang, Ross Girshick, Abhinav Gupta, and Kaiming He, Non-local Neural Networks. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2018.
Zhenqin Wu, Bharath Ramsundar, Evan N Feinberg, Joseph Gomes, Caleb Geniesse, Aneesh S Pappu, Karl Leswing, and Vijay Pande. MoleculeNet: A Benchmark for Molecular Machine Learning. Chemical Science, 9(2):513-534) 2018
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How Powerful are Graph Neural Networks? In Proceedings of the 7th International Conference on Learning Representations (ICLR), 2019.
10

---

Under review as a conference paper at ICLR 2020
Kevin Yang. Kyle Swanson, Wengong Jin, Connor Coley, Philipp Eiden, Hua Gao, Angel Guzman- Perez, Timothy Hopper, Brian Kelley, Miriam Mathea, Andrew Palmer, Volker Settels, Tommi Jaakkola, Klavs Jensen, and Regina Barzilay. Are Learned Molecular Representations Ready For Prime Time? Xrv, pp. 1904.01561, 2019.
Rex Ying, Jiaxuan You, Christopher Morris, Xiang Ren, William L. Hamilton, and Jure Leskovec. Hierarchical Graph Representation Learning with Differentiable Pooling. In Advances in Neural Information Processing Systems 31 (Proceedings of NeurlPS), 2018.
A OUR FORMULATION OF RGAT
Apart from the original RGAT (Busbridge et al. 2018), we have developed similar GNN in slightly different formulation. Followings are our RGAT Tormulation:
be41, tanh (Wiconcat/_hee..
(10)
heiik Finhee, Σ
(11)
JEN,
Oi.j. = softmax (a (he,s he,5 Ack,,r,))
(12)
(bea, he Acteess) =11,0 Ar
(13)
W, F, G, A are the coefficient matrix to be tuned. l is the index of the layer up to L,, k is the index of the attention head up K, i, are the index of the nodes up N, 6, = the index the edge type up to R.
The main point is the edge type information in Eq 13 matrix of the attention similarity function, a. This means that the associations between nodes should be computed dependent on the edge type. This a natural assumption for chemical molecular have multiple bond types between nodes atoms: single-bond, bound,
The edge type Ei,y = r switches the weight
graphs. Typically triple-bond, and the aromatic ring. natural to assume that interactions between atoms are affected by the bond types among the atoms.
The main differences from the original RGAT lie in the 1q The original RGAT assumes that the weight matrix G' is also dependent the edge type (G2,k.es.)) while omit this dependency. Also, the original RGAT does not provide a self-link weight matrix FF while we do. We made these changes based on our preliminary experiments. We found our formulation is better than the original RGAT formulation in the MoleculeNet dataset, in terms of the training stability and the generalization performances.
Another difference is the choice of the attention function. In our formulation, the attention sim- ilarity measure a(-) is defined by the general attention in (Luong et al. (0055) while the original GAT (Veličković et al. 2018) and the RGAT (Busbridge et al. 2018) employed simpler concat attention.
B OUR IMPLEMENTATION OF GIN
We implement the simplest GIN: 2-layer MLP with ReLU activation for each layer and bias parameter e fixed at O. We regularize GIN with dropout (Srivastava et al., 2014), instead batch- normalization (Ioffe & Szegedy 2015). This is because the batch-normalizztioo of the Chainer- Chemistry library did not correctly treat the padded node elements in the minibatches when we conducted the experiments.
C EXPERIMENTS DETAILS: GENERAL ISSUES
C.1 GRAPH DATA REPRESENTATION
All datasets used in our experiments are taken from the We et al., 2018). Four used datasets provided in the SMILES format. A SMILES format is line notation for describing the
11

---

Under review as a conference paper at ICLR 2020
structure of chemical compounds. We decode SMILES molecular data into graph representation of the molecule. A node the graph corresponds to an atom. Each atom node is associated with the bond between
symbolic label of the atom name ("H", "C", ..). An edge in the graph corresponds atoms. Each bond edge is associated with the bond type information (single- double-....).
Given the graph, extract input feature vectors for node x; and that of supernode x'. I,, the input feature vector for the node / is a D-dimensional continuous vector, which is an embedded vector of the one-hot atom label vector with trainable linear transformation. X', The input feature vector for the supernode is a D' D'-dimensional continuous vector, which again is an embedded vector of some graph-global features with trainable linear transformation. Choices for the graph-giobal features are detailed in the following section.
The edge information is converted an adjacency matrix, d/
C.2 EXPLICIT FEATURES FOR SUPERNODE
Since the supernode does not exist in the original graph G, we have observable for the supernode. For simplicity, we propose to use an aggregation of node features, such as:
Histograms of discrete labels attached to original nodes
Averages, maximums, minimums, or medians of numerical attributes attached to original nodes
Histograms of edge types if the graph is multiple relational graph.
Number of nodes, graph diameters, modularity, and other simple statistics for graph structure.
We can augment the super feature vector x' some additional information about the graph is provided. Essentially, these simple aggregations of the feature vectors do not bring new information into the network. However we found that the graph-wise super feature input boosts the performance of the learned network model.
C.3 DATA SPLITS
In chemical datasets, totally random shuffling of samples into train/val/test subsets is not always a valid way of data splitting. Therefore MoleculeNet provides several ways of data splitting. The "random" split is the random sample shuffling that are most familiar to the machine learning commu- nity. The "scaffold" split separate samples based on the molecular wwo-dimensional structure. Since the scaffold split separates structurally different molecules into different subsets, "it offers a greater challenge for learning algorithms than the random split" (Wu et al. 2018). Throughout the paper, we adopt the scaffold split to assess the full potential of the GWM-attaching GNNs.
The actual construction of the scaffold split train/validation/esst subsets has a freedom of algorithm choices. We basically adopted the algorithm provided by the deepchen@libraa which is the standard split algorithm for many papers. However, for the experiment of train/test loss comparison, we adopted the algorithm provided by the Chainer Chemistry library.
C.4 READOUT LAYER
In many applications of GNNs users may expect single fixed-length vector representing the characteristics of the graph G. So we add the 'readout' layer to aggregate the original node hidden states {Hę) and the global node hidden states {ge}-
A main issue in the readout unit is how to aggregate the original nodes, whose number varies for each graph. A simple way take an arithmetic average (sum) of the hs the L-th layer. but we can also use a DNN to compute (non-linear) "average" of hss (Li et al. 20116] Gilmer et al. 2017). After the aggregation of the node hidden states, we simply concatenate with gs and apply some transformations to achieve the readout vector, r:
1 1 (concat DNN,22 (Hz).gt.)
(14)
hhtps://deepchem.ia
12

---

Under review as a conference paper at ICLR 2020
In the above equation, DNN,1 is multi-layer perceptron fully connected layer to mix the concatenated hidden vectors. We adopted a simple fully-connen culny-coneetted layer for DNN,1 in this paper. D is specific readout unit accompanied with original GNN to aggregate variable-length H,..
C.5
OPTIMIZER
All models were trained with Adam (Kingma & Ba 2015 a 0.001, 31 = 0.9, and B2 = 0.999.
C.6 ABLATION MODELS FORMULATION
Here we detail the formulation of the ablation models used in the main comparison experiments (Sec. 4.4).
As written in the main manuscript, we formulate the two ablation models (Table T as follows. A simple supernode model is an ablation model without attentions, gates, nor the GRUs, and it can be considered variant of (Li al., [017 Pham et al., 2017) with bidirectional communication between the supernode and the bulk, omitting all attentions and gates. First, there is no attention for the Transmitter. So the message from the main nodes the supernode is just a sum of hidden vectors, h
he
main-tsupee tanh
We
Σ
h
he
E ERD
(15)
Originally there no messages from the supemode to the main GNN in (i et 2017). but allow such a simple message in this ablation model:
super-mmain = tanh (Fege-1) E ER'.
(16)
Messages are merged by simple linear combinations, instead of gates and GRUs, following (Li et al. 2017
h Zenhhes Ze
Supeer -aiin
E R
(17)
Z 11
(8) main-tsuper super + 2 (S)
1.2 90 E
DD
(18)
We find it difficult to fully recover the supernode of (Gilmer al., 2017) since their description on the supernode is quite limited. Thus, a NoGate GWM model, which surrogates (Gilmer et al. (017). only capture the essence of their supernode: no gatings for merger, and GRU is not installed for the nodes of the main GNN. In this model, we use the same attention-basee based Transmitter unit in (2-6). We reduce the adaptive gatings in the Warp unit by a simple averaging, and omit the GRU for hggi.
h2 = Ze,1he-1 + Zr,29
super main
R
(19)
68 ( h mmiin suuper 1 ge R
(20)
he = 11 E D
(21)
ge GRU (gc-1, 92) E D
(22)
C.7 HYPERPARAMETER
We fix a part of hyperparameters throughout the experiments, which does not influence performances so much: the number of heads in all multi-head attention mechanisms K 8, and used R 4 edge types for the multi-relational mechanism in all models. Also, at every layer, we set the dimension of the supernode feature to be the same as that of the features of the nodes the main GNN.
In the next section, we list the other hyperparameters (the number of layers L,, the dimension of feature vectors ( D')) used in several experiments/figures, as well other experinenta/immleennttiiii details.
C.8 COMPUTATIONAL ENVIRONMENT
We use a single GPU (mainly nvidia Tesla V100) for an experimental run. run roughly takes I hour to day, depending on the hyperparameters and the GNN models.
13

---

Under review as a conference paper at ICLR 2020
0.20
Training Loss
GGNN
GGNN-GWM
GIN GIN-GWM
0.10
RSGON - RSGCN-GWM
NFP NFP-GWM
0.00
5 6 10
10
100
1000
Layers
Dimensions
Figure 4: Training losses of various GNN models molecule graph dataset (Tox21). The horizontal axis denotes the number of GNN layers (the left panel) or the dimension of the node feature vectors (the right panel). Color denotes the GNN model. Thinner dashed lines are the losses of the vanilla GNNs, while thicker solid lines show the losses of the GNNs attached with the proposed Graph Warp Module ("GWM"). Scores partially unavailable due to memory shortages.
D EXPERIMENTS DETAILS: FOR EACH EXPERIMENT
In this section, we report details for each experiment, including the chosen hyperparameters and additional results.
D.I TRAINING LOSS BEHAVIORS OVER NUMBER LAYERS AND EMBEDDING DIMENSIONS
We used the Tox21 dataset to confirm behaviors of training losses, mentioned in the first paragraph of the introduction (Fig. 4
To study the effect of the number of layers L (the left panel), we fixed the dimension D = 32. To study the effect of the feature vector dimensions D (the right panel), we fixed the number layers L= 4.
All models are trained for 30 epochs.
Thinner dashed lines in Fig. plott the training losses of original networks (w/o GWM). Thick solid lines in Fig. plot the training losses of networks augmented with GWM. In general, GWM has an effect of decreasing the training loss for most choices of numbers of layers and dimensions of the node feature vectors, for all GNNs. In general, similar results are obtained from the other three datasets we omit these figures.
D.2 SECTION 4.3: TRAIN AND TEST LOss REDUCTION
In the experiment in the Section 4.3 (Fig. 1), we align the hyperparameters including L, D among vanilla GNN and its GWM-installed counterpart, compare the loss function values. We used four datasets. For each dataset, we fixed L and D for all GNN models to compare the loss reduction performances.
In Fig. L = 3 and D = 50 for all GNNs and datasets. In preliminary experiments, we manually changed I.s (E (2. 3. 4)) and Ds (E 32, 50, 100, 150) in some extent, but found the overall tendency of the scatter plots does not dramatically change. This is partially understood from the Figure I in main manuscript: the loss curves of the vanilla GNNs and their GWM-augmented counterparts evolve in roughly parallel. This implies the ratios of loss reductions are not much dependent on the hyperparameter choices.
Here show the results of other hyperparameter settings. All cases observe the similar plot patterns.
Fig. 5 is the scatter plot of the (Ftrain Floss), L = 3 and D 32. In this case, two HIV dataset plus one case for QM9 reported the increase the training loss. For other 13 pairs, the GWM successfully reduce the training losses. It is remarkable that all 16 plots have positive test loss reduction rates: namely, the generalization performances are improved by the GWM in this choice of the hyperparameters.
14

---

Under review as a conference paper at ICLR 2020
Reduction Ratio Test Loss
1.04
0.6
0.4-
GNN
Dataset
GGNN
Tox21
0.2
GIN
+ GMP
-0.2
0.0
0.2
0.4
0.6
0.8
1.0
Reduction Ratio of Train Loss
Figure 5: Train (horizontal) and test (vertical) loss reduction ratios on various pairs of GNN models and datasets. L = 3, D 32. Each plot presents the rational train/test loss reductions induced by the GWM attachment for specific pair of (datase((symbol), GNN(color)).
Reduction Ratio of Test Loss
1.0
GNN
Dataset
NFP
HIV
0.8
GGNN
Tox21
RSGON
* Lpo
0.6
GIN
+
0.4
0.2
0.0
0.0
0.4
0.8
Reduction Ratio of Train Loss
Figure 6: Train (horizontal) and test (vertical) loss reduction ratios on various pairs of GNN models and datasets. L 4, D 100. Each plot presents the rational train/test loss reductions induced by the GWM attachment for specific pair of (dataset(symbol), GNN(color)).
Fig. is the scatter plot of the (Frrain, Ploss), L and D = 100. In this case, only two cases of the HIV datasets record the negative Firain values. For other 14 pairs, the GWM successfully reduce the training losses and 13 out of these 14 pairs have positive FrestS.
All models were trained for 30 epochs.
D.3 SECTION 4.4: THE FULL COMPARISON
For the experiments in the Section 4.3 (Table 2 and in the main manuscript). we employed the dataset and a GNN model. The
Bayesian optimization to tune and 1 for each combinations Bayesian optimization (BO) trials were conducted by the Optuna library, with 200 sampling (searches) < 512.
for each combination. Ranges of the BO search is: 2 < < 8,
Chosen Ls and Ds are presented in the Table 4
Note that GWM-attached GNNs tend to perform better with deeper architecture (i.e larger L), when the vanilla GNNs without GWM do not change its overall performance with the number of layers (Table 4b. The performance of the GWM-attached models often improve with the number of layers until the model depth reaches quite large number 8) (e.g. [QM9, GGNN+Proposed GWMJ, [HIV, GIN+Proposed GWM]). Out of 24 pairs (=4 datasets x 6 GNN models), pairs perform better with deeper architecture when deployed with GWM.
For the QM9 dataset, we trained the models for 50 epochs. For the HIV and the Tox21 dataset, we trained the models for 100 epochs. For the LIPO dataset, trained the models for 200 epochs.
15

---

Under review as a conference paper at ICLR 2020
Dataset
GNN model
NEP
WeaveNet
RGAT
GGNN
RSGCN
GIN
vanilla GNN
(4, 232)
(3, 50)
(4, 9)
(4, 32)
(5, 19)
(6. 19)
LIPO
Simple Supernode
(2, 71)
(3, 9)
(3, 8)
(4, 18)
(2,
(3. 14)
+ +NoGate GWM
(2, 65)
(2. 14)
(3, 12)
(5)
(2)
(6. 9).
Proposed GWM
(5, 23,)
(4. 15)
(5. 19)
(6, 127)
(4,
(2. 26)
vanilla GNN
(5, 86)
(3, 22)
(4, 40)
(5, 71)
(4, 100)
(4, 250)
QM9
Simple Supernode
44)
(3, 111)
(6, 34)
(5, 60)
69)
(3, 22)
+ +NoGate GWM
(4, 48)
(5, 250)
(4, 31)
(6, 50)
(4,
(4. 34)
Proposed GWM
72)
(3, 104)
(4, 156)
(8, 50)
(4.36)
(4. 23)
vanilla GNN
(6, 213)
(2, 65)
(3, 28)
(6, 29)
(7,
(2, 72)
HIV
Simple supernode
(3,
(3. 39)
(2,
(4, 54)
(4, 255)
(5, 93)
+NoGate GWM
(6)
(3,
(2, 12)
(2, 51)
(3,
(3, 60)
Proposed GWM
200)
(3, 92)
(3, 23)
135)
(3. 106)
(8. 38)
vanilla GNN
(3, 204)
(5, 90)
(3, 19)
(6, 36)
(5, 70)
(5, 103)
Tox21
Simple supernode
(5, 129)
(6, 31)
(2, 36)
(6, 136)
(5, 119)
(5, 157)
+ NoGate GWM
(2, 123)
(2, 157)
(3, 43)
(5, 79)
(4, (1)
(6, 117)
Proposed GWM
(3, 106)
(4. 19)
(3,
(7. 48)
(8)32)
(6, 102)
Table 4: Hyperparameters L and D for the experiment in Section 4.4. The format of the table cells is: (I, D).
Tables and standard deviation values in parentheses.
and are the full lists of the main comparison experiments in the main manuscript, with the
16

---

Under review as a conference paper at ICLR 2020
Table 5: MAES on the LIPO dataset and QM9 dataset. The number of layers and the dimension of the feature vectors are defined via Bayesian Optimization for each method and each dataset. Averages (standard deviations) over 10 random runs, Smaller values are better.
Table 6: POC-AUCs on the HIV datasel and Tox21 dataset, The number of layers and the dimension of feature vectors are defined via Bayesian Optimization for
.844 (.026)
.819 (.023)
.847 (.048)
.784* (.012)
14.0 (2.47)
11.5 (.755)
12.9 (1.32)
11.8 (1.21)
.729 (.020)
.729 (.025)
.739 (.012)
.755* (:009)
.740 (.007)
.763 (.006)
.766 (008)
.768* (.007)
GIN
GIN
.801 .801 (.014)
.775 (.011)
.787 (.048)
.752* (.014)
15.2 15.22 (1.05)
14.6 (1.71)
12.5 (.564)
11.9 (1.48)
.728 (.009)
.742 (.019)
.758* (.022)
.760 (.005)
.770 (006)
.759 (.007)
.769 (.006)
RSGCN
RSGCN
.746
.582 (,022)
.604 (.027)
.576 (.02))
.569 (.022)
4.92 (.145)
5.41 (.1
5.52 (.344)
4,88 (.231)
.746 (.018)
.764 (.014)
.744 (.006)
.762 (.016)
.764 (.009)
.790 (.006)
.792* (.008)
.785 (.00)
each method and each dataset. Averages (standard deviations) over 10 random runs, Larger values are better.
GGNN
GGNN
.753 (.045)
.740 (.032)
.688 (.013)
.659 (.016)
8.96 192)
9.00 (.399)
9.21 (.368)
8.39 (.219)
.707(.03))
.704 (.019)
.726 (.024)
.748 (.019)
.744 744 (.008)
.787* (.005)
.786 (.009)
.787 (.009)
RGAT
RGAT
WeaveNet
1.19 (.327)
1.01 (.208)
.721 (.225)
.688 (.105)
6.38 (.289)
5.51 (.248)
5,40 (.238)
5.90 (.248)
WeaveNet
.670 (.020)
.676 (.031)
.680 (.043)
.681 (.010)
.710 .710 (.029)
.750 (020)
.764 (.011)
.767 (.021)
.677 (.040)
.693 (.027)
.675 (.017)
.672 (.040)
66 (.231)
7.68 (.316)
6.84 (.348)
6.64 (.360)
.724 (.017)
.707 (.023)
.714 (.018)
.731 (.020)
.763 .763 (.004)
.770 (.008)
.775* (.007)
.769 (.007)
NFP
NFP
+ Simple Supernode
+NoGate GWM
Proposed GWM
Simple Supernode
+NoGate GWM
Prropssed GWM
+-Smplle supemode
+NoGate GWM
+Proposed GWM
+-imple superode
+NoGate GWM
+Proposed GWM
GNN model
vanilla GNN
vanilla GNN
GNN model
vanilla GNN
vamilla GNN
Dataset
Dataset
LIPO
QM9
HIV
Tox21
17

---

Under review as a conference paper at ICLR 2020
Finally present the detailed reports the sub-tasks of QM9 and Tox21. These datasets consists of 12 sub-tasks and so far we reported the sub-task-averagee scores. Below we report the scores changes brought by GWM-attached GNNs in Table (QMM) and Table o
(Tox21). For QM9, report the relative reductions of MAE in percentage %. For Tox21, we report the improvements of binary classification accuracy in percentage (%). Numbers of sub-tasks improved by GWM (positive value slots) are in good accordance with the score gains in Table 2 and Table
GNN/Tasks
mu
alpha
HOMO
LUMO
gap
12
zpve
CV
u0
u298
h298
g298
GGNN
-0.6
-0.9
1.0
1.6
4.6
2.9
19
0.3
-3.9
-4.8
-8.3
-4.2
WeaveNet
-2.2
-0.3
27
-35
52
6.7
107
-11
84
91
86
87
Table 7: Relative reduction of MAE (%) GWM, 12 tasks in QM9. Larger values are better.
GNN/Tasks
Task1
Task2
Task3
Task4
Task5
Taskó
RGAT
0.9
0.2
0.7
0.5
1.2
0.2
GGNN
0.0
3.7
2.1
1.1
1.5
16
GNN/Tasks
Task7
Task8
Task9
Task
Task
Task12
RGAT
-0.5
1.2
-0.0
0.2
1.9
0.4
GGNN
5.9
2.8
0.7
1.5
Table 8: Improvements of binary classification accuracy (%) by GWM, 12 tasks in Tox21. Larger values are better.
18


Reviews:
[{"review_id": "rJlciCPDtS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "= Summary\nA Graph Neural Network extension integrating a global supernode explicitly into the message passing process. Concretely, message passing along the graph edges is alternated with message passing to/from a fresh super-node. Experiments show that this improves results of a number of common GNN architectures on four datasets.\n\n= Strong/Weak Points\n+ Simple but useful extension of the existing super-node idea\n+ Experiments on a number of datasets and baseline GNN architectures, providing ample experimental evidence of the usefulness of the method.\n- Writing is overcomplicated and uses a lot of jargon (\"transmitter unit\", \"warp gate\", \"intermodule hyperspace\"). I found the text entirely impenetrable and instead simply focused on Fig. 3 + the actual equations.\n\n= Recommendation\nThis is a nice contribution of minor novelty, with empirical evidence of its usefulness. I believe the paper should be accepted to a large conference such as ICLR.\n\n= Minor Comments\n- Fig. 3: Inconsistent \"intra-module\" (top) vs. \"intra module\" (bottom) \n- Concurrent work in https://openreview.net/forum?id=B1lnbRNtwr discusses a \"sandwich\" model which alternates graph message passing with (essentially) a Transformer layer applied to all nodes. This idea seems related (in that it alternates local and global information exchange).", "extra_scores": {"experience_assessment": "I have published in this field for several years.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "Hkg526KzqB", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Graph Neural Networks is a popular architecture for the analysis of chemical molecules. The authors propose an auxiliary module that can be attached to a GNN that can boost the representation power of GNNs. The auxiliary module has three building blocks: 1. a supernode, 2. a transmitter unit and 3. a warp gate unit. The authors show through carefully designed experiments that these additions can be attached to any type of GNN and that they are successful in reducing both the training error and test error. A variety of graph regression and graph classification tasks are chosen to show the efficacy of the method.\n\nThe paper is well written and easy to follow. The modification suggested by the authors is novel and useful. Experiments are well designed. An aspect that is not clear from the paper is the ability of these models to overfit the data as you increase the representation power of the network. While the authors claim that to be one of the shortcomings of existing GNNs, it is not clear whether the proposed method solves that problem. For example, in figure 4. the training loss hardly decreases as the number of layers are increased. It would be good if the authors can share any insights on this point. \n\nOverall, I think this is a good paper that the community will benefit from.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "N/A", "review_assessment:_checking_correctness_of_experiments": "I carefully checked the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}, {"review_id": "HJeaT6S-5B", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes an auxiliary module for GNNs to boost the representation power. The new module consists of virtual supernode, attention unit, and gating unit, each of which is demonstrated useful in the experiments. The module can be applied to various types of GNNs. \n\nThis work can be seen as an improvement to previous virtual supernode based methods. Adding the attention units and gating units is rational, and the effectiveness is also proved in the ablation studies. However, the claimed contribution of improving the representation power may mainly come from the idea of supernodes (instead of the attention and gating). This largely reduces the novelty of this paper and make it incremental, because using virtual supernodes is not this paper’s original idea.\n\nThe paper is generally well written. However, the comparison with previous supernode based models is not described clearly enough. The authors listed the difference from (Glimer et al. 2017) and (Li et al. 2017) in Table 1, but ignored (Pham et al. 2017) and (Battaglia et al. 2018), which were also cited in the related work. Moreover, (Li et al. 2017)’s method is actually different from the simple supernode baseline, in that it is not a bidirectional message passing between supernode and the main network. Table 1 does not contain this property.\n", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I did not assess the derivations or theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents an auxiliary module to boost the representation power of GNNs. The new module consists of virtual supernode, attention unit, and warp gate unit. The usefulness of each component is shown in well-organized experiments.
This is the very borderline paper with split scores. While all reviewers basically agree that the empirical findings in the paper are interesting and could be valuable to the community, one reviewer raised concern regarding the incremental novelty of the method, which is also understood by other reviewers. The impression was not changed through authors’ response and reviewer discussion, and there is no strong opinion to champion the paper. Therefore, I’d like to recommend rejection this time. 

Decision: Reject