Title: MOLECULE PROPERTY PREDICTION AND CLASSIFICA-TION WITH GRAPH HYPERNETWORKS

Abstract: Graph neural networks are currently leading the performance charts in learningbased molecule property prediction and classification. Computational chemistry has, therefore, become the a prominent testbed for generic graph neural networks, as well as for specialized message passing methods. In this work, we demonstrate that the replacement of the underlying networks with hypernetworks leads to a boost in performance, obtaining state of the art results in various benchmarks.
A major difficulty in the application of hypernetworks is their lack of stability. We tackle this by combining the current message and the first message. A recent work has tackled the training instability of hypernetworks in the context of error correcting codes, by replacing the activation function of the message passing network with a low-order Taylor approximation of it. We demonstrate that our generic solution can replace this domain-specific solution.

1. INTRODUCTION:

The field of learning-based prediction of molecule properties holds the promise of delivering accurate predictions at a fraction of the complexity that is required by the Density Functional Theory (DFT) models, while not being tied to the assumptions and approximations of this theory. This order of magnitude reduction in runtime supports not only the rapid screening of molecule banks, with important applications in medicine, manufacturing, and environmental science, but also the automatic design of new materials.

Molecules are often represented as graphs. Similarly to other application fields, such as computer vision, computational chemistry has benefited both from the development of powerful generic (graph) neural networks, as well as the development of specialized methods that are developed for the specific prediction tasks. For example, the state of the art NMP-Edge method of Jørgensen et al. ( 2018) is a sophisticated domain-specific method, with many significant algorithmic choices, which generalizes the method of Schütt et al. (2017a) and incorporates ideas from the work of Gilmer et al. (2017) and Kearnes et al. (2016) .

In this work, we propose a generic way to improve graph neural networks and demonstrate that it is able to improve molecule property prediction and classification in both specialized networks and in more generic methods that are applied to computational chemistry datasets. The value of our scheme stems from its ability to improve upon a diverse set of already optimized state of the art methods.

Our method employs hypernetworks, also known as dynamic networks. In such neural networks, the weights of at least some of the layers vary dynamically based on the input. A hypernetwork can be seen as a composite network in which one network predicts the weights of another network. In our case, both networks receive the messages that are passed in the graph neural network as inputs.

Since the weights of the message generating network in the hypernetworks change dynamically during inference, training it is a challenge. We tackle this with a specific way of incorporating the incoming messages into the hypernetwork. Instead of passing the current message, we pass a linear combination of the current message and the first message. This simple modification is enough to ensure an improvement in performance. Without it, the hypernetwork would not typically outperform the original network.

Our experiments show that the same scheme is able to improve the predictions provided by three state of the art methods: the NMP-Edge network, the Invariant Graph Network of Maron et al. (2019a) , and the Graph Isomorphism Network of Xu et al. (2019) . In addition, we evaluate our method in the domain of error correcting codes, in which a very recent contribution by Nachmani & Wolf (2019) employed hypernetworks to improve the accuracy of a message passing scheme. We are able to show that our modification of the input messages is able to replace the method used there to stabilize the network. Taking both methods together, the results further improve.

2. RELATED WORK:

Graph Networks The topic of graph neural networks has drawn considerable attention, both in the context of specific applications, such as text analysis (Socher et al., 2013) and computer vision (Johnson et al., 2018) and as a generic tool. Earlier work employed recursive neural networks (Goller & Kuchler, 1996; Gori et al., 2005; Scarselli et al., 2008; Li et al., 2016) , where the information flows once in a network that is generated based on the graph. Most current methods, including graph spectra methods (Bruna et al., 2013; Defferrard et al., 2016; Kipf & Welling, 2016) can be cast as message passing algorithms (Gilmer et al., 2017; Duvenaud et al., 2015; Li et al., 2016; Battaglia et al., 2016; Kearnes et al., 2016; Schütt et al., 2017b; Xu et al., 2019) .

The generic message passing methods employ three networks: one that pools the hidden states from the neighborhood graph vertices, another that updates the hidden states based on the aggregated representation of the neighbouring vertices, and one that reads the information from the entire graph in order to generate the final classification. In such a network, the messages are the hidden states of the nodes. In molecule prediction, conditioning the messages also on the receiving graph node and storing a hidden state for the linking edge improves performance (Kearnes et al., 2016; Gilmer et al., 2017; Jørgensen et al., 2018) .

An alternative to message passing techniques, is presented by permutation equivariant operators on the tensors that represent k-order interactions between graph nodes (Kondor et al., 2018; Maron et al., 2018; Murphy et al., 2019; Maron et al., 2019a) . We are able to demonstrate that applying our method to both message passing methods and equivariant operator methods improved performance.

Molecule property prediction While in the past, feature engineering was the main route to applying machine learning in chemistry (Rogers & Hahn, 2010; Rupp et al., 2012; Montavon et al., 2012; Hansen et al., 2015; Huang & Von Lilienfeld, 2016) , neural networks have become increasingly popular. The NMP-Edge model of Jørgensen et al. (2018) described in Sec. 3.1 is an example of a specialized model. It follows the basic architecture of Gilmer et al. (2017) , in which the messages are conditioned on the nodes across both sides of the edge and on the hidden representation of the edge. In the NMP-Edge model, however, similar to SchNet (Schütt et al., 2017a) , the message is an elementwise product of a network that encodes the sending node and a network that encodes the edge (not encoding the receiving edge directly). Also similar to SchNet, an RBF initialization and a soft-plus activation are used. Unlike SchNet, the edge embedding is being updated in time, following the Weave network proposed by Kearnes et al. (2016) .

An example of a generic graph network solution that also excels on the popular QM9 benchmark (Ramakrishnan et al., 2014; Ruddigkeit et al., 2012) is the Invariant Graph Network (IGN) of Maron et al. (2019a) , presented in Sec. 3.2.

Hypernetworks:

Dynamic layers, also known as gating layers, are layers in which the weights are determined by a separate neural network. Such networks were introduced by Klein et al. (2015) ; Riegler et al. (2015) for visual tasks that require an adaptation of the input image. More recently, the term hypernetworks was coined to refer to a composite neural network in which a network f is trained to predict the weights θ g of another network g. The shift from specific layers to entire networks was presented by Jia et al. (2016) , who employed hypernetworks for video frames and stereo views prediction. The usage of hypernetworks for recurrent neural networks was presented by Ha et al. (2016) . Krueger et al. (2017) have presented a Bayesian formulation of hypernetworks, and such networks have become prominent in meta-learning following Bertinetto et al. (2016) , who studied transfer learning between multiple few-shot learning tasks.

Since the weights of network g are generated instantaneously by network f , Brock et al. (2018) have used hypernetworks for searching over the space of possible network architectures. In this case, a lengthy backpropagation optimization is replaced by the feed forward prediction of network f . More related to our work is that of Zhang et al. (2019) , who use hypernetworks on graphs, also in the domain of network architecture search. In this work, the weight generating network f is a graph network that operates on the graph that captures the generated architecture.

Another recent application of hypernetworks to graphs is the work of Nachmani & Wolf (2019) , where an MLP generates the weights of a message passing network that decodes error correcting codes. This generalizes earlier attempts in the domain of network decoders including (Nachmani et al., 2016; Kim et al., 2018; Gruber et al., 2017; Teng et al., 2018; Cammerer et al., 2017; Vasić et al., 2018) and is shown to improve performance. The input to both the weight generating network f and the message generation network g is the incoming message, where for the first network, the absolute value is used. It is shown that training hypernetworks suffers from severe initialization challenges and would often lead to the explosion of the weights. Nachmani & Wolf (2019) , therefore, present a new activation function that is more stable than the arctanh activation typically used in message passing decoders. In our work, we employ conventional activations, and do not employ the absolute value for molecule prediction. We demonstrate that a combination of the initial message (from the first iteration) with the last message is an effective way to stabalize the training of the graph hypernetwork and do not employ dedicated activation functions.

3. GRAPH HYPERNETWORKS:

We extend three leading architectures for graph neural networks that were either designed for the molecule inference task or shown to excel on it. In each case, we add a hypernetwork scheme in which the input is a linear combination of the first message passed in the network and the current message. As our experiments show, in all three cases, sizable gains in performance are obtained, in comparison to the underlying method. In addition, in order to compare ourselves with a recent hypernetwork message passing scheme, we modify the decoding method of Nachmani & Wolf (2019) .

3.1. EXTENDING THE NMP-EDGE NETWORK BY JØRGENSEN ET AL. (2018):

In order to describe how hypernetworks are applied to the NMP-Edge network, we rely on the original notation of Jørgensen et al. (2018) . Let h t v be the hidden state of a node associated with a specific atoms at iteration t, and e t vw be the hidden state representation of an edge, which denotes either a chemical link between atom or spatial proximity. The hidden states of the atoms h 0 v are initialized using a look-up table and the hidden state of the edges e 0 vw are initialized using an RBF function with multiple scales, following Jørgensen et al. ( 2018); Schütt et al. (2017a) .

The message passing scheme of the original NMP-Edge network takes the form:

EQUATION

EQUATION

where m t v are the messages aggregated at node v at time t, and M t , S t are the message and transition networks for iteration t. These networks are dynamic (vary between iterations) but are independent of the inputs. The earlier work by Schütt et al. (2017b) uses a similar set of networks which do not change between the iterations.

In our modified network, we replace the state transition function with the hypernetwork f and g as follows:

θ

t g = f c • h 0 v + (1 -c) • h t v (3) h t+1 v = h t v + g θ t g m t+1 v (4)

where c is a learned damping factor, which is clipped to be in the range [0,1] and is initialized with a uniform distribution, and the weights of network g are given by θ t g . For t = 0 , Eq. 3 becomes 

θ 0 g = f h 0 v . Note

F = m • h • B d • ... • B 2 • B 1

where, h is an invariant layer (Maron et al., 2019b) , m is a MLP, and the blocks B i are defined as:

EQUATION

where X i ∈ R n×n×a denote the input tensor to the block, denotes element-wise multiplication, the square brackets denote concatenation along the last tensor dimension, and the three MLPs m 1 , m 2 : R a → R b and m 3 : R a → R b are applied to each of the n × n elements of the input tensor individually along the third tensor dimension. The dimension of the block's output

Y i is, therefore, n × n × (b + b).

The modified IGN network has the form:

EQUATION

where H 1 , ..., H d-1 are hyper blocks. Let Y i = B i (X i ) be the input tensor to the hyper block H i .

The hyper block H i performs the following computation:

EQUATION

EQUATION

where the damping factor c is a learned parameter, initialized from the uniform [0,1] distribution and clipped to remain in this range. The input tensors for f and g are an aggregation of Y 0 and Y i with the damping factor c. Each layer in g is applied to each feature of the input tensor independently along the third dimension. As before, f and g are neural networks with the tanh activation with f having four layers and g two.

We use the same suffix networks per benchmark as Maron et al. (2019a) . For QM9 h of Eq. 6 is an invariant max pooling, which is followed by a MLP m with three layers. For the classification datasets, h is an invariant max pooling layer from every block B i output H i (Y i ) (Y i in the original work) followed by a single layer. These outputs are then summed to produce the network output.

3.3. EXTENDING THE GRAPH ISOMORPHISM NETWORK OF (XU ET AL., 2019):

We now turn to the notation used in (Xu et al., 2019) to introduce the modified GIN model. G(V, E) is the graph with node feature vector X v for vertices v ∈ V and edges E. In the graph classification problem, one is given a set of graphs with matching labels {(G 1 , y 1 ), (G 2 , y 2 ), ..., (G N , y N )}. The GNN model calculates representation vectors h v for each node v in an iterative manner. After convergence, a readout function calculates the global graph embedding h G , from which the label is predicted ȳG = M (h G ) for a MLP M . In GIN, the readout takes the form of a summation followed by concatenation:

EQUATION

)

where K is the final iteration used for prediction. The update function of the hidden node representation for iteration k is given by a MLP that is specific to this iteration:

h (k) v = MLP (k) 1 + (k) • h (k-1) v + u∈N (v) h (k-1) u (10)

where (k) is either a learned parameter or a fixed scalar, depending on the experiment.

The modified GIN model we propose modifies the update step, without changing the final readout:

θ k g = f c • h 0 v + (1 -c) • h (k-1) v + u∈N (v) h (k-1) u (11) h (k) v = g θ k g c • h 0 v + (1 -c) • h (k-1) v + u∈N (v) h (k-1) u (12)

where h 0 v is calculated from Eq. 10 with k = 0 as h

(0) v = MLP (0) 1 + (0) • x v + u∈N (v)

x u , where x v is the vector of input features of node v. c is a damping factor that is learned during training. f and g are neural networks with three and two hidden layers respectively with the tanh activation. Note that the network g changes between iterations and across nodes, depending on the input to f . However, the entire hypernetwork is fixed between the iterations.

3.4. EXTENDING THE DECODING HYPERNETWORK OF NACHMANI & WOLF (2019):

Since Nachmani & Wolf (2019) have proposed to extend an existing network using a hypernetwork, we modify their work in order to compare our way of converting a graph network to a hypernetwork with theirs. Specifically, it is reported that hypernetworks cannot train for the task of decoding error correcting codes, unless a dedicated activation function is used, since any other activation function attempted in their experiments leads to a divergence of the weights.

Nachmani & Wolf (2019) modify the belief propagation algorithm of Nachmani et al. (2016) , which is given by:

x j e = x j (c,v) =    tanh 1 2 l v + e ∈N (v)\{e} w e x j-1 e , j is odd 2arctanh e ∈N (c)\{e} x j-1 e , j is even (13)

where l v is the log likelihood ratios of the input bits, x j e is the computed edge message for the edge e = (c, v) in a Tanner graph, which is a bidirectional graph that has variable nodes v on one side and check nodes c on the other. Let H be the parity check matrix. Each variable node is indexed by an edge e = (c, v) on the Tanner graph and N (v) = {(c, v)|H(c, v) = 1}, i.e, the set of all edges in which v participates. x j-1 e is the message from the previous iteration.

The hypernetwork model of Nachmani & Wolf (2019) has the following update equations for odd j:

EQUATION

x

EQUATION

where N (v, c) is a vector that contains the elements of x j that correspond to the indices N (v) \ {(c, v)}. For even j the update equation takes the form:

x j e = x j (c,v) = 2 q m=0 1 2m + 1   e ∈N (c)\{(c,v)} x j-1 e   2m+1 (16)

where q is the degree of the Taylor approximation of arctanh.

We modified the model of Nachmani & Wolf (2019) with the following update equations. For odd j:

EQUATION

where x 0 is the output of one iteration from Eq. 13, and c is the damping factor which is learned during training.

For an even j we either use Eq. 16 (Taylor approximated arctanh), or consider the conventional arctanh activation, as in Eq. 13. The readout function is not modified.

4. EXPERIMENTS:

We evaluate our model on regression and classification for predicting molecule proprieties and for decoding linear block codes. For regression we use the Quantum Machines 9 (QM9) dataset (Ramakrishnan et al., 2014; Ruddigkeit et al., 2012) For classification we employ the four bioinformatics dataset of Yanardag & Vishwanathan (2015) , which contains protein structures or chemical compounds: MUTAG, PROTEINS, PTC and NCI1. We use the original train folds, which are also used by Maron et al. (2019a) and Xu et al. (2019) .

For decoding error correcting codes, we use the parity check metrics of Helmling et al. (2019) .

We use three classes of linear block codes: Low Density Parity Check (LDPC) codes (Gallager, 1962) , Polar codes (Arikan, 2008) and Bose-Chaudhuri-Hocquenghem (BCH) codes (Bose & Ray-Chaudhuri, 1960) .

We compare with various baseline method on top of the methods that we modify. For the QM9 and OQMD datasets we compare with V-RF (Ward et al., 2017) , SchNet (Schütt et al., 2017a) , enn-s2s (Gilmer et al., 2017) , Cormorant (Anderson et al., 2019) , Incidence (Albooyeh et al., 2019) , 123-gnn (Morris et al., 2019) and the two methods by Wu et al. (2018) : DTNN and MPNN. The baseline method for the classification datasets are WL subtree (Shervashidze et al., 2011) , DCNN (Atwood & Towsley, 2016) , PATCHY-SAN (Niepert et al., 2016) , DGCNN (Zhang et al., 2018) , AWL (Ivanov & Burnaev, 2018) , GCN (Kipf & Welling, 2016) , and GraphSAGE (Hamilton et al., 2017) .

Implementation details:

The various hyperparameters were selected based on the validation set (where vary between experiments) or set arbitrarily based on the underlying architecture (where fixed). NMP-edge network For QM9, we trained the models with the ADAM optimizer, with a learning rate set to 1e -4. The number of iterations was 4. The number of neurons in network f was 64 and the number of neurons in network g was 128. The learning rate decreases by a factor of 0.96 every 100, 000 gradient steps. The minibatch size was 32. The node embedding size was 256 to all the parameters. For OQMD, we use an ADAM optimizer, with a learning rate set to 1e -4. The learning rate decreases by a factor of 0.96 every 400, 000 gradient steps. We use a minibatch of 32 examples. The number of iterations was 3. The number of neurons per layer in network f was 64 and that number in network g was 128. Invariant graph network For QM9, we use the same configuration as Maron et al. (2019a), except for the following hyper parameters. When training one model to predict all molecule parameters f has four layers with 128 neurons, whereas g has two layers with 128 neurons. When we trained a separate model for each molecule parameter, which calls for a smaller capacity, f has four layers with 64 neurons and g has two layers with 64 neurons. For the classification datasets we trained the models with the following hyperparameters, learning rate was 5e -5 for MUTAG, PTC and NCI1, and was 1e -3 for PROTEINS. The number of channels in blocks B i was 400 for all datasets except for PROTEINS which has 128 channels. The number of layers in the MLP m was 2 for all datasets. For all datasets the number of neurons in network f was 64 and the number of neurons in network g was 64, except for PROTEINS which has 32 neurons for f and g. Graph isomorphism network For the classification datasets, we use the same training procedure as Xu et al. (2019) , who train the model for 10 folds and choose the number of epochs based on the cross validation accuracy over the folds. We train the models with the following hyper-parameters. Learning rate was 5e -3, the models run for 5 iterations and the number of epochs was 180 for all datasets. The minibatch sizes were 512, 256, 32 and 16 for NCI1, PTC, MUTAG and PROTEINS, respectively. In all datasets, f has three layers and 64 neurons, g has two layers with 64 neurons each. Decoding hyper-network We use the same hyper-parameters as in Nachmani & Wolf (2019) .

4.1. RESULTS:



Graph regression:

The results for the QM9 dataset are reported in Tab. 1 for the NMP-Edge compatible splits and units and in Tab. 3 for the benchmark version used by IGN. Our model based on the NMP-Edge architecture achieves state of the art performance on 9 out of 12 parameters, and in only one parameter it is outperformed by the original NMP-Edge model (and another tie).

The result for formation energy predictions OQMD, based on the NMP-Edges architecture, is provided in Tab. 2. We obtain state of the art performance in this benchmark as well. 

Graph classification:

The results for the classification datasets are provided in Tab. 4. As can be observed, our modified versions of IGN and GIN improve the baseline IGN and GIN models in almost all cases (in one case we tie). Note that the GIN model has many variants and our modification is based on the GIN-model. There is no Graph Neural Network model that outperforms our results, and for the PTC and PROTEINS datasets, our method outperforms all literature baselines.

Error Correcting Codes results:

In Fig. 1 we provide the BER-SNR results for multiple linear block codes. Our method improves on Nachmani & Wolf (2019) across codes. We get an improvement range between 0.09dB and 0.12dB for large SNR. Moreover, in all three cases, we are able to improve the baseline results, even without the Taylor approximation of Nachmani & Wolf (2019) .

Since Nachmani & Wolf (2019) fail to train without the approximation (using arctanh their runs always diverge), this shows that our method stabilizes the training process for error correcting codes.

We can also observe that our results with and without this approximation are almost identical.

4.2. ABLATION ANALYSIS:

In Tab. 5 we provide an ablation analysis on the QM9 benchmark. For the NMP-Edge model, we can observe degradation of 11 out of 12 parameters when training without h 0 v in Eq. 3 and the associated damping factor. Moreover, when training without the hypernetwork, but with h 0 v (Eq. 2 becomes

h t+1 v = c • h 0 v + (1 -c) • S t h t v , m t+1 v

) we get a degradation of 7 out of 12 parameters.

For the IGN model, we can observe degradation in 9 out of the 12 parameters when training without the damping factor and Y 0 in Eq. 7, 8. Moreover, when training without the hypernetwork but with the added first message (X i become c • X 0 + (1 -c) • X i in Eq. 5), we get a degradation of 8 out of the 12 parameters.

5. CONCLUSIONS:

Graph neural networks are becoming the dominant tool in molecule prediction and classification tasks. Here we show that by employing hypernetworks with a stabilization mechanism, significant performance gains are obtained. In order to demonstrate the advantage of our stabilizing mechanism over a recently proposed hypernetwork scheme, we also show improved performance in the field of decoding linear block codes. 

:



Figures and Tables:

Figure 1: BER for various values of SNR for various codes. (a) BCH (63,51), (b) POLAR(64,48), (c) LDPC ARRAY (121,80).

architecture employ a shifted-soft-plus network, followingSchütt et al. (2017a), while f, g employ tanh. Bias terms are not used. The number of layers is two in both g and S t . f has four layers (we believe that the architecture of S t is locally optimal and adding layers to it did not improve the accuracy in our experiments).3.2 EXTENDING THE INVARIANT GRAPH NETWORK OF MARON ET AL. (2019A)We follow the original notations of IGN, in order to enable a quick reference to the original IGN work. The original model with d blocks has the form

Table content: <table/>

and Open Quantum Materials Database (OQMD)Saal et al. (2013);Kirklin et al. (2015). The QM9 dataset has 133, 885 molecules. Each molecule has 12 properties for predicting. When comparing our results to NMP-Edge and IGN methods, we use the same train-validation-test split asJørgensen et al. (2018) orMaron et al. (2019a), respectively. While based on the same dataset, these two benchmarks cannot be directly compared for many of the properties. The OQMD dataset contain 435, 582 inorganic structures. We use the same train-validation-test split asJørgensen et al. (2018).

Table content: <table/>

Mean absolute error for QM9 molecule parameters prediction, using the NMP-Edge splits, units and our modified architecture. The lowest error is in bold. The results of the Incidence and Cormorant networks are fromAlbooyeh et al. (2019) andAnderson et al. (2019), respectively. The rest of the results fromJørgensen et al. (2018).

Table content: <table><tr><td>Target</td><td>Unit</td><td colspan="6">SchNet enn-s2s NMP-Edge Cormorant Incidence Ours</td></tr><tr><td colspan="2">ε HOM O meV</td><td>41</td><td>43</td><td>36.7</td><td>36</td><td>89</td><td>26.56</td></tr><tr><td colspan="2">ε LU M O meV</td><td>34</td><td>37</td><td>30.8</td><td>36</td><td>49</td><td>23.47</td></tr><tr><td>∆ε</td><td>meV</td><td>63</td><td>69</td><td>58.0</td><td>60</td><td>68</td><td>41.91</td></tr><tr><td>ZPVE</td><td>meV</td><td>1.7</td><td>1.5</td><td>1.49</td><td>1.982</td><td>8</td><td>1.49</td></tr><tr><td>µ</td><td>Debye</td><td>0.033</td><td>0.030</td><td>0.029</td><td>0.130</td><td>0.04</td><td>0.031</td></tr><tr><td>α</td><td>Bohr 3</td><td>0.235</td><td>0.092</td><td>0.077</td><td>0.092</td><td>0.03</td><td>0.066</td></tr><tr><td>R 2</td><td>Bohr 2</td><td>0.073</td><td>0.180</td><td>0.072</td><td>0.673</td><td>0.017</td><td>0.057</td></tr><tr><td>U 0</td><td>meV</td><td>14</td><td>19</td><td>10.5</td><td>28</td><td>8</td><td>7.27</td></tr><tr><td>U</td><td>meV</td><td>19</td><td>19</td><td>10.6</td><td>-</td><td>7</td><td>6.99</td></tr><tr><td>H</td><td>meV</td><td>14</td><td>17</td><td>11.3</td><td>-</td><td>8</td><td>7.17</td></tr><tr><td>G</td><td>meV</td><td>14</td><td>19</td><td>12.2</td><td>-</td><td>8</td><td>7.99</td></tr><tr><td>C v</td><td colspan="2">cal/molK 0.033</td><td>0.040</td><td>0.032</td><td>0.031</td><td>0.028</td><td>0.026</td></tr></table>

OQMD -NMP-Edge. Mean absolute error for formation energy predictions. The results of the various baselines are fromJørgensen et al. (2018)

Table content: <table><tr><td>Method:</td><td colspan="4">V-RF SchNet NMP-Edge Ours</td></tr><tr><td colspan="2">meV/atom 74.5</td><td>27.5</td><td>14.9</td><td>13.5</td></tr></table>

Mean absolute error for QM9 molecule parameters prediction for the IGN splits, units, and our modified architecture of it. The results of the various datasets are taken from(Maron et al.,  2019a). For IGN and our modifications, we show results of a single network predicting all values and of dedicated networks.

Table content: <table><tr><td>Target</td><td>DTNN</td><td>MPNN</td><td colspan="5">123-gnn IGN -all IGN -single Ours -all Ours -single</td></tr><tr><td>µ</td><td>0.244</td><td>0.358</td><td>0.476</td><td>0.231</td><td>0.0934</td><td>0.157</td><td>0.0883</td></tr><tr><td>α</td><td>0.95</td><td>0.89</td><td>0.27</td><td>0.382</td><td>0.318</td><td>0.325</td><td>0.303</td></tr><tr><td>homo</td><td colspan="3">0.00388 0.00541 0.00337</td><td>0.00276</td><td>0.00174</td><td>0.00203</td><td>0.00178</td></tr><tr><td>lumo</td><td colspan="3">0.00512 0.00623 0.00351</td><td>0.00287</td><td>0.0021</td><td>0.00228</td><td>0.0020</td></tr><tr><td>∆</td><td>0.0112</td><td>0.0066</td><td>0.0048</td><td>0.00406</td><td>0.0029</td><td>0.00306</td><td>0.0027</td></tr><tr><td>R 2</td><td>17</td><td>28.5</td><td>22.9</td><td>16.07</td><td>3.78</td><td>13.9</td><td>7.56</td></tr><tr><td colspan="4">ZP V E 0.00172 0.00216 0.00019</td><td>0.00064</td><td>0.000399</td><td>0.00049</td><td>0.000396</td></tr><tr><td>U 0</td><td>2.43</td><td>2.05</td><td>0.0427</td><td>0.234</td><td>0.022</td><td>0.093</td><td>0.018</td></tr><tr><td>U</td><td>2.43</td><td>2</td><td>0.111</td><td>0.234</td><td>0.0504</td><td>0.092</td><td>0.0174</td></tr><tr><td>H</td><td>2.43</td><td>2.02</td><td>0.0419</td><td>0.229</td><td>0.0294</td><td>0.093</td><td>0.0193</td></tr><tr><td>G</td><td>2.43</td><td>2.02</td><td>0.0469</td><td>0.238</td><td>0.024</td><td>0.093</td><td>0.017</td></tr><tr><td>C v</td><td>0.27</td><td>0.42</td><td>0.0944</td><td>0.184</td><td>0.144</td><td>0.180</td><td>0.146</td></tr></table>

Test set classification accuracies (%) for MUTAG, PROTEINS, PTC, NCI1 datasets. The highest results on bold. The results of IGN are by (Maron et al., 2019a), the rest of the results are from Xu et al. (2019). OURS BASED ON IGN 91.66 ± 6.54 77.8 ± 5.93 68.23 ± 10.07 81.99 ± 2.08 IGN 90.55 ± 8.7 77.2 ± 4.73 66.17 ± 6.54 83.19 ± 1.11

Table content: <table><tr><td/><td>Datasets</td><td>MUTAG</td><td>PROTEINS</td><td>PTC</td><td>NCI1</td></tr><tr><td>Datasets</td><td># graphs # classes Avg # nodes</td><td>188 2 17.9</td><td>1113 2 39.1</td><td>344 2 25.5</td><td>4110 2 29.8</td></tr><tr><td/><td>WL subtree</td><td>90.4 ± 5.7</td><td>75.0 ± 3.1</td><td>59.9 ± 4.3</td><td>86.0 ± 1.8  *</td></tr><tr><td>Baselines</td><td>DCNN PATCHYSAN DGCNN</td><td>67.0 92.6 ± 4.2  *  85.8</td><td>61.3 75.9 ± 2.8 75.5</td><td>56.6 60.0 ± 4.8 58.6</td><td>62.6 78.6 ± 1.9 74.4</td></tr><tr><td/><td>AWL</td><td>87.9 ± 9.8</td><td>-</td><td>-</td><td>-</td></tr><tr><td/><td>OURS BASED ON GIN</td><td colspan="2">90.55 ± 5.4 76.90 ± 2.24</td><td>69.68 ± 5.5</td><td>82.7 ± 2.0</td></tr><tr><td/><td>GIN-0</td><td>89.4 ± 5.6</td><td>76.2 ± 2.8</td><td>64.6 ± 7.0</td><td>82.7 ± 1.7</td></tr><tr><td>GNN variants</td><td>GIN-GIN-SUM-1-LAYER GCN-MEAN-1-LAYER GRAPHSAGE-MAX-1-LAYER</td><td>89.0 ± 6.0 90.0 ± 8.8 85.6 ± 5.8 85.1 ± 7.6</td><td>75.9 ± 3.8 76.2 ± 2.6 76.0 ± 3.2 75.9 ± 3.2</td><td>63.7 ± 8.2 63.1 ± 5.7 64.2 ± 4.3 63.9 ± 7.7</td><td>82.7 ± 1.6 82.0 ± 1.5 80.2 ± 2.0 77.7 ± 1.5</td></tr></table>

Ablation analysis on the two QM9 dataset views. Mean absolute error is reported in both. NMP-Edge split,units, and architecture IGN split, units, and architecture

Table content: <table><tr><td>Target</td><td>Full</td><td>No h 0 v</td><td>No hypernetwork</td><td>Full</td><td>No X 0</td><td>No hypernetwork</td></tr><tr><td>µ</td><td>0.031</td><td colspan="2">0.033 0.024</td><td>0.157</td><td>0.163</td><td>0.171</td></tr><tr><td>α</td><td>0.066</td><td colspan="2">0.455 0.065</td><td>0.325</td><td>0.332</td><td>0.318</td></tr><tr><td>ε HOM O</td><td>26.5</td><td>26.7</td><td>25.9</td><td>0.00203</td><td colspan="2">0.00223 0.00218</td></tr><tr><td>ε LU M O</td><td>23.4</td><td colspan="2">23.99 22.5</td><td>0.00228</td><td colspan="2">0.00229 0.00244</td></tr><tr><td>∆ε</td><td colspan="3">41.916 34.44 42.934</td><td>0.00306</td><td colspan="2">0.00311 0.00341</td></tr><tr><td>R 2</td><td>0.057</td><td colspan="2">0.183 0.197</td><td>13.9</td><td>13.6</td><td>13.8</td></tr><tr><td>ZPVE</td><td>1.49</td><td>2.16</td><td>1.51</td><td>0.00049</td><td colspan="2">0.00045 0.00045</td></tr><tr><td>U 0</td><td>7.27</td><td>7.71</td><td>9.09</td><td>0.093</td><td>0.099</td><td>0.102</td></tr><tr><td>U</td><td>6.99</td><td>7.73</td><td>9.64</td><td>0.092</td><td>0.0986</td><td>0.103</td></tr><tr><td>H</td><td>7.17</td><td>7.40</td><td>9.36</td><td>0.093</td><td>0.096</td><td>0.104</td></tr><tr><td>G</td><td>7.99</td><td>9.05</td><td>9.58</td><td>0.093</td><td>0.0997</td><td>0.102</td></tr><tr><td>C v</td><td>0.026</td><td colspan="2">0.043 0.025</td><td>0.180</td><td>0.163</td><td>0.180</td></tr></table>

Peer review (iclr2020):
~The authors propose methods to apply hypernetworks to graph neural networks. They also introduce a Taylor approximation to one model with the hypernetwork to allow for stable training.~ I think this work is a very interesting application of hypernetworks, but the contributions by the authors is limited in scope. Moreover, the paper is not clearly written and the results for improvement of the method are not conclusive. Questions about the work in decreasing importance: How many parameters do your hypernetworks have relative to the control models? How much does one need to tune the hypernetwork parameters? The tables shown on pages 8 and 9 need considerable description that should make this work self-sufficient--someone shouldn’t have to read another paper to understand your tables. Table 1 and Table 3: -What are the other Cormorant, Incidence, SchNet, and enn-s2s networks? -What are the Targets that you are predicting? Which one is challenging, or should we really care about improving? Table 4: -Almost all of your improvements are within the standard error of other models. Table 5: -Why don’t these predictions have standard errors associated with them? I am still confused on the need for the Taylor approximation to the arctanh function. This needs to be explained in much further detail. What am I supposed to glean from Figure 1? What is BCH, POLAR, and LDPC ARRAY? What is the x-axis (EbNo(dB) and y-axis (BER)? In section 3.1, what was the decision for the choice of hyperparameters and modeling decisions on page 4? Why is “c”, the learned damping factor used throughout the paper learned as a scalar value, and not as the output of the neural network that employs a sigmoid activation as an output? In the abstract and introduction, I am confused by the phrase: “We tackle this by combining the current message and the first message”, especially since the word “message” has not been described in the abstract. Please clarify. Other notes that did not impact paper review: Throughout the paper, the word “parameter” is used instead of “metric”, “statistic”, or condition, e.g. from page 6: “...the NMP-Edge architecture achieves state of the art performance on 9 out of 12 parameters, and in only one parameter it is outperformed by the original NMP-Edge model. If the hypernetwork is changing due to input molecules or proteins, it would be great to show how the weights change to different .