GNN-FiLM: Graph Neural Networks with Feature-wise Linear Modulation
Authors: Marc Brockschmidt
Venue:  (2020)
Keywords: Graph Neural Networks

Abstract: This paper presents a new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM). Many standard GNN variants propagate information along the edges of a graph by computing ``messages'' based only on the representation of the source of each edge. In GNN-FiLM, the representation of the target node of an edge is additionally used to compute a transformation that can be applied to all incoming messages, allowing feature-wise modulation of the passed information.

Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods. Hyperparameters for all methods were found using extensive search, yielding somewhat surprising results: differences between baseline models are smaller than reported in the literature. Nonetheless, GNN-FiLM outperforms baseline methods on a regression task on molecular graphs and performs competitively on other tasks.


Under review as a conference paper at ICLR 2020
GNN-FILM: GRAPH NEURAL NETWORKS WITH FEATURE-WISE LINEAR MODULATION
Anonymous authors Paper under double-blind review
ABSTRACT
This paper presents new Graph Neural Network (GNN) type using feature-wise linear modulation (FiLM). Many standard GNN variants propagate information along the edges of a graph by computing "messages" based only the represen- tation of the source of each edge. In GNN-FiLM, the representation the target node of edge is additionally used to compute transformation that be ap- plied to all incoming messages, allowing feature-wise modulation of the passed information.
Results of experiments comparing different GNN architectures on three tasks from the literature are presented, based on re-implementations of baseline methods. Hy- perparameters for all methods were found using extensive search, yielding some- what surprising results: differences between baseline models are smaller than re- ported the literature. Nonetheless, GNN-FiLM outperforms baseline methods regression task on molecular graphs and performs competitively on other tasks.
INTRODUCTION
Learning from graph-structured data has seen explosive growth over the last few years, as graphs are a convenient formalism to model the broad class of data that has objects (treated as vertices) with some known relationships (treated as edges). Example usages include reasoning about physical and biological systems, knowledge bases, computer programs, and relational reasoning in computer vision tasks. This graph construction is a highly complex form of feature engineering, mapping the knowledge of domain expert into graph structure which can be consumed and exploited by high-capacity neural network models.
Many neural graph learning methods can be summarised as neural message passing (Gilmer et al., 2017): nodes are initialised with some representation and then exchange information by transform- ing their current state (in practice with a single linear layer) and sending it as a message to all neighbours in the graph. At each node, messages are aggregated in some way and then used to update the associated node representation. In this setting, the message is entirely determined by the source node (and potentially the edge type) and the target node is not taken into consideration. A (partial) exception to this is the family of Graph Attention Networks (Veličković et al., 2018). where the agreement between source and target representation of edge is used to determine the weight of the message in an attention architecture. However, this weight is applied to all dimensions of the message at the same time.
A simple consequence of this observation may be to simply compute messages from the pair of source and target node state. However, the linear layer commonly used compute messages would only allow additive interactions between the representations of source and target nodes. More com- plex transformation functions are often impractical, as computation in GNN implementations dominated by the message transformation function.
However, this need for non-trivial interaction between different information sources a common problem in neural network design. A recent trend has been the use of hypernetworks (Ha et al., 2017), neural networks that compute the weights of other networks. In this setting, interaction be- tween two signal sources is achieved by using one of them the input other as input to the computed network. While an intellectually pleasing approach, it is often imprac- tical because the prediction of weights of non-trivial neural networks is computationally expensive.
hypernetwork and the

---

Under review as a conference paper at ICLR 2020
Approaches to mitigate this exist (e.g., Wu et al. (2019) handle this in natural language processing), but are often domain-specific.
A more general mitigation method is to restrict the structure of the computed network. Recently, "feature-wisc linear modulations" (FiLM) were introduced in the visual question answering do- main (Perez et al., 2017). Here, the hypernetwork is fed with encoding of question and produces an element-wise affine function that is applied to the features extracted from a picture. This can be adapted the graph message passing domain by using the representation of the target node com- pute the affine function. This compromise between expressiveness and computational feasibility has been very effective in some domains and the results presented in this article indicate that it is also a good fit for the graph domain.
This article explores the use of hypernetworks in learning on graphs. Sect. first reviews existing GNN models from the related work to identify commonalities and differences. This involves gen- eralising number of existing formalisms to new formulations that are able handle graphs with different types of edges, which are often used to model different relationship between vertices. Then, two new formalisms are introduced: Relational Graph Dynamic Convolutional Networks (RGDCN), which dynamically compute the neural message passing function linear layer, and Graph Neural Networks with Feature-wise Linear Modulation (GNN-FiLM), which combine learned message passing functions with dynamically computed element- wise affine transformations. In Sect. 3, range of baselines are compared in extensive experiments on three tasks from the literature, spanning classification, regression and ranking tasks small and large graphs. Experiments were performed on re-implementations of existing model architectures in the same framework and hyperparameter setting searches were performed with the same computational budgets across all architectures. The results show that differences between baselines are smaller than the literature suggests and that the new FiLM model performs well on number of interesting tasks.
2 MODEL.
Notation. Let L be finite (usually small) set of edge types. Then, a directed graph G = (V,E) has nodes V and typed edges & C node of type f, usually written as 4 c.
V, where (u, E, v) E E denotes an edge from node u to
Graph Neural Networks. As discussed above, Graph Neural Networks operate by propagating information along the edges of a given graph. Concretely, each node is associated with initial representation h (for example obtained from the label of that node, or by some other model com- ponent). Then, a GNN layer updates the node representations using the node representations of its neighbours in the graph, yielding representations h.") This process can be unrolled through time by repeatedly applying the same update function, yielding representations h .. Alterna- tively, several GNN layers can be stacked, which is intuitively similar to unrolling through time, but increases the GNN capacity by using different parameters for each timestep.
(T)
In Gated Graph Neural Networks (GGNN) (Li et al., 2016), the update rule uses one linear layer We per edge type to compute messages and combines the aggregated messages with the current representation of a node using a recurrent unit r (e.g., GRU or LSTM cells), yielding the following definition.
((+1)) =r h (2.
Σ
(eh 0,
The learnable parameters of the model are the edge-type-dependent weights Wt and the recurrent cell parameters 6,.
In Relational Graph Convolutional Networks (R-GCN) (Schlichtkrull et al., 2018), the gated unit is replaced by a simple non-linearity a (e.g., the hyperbolic tangent).
(1+1) h
Σ
(2)
" u--UEE Co.e
Here, Cu is a normalisation factor usually set to the number of edges of type ending in v. The learnalle parameters of the model are the edge-type-dependent weights W.. It is important note
2

---

Under review as a conference paper at ICLR 2020
that in this setting, the edge type set L is assumed to contain special edge type 0 for self-loops V I v, allowing state associated with a node to be kept.
In Graph Attention Networks (GAT) (Veličković et al., 2018), new node representations are com- puted from a weighted sum of neighbouring node representations. The model can be generalised from the original definitional to support different edge types follows (we will call this R-GAT below).
6,,,v = Leaky (Weh (1 ( a, softmax u & EE
h ((+1) Σ QUEE
(av)uso Weh!
Here, ae is a learnable row vector used to weigh different feature dimensions in the computation of an attention ("relevance") score of the node representations, æly is the concatenation of vectors and y, and (a.). refers to the weight computed by the softmax for that edge. The learnable parameters of the model are the edge-type-dependee In practice, GATs usually employ several attention heads that independently implement the mech- anism above in parallel, using separate learnable parameters. The results of the different attention heads are then concatenated after each propagation round to yield the value of h (t+1)
weights We and the attention parameters oz-
More recently, Xu et al. (2019) analysed the expressiveness of different GNN types, comparing their ability to distinguish similar graphs with the Weiseeiler-Lehman (WL. graph isomorphism test. Their results show that GCNs and the GraphSAGE model Hamilton al. (2017) are strictly weaker than the WL test and hence they developed Graph Isomorphism Networks (GIN) (Xu et al., 2019), which are indeed as powerful as the WL test. While the GIN definition is limited to a single edge type, Corollary 6 of Xu et al. (2019) shows that using the definition
(++1) = (a+0)) + Σ ((N h
SIVEE
there are choices for e, o0 and f such that the node representation update is sufficient for the overall network to be as powerful as the WL test. In the setting of different edge types, the function f in the sum over neighbouring nodes needs to reflect different edge types to distinguish graphs such as U 1s 2 w and v 2 u 4 w from each other. Using different functions fe for different edge types makes it possible to unify the use of the current node representation h. with the use of neighbouring node representations by again using a fresh edge type O for self-loops v 4 v. In that setting, the factor (1 + e) can be integrated into fo- Finally, following Xu et al. (2019), y and f at subsequent layers can be "merged" into a single function which can be approximated by multilayer perceptron (MLP), yielding the final R-GIN definition
argument similar to
( Σ MLP(h WEEEE
(1+1) = h(+11)=
(9) 0.)
(4)
The learnable parameters here are the edge-specific weights O,. Note that Eq. (4) is very similar to the definition of R-GCNs (Eq. (2)). only dropping the normalisation factor cu.e and replacing linear layers by an MLP.
While many more GNN variants exist, the four formalisms above are broadly representative of general trends.
notable that all of these models, the information passed from one node to another is based on the learned weights and the representation of the source of an edge. In contrast, the representation of the targer of an edge is only updated (in the GGNN case Eq. (1)). treated as another incoming message (in the R-GCN case Eq. (2) and the R-GIN case Eq. (4)), used to weight the relevance of an edge (in the R-GAT case Eq. (3)). Sometimes unnamed GNN variants of the above are used (e.g.. by Selsam al. (2019): Paliwal et al. (2019)), replacing the linear layers to compute the messages for each edge by MLPs applied to the concatenation of the representations
Note that this is similar to the ARGAT model presented by Busbridge et al. (2019), but unlike the models studied there (and like the original GATs) uses single linear layer to compute attention scores Cu instead of simpler additive multiplicative variants.
3

---

Under review as a conference paper at ICLR 2020
of source and target nodes. In the experiments, this will be called GNN-MLP, formally defined as follows.
h h(++)) h(th) = E
MLP (1 0t)
(5)
NM-BUEE Cofe
Below, we will instantiate the MLP with a single linear layer to obtain what we call GNN-MLPO, which only differs from R-GCNs (Eq. (2)) that the message passing function applied to the concatenation of source and target state.
2.1 GRAPHI HYPERNETWORKS
Hypernttorks (i.e., neural networks computing the parameters of another neural network) (Ha al., 2017) have been successfully applied to number of different tasks; naturally raising the question if they are also applicable in the graph domain.
Intuitively, a hypernetwork corresponds to higher-order function, i.e., be viewed as func- tion computing another function. Hence, a natural idea would be to use the target a message propagation step to compute the function computing the message; essentially allowing focus on features that are especially relevant for the update of the target node representation.
Relational Graph Dynamic Convolutional Networks (RGDCN) A first attempt would be to adapt (2) to replace the learnable message transformation W₁ by the result of some learnable func- tion f that operates on the target representation:
(
h (1+1) = =
Σ LUEE
f(h 1
However, for a representation size D, f would need to produce a matrix of size D2 from D inputs. Hence, implemented simple linear layer, would have on the order of O(D3) parameters, quickly making it impractical in most contexts.
This can be somewhat mitigated by splitting the node representations h into C' "chunks" h.) of dimension K C:
WL,,, = f(h (.,,,c
(E
((+1) h
Σ
Wet.e,cchl
(6)
Isese
The number of parameters of the model can now be reduced by tying the value of some instances of f.t.c- For example, the update function for a chunk e can be computed using only the corresponding
chunk of the node representation (t()) or the same update function can be applied to all "chunks" by setting f = work parameters 0 f.e.c- This is somewhat less desirable than the related idea of Wu et al. (2019), which operates on sequences, where sharing between neighbouring elements of the sequence has an intuitive interpretation that is not applicable in the general graph setting.
0 s.e.c- The learnable parameters of the model are only the hypernet-
Graph Neural Networks with Feature-wise Linear Modulation (GNN-FiLM) In (6), the mes- sage passing layer linear transformation conditioned on the target node representation, focusing on separate chunks of the node representation at time. In the extreme case in which the dimen- sion of each chunk 1, this method coincides with the ideas of Perez et (2017), who propose to use layers of element-wise affine transformations to modulate feature maps in the visual ques- tion answering setting; there, a natural language question the input used compute the affine transformation applied to the features extracted from picture.
In the graph setting, we can use each node's s representation as an input that determines an element- wise affine transformation of incoming messages, allowing the model to dynamically up-weight
2 These are similar to R-GIN, but apply an MI.P the concatenation of source and target state for each message.
4

---

Under review as a conference paper at ICLR 2020
and down-weight features based on the information present at the target node of an edge. This yields the following update rule, using a learnable function & to compute the parameters the affine transformation.
( 18,0 = 0g.e)
h ((+1) =0
Σ UEEE
Ye o Wen
B
The learnalle parameters of the model are both the hypemetwork parameters ,g, and the weights We. In practice, implementing g as a single linear layer works well.
In the case of using single linear layer, the resulting message passing function is bilinear in source and target node representation, as the message computation is centred around h ()) c (Wh!!). This is the core difference to the (linear) interaction of source and target node representations in models that use W (
simple toy example may illustrate the usefulness of such a mechanism: assuming a graph of nodes VA and VB and edge types 1 and 2, task may involve counting the number of 1-neighbours of VA nodes and 2-neighbours of VB nodes. By setting 71, = 1, V2...
for t VA and
71.0 = 0, 72.2, 1 for ts VB. GNN-FiLM can solve this in single layer. Simpler approaches can solve this by counting A/1, A/2, B/1 and B/2 neighbours separately in one layer and then projecting the correct counter, but require more feature dimensions and layers for this. As this toy example illustrates, a core capability of GNN-FiLM is to learn to ignore graph edges based on the representation of target nodes.
Note that the featurewise modulation can also be viewed of an extension of the gating mechanism of GRU LSTM cells used in GGNNs. Concretely, the "forgetting" of memories in GRU/LSTM is similar to down-weighting messages computed for the self-loop edges and the gating of the cell input is similar to the modulation of other incoming messages. However, GGNNs apply this gating to the sum of all incoming messages (cf. Eq. (1), wheras in GNN-FiLM the modulation additionally depends on the edge type, allowing for a more fine-grained gating mechanism.
Finally, small implementation bug brought focus to the fact that applying the non-linearity after summing up messages from neighbouring nodes can make it harder to perform tasks such as counting the number of neighbours with certain feature. In experiments, applying the non-linearity before aggregation as in the following update rule improved performance.
1 (1+1) = -(E Σ a (7h. We (9) + BB..) 0:
(8)
However, this means that the magnitude of node representations is dependent on the degree of nodes in the handled graph. This can sometimes lead to instability during training, which can in turn be controlled by adding an additional layer / after message passing, which can be simple bounded nonlincarity (e.g. tanh), a fully connected layer, or layer normalisation (Ba et al., 2016), or any combination of these.
EVALUATION
3.1 GNN BENCHMARK TASKS
Due to the versatile nature of the GNN modelling formalism, many fundamentally different tasks studied in the research area and it should be noted that good results task often do not transfer over to other tasks. This is due to the widely varying requirements of different tasks, as the following summary of tasks from the literature should illustrate.
Cora/Citeseer/Pummed (Sen et al., 2008): Each task consists of a single graph of ~ 10 nodes corresponding documents and undirected (sic!) edges corresponding references. The sparse ~ 000 node features are a bag of words representation of the corresponding documents. The goal is to assign subset of nodes to small number of classes. State the art performance on these tasks is achieved with two propagation steps along graph edges.
5

---

Under review as a conference paper at ICLR 2020
PPI (Zitnik & Leskovec, 2017): A motein-proteiin interaction dataset consisting of 24 graphs of ~ 2 500 nodes corresponding to different human tissues. Each node has 50 features selected by domain experts and the goal is node-level classification, where each node may belong to several of the 121 classes. State of the art performance on this task requires three propagation steps.
QM9 property prediction (Ramakrishnan et al., 2014): 130 graphs of 8 nodes represent molecules, where nodes are heavy atoms and undirected, typed edges are bonds between these atoms, different edge types indicating single/double/ttc. bonds. The goal is to regress each graph to a number of quantum chemical properties. State of the art performance on these tasks requires at least four propagation steps.
VarMisuse (Allamanis al., 2018): 235 000 graphs of 2500 nodes each represent program fragments. where nodes are tokens in the program text and different edge types represent the program's 's abstract syntax tree, data flow between variables, etc. The goal is to select one of a set of candidate nodes per graph. State of the art performance requires at least six propagation steps.
Hence, tasks differ in the complexity of edges (from undirected and untyped directed and many- typed), the size of the considered graphs, the size of the dataset, the importance of node-level vs. graph-level representations, and the number of required propagation steps.
This article includes results on the PPI, QM9 and VarMisuse tasks. Preliminary experiments on the citation network data showed results that were at best comparable to the baseline methods, but changes of random seed led to substantial fluctuations (mirroring the problems with evaluation on these tasks reported by Shehur et al. (2018)).
3.2 IMPLEMENTATION
To allow for a wider comparison, the implementation of GNN-FiLM is accompanied by imple- mentations of a range of baseline methods. These include GGNN (Li et al., 2016) (see Eq. (1)), R-GCN (Schlichtkrull
al., 2018) (see Eq. (2)). R-GAT (Veličković et al., 2018) (see Eq. (3))
and R-GIN (Hamilton et al., 2017) (see Eq. (4)) Additionally, GNN-MLPO is a variant of R-GCN using a single linear layer to compute the edge message from both source and target state (i.e., Eq. (5) instantiated with an "MLP" without hidden layers), and GNN-MLPI is the same with a single hidden layer. The baseline methods were re-implemented in TensorFlow and individually tested to reach performance equivalent to results reported in their respective source papers. All code for the implementation of these GNNs is released on https: attps://reveal h/reeeealee edd offter/doubl lifted, together with implementations of all tasks and scripts necessary to reproduce the results reported in this paper. This includes the hyperparameter settings found by search, which are stored in tasks/ defauult t. hypers/ / and selected by default the respective tasks. The code is designed to facilitate testing new GNN types on existing tasks and easily adding new tasks, allowing for rapid evaluation of new architectures.
/bl ind/
Early on in the experiments, it became clear that the RGDCN approach (Eq. (6)) as presented is infeasible. It is extremely sensitive to the parameter initialisation and hence changes the random seed lead to wild swings in the target metrics. Hence, no experimental results are reported for it in the following. It is nonetheless included in the article (and the implementation) to show the thought process leading to GNN-FiLM, as well as to allow other researchers to build upon this. In the following, GNN-FiLM refers to the formulation of Eq. (8), which performed better than the variant of Eq. (7) across all experiments. Somewhat surprisingly, the same trick (of moving the non-linearity before the message aggregation step) did not help the other GNN types. For all models, using each layer only for single propagation step performed better than using fewer layers with several propagation steps.
In all experiments, models were trained until the target metric did not improve anymore for some additional epochs (25 for PPI and QM9, 5 for VarMisuse). The reported results on the held-out test data averaged across the results of a number of training runs, each starting from differrnt random parameter initializations.
Note that Fq. (3) and Eq. (4) define generalisations to different edge types not present in the original papers.

---

Under review as a conference paper at ICLR 2020
3.3 EXPERIMENTAL RESULTS
3.3.1 PROTEIN-PROTEI INTERACTIONS (PPI)
The models are first evaluated on the node-level classification PPI task (Zitmik & Leskovec, 2017), following the dataset split from earlier papers. Training hence used a set of 20 graphs and validation and test sets of two separate graphs each. The graphs use two edge types: the dataset- tataset-ppooided un- typed edges as well as a fresh "self-loop" edge type to allows nodes to keep state across propagation steps.
Hyperparameters for all models were selected based on results from earlier papers and a small grid search of number of author-selected hyperparameter ranges (see App. A for details). This resulted in three (R-GAT), four (GGNN, GNN-FiLM, GNN-MLPI, R-GCN), or five (GNN-MLPO, R-GIN) layers (propagation steps) and node representation size of 256 (GNN-MLPO, R-GIN) or 320 (all others). All models use dropout on the node representations before all GNN layers, with a keep ratio of 0.9. After selecting hyperparameters, all models were trained ten times with different random seeds on a NVidia V100.
Tab. 1 shows the micro-averaged F1 score the classification task on the test graphs, with standard deviations and training times in sec- onds computed over the ten runs. The results for all re-implemented models are better than the results reported by Veličković et al. (2018) for the GAT model (without edge types). A cur- sory exploration of the reasons yielded three factors. First, the generalisation to different edge types (cf. Eq. (3)) and the subsequent use of a special self-loop edge type helps R-GAT (and all other models) significantly. Second, using dropout hetween layers significantly im- proved the results. Third, the larger node representation sizes (compared to 256 used by Veličković et al. (2018)) improved the results again. However, the new GNN-FiLM improves slightly over these four baselines from the literature, while converging substantially faster than all baselines, mainly be- cause it converges in significantly fewer training steps (approx. 240 epochs compared to 400-600 epochs for the other models).
Table 1: GNN results on PPI task. GAT result taken from Veličković et al. (2018).
Model
Avg. Micro-F1
Time (s)
GAT
0.973 +-0.002
n/a
GGNN
0.990 10.001
432.6
R-GCN
0.989 #0.000
759.0
R-GAT
0.989 00.001
782.3
R-GIN
0.991 #0.001
704.8
GNN-MLPO
0.992=000000
556.9
GNN-MLPI
0.9921100001
479.2
GNN-FiLM
0.992
308.1
3.3.2 QUANTUM CHEMISTRY (QM9)
All models were additionally evaluated on graph-level regression tasks on the QM9 molecule data set (Ramakrishnan et al., 2014). considering thirteen different quantum chemical properties. The ~130k molecular graphs in the dataset were split into training, validation and test data by randomly selecting 10000 graphs for the latter two sets. Additionally, another data split without test set was used for the hyperparameter search (see below). The graphs use five edge types: the dataset- provided typed edges (single, double, triple and aromatic bonds between atoms) as well as a fresh "self-loop" edge type that allows nodes to keep state across propagation steps. The evaluation differs from the setting reported by Gilmer et al. (2017), as no additional molecular information is encoded as edge features, nor are the graphs augmented by master nodes additional edges.*
Hyperparameters for all models were found using a staged search process. First, 500 hypeparrameter configurations were sampled from an author-provin on the first three regression tasks. The top three configurations for each of these three tasks were then run all thirteen tasks and the final configuration was chosen average mean absolute error across all properties, as evaluated on the validation data of that dataset split. This process led to eight layers / propagation steps for all models but GGNN and R-GIN, which showed best performance with six layers. Furthermore, all models used residual connections connecting every second layer and GGNN, R-GCN, GNN-FiLA and GNN-MLPO additionally used layer normalisation (as in Eq. (8)).
search space (see App. A for details) and run
the with the lowest
Adding these features is straightforward, but orthogonal the comparison of different GNN variants.
7

---

Under review as a conference paper at ICLR 2020
Table 2: GNN average error rates and standard deviations on QM9 target values.
Pruperty
GGNN
R-GCN
R-GAT
R-GIN
GNN-MLPO
GNN-MLPI
GNN-FiLM
mu
3.85 20.16
3.21 40.06
2.68 20.06
2.64 10.11
2.36 10.04
2.44 #0.12
2.38 #0.13
alpha
502260.86
4.22 +0.45
4.65 00.44
4.67 #0.52
4.27 #0.36
4.63 00.54
3.75
HOMO
1.67
10.01
10.03
1442 10.01
1.25 10.04
1.22 10.07
LUMO
10.06
1.62 10.04
1.33 40.07
1.50 10.09
1.35 10.04
1.50 10.19
1.30 10.05
gap
20.000006
2.42 #0.14
2.31 +0.06
2.27 +0.09
2.04 +0.05
2.06 +0.10
1.96 +0.06
R2
35.94 +35.68
16.38 +0.49
52.39
+:2.58
15.63 +1.40
14.86
+1.02
15.81 1.42
15.59 +.338
ZPVE
17.84
17.40 43.56
14.87 42.88
12.93 11.81
12.00
11.66
14.12 11.10
11.00 10.74
8.65 #2.46
7.82 20.80
7.61
5.88 +1.01
5.55 +0.38
6.94 00664
5.43 10.96
U
9.22
8.24
6.86 +0.53
18.71 +23.36
6.20 +0.88
7.00
5.95 +0.46
H
9.35 +0.96
9.05 +1.21
7.64 +0.92
5.62 +0.81
5.96 +0.45
7.98 +0.88
5.59 +0.57
G
7.14
7.00 #1.61
6.54 +0.36
5.38 +0.75
5.09 +0.57
7.14 +0.51
5.17 11113
CV
8,86 #9.07
3.93 +0.48
4.11 +0.27
3.53 #0.37
3.38 00020
4.60 00.74
3.46 +0021
Omega
1.57 +0.53
1.02 +0.05
1.48 +0.87
1.05 +0.11
0.84 +0002
5.60 +-8.2
0.98 +0006
Table 3: Accuracy VarMisuse task. GGNN result taken from appendix of Allamanis et al.
(2018).
Model
TRAIN
VALID
SEENPROJTEST
UNSEENPROJTEST
GGNN
n/a
84.0 n/a
74.1
GGNN
87.5
$2.1
85.7 +0.5%
79.3 #1.2%
R-GCN
88.7
8..7.1.6%
87.2%
81.44223%
R-GAT
90.4+3.9%
84.2+1.0%
86.9 +0.7%
81.2 +0.9%
R-GIN
93.4+1.8%
84.2+100%%
87.1 #0.1%
81.1 +0.9%
GNN-MLPO
9..4%
83.4%
86.5 +0.2%
80.5 11.4%
GNN-MLP1 11.2%
84.44
86.9 +0.3%
40.7%
GNN-FiLM
94.3+1.0%
84.6
87.0 #0.2%
81.3 #0.9%
Each model was trained for each of the properties separately five times using different random seeds on compute nodes with NVidia P100 cards. The average results of the five runs are reported in Tab. 2, with their respective standard deviations. The results indicate that the new GNN-FiLM model outperforms the standard baselines on all tasks and the usually not considered GNN-MLP variants on the majority of tasks.
3.3.3 VARIABLE USAGE IN PROGRAMS (VARMISUSE)
Finally, the models were evaluated the VarMisuse task of Allamanis et al. (2018). This task requires to process a graph representing an abstraction of a program fragment and then select one of few candidate nodes (representing program variables) based on the representation of another node (representing the location to use a variable in). The experiments are performed using the released split of the dataset, which contains 130k training graphs, 20k validation graphs and test sets: SHEENPROTEST which contains ~ 55k graphs extracted from open source projects that also contributed data to the training and validation sets, and UNSEENPROIT'RST. which contains 30k graphs extracted from completely unseen projects.
Due to the inherent cost of training models on this dataset (Balog al. (2019) provide an in-depth performance analysis), limited hyperparameter grid search was performed, with only ~ 30 can- didate configurations for each model (see App. for details). For each model, the configuration yielding the best results on the validation data set fold was selected. This led to six layers for GGNN and R-GIN, eight layers for R-GAT and GNN-MLPO, and ten layers for the remaining models. Graph node hidden sizes were 128 for all models but GGNN and R-GAT, which performed better with 96 dimensions.
The results, shown in Tab. 3, are somewhat surprising, they indicate different ranking of model architectures as the results on PPI and QM9, with R-GCN performing best. All re-implemented
'Not that training sometimes did not converge (as visible in the large standard deviation). Removing these outliers, GGNN achieved 18.11(#1.62) and R-GAT achieved 17.66(+1.22) on R2; R-GIN has an average rate of 7. 7.04(+1.41) U, and GNN-MIPP1 result the Omega task is 1..088).
8

---

Under review as a conference paper at ICLR 2020
baselines beat the results reported by Allamanis al. (2018), who also reported that R-GCN and GGNN show very similar performance. This is in spite of a simpler implementation of the task than in the original paper, it only uses the string labels of nodes for the representation and does not use the additional type information provided in the dataset. However, the re-implementation of the task uses the insights from Cvitkovic et al. (2019), who use character CNNs encode node labels and furthermore introduce extra nodes for subtokens appearing in labels of different nodes, connecting them their sources (e.g,, nodes labelled openžul frax and closeWul frax are both connected to a fresh Wull frax node).
deeper investigation results showed that the more complex models seem to suffer from significant overfitting to the training data, as can be seen in the results for training and validation accuracy reported Tab. brief exploration of more aggressive regularisation methods (more dropout, weight decay) showed no improvement and a deeper understanding of the cause of these results remains for future work.
Furthermore, the large variance in results on the validation set (especially for R-GCN) makes it likely that the hyperparameter grid search with only one training run per configuration did not yield the best configuration for each model.
4 DISCUSSION & CONCLUSIONS
After review of existing graph neural network architectures, the idea of using hypernetwork- inspired models in the graph setting was explored. This led to two models, Graph Dynamic Convo- lutional Networks and GNNs with feature-wise linear modulation, were presented. While GDCNs seem to be impractical to train, experiments show that GNN-FiLM competitive with or improving on baseline models three tasks from the literature.
The extensive experiments also show that number of results from the literature could benefit from more substantial hyperparameter search and are often missing comparisons number of obvious baselines:
The results in Tab. 1 indicate that GAT' have no advantage over GGNNs or R-GCNs on the PPI task, which does not match the findings by Veličković et al. (2018).
The results in Tab. indicate that R-GCNs are outperforming GGNNs substantially on the VarMisuse task, contradicting the findings of Allamanis et al. (2018).
The GNN-MLP models are obvious extensions that often alluded but part of the usually considered set of baseline models. Nonetheless, experiments across all three tasks have shown that these methods outperform better-published techniques such as GGNNs, R-GCNs and GATs, without a substantial runtime penalty.
These results indicate that there is substantial value in independent reproducibility efforts and com- parisons that include "obvious" baselines, matching the experiences from other areas of machine learning as well as earlier work by Shchur et al. (2018) on reproducing experimental results for GNNs on citation network tasks.

---

Under review as a conference paper at ICLR 2020
REFERENCES
Miltiadis Allamanis, Mare Brockschmidt, and Mahmoud Khademi. Learning to represent programs with graphs. In International Conference on Learning Representations (ICLR), 2018.
Lei Jimmy Ba, Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. CoRR, abs/ abs/1607.064500 2016.
Matej Balog, Bart van Merriënboer, Subhodeep Moitra, Yujia Li, and Daniel Tarlow. Fast training of sparse graph neural networks on dense hardware. CoRR, abs/1 906.1 1786, 2019.
Dan Busbridge, Dane Sherburn, Pietro Cavallo, and Nils Hammerla. Relational graph attention networks. CoRR, abs/1904.05811 2019.
Milan Cvitkovic, Badal Singh, and Anima Anandkumar. Open vocabulary learning on source code with graph-structured cache. In International Conference Machine Learning (ICML), 2019.
Justin Gilmer, Samuel S Schoenbolz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neu- ral message passing for quantum chemistry. In International Conference on Machine Learning (ICML), 2017.
David Ha, Andrew M. Dai, and Quoc V. Le. HyperNetworks. International Conference on Learning Representations (ICLR), 2017.
William I. Hamilton, Rex Ying, and Jure Leskovec. Inductive representation learning on large graphs. Advances in Neural Information Processing Systems (NeurIPS), 2017.
Yujia Li, Daniel Tarlow, Mare Brockschmidt, and Richard Zemel. Gated graph sequence neural networks. In International Conference on Learning Representations (ICLR), 2016.
Aditya Paliwal, Sarah M. Loos, Markus N. Rabe, Kshitij Bansal, and Christian Szegedy. Graph representations for higher-order logic and theorem proving. CoRR, 000/1905.0000 2019.
Ethan Perez, Florian Strub, Harm de Vries, Vincent Dumoulin, and Aaron C. Courville. FiLM: Visual reasoning with a general conditioning layer. In AAAI Conference on Artificial Intelligence, 2017.
Raghunathan Ramakrishnan, Pavlo O. Dral, Matthias Rupp. and O. Anatole Von Lilienfeld. Quan- tum chemistry structures and properties of 134 kilo molecules. Scientific Data, 1, 2014.
Michael Schlichtkrull, Thomas N. Kipf, Peter Bloem, Rianne van den Berg, Ivan Titov, and Max Welling. Modeling relational data with graph convolutional network. In Extended Semantic Web Conference (ESWC), 2018.
Daniel Selsam, Matthew Lamm, Benedikt Bünz, Percy Liang, Leonardo de Moura, and David L. Dill. Learning SAT solver from single-bit supervision. In International Conference on Learning Representations (ICLR), 2019.
Prithviraj Sen, Galileo Namata, Mustafa Bilgic, Lise Getoor, Brian Galligher, and Tina Eliassi-Rai Collective classification in network data. Al magazine, 29, 2008.
Oleksandr Shchur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. CoRR, abs/181 1.05868. 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pictro Liò, and Yoshua Bengio. Graph Attention Networks. International Conference on Learning Representations (ICLR), 2018.
Felix Wu, Angela Fan, Alexei Baevski, Yann Dauphin, and Michael Auli. Pay less attention with lightweight and dynamic convolutions. In International Conference on Learning Representations (ICLR), 2019.
Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? In International Conference on Learning Representations (ICLR), 2019,
Marinka Zitnik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33, 2017.
10

---

Under review as a conference paper at ICLR 2020
A HYPERPARAMETER SEARCH SPACES
A.1
PPI
For all models, full grid search considering all combinations of the following parameters was performed:
h bideensize E (192, 256,320) size of per-node representations.
graph_num_layers (2,3, 4,5) number of propagation steps / layers.
graghh.eyee..hhpttttrru......ep fore propagation steps.
{0.8, 0.9, 1.03 dropout applied be-
A.2 QM9
For all models, 500 configurations were considered, sampling hyperparameter settings uniformly from the following options:
siddenssize {64, 96, 128) size of per-node representations.
graph num layers E [4, 6, 8} number of propagation steps / layers.
{0.8, 0.9, 1.0) dropout applied be-
fore propagation steps.
layer norm E {True, False) decided if layer norm is applied after each propagation step.
dense_ dense_ (1,2,32) insert fully connected layer applied node representa- tions between every dense.layers propagation steps. (32 effectively turns this off)
res connection E [1)2,32] insert a residual connection between every res connection propagation steps. (32 effectively turns this off)
graph_act sishhaactiatee wwtecttvatioo_fuuuueeoeo crciioonnnnitioooo
(relu, leaky_reeie elu, gelu, tonh] non-linearity
applied after message passing.
optimizer {RMSProp, Adam) optimizer used (with TF 1.13.1 default parameters).
1r E [0.0005, 0.001] learning rate.
cell E {RNN, GRU, LSTM) gated cell used for GGNN (only part of search space for GGNN).
num heads {4, 8, 16) number of attention heads used for R-GAT (only part of search space for R-GAT).
A.3 VARMISUSE
For all models, a full grid search considering all combinations of the following parameters was performed:
h diden size (64, 96, 128) size of per-node representations.
graph_num,layers E [6, 8, 10) number of propagation steps layers.
e {0.8, 1.0) dropout applied be-
fore propagation steps.
cell E {GRU, LSTM} gated cell used for GGNN (only part of search space for GGNN).
num_heads e {4,8) number of attention heads used for R-GAT (only part of search space for R-GAT).
11


Reviews:
[{"review_id": "S1eellpeoH", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper introduces a new type of Graph Neural Network (GNN) that incorporates Feature-wise Linear Modulation (FiLM) layers. Current GNNs update the target representations by aggregating information from neighbouring nodes without taking into the account the target node representation. As graph networks might benefit from such target-source interactions, the current work proposes to use FiLM layers to let the target node modulate the source node representations. The authors thoroughly evaluate this new architecture — called GNN-FiLM—-on several graph benchmarks, including Citeseer, PPI, QM9, and VarMisuse. The proposed network outperforms the other methods on QM9 and is on par on the other benchmarks. \n\nStrengths\n- The literature review of existing work on GNN was a pleasure to read and provided a good motivation for the proposed GNN-FiLM architecture.\n- The authors put significant effort into reproducing other GNN baselines. Perhaps the most surprising result of this work is that all GNNs perform remarkably similar (contrary to what previous work has reported)\n\nWeaknesses\n- There seems to be a much tighter relationship between GNN-FiLM and Gated GNNs than currently discussed. If you actually write down the equations of the recurrent cell $r$ in Eq 1), you’ll notice that there are feature-wise interactions between the target node representations and the (sum of) source node representations. The paper should discuss in more depth what the exact differences are. Some visualizations would also help here. \n- Related to the previous point, I’d like to see a bit more discussion on *why* tight interactions between target and source nodes are helpful. For example, one could perhaps provide a toy example for which that’s obviously the case. \n\nAll in all, I believe the ideas and results of this paper are promising but insufficient for publication in its current form. The paper tries to communicate two messages: 1) a model paper arguing for GNN-FiLM, 2) an unbiased evaluation of existing GNN models, showing that their performance is surprisingly similar on a number of benchmarks (with equal hyperparameter search). Both points are interesting but are not worked out sufficiently to pass the bar. For 1), I’d like to see an in-depth discussion on the benefits of target-source interactions, providing more insights into why this might be beneficial. Your current experiments report very minimal gains for your proposed network, questioning why such interactions might be necessary in the first place. For 2), I’d suggest to rewrite the paper from a slightly different angle and add more graph benchmarks if available (disclaimer: I’m not in the graph network community, so I can’t fully evaluate how significant these results are)\n\n\nTypos\n——-\nLast paragraph of intro: “two two” - > two\n\n*EDIT\nAfter reading the rebuttal and revised paper, I've updated my score to \"Weak Accept\". The toy example and connection to GGNN are important additions which makes the paper more complete, though I believe there's room for further improvement here (see comments below). All in all, I weakly recommend to accept the paper. ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A"}, "extra_text": {}}, {"review_id": "ByxVgoKiFB", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper performs a concise experimental survey of graph neural network method, and introduce GNN-FiLM as a new approach. They reproduce all the baseline and show that GNN-FiLM is in line with SOTA models. \n\nThe paper is pretty easy to follow, and the intuition is clearly explained. The idea is pretty simple, and it is a natural extension of FiLM to the graph setting. The results are tested on 4 different datasets, and the experimental protocol is clearly explained, and the results are convincing.) \n\nHowever, I am missing a discussion of the proposed methods. For instance, what are the FiLM clusters? What are the key errors with GNN-FiLM, GNN-MLP0? Are the models complementary (e.g., by using a mixture of experts)? Can we combine GNN-MLP0 and GNN-FiLM? \n\nRemarks:\n - The bibliography work is incomplete. For instance, the authors do not cite the Hypernetwork paper [0], when stating that several GNN variant exists, we expect to have some references, idem when the authors mention that [FiLM] has been very effective in several domain (GAN [1], Meta-learning [2] Sound [3], language-vision task [4]). As a hint, it is pretty rare to have empirical papers with only 16 citations. I would say that 20-25 is the bare minimum. \n - Adding a visual sketch of the different models may provide additional intuition \n - I would encourage the author to define the dimension of W_l, h_u, etc. clearly.\n - On a personal note, I like informal writing and paper honesty, but I would not always recommend it for submission.\n - As the authors reproduce the experiments, It would have been useful to add the original results in the table whenever it is possible\n - The paper gives a feeling that it lacks some rigor (missing ref, detected bug, mathematical formalism is light); it makes me a bit skeptic regarding some experimental conclusion. It would have been helpful to look at the code (at least to see if it can be easily parsed and analyzed.)\n\nI think the paper is valuable for the community as it provides a simple survey of the current methods, the code is available, and the authors provide a lot of intuition. However, the overall level is slightly below the ICLR level for the following reason: incomplete bibliography, lack of complementary experiments, lack of discussion. The overall writing style is quite unusual, but It is not a negative point from my perspective. In its current state, it is a strong workshop submission but a not-good-enough paper for ICLR.\n\nHowever, I am open to discussion, especially if the authors have complementary discussions and results.\n\nPS: the reviewer is very familiar with the modulation and multi-modal literature, but he has only basic knowledge in graph networks.\n\n \n[0] Ha, David, Andrew Dai, and Quoc V. Le. ICLR 2017.\n[1] Brock, Andrew, Jeff Donahue, and Karen Simonyan. \"Large scale gan training for high fidelity natural image synthesis.\" ICLR 2019 \n[3] Jiang, X., Havaei, M., Varno, F., Chartrand, G., Chapados, N., & Matwin, S.. Learning to learn with conditional class dependencies. ICLR 2019\n[3] Abdelnour, Jerome, Giampiero Salvi, and Jean Rouat. \"From Visual to Acoustic Question Answering.\" arXiv preprint arXiv:1902.11280 (2019).\n[4] Strub, F., Seurin, M., Perez, E., De Vries, H., Mary, J., Preux, P., & CourvilleOlivier Pietquin, A.. Visual reasoning with multi-hop feature modulation. ECCV 2018", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "Bkgk9p1HtS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper proposes a new Graph Neural Network (GNN) architecture that uses Feature-wise Linear Modulation (FiLM) to condition the source-to-target node message-passing based on the target node representation. In this way, GNN-FiLM aims to allow a GNN's message propagation to \"focus on feature that are especially relevant for the update of the target node.\" The authors clearly describe prior GNN architectures, showing that the do not incorporate such forms of message propagation. The authors then describe several intuitive ways of adding such a form of message propagation, before describing why those approaches do not work in practice. Finally, the authors introduce GNN-FiLM, which is computationally reasonable and works well in practice, as evaluated according to several GNN benchmarks. The GNN-FiLM model is also quite simple and elegant, which makes me think it is likely to work on more tasks than the authors experiment on.\n\nThe paper is clear and easy to follow. The authors' description of other GNNs architectures is clear, and their own approach seems well motivated and clearly described in relation to previous GNNs.\n\nThe authors have released the code for method, where they have also reimplemented several popular GNN methods. The open-source codebase also seems to be valuable contribution for future research and reproducibility in work on GNNs.\n\nThe empirical evaluation seems thorough. GNN-FiLM works well on 3 graph tasks (PPI, QM9, VarMisuse) with different properties. To compare models, the authors conduct a search of hyperparameter ranges for each model. The authors even improve several of the baseline methods, generalizing some approaches to include different edge types and to add self-loops, as well as using better networks (adding dropout and increasing hidden dimensions). The paper even finds that one existing/obvious GNN architecture (GNN-MLP) is underrated. The paper reads like an honest analysis of existing methods, even though it also introduces its own, new method that works better.\n\nQuestions:\n* Do you have any intuition about why the Eqn. 5 model is less stable than GNN-FiLM?\n* On GNN-FiLM's training stability and regularization: I am interested in the negative result described in the following sentence: \"Preliminary experiments on the citation network data showed results that were at best comparable to the baseline methods, but changes of a random seed led to substantial fluctuations (mirroring the problems with evaluation on these tasks reported by Shchur et al. (2018))\" Do the authors have any intuition about why the results are highly dependent on the random seed? The results on other tasks seems to not have much variance. It could be interesting to try various techniques to stabilize learning on that task. A simple approach like gradient clipping might work, or there are perhaps other techniques. For example, in \"TADAM: Task dependent adaptive metric for improved few-shot learning\", the authors find it important to regularize FiLM parameters \\gamma and \\beta towards 1 and 0, respectively, which may make learning more stable here. In general, previous work seems to find it important to regularize the parameters that predict FiLM parameters, which may also fix GNN-FiLM's overfitting on VarMisuse. Exploring such approaches could make it easier for future work to use FiLM with GNNs.\n* On potential related work: GNN-FiLM is a \"self-conditioned\" model which learns to apply feature-wise transformations based on the activations at the current layer. If I am not mistaken, the self-conditioning aspect of GNN-FiLM makes it related to the self-conditioned models described in \"Feature-wise Transformations\" ([Feature-wise transformations](https://distill.pub/2018/feature-wise-transformations/)) - for example, see the \"Image Recognition\" section (Squeeze-and-Excitation Networks and Highway Networks) and the \"Natural Language Processing\" section (LSTMs, gated linear units, and gated-attention reader). Do the authors see a connection with such models? If so, it would be interesting to hear these works discussed (in the rebuttal and paper) and the exact connection described.", "extra_scores": {"experience_assessment": "I have published one or two papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "N/A", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly."}, "extra_text": {}}]

Author rebuttal:

Meta-review: While considerable effort went into improving the paper during the author response period, the concerns outlined by reviewer 2 remain and the aggregate score across reviewers reflects this issue. The AC recommends rejection with strong encouragement to resubmit this work to another high quality venue upon further revision to the work.
Decision: Reject