Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: This paper introduces a new type of Graph Neural Network (GNN) that incorporates Feature-wise Linear Modulation (FiLM) layers. Current GNNs update the target representations by aggregating information from neighbouring nodes without taking into the account the target node representation. As graph networks might benefit from such target-source interactions, the current work proposes to use FiLM layers to let the target node modulate the source node representations. The authors thoroughly evaluate this new architecture — called GNN-FiLM—-on several graph benchmarks, including Citeseer, PPI, QM9, and VarMisuse. The proposed network outperforms the other methods on QM9 and is on par on the other benchmarks. 

Strengths
- The literature review of existing work on GNN was a pleasure to read and provided a good motivation for the proposed GNN-FiLM architecture.
- The authors put significant effort into reproducing other GNN baselines. Perhaps the most surprising result of this work is that all GNNs perform remarkably similar (contrary to what previous work has reported)

Weaknesses
- There seems to be a much tighter relationship between GNN-FiLM and Gated GNNs than currently discussed. If you actually write down the equations of the recurrent cell $r$ in Eq 1), you’ll notice that there are feature-wise interactions between the target node representations and the (sum of) source node representations. The paper should discuss in more depth what the exact differences are. Some visualizations would also help here. 
- Related to the previous point, I’d like to see a bit more discussion on *why* tight interactions between target and source nodes are helpful. For example, one could perhaps provide a toy example for which that’s obviously the case. 

All in all, I believe the ideas and results of this paper are promising but insufficient for publication in its current form. The paper tries to communicate two messages: 1) a model paper arguing for GNN-FiLM, 2) an unbiased evaluation of existing GNN models, showing that their performance is surprisingly similar on a number of benchmarks (with equal hyperparameter search). Both points are interesting but are not worked out sufficiently to pass the bar. For 1), I’d like to see an in-depth discussion on the benefits of target-source interactions, providing more insights into why this might be beneficial. Your current experiments report very minimal gains for your proposed network, questioning why such interactions might be necessary in the first place. For 2), I’d suggest to rewrite the paper from a slightly different angle and add more graph benchmarks if available (disclaimer: I’m not in the graph network community, so I can’t fully evaluate how significant these results are)


Typos
——-
Last paragraph of intro: “two two” - > two

*EDIT
After reading the rebuttal and revised paper, I've updated my score to "Weak Accept". The toy example and connection to GGNN are important additions which makes the paper more complete, though I believe there's room for further improvement here (see comments below). All in all, I weakly recommend to accept the paper. 
review_title: Official Blind Review #4

final_score: rating: 3: Weak Reject
review_content: This paper performs a concise experimental survey of graph neural network method, and introduce GNN-FiLM as a new approach. They reproduce all the baseline and show that GNN-FiLM is in line with SOTA models. 

The paper is pretty easy to follow, and the intuition is clearly explained. The idea is pretty simple, and it is a natural extension of FiLM to the graph setting. The results are tested on 4 different datasets, and the experimental protocol is clearly explained, and the results are convincing.) 

However, I am missing a discussion of the proposed methods. For instance, what are the FiLM clusters? What are the key errors with GNN-FiLM, GNN-MLP0? Are the models complementary (e.g., by using a mixture of experts)? Can we combine GNN-MLP0 and GNN-FiLM? 

Remarks:
 - The bibliography work is incomplete. For instance, the authors do not cite the Hypernetwork paper [0], when stating that several GNN variant exists, we expect to have some references, idem when the authors mention that [FiLM] has been very effective in several domain (GAN [1], Meta-learning [2] Sound [3], language-vision task [4]). As a hint, it is pretty rare to have empirical papers with only 16 citations. I would say that 20-25 is the bare minimum. 
 - Adding a visual sketch of the different models may provide additional intuition 
 - I would encourage the author to define the dimension of W_l, h_u, etc. clearly.
 - On a personal note, I like informal writing and paper honesty, but I would not always recommend it for submission.
 - As the authors reproduce the experiments, It would have been useful to add the original results in the table whenever it is possible
 - The paper gives a feeling that it lacks some rigor (missing ref, detected bug, mathematical formalism is light); it makes me a bit skeptic regarding some experimental conclusion. It would have been helpful to look at the code (at least to see if it can be easily parsed and analyzed.)

I think the paper is valuable for the community as it provides a simple survey of the current methods, the code is available, and the authors provide a lot of intuition. However, the overall level is slightly below the ICLR level for the following reason: incomplete bibliography, lack of complementary experiments, lack of discussion. The overall writing style is quite unusual, but It is not a negative point from my perspective. In its current state, it is a strong workshop submission but a not-good-enough paper for ICLR.

However, I am open to discussion, especially if the authors have complementary discussions and results.

PS: the reviewer is very familiar with the modulation and multi-modal literature, but he has only basic knowledge in graph networks.

 
[0] Ha, David, Andrew Dai, and Quoc V. Le. ICLR 2017.
[1] Brock, Andrew, Jeff Donahue, and Karen Simonyan. "Large scale gan training for high fidelity natural image synthesis." ICLR 2019 
[3] Jiang, X., Havaei, M., Varno, F., Chartrand, G., Chapados, N., & Matwin, S.. Learning to learn with conditional class dependencies. ICLR 2019
[3] Abdelnour, Jerome, Giampiero Salvi, and Jean Rouat. "From Visual to Acoustic Question Answering." arXiv preprint arXiv:1902.11280 (2019).
[4] Strub, F., Seurin, M., Perez, E., De Vries, H., Mary, J., Preux, P., & CourvilleOlivier Pietquin, A.. Visual reasoning with multi-hop feature modulation. ECCV 2018
review_title: Official Blind Review #2

final_score: rating: 8: Accept
review_content: The paper proposes a new Graph Neural Network (GNN) architecture that uses Feature-wise Linear Modulation (FiLM) to condition the source-to-target node message-passing based on the target node representation. In this way, GNN-FiLM aims to allow a GNN's message propagation to "focus on feature that are especially relevant for the update of the target node." The authors clearly describe prior GNN architectures, showing that the do not incorporate such forms of message propagation. The authors then describe several intuitive ways of adding such a form of message propagation, before describing why those approaches do not work in practice. Finally, the authors introduce GNN-FiLM, which is computationally reasonable and works well in practice, as evaluated according to several GNN benchmarks. The GNN-FiLM model is also quite simple and elegant, which makes me think it is likely to work on more tasks than the authors experiment on.

The paper is clear and easy to follow. The authors' description of other GNNs architectures is clear, and their own approach seems well motivated and clearly described in relation to previous GNNs.

The authors have released the code for method, where they have also reimplemented several popular GNN methods. The open-source codebase also seems to be valuable contribution for future research and reproducibility in work on GNNs.

The empirical evaluation seems thorough. GNN-FiLM works well on 3 graph tasks (PPI, QM9, VarMisuse) with different properties. To compare models, the authors conduct a search of hyperparameter ranges for each model. The authors even improve several of the baseline methods, generalizing some approaches to include different edge types and to add self-loops, as well as using better networks (adding dropout and increasing hidden dimensions). The paper even finds that one existing/obvious GNN architecture (GNN-MLP) is underrated. The paper reads like an honest analysis of existing methods, even though it also introduces its own, new method that works better.

Questions:
* Do you have any intuition about why the Eqn. 5 model is less stable than GNN-FiLM?
* On GNN-FiLM's training stability and regularization: I am interested in the negative result described in the following sentence: "Preliminary experiments on the citation network data showed results that were at best comparable to the baseline methods, but changes of a random seed led to substantial fluctuations (mirroring the problems with evaluation on these tasks reported by Shchur et al. (2018))" Do the authors have any intuition about why the results are highly dependent on the random seed? The results on other tasks seems to not have much variance. It could be interesting to try various techniques to stabilize learning on that task. A simple approach like gradient clipping might work, or there are perhaps other techniques. For example, in "TADAM: Task dependent adaptive metric for improved few-shot learning", the authors find it important to regularize FiLM parameters \gamma and \beta towards 1 and 0, respectively, which may make learning more stable here. In general, previous work seems to find it important to regularize the parameters that predict FiLM parameters, which may also fix GNN-FiLM's overfitting on VarMisuse. Exploring such approaches could make it easier for future work to use FiLM with GNNs.
* On potential related work: GNN-FiLM is a "self-conditioned" model which learns to apply feature-wise transformations based on the activations at the current layer. If I am not mistaken, the self-conditioning aspect of GNN-FiLM makes it related to the self-conditioned models described in "Feature-wise Transformations" ([Feature-wise transformations](https://distill.pub/2018/feature-wise-transformations/)) - for example, see the "Image Recognition" section (Squeeze-and-Excitation Networks and Highway Networks) and the "Natural Language Processing" section (LSTMs, gated linear units, and gated-attention reader). Do the authors see a connection with such models? If so, it would be interesting to hear these works discussed (in the rebuttal and paper) and the exact connection described.
review_title: Official Blind Review #3

Meta-review: While considerable effort went into improving the paper during the author response period, the concerns outlined by reviewer 2 remain and the aggregate score across reviewers reflects this issue. The AC recommends rejection with strong encouragement to resubmit this work to another high quality venue upon further revision to the work.
Decision: Reject