Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 1: Reject
review_content: This work empirically  study the behavior of Graph Neural Networks in various topological contexts. Four sets of experiments are provided follow the setting in [1].
Pros:
1. The research problem studied in the paper is important.
2. Authors conduct extensive experiments on multiple dataset.
Cons:
1. The paper lacks formal justifications on the raised claims. They look intuitive and post-justified by experiments and not by rigorous arguments.
2. All the experiments are conducted based on the convolutional graph neural network based methods. It is suggested to evaluate on the other types of graph neural network, e.g. Recurrent Graph Neural Networks, Graph Autoencoder.
3. The writing needs to be significantly improved.

[1] Shchur et al. "Pitfalls of Graph Neural Network Evaluation",  arXiv preprint arXiv:1811.05868 (2018)
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: The paper presents four experimental set-ups to get a better understanding how the performance of Graph Neural Networks (GNN) depend on topological and/or nodal information.

As the paper is not really in my research area, I would have liked the paper to be a bit more self-contained, but the writing of the paper is generally clear. 

The contributions of the paper are mainly experimental and show different "surprising" aspects about GNNs. I found the experimental results to be interesting, especially those presented in Section 5 about decoupling attribute and topological information. However, I think the presentation may be improved and more information (e.g. about how the experiments were conducted or more plots) should be reported in the main paper or the appendix.

Regarding the experiments, the authors made choices about which subsets of datasets (Table 2) or which subsets of measures (Table 1) to use. It would be nice to explain how those choices are made and if they are well-justified. Otherwise, it feels that the presented experimental results have been hand-picked.

Regarding Section 3, I guess that Figure 1 presents averaged results over datasets or over models. I think it would be important to share the results per datasets or per models, as averaging may hide some key aspects.

On page 3, the third paragraph could be illustrated with some plots. Besides, the its first sentence seems to contradict the last sentence of the paragraph before Section 4. Moreover, the text in the fourth paragraph doesn't seem to fit well Fig. 2b for Amazon computers. Am I misreading this plot?

Regarding Section 4, I think it would have been interesting to report the metrics about the topological features for the perturbed graphs. Are they very different from those for the orignal graphs?

Regarding Section 6, the sentence "In the previous sections, we have demonstrated that GNNs .. are robust to perturbations to it" on page 7 seems to contradict the conclusion of Section 4.
I found the experiments in this section to be less convincing. The edge addition technique seems to be ad-hoc and it is not clear why it would improve the performance or not. For instance, how was the value 6°*density-G chosen?

Overall, although I found some of the experimental results very intriguing, I think the paper may not be ready for publication in its current state.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper analyzes the properties of graph neural networks (GNNs). It shows that several hypothesis that one might intuitively make about the behavior of GNNs, do not actually hold. In fact, some observations are even contradictory, indicating that GNNs' performance is not robust, and care needs to be taken when using them. In particular, the authors analyze what happens when topology is altered by computing correlations between topology metrics and accuracy, dropping connections, adding extraneous connections, etc. The authors examine performance is a function of graph connectedness, and find a weak correlation. 

Some concerns: 
a) The part about attributes and topology is not very clear to me. What are the attributes? What's "decoupling by shuffling"? 
b) Figures could benefit from a brief "so what" explanation in the caption. 
c) While the work is important because GNNs are a rising trend, it is a bit disappointing that there is no discussion of "how do we fix GNNs" and "what's next". 
d) Topology is one important feature of graphs, but could it be examined in terms of what kinds of edges are added based on learned similarity metrics, etc? Learning the adjacency matrix is one important step in GNN methods and it would be useful to examine robustness to different ways of learning that matrix.
review_title: Official Blind Review #1

Meta-review: The paper empirically investigates the behaviour of graph neural networks, as a function of topology, structural noise, and coupling between nodal attributes and structure. While the paper is interesting, reviewers in general felt that the presentation lacked clarity and aspects of the experiments were hard to interpret. The authors are encouraged to continue with this work, accounting for reviewer comments in subsequent versions.

Decision: Reject