Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 6: Weak Accept
review_content: The paper dissects the importance of two parts in GCN: 1) nonlinear neighborhood aggregation; 2) nonlinear set function by linearizing the two parts and resulting in Graph Feature Network (GFN) and Graph Linear Network (GLN). It shows empirically that GFN achieves almost the same performance while GLN is much worse, suggesting the nonlinear graph neighborhood aggregation step may be unnecessary. Extensive ablation studies are conducted to single out the effects of various factors.

The paper studies an interesting problem and sets out a good plan of experiments to verify the hypotheses. The results are interesting: merely constructing graph neighborhood features alone is enough to get comparable performance with GCN since the nonlinearity in the set function is strong enough. The experiments are designed nicely: 1) it compares with various baselines on a variety of popular benchmarks; 2) ablation studies single out the importance of different graph features, such as degree, and multi-hop averages; 3) verifying whether the good performance GFN comes from easier optimization.

The paper is also clearly written, with clean notations, and well-structured sections.

I think the experiment can be improved by comparing on larger, more complex datasets. Figure 1 seems to suggest GCN is overfitting compared to GFN due to its extra capacity--significantly better training accuracy but slightly worse test accuracy. It is usually the case that larger and more complex datasets require more sophisticated models. But the paper makes a good case for GFN in these datasets for the graph classification task.
review_title: Official Blind Review #2

final_score: rating: 6: Weak Accept
review_content: This paper presents a dissection analysis of graph neural networks by decomposing GNNs into two parts: a graph filtering function and a set function. Although this decomposition may not be unique in general, as pointed out in the paper, these two parts can help analyze the impact of each part in the GNN model. Two simplified versions of GNN is then proposed by linearizing the graph filtering function and the set function, denoted as GFN and GLN, respectively. Experimental results on benchmarks datasets for graph classification show that GFN can achieve comparable or even better performance compared to recently proposed GNNs with higher computational efficiency. This demonstrates that the current GNN models may be unnecessarily complicated and overkill on graph classification. These empirical results are pretty interesting to the research community, and can encourage other researchers to reflect on existing fancy GNN models whether it's worth having more complex and more computationally expensive models to achieve similar or even inferior performance. Overall, this paper is well-written and the contribution is clear. I would like to recommend a weak accept for this paper. If the suggestions below can be addressed in author response, I would be willing to increase the score.


Suggestions for improvement:

1) Considering the experimental results in this paper, it is possible that the existing graph classification tasks are not that difficult so that the simplified GNN variant can also achieve comparable or even better performance (easier to learn). This can be conjectured from the consistently better training performance but comparable testing performance of original GNN. Another possibility is that even the original GNN has larger model capacity, it is not able to capture more useful information from the graph structure, even on tasks that are more challenging than graph classification. However, this paper lacks such in-depth discussions;

2) Besides the graph classification task, it would be better to explore the performance of the simplified GNN on other graph learning tasks, such as node classification, and various downstream tasks using graph neural networks. This can help demystify the question raised in the previous point; 3) The matrix \tilde{A} in Equation 5 is not well explained (described as "similar to that in Kipf and Welling (2016)"). It would be more clear to directly point out that it is the adjacency matrix, as described later in the paper.
review_title: Official Blind Review #3

final_score: rating: 3: Weak Reject
review_content: This paper tries to study the importance of different components of GNNs. This paper studies two components 1) graph filtering: aggregation of neighboring features and 2) the aggregation function for the output.

To study this problem, this paper proposes two models, Graph Feature Network (GFN) and Graph Linear Network (GLN). GFN first uses the adjacency matrix to create several layers of features, then applies a multi-layer fully-connected neural network. GLN is a special case of GFN with the fully-connected neural network being linear.

This paper conducts experiments on graph classification task and finds GFN gives a reasonable performance, whereas GLN's performance is weaker.



Comments:
This paper studies an important problem in GNN, and the proposed method is interesting. However, I cannot accept the paper in the current form because of the following reasons.

1. There is no theoretical analysis in the paper. For example, on some datasets, GFN, GLN, and GNN's performances are close while on other datasets, there are gaps. The current paper does not provide insight.

2. GNN also contains non-linearity in the middle layers. However, the methodology in this paper cannot account for the importance of non-linearity in the middle layers.

3. The experiment section ignores some recent results on graph classification tasks. See:
https://arxiv.org/abs/1809.02670
https://arxiv.org/abs/1905.13192
review_title: Official Blind Review #1

Meta-review: This paper proposes to split the GNN operations into two parts and study the effects of each part. While two reviewers are positive about this paper, the other reviewer R1 has raised some concerns. During discussion, R1 responded and indicated that his/her concerns were not addressed in author rebuttal. Overall, I feel the paper is borderline and lean towards reject.
Decision: Reject