Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: 

Summary:

	The authors propose two new graph-aware normalization techniques for training Graph Neural Networks, viz., (I) adjacency normalization (GN_a), and (ii) graph normalization (GN_g) and study them along with existing (iii) Node-wise (Layer Norm) and (iv) Batch normalization. Motivated by their experimental observation that different tasks work well with different normalization techniques, they propose an attention component that learns to weigh and combine the above four normalizations to best suit the end task. 

——
Pros:

        - Paper is well written
        - Results reported on multiple datasets and tasks
	- The proposed Graph level normalization seems useful for node classification tasks over multiple graphs.     
——
Concerns:

Major:

	- The paper lacks a clear model wise analysis of the usefulness of different normalization techniques
	- Though the paper mentions that the importance of normalization techniques varies with tasks, there seems to be no consistent model wise improvement under each task. A precise summary would help understand the pattern in the results.  
	- Need additional results on real-world datasets - Open Graph Benchmark, standard citation networks, etc.,.
	- Only the datasets in the Graph Classification/Regression task reported in Table 4 have all real-world datasets. Either of the existing Layer norm or Batch norm technique is the best or second-best normalization technique across all models on all three datasets. 
	- It is not clear whether the difference with the proposed variants is statistically significant. Thus, a significance test on the same for all results would be helpful in understanding the usefulness of the proposed variants.
	- A GNN baseline with existing Layer norm and Batch norm combined is missing.
	- It not clear why the AGN model with GN_a didn't scale for the COLAB dataset 
	- As the usefulness of GN_a and GN_g is not clear from the results, the need for the AGN model could not be appreciated. Albeit, the AGN model's importance is limited as there is no demanding need to learn the importance of a lot of components. Since AGN is not clearly the winner in most cases and as the results don't significantly improve with AGN, it might be better to test the different normalization techniques on a validation set and choose the one that works. Note that if the usefulness of GN_a and GN_g is clearly established, then the need for an attentive model is straightforward.
	
Detailed concerns: 

	(i) Node classification results are limited to datasets with multiple graphs. 
	No results on standard single graph benchmarks like Citation, Co-Purchase networks, Social networks, etc.. 
	Would the proposed adjacency-based normalization do better the standard Layer norm and Batch norm?
	
	(ii) For node classification on multiple graphs, if the batch size is restricted to nodes from only one graph, how useful is adjacency normalization compared to layer norm and batch norm?

	(iii) Adjacency-Normalization (GN_a) performance is poor in the real dataset 
		- From Table 2 on the SROIE dataset, the performance of GNN with GN_a drops ~8% points compared to the GNN with no Norm.

	(iv) Even on the artificial datasets, the performance gain with GN_a is not consistent 
		- The GN_a + GCN model with 16 layers sees a drop of ~6% points 

	(v) The baseline numbers are different than what is reported in Dwivedi et al. 
		- 16 layers GCN+BN performance is lower than what is reported in Dwivedi et al. 
		- While there is performance gain on 
	
review_title: Good observation, extensive experiments but results are not conclusive 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper provides a unified view over different layer/batch normalization mechanisms for graph neural networks and formalizes an attention-driven approach to adaptively integrate the different forms of normalizations. An extensive empirical analysis is provided, spanning graph classification, link prediction and node classification tasks.

On the positive side, the paper is very well written and organised. Extremely clear while remaining concise and focused on the point. The idea of integrating several normalization mechanisms through an attention layer makes perfect sense. Also, I appreciate the attempt to provide an articulated empirical analysis spanning graphs and tasks of different nature.

On the negative side, the first issue is that of limited novelty. The original contributions of the paper seem to be: (i) graph-wise normalization and (ii) the attention-based integration mechanism. The former contribution seems to provide some form of empirical advantage (more on this later) but it is quite derivative and does not seem to provide any substantial novel insight on the quality of the learned representations nor on the representational power. The second contribution (AGN) has some additional novelty, but still marginal, as it is a straightforward application of standard attention. More importantly, the claim of AGN being an effective method to integrate contributions from different normalization mechanism is not supported by the empirical results.

Which brings me to the second key issue in the paper: the empirical results do not seem to strongly support the claims in the paper. First, when considering the predictive performance, one can see that the radically different normalization mechanisms have different performances (on the same dataset) only for   GCN and GAT. When considering GatedGCN (for the same number of layers), the differences between the attention mechanisms are minor. Hence it seems to suggest that the neural architecture is far more impacting than the choice of the attention mechanism. In this respect, the choice of GatedGCN to show the attention weight results in Figure 2 seems somehow driven to confirm the hypothesised “stability” of weight distributions. I would have liked to see the same analysis performed on the rather less stable GCN and GAT.

Additional issues as concerns the empirical analysis are:
-	Only training/test results are reported: where is the validation set? If no validation set is used (I hope not), how are model selection and training convergence choices taken in a robust way? Figure 4 and 5 show that different methods are stopped at different epochs: so how is this choice taken without looking into the test?
-	Given the number of configurations and datasets considered it is difficult to gain a high-level insight into which approach is truly performing best. The paper should include some sort of summarised statistics, like the average rank of each approach across the different tasks. This way one can also run some tests of statistical significance to tell which ranks are statistically different. 
-	It is unclear to me why GN_a fails with out of memory on COLLAB while other methods don’t. Where is the added memory complexity of GN_a that I am missing?

I have one final concern as regards the definition of adjacency-wise normalization. Why is it using a feature-wise statistic (i.e. summing on the embedding features) instead of computing the actual first order (and second order) statistics w.r.t to the 1-hop neighbourhood of a node? While feature-wise statistics seems the only choice for node-wise normalization, I do not see it as a necessary case for the adjacency based one. There should be more discussion on this choice in the paper.
review_title: Attention-based integration of different batch/layer normalization approaches in graph neural networks with an articulated empirical benchmarking

final_score: rating: 3: Clear rejection
review_content: ===== Summary =====

This paper proposes and evaluates different normalization techniques for graph neural networks. Also, the authors argue that the best normalization technique is task dependent, so they propose to use a weighted average of different normalizations that is learned during training, called AGN. In the paper they propose 4 different normalizations some of which are structure-dependent, and compare the performance of GCN, GAT and GatedGCN with and without these normalizations, and the learned combination of all of them.

===== Pros =====
* The paper explores normalization for Graph Neural Networks, which is a relevant topic.
* In the experiments section the authors follow the benchmark proposed by Dwivedi et al. 2020, which uses standard tasks so that comparison with other methods is easier.

===== Cons =====
* The main concern about this paper is that the authors provide no justification about why the proposed normalizations are needed, or an explanation about why any of the proposed normalization techniques should perform better than the other techniques presented in this or other papers. Moreover, the authors only evaluate their proposed normalization methods without experimenting with other methods such as PairNorm or MsgNorm [2] which is also not cited.
* Node-wise normalization is normalizing the feature vector of each node in each graph independently of all the other nodes in the same graph, therefore it is not equivalent to Layer Normalization as the authors say, because Layer Normalization would normalize all nodes in the same graph together.
* Additionally, the Node-wise normalization has already been proposed as NodeNorm in [1]
* The authors claim that adjacency-wise normalization and graph-wise normalization are both structure aware. While that is true for the former, since it normalizes nodes according to its neighbours, it isn’t true for the graph-wise normalization. The reason is that as explained by the authors in the beginning of section 2 each sample in the batch is a single graph, so normalizing across the graph is the same as normalizing across the whole sample, which doesn’t use the graph structure. It looks like this case is equivalent to Layer Normalization.

==== Questions =====

* I would like the authors to explain the motivation for their proposed normalization techniques beyond the empirical results. Why should they improve the vanilla GCN, GAT and GatedGCN models?
* Specifically since it is a novel normalization technique presented in this paper, the authors should explain what is the motivation for the adjacency-wise normalization. Making an analogy to images, the proposed normalization would be the same as normalizing one unit in a convolutional layer by the value of its direct neighbours. Why is that useful for graphs?

===== Minor comments =====

Some sections of the paper are not very well written. For example the following sentence “On other hand, while multiple normalization methods are available for training GNNs and it is still hard to know in advance which normalization method is the most suitable to a specific task.” is not very clear.

It isn’t clear if AGN stands for “Automatic Graph Normalization” or “Attentive Graph Normalization” as both are used across the paper.

===== Reasons for score =====

Because of the lack of motivation and explanation of the proposed normalization methods I vote for rejection. In my opinion the paper lacks justification and explanations of the proposed methods, and it isn’t clear why they are better than other normalization approaches.

===== References =====

[1] Zhou, K., Dong, Y., Lee, W. S., Hooi, B., Xu, H., & Feng, J. (2020). Effective training strategies for deep graph neural networks. arXiv preprint arXiv:2006.07107.
[2] Li, G., Xiong, C., Thabet, A., & Ghanem, B. (2020). Deepergcn: All you need to train deeper gcns. arXiv preprint arXiv:2006.07739.


review_title: Not enough justification and explanation for the proposed normalization techniques

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary and Strength: The authors propose two graph-wise normalization ways for graph neural network based applications. Combined with other two normalization strategies, a combination framework is proposed for weight learning of four normalizations. The experimental results  demonstrate the performance improvements of the proposed framework.

Concerns:
1. The novelty of graph-wise normalization is too thin. More insights are needed. For example, the authors might want to deliver why the normalization can bring the performance improvement, and the advantages and disadvantages of different normalization ways.
2. The notations seem a little messy. It might be better if the authors use superscripts in Section 2.
3. Combining with four different normalizations in Eq. (11), higher performance is expected due to more parameters. Without insights, the engineering work of this paper hardly inspires the readers. 
4. Some time complexities should be discussed.
5. Significant tests are suggested to verify the gained improvements.
6. How to interpret Figure 2?

This paper proposed different normalization ways for graph neural networks, unfortunately fails to answer the key questions that which is the best or default one. Instead, the authors combine them together with a weighted strategy, which lowers the technical contribution. 
review_title: Lack of insights

Meta-review: Four knowledgeable referees have indicated reject mainly because of limited motivation [R1,R3,R4], limited insights on the proposed approach [R1,R2,R3,R4], and inconclusive results [R1,R2,R3,R4]. The claims of the paper could have been strengthened by e.g. discussing currently missing experimental details [R1], performing statistical significance tests [R2,R4], and including comparisons to baselines/previously introduced normalization strategies [R2,R3]. Unfortunately, there was no rebuttal. The paper is therefore rejected.
Decision: Reject