Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a Dirichlet energy-based method to alleviate the over-smoothing and over-separating problems which are commonly observed in deep GNNs. Different from existing methods that rely on techniques from CNNs or heuristic strategies, this work provides theoretical proof of the lower and upper bound of the Dirichlet energy in GNNs and proposes EGNN which enables deeper GNNs. The effectiveness of EGNN is evaluated on commonly used benchmarks.
main_review: Pros:
1.	The paper points out that the Dirichlet energy of each layer of GNN should not too large or too small.
2.	According to the theoretical findings, the paper proposes constraints on Dirichlet energy. Especially, it smartly binds the lower bound of the Dirichlet energy to avoid over-smoothing.
3.	The experimental results show the great impact of Dirichlet energy constraints for deep GNNs.

Cons:
1.	In deep graph learning, solving the Dirichlet Energy of embedding matrix is equivalent to a set of linear equations in terms of Laplacian matrix and embedding matrix. Therefore, the theoretical findings of Lemma 2 and Lemma 3 are equivalent to the theoretical findings of Theorem 3 in [1]. In [1], it also proves the upper bound of $s_max$ in terms of node size. Therefore, the theoretical contribution is not significant. 
2.	[2] also constrains the Dirichlet energy by minimizing it although it does not constrain the lower bound of the Dirichlet energy (see Eq. 5 in [2]). However, the paper does not discuss with [2].
3.	The performance improvement is insignificant compared with GCNII [27]. Furthermore, the classification accuracy of GCNII shown in Table 1 is not consistent with the results reported in the original paper [27]. For example, GCNII achieves 80.2 on Pubmed with num_layer=16 [27], while Table 1 reports 79.8. 

Questions:
1.	What is your main Theorem? There are only Lemmas in the paper.
2.	In Figure 1, why the Dirichlet energy of GCN is zero at the very beginning? It is supposed to be decreasing to zero after several layers.

[1] Oono, Kenta, and Taiji Suzuki. "Graph Neural Networks Exponentially Lose Expressive Power for Node Classification." International Conference on Learning Representations. 2019.
[2] Deng, C., et al. "GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding." The International Conference on Learning Representations (ICLR). 2020.

limitations_and_societal_impact: see above
review_title: null

final_score: rating: 7: Good paper, accept
review_content: summary: This paper proposes Dirichlet energy constrained learning as a generalizable principle to guide the training of deep GNNs through regularizing Dirichlet energy at each layer. Following this principle, a novel deep architecture called Energetic Graph Neural Networks (EGNN) is proposed, which could learn an optimal Dirichlet energy to enable effective training of deep networks. Empirical results demonstrate that EGNN can be trained to achieve better results with more layers and outperform other GNN deepening approaches. 
main_review: This paper leverages a metric of Dirichlet energy of node embeddings to provide a better understanding of the over-smoothing and over-separation issues of deep GNNs. The new Dirichlet energy constrained learning framework is well presented to explain the key constraints in training deep GNNs. Three learning strategies (orthogonal weight controlling, lower-bounded residual connection, and shifted ReLU activation) are proposed to ensure that the training of the proposed EGNN model satisfies the constrained learning. Theoretical proofs and empirical results on node classification both demonstrate the effectiveness of the proposed solution. This paper is well-written and has made solid technical contributions towards effectively training deep GNNs. Its main claims are theoretically and experimentally supported. 

I have carefully read the authors' responses -- Although there are a few potential issues raised by other reviewers, I think overall this is a pretty good paper in my batch, so I retain my recommendation.
limitations_and_societal_impact: No.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: In order to solve the over-smoothing issue for graph neural network training, this paper propose a way with lower and upper constraints in terms of Dirichlet energy at each layer.  According to their experiment results, they achieve state-of-the-art performance by using deep layers in this Energetic Graph Neural Networks (EGNN) framework.
main_review: Originality:

In this work,  lower and upper constraints in terms of Dirichlet energy at each layer are used for solving over-smoothing issue. This is a new and interesting approach. 

Quality:

 Claims are reasonably supported. The initializations satisfy the lower and upper constraints. However, learning with strong regularizations respect to initializations does not mean that the lower and upper constraints hold after the parameters are updated.

Clarity:

The writing could be improved. For example, the abbreviation "EGNN" appear in the abstract, however, it does not state that it means "Energetic Graph Neural Networks".

Significance:

The results seems promising. However, learning with strong constraints respect to the distance with initializations  could limit the ability of graph neural network. Some discussions could be done.


limitations_and_societal_impact: Limitations from previous reviewers are almost solved.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: The authors suggest a way to prevent over smoothing in many graph neural networks that use the graph Laplacian as the spatial convolution kernel. The idea is to keep the Dirichlet Energy under control throughout the network, and hence not allow the network to oversmooth. 
main_review: The paper is interesting with several theoretical results that look correct as far as I can tell. The results are nice but a bit limited to four data sets only of the same type, and the method still relies on the x^0 input term of GCNII at every layer, which looks specific to this type of data sets. More specific details are given below:

1) "Orthogonal initialization": the authors apply a diagonal initialization, probably with diagonal entries of +-1. Is that right? The writing is a bit confusing. What happens when the matrices W are not square? 

2) Eq. (7) the authors regularize W to be a scaled identity. I would not necessarily call it "orthogonal". In orthogonal regularization, I would expect something like ||W^T*W - c*I||. Why call it or even regularize it this way? Am I missing something? 

3) In continuation to the previous remark, the authors say that for three of the four data sets (e.g. cora, pubmed and Coauthors-Physics), they use a high regularization parameter gamma. It means that many W's are close to the identity. That also means that the network computes a lot of FLOPs but the weights W are not so meaningful. Is that a good network design? What happens if there are no W's at all (just W=c*I)?

4) In continuation to the previous points, you penalize the energy, not necessarily constrain it. For a low gamma, the energy can be high. Am I missing something? In your fourth experiment, you use a very small gamma. Is the energy bounded there?

5) Page 9: overwhelming residual information. This is quite obvious. By setting cmin=0.95 you more or less ignore the Laplacian and weight, and just propagate the input through the network without doing anything. I'm not so sure this is even worthy of discussing.

6) In figure 1, it looks like the legend is incorrect. "GCN" and "Lower" are flipped. Am I wrong?

7) You use gamma=20 for the three data sets and the gamma=10^-4 for the last one. That is a huge difference. Can you comment on why is that? 

8) "We implement our EGNN upon GCN". The additions in (8) look very similar to those applied in GCNII, so it looks like most of the performance boost is coming from there. Now, the beta term in GCNII (having the input x^0 in all layers) is a bit specific to these data sets (i.e. not used in other scenarios like in standard CNNs or other GNN tasks). How does the method perform without beta? (which is learnable, right?)  Also - in Fig 1 it looks like GCNII does not lose energy because of such additions - so why do we still need to penalize the energy?

9) Are the results for the semi-supervised or fully supervised node classification. I guess the former, but this is not written in the paper.


Post rebuttal:
==========
I upgrade my score to 6. 

 
limitations_and_societal_impact: No mentioning.
review_title: null

Meta-review: Congratulations! Your paper is accepted to NeurIPS 2021. 
Please incorporate the edits and corrections as discussed in the rebuttal. 
Decision: Accept (Poster)