Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: The authors study the gradient flow of deep GCNs and derive a topology-aware isometric initialization scheme for vanilla-GCNs. Then, they propose to use Dirichlet Energy for dynamic rewiring of vanilla-GCNs with skip-connections to increase the expressive power of GCN. 

strengths_and_weaknesses: ## Strengths: 
1. The writing is easy to follow.

2. The experiments are comprehensive.


## Weaknesses: 

1. The method and results have some novelty but not significant.

2. Some plots are messy and some arguments are not well support by enough evidence.

3. Some Missing literature on the gradient flow and trainability of GNNs. [1,2]

[1] Luan S, Zhao M, Chang X W, et al. Training matters: Unlocking potentials of deeper graph convolutional neural networks[J]. arXiv preprint arXiv:2008.08838, 2020.

[2] Cong W, Ramezani M, Mahdavi M. On provable benefits of depth in training graph convolutional networks[J]. Advances in Neural Information Processing Systems, 2021, 34: 9936-9949.


questions: 1).  Line 76, “we leverage Dirichlet Energy (which measures the expressiveness of feature embedding)…” I cannot see why Dirichlet Energy measures the expressiveness of feature embedding? 

2). In equation (4), why is the gradient flow defined as the mean of the p-norm of the gradients for all the L layers.

3). What is the standard deviation of the results in table 2.

4). What is the “Dirichlet Energy guided rewiring” looks like? Write it in math.

Writing:

1). The position of Figure 1 need to be adjusted.

2). Figure 3 is messy, I suggest only keeping one set of legend and moving it outside the plot.

3). To simplify the computation, in equation (6), you can write $y_i = WX(A+I)e_i$, where $e_i$ is the column one-hot vector with 1 in I-th element and 0 for others.

limitations: The authors adequately addressed the limitations and potential negative societal impact of their work
review_title: null

final_score: rating: 5: Borderline accept: Technically solid paper where reasons to accept outweigh reasons to reject, e.g., limited evaluation. Please use sparingly.
review_content: summary: This paper tries to improve the performance of the deep GNN model by improving the gradient flow during training. Specifically, the authors propose two methods, topology-
aware isometric initialization and dynamic rewiring by Dirichlet Energy. Topology-
aware isometric initialization aims to provide a good initialization with isometric property, while dynamic rewiring is to dynamically add the skip connections based on the decrease of  Dirichlet energy of each GCN layer.

strengths_and_weaknesses: Strengths:
The idea of improving gradient flow is insightful. I agree that gradient flow is a key factor in improving the performance of Deep GNNs. 
It's easy to read and understand. I enjoyed reading this paper. 

Weaknesses:
1. I think the observations in this paper are insightful. The proposed solution generally works.  The main weakness lies in the theoretical analysis. In the isometric initialization part, the authors only give the construction of the initialization (Eq 14) without any further justification. Then, in the rewiring part, the authors directly apply the Dirichlet energy as a measure to control the generation of skip connections. It makes use of the property of Dirichlet energy but is too artificial.  Meanwhile, the relationship between Dirichlet energy and gradient flow is not well elaborated. 

2. I like the nice figures and rich empirical evaluations in this paper.  Here are my suggestions:

- Since the gradient flow is defined on gradients rather than graphs, I’m curious whether the proposed isometric initialization and dynamic rewiring can improve the gradient flow on the MLP model or not. 

- As stated in the paper, the dynamic rewiring needs to update Dirichlet energy $Tr(XLX^{T})$ for each GCN layer during training. It looks like the calculation of  $Tr(XLX^{T})$ is expensive, especially for a large graph. I want to see the impact of this dynamic rewiring in terms of training time.

 - Is isometric initialization also work for other GNN backbones, such as GAT, and APPNP?

questions: Please refer  to the Strengths And Weaknesses.
limitations: None
review_title: null

final_score: rating: 4: Borderline reject: Technically solid paper where reasons to reject, e.g., limited evaluation, outweigh reasons to accept, e.g., good evaluation. Please use sparingly.
review_content: summary: The work focuses on improving the trainability of vanilla-GCN. Starting from the initial observation of the gradient flow issue, the authors propose two techniques, namely topology-aware isometric initialization and Dirichlet energy guided rewiring to improve the optimization. Empirically, on several benchmark datasets, authors show that the propose two techniques do improve the training of vanilla-GCN, leading to a competitive model quality (though not SOTA).
strengths_and_weaknesses: The paper is written in a very logical way and the proposed methods are well motivated. The empirical evaluation is systematic and convincing.

However, it seems the trainability issue generally happens when the number of layers is large. But even when the proposed methods are used, increasing the number of layers generally does not improve the model quality at all. Hence, is it possible that the trainability issue comes from the fact that the datasets are too small and/or too noisy such that higher model capacity does not help? 

Another question I have is that would the proposed two techniques be helpful to other more complicated SOTA GNNs with e.g. attention  & normalization? If so, the proposed techniques would be more helpful in practice.

Finally, while adaptive rewiring based on Dirichlet energy is quite effective, it always uses the output feature matrix of first layer as the skip feature. Compared to the standard skip connection used in ResNet, this effectively makes the network relatively shallower, which can be seen as trading model complexity for trainability. Then, why not just use shallower networks instead then?
questions: Can the proposed two techniques improve SOTA GNNs not just GCNs?

limitations: Nothing particular.
review_title: null

final_score: rating: 8: Strong Accept: Technically strong paper, with novel ideas, excellent impact on at least one area, or high-to-excellent impact on multiple areas, with excellent evaluation, resources, and reproducibility, and no unaddressed ethical considerations.
review_content: summary: This work adds insight into why Graph Convolutional Networks (GCNs) have had trouble going deep. In particularly, it extends the intuition of "over-smoothening" and "information bottlenecks" that plague other neural nets (which were solved by gradient flow) and uses techniques to improve gradient flow for GCNs. In particular, it does better initialization (topology-aware isometric initialization) and rewiring (Dirichlet Energy Guided Dynamic Rewiring). By having a better distribution of initial weights and adding skip connections to propagate gradients into needed areas on where needed, the authors demonstrate very competitive (usually top-2 of 14) performance by enhancing a mere vanilla-GCN.
strengths_and_weaknesses: Strengths
+ This is a very well-motivated work. The discussion of gradient flow solutions for other neural networks very cleanly tees up the "obviousness" of gradient flow solutions for GCNs.
+ The experiments to measure gradient flow in GCNs (eg Figure 2) are useful and interesting.
+ Really enjoyed the simple insight applied by this paper to demonstrate big gains. Particularly impressed by the dynamic skip connections.
+ The writing for this paper was very clear (e.g. description of how GCNs work in section 2.2, underlining certain phrases for emphasis, approachable tone of describing other methods as "fancy state-of-the-art methds")

Weaknesses
- It wasn't always clear whether the demonstrated issues with deep networks were primarily from gradient flow, per se, as opposed to generally just overfitting. Deeper networks are higher-capacity, but the motivating experiments did not disentangle that.
- You reference "Appendix 1" on line 158, but this work does not have an appendix.

Suggested Tweaks
- Format-wise, the figures were a little scattered. Figure 1 is listed on page 2 but not described until page 4.
- There are a few typos (e.g. "bolltleneck" on line 32, plural complex numbers* on line 138) or seemingly wrong words (eg on page 5 you miight have meant "improvises" instead of "improves" gradient flow, though I don't think the next word is correct when you say "Our dynamic rewiring ... retain Dirichlet Energy").
- In Table 4, imo you should only bold the best entry in each column. If you really want to draw attention to the second best entry, perhaps you could underline them?
questions: Under what conditions is the sparsity assumption for the structure of a_{i}a_{j} no longer true?
limitations: n/a
review_title: null

Meta-review: metareview: The paper proposes to promote the training of deep GCNs from the gradient flow perspective by introducing topology-aware isometric initialization and Dirichlet energy guided rewiring. The intuition of looking into healthy gradient flow is innovative and the observations are insightful. The experiments generally perform good, which demonstrate the effectiveness of proposed method. The reviewers have raised several concerns about the technical and experiments details, such as the use of Dirichlet energy and the discussion of trainability issue. The authors provide a nice rebuttal, and the discussions should be included in the revision to better present the research work. 
confidence: Less certain
Decision: Accept