Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: This work performs a systematic study to analyze the main issues of the difficulty in training deep GNNs by disentangling the effects of embedding propagation (EP) and embedding transformation (ET). They find that the large $D_t$ is the root cause for the failure of deep GNNs. Node-adaptive combination mechanism and residual connections between ET operations are proposed to train deep GNNs.
main_review: (+) The paper is well-organized and easy to follow. 

(+) This work performs a systematic study to analyze the main issues of the difficulty in training deep GNNs by disentangling the effects of embedding propagation (EP) and embedding transformation (ET). I enjoy reading the design of the analysis.

Concerns:

* The authors claim model degradation is the main cause of performance degradation of deep GNNs. However, what is model degradation is not defined formally.

* It seems the proposed Node Smoothing Level is a measure of **smoothness** instead of **over-smoothness**. Claiming over-smoothing is not a major issue using Node Smoothing Level is not very well-justified. I think a better metric should be used. I recommend the authors check Group Distance Ratio and Instance Information Gain proposed in [1].

* Why is the essential difference between the proposed “ResGCN” and “DenseGCN” in Figure 3 (b) and “ResGCN” and “DenseGCN” in [2].

* What is the difference between the findings of adding skip connections to GNNs in [2] [3] and the findings in Section 5.3? This should be discussed.

* It is unclear how the node-adaptive combination mechanism and residual connections between ET operations help with training deep GNNs. An ablation study needs to be carried out.

* What is the reason that the performance of DGMLP on ogbn-product is far from the leading methods on the OGB leaderboard?

* Why is the performance of baseline SIGN [4] is lower than the originally reported results (0.6568 ± 0.0006) on the leaderboard?

* What are the benefits of disentangled graph convolution over entangled graph convolution rather than scalability?

[1] Zhou, Kaixiong, et al. "Towards Deeper Graph Neural Networks with Differentiable Group Normalization." NeurIPS 2020.

[2] Li, Guohao, et al. "Deepgcns: Can gcns go as deep as cnns?." ICCV 2019.

[3] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv 2020.

[4] Frasca, Fabrizio, et al. "SIGN: Scalable Inception Graph Neural Networks." arXiv 2020.
summary_of_the_review: (+) The systematic study of training deep GNNs of this work is interesting.

(-) Some claims are not well-justified. Some related works are not discussed properly. Alations need to be done for the proposed methods. The performance of DGMLP is not very strong.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper considers the application of GNNs to semi-supervised node classification tasks. The problem arises in big data problems such as modeling citation databases.  In this line of work it is assumed that the data follows a graph structure, i.e., that locally connected nodes are likely to have the same label. Each node has a d-dimensional feature vector, with graph degree matrix A. Various GNN architectures have been proposed and studied for this problem, often using citation databases as the driving application. The paper considers two aspects of this problem.  First, the authors consider various choices of the hyper-parameters, including the GNN iterations and a (typically) fully connected layer at the output.  It is well known that iterating the graph convolutions leads to local averaging that will asymptotically converge to some form of smoothing, which can result in poor classification.  Here, the authors decouple the GNN smoothing and the follow on network, and refer to this as “unentangled” processing architecture (also called decoupled in the numerical results portion).  They further propose a revised GNN architecture (DGMLP) that incorporates node-adaptive weighting and skip-connections in the second stage.  Numerical experiments complete the work, showing that with appropriate choice of parameters the DGMLP is flexible and can achieve the state of the art results, and good scalability.


main_review: The exploration of hyper-parameters for this problem is well founded, especially given the many different architectures that have been proposed.  The paper characterizes this through two primary parameters, Dp and Dt, and compares the DGMLP with varying values.  It isn’t clear why the authors don’t refer to these as hyper-parameters (except buried in the appendix). Perhaps it is not surprising that various papers have shown that the unentangled approach can outperform the entangled one.

The main body of the paper seems repetitive at times, stating repeatedly the basic points about the tradeoffs with Dp and Dt.  In the reviewers opinion, the meaningful portions of the paper are in the appendices, especially A and B.  It isn’t clear why these don’t make up the major portion of the main paper.  It seems that the paper spends a large amount of space with discussion and generality, often spending too much effort on criticism.  

The results show run-time and GPU memory versus graph size, and the DGMLP has good scaling properties.  

The node-adaptive weighting mechanism (Section 6.1) is interesting.  However, the influence here wasn’t entirely clear.

The paper would be significantly improved with a statement of the DGMLP in some kind of Table or Algorithm statement.

The DGMLP is able to achieve state of the art results compared with other algorithms.  So, the value here is the flexibility of the DGMLP, and also perhaps complexity reduction and scalability. It wasn’t entirely clear what is the overall complexity in comparison with other algorithms, especially with respect to choice of the hyper-parameters.  Perhaps the authors could add some further complexity results that include the cost of hyper-parameter exploration with the various algorithms to make this clear to the reader. 

summary_of_the_review: The theme of the paper is “Hyper-Parameter Selection and a Modified GNN for Semi-Supervised Node Classification”.  Perhaps that’s a better title.  

Overall the paper builds on the many other GNN works for this problem, incorporates the best known practices, and provides a flexible approach to achieving the state of the art results when compared to other algorithms.  However, the presentation could be significantly strengthened as outlined in the Main Review.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper points out the dominant influence on deep GNNs, i.e., the embedding transformation, with careful and extensive studies. Based on these studies, several guidelines are proposed to design deep model named DGMLP. 
main_review: **Strengths:**

(1) The concepts of embedding propagation and transformation depths are formally defined to study their impacts on deep GNNs.
(2) The series of empirical studies on these two depths are given.
(3) The paper is well organized.


**Weakness:**

(1) The empirical guidelines 1-3, and the experimental analysis (in Section 4) are not novel at all to the community of deep GNNs. Specifically, the decoupling of embedding propagation and transformation, adaptive weighting, and skip connection have already been studied in literature. I know the authors derive these guidelines based on formulating the concepts of D_p & D_t and conducting the extensive experiments to get Findings 1. However, these knowledges have been implicitly explained in prior efforts. For example, the decoupling of EP and ET was proposed in DAGNN [1], based on the empirical study of SGC similar to the one in this paper. To avoid the harmful impact of embedding transformation, the identity mapping is incorporated in GCNII [4] to avoid the overly feature change. In other word, the dominant influence of embedding transformation in deep GNNs is not a news.

(2) The proposed model, named DGMLP, is incremental comparing with the existing works, including DAGNN, SGC, JK-Net, etc. The only difference is the usage of skip connection in the final MLP module. 

(3) Although the experimental results show the superiority of DGMLP, most of the important baselines are missing in Table 1, which easily leads to misunderstanding. For example, GCNII has good or comparable performances in Cora, Citeseer, and ogbn-arxiv. In obgn-products, SAGN [5] and DeeperGCN [6] with the similar architecture to the proposed model shows outperforming results. Furthermore, SAGN also has better results in ogbn-100M. But all these important baselines are missing.

(4) Some of the claims made in this paper is not correct. See below for the detailed comments.


**Questions:**
(1) Under Eq. (4), it is stated that ``Under this scenario, the neighborhood information is fully corrupted, resulting in catastrophic node classification accuracy.” I’m concerning on the correctness of this statement: why neighborhood information will be corrupted if the neighborhood influence is determined by the degree? The specific neighbor features could still distinguish them. As shown in Figure 4 of reference [1], by stacking many EP layers, the node classification accuracy is still good enough, which empirically demonstrates that neighbor information is not corrupted. 

(2) The claim of ``As the over-smoothing issue is only introduced by the EP operation rather than the ET operation” below Eq. (4) is not correct. As theoretically demonstrated in references [2,3], the over-smoothing issue is also correlated to the singular values of trainable weights and non-linear activations involved in ET. 

(3) For the example study of Figure 2(c), it is unclear how to formulate GCN model with D_t=2 and the adjacency matrix powered by D_p/2. Could you illustrate more with the formal expression or model figure?

(4) In Section 4.2, please formally define the residual connection and dense connection used in the empirical study.

(5) The guideline 2 seems not novel at all. It is well-known that the optimal convolution depths for the diverse nodes are different. For example, DAGNN in [1] applies attention module to learn the optimal combination weight for each specific node.

[1] Liu, Meng, Hongyang Gao, and Shuiwang Ji. "Towards deeper graph neural networks." Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining. 2020.

[2] Cai, Chen, and Yusu Wang. "A note on over-smoothing for graph neural networks." arXiv preprint arXiv:2006.13318 (2020).

[3] Oono, Kenta, and Taiji Suzuki. "Graph neural networks exponentially lose expressive power for node classification." arXiv preprint arXiv:1905.10947 (2019).

[4] Chen, Ming, et al. "Simple and deep graph convolutional networks." International Conference on Machine Learning. PMLR, 2020.

[5] Sun, Chuxiong, and Guoshi Wu. "Scalable and Adaptive Graph Neural Networks with Self-Label-Enhanced training." arXiv preprint arXiv:2104.09376 (2021).

[6] Li, Guohao, et al. "Deepergcn: All you need to train deeper gcns." arXiv preprint arXiv:2006.07739 (2020).
summary_of_the_review: This paper formally define the concepts of embedding propagation and transformation depths, and distinguish their impacts on the deep GNNs. Based on the empirical studies, some well-known guidelines are proposed to design deep GNNs. 

However, I cannot find much novelty from this paper, which seems to ensemble the existing knowledges. Some of important baselines are missing to validate the effectiveness of the proposed methods.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: In this work, the authors performed an experimental evaluation on several GNNs in order to understand what aspects of the current architecture designs that leads to the compromised performance of deep GNNs. They claimed to find the root causes: large propagation depth leads to the over-smoothing issue and large transformation depth leads to the model degradation issue. Then they proposed guidelines for designing deep GNNs and a Deep Graph Multi-Layer Perceptron (DGMLP) for implementing the guidelines. Experimental results demonstrate performance of DGMLP in accuracy, flexibility, scalability, and efficiency.
main_review: The definitions of Node Smoothing Level and Graph Smoothing Level compare the representations of the same node across different EP steps. However, the oversmoothing problem is across nodes. Considering the deviations, the performance of DGMLP appears to be comparable to the state-of-the-art. It is claimed that DGMLP can support large transformation depth D_t. But Fig6 (b) only shows a very stable accuracy as D_t increases. It would be more convincing if the performance of DGMLP improves with a larger D_t. Otherwise, why bother to increase D_t? Larger D_t means more parameters, and hence may need to train models longer (and with better training settings). Is it possible that large D_t leads to undertrain of GNN? The proposed node-adaptive weighting mechanism is interesting. It could helpful to see experiments that validate the importance of the node-adaptive weighting mechanism. 
summary_of_the_review: The experiments, which compare various GNNs, provide useful information. The proposed DGMLP contains interesting ideas. However, experimental results are mixed and do not clearly support all claims made in this manuscript. The proposed smoothing level metrics do not measure smoothing across nodes.
review_title: null

Meta-review: The paper studies why existing deep GCNs suffer from poor performance and propose DGMLP to improve over existing GCNs. However, the reviewers think there are still many unjustified claims and the paper. Further, several reviewers question about the novelty of the proposed method, which seems to be a combination of existing approaches. 

I suggest the authors to revise the paper by defining terms like model degradation and smoothness mathematically and try to justify each claim (e.g., the effect of disentangling) with solid experiments. These will significantly improve the analysis part and make the conclusions stronger.
Decision: Reject