Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors propose a Data-Dependent GCN framework D$^2$-GCN which integrates data-dependent node-wise, edge-wise and bit-wise skipping to save the cost for inference while does not sacrifice prediction accuracy.
main_review: 
Strength:

The writing is easy to understand and the idea is clearly presented. The experiment results show good empirical inference speed compared to the existing methods.


Weakness&Advice:

1. The description of “The inference process of one GCN layer can be viewed as two separated phases: Aggregation and Combination. …” for equation (1) is not precise. See page 52 in [1]. Instead of being viewed as two separated phases, “we no longer need to define an explicit update function, as the update is implicitly defined through the aggregation method”.

2. “we hypothesize that explicit or implicit regularizations during GCN training can improve GCNs’ scalability towards going deeper,”  Randomly dropping being able to help with deeper GCN does not mean all regularization methods can. This hypothesis is over-claimed and groundless.

3. What is the training speed of your model and the three-stage training pipeline compared to the existing methods?

4. "$L_{comp}$  is the computational cost determined by the decisions made by the gating functions” What does this sentence means? It’s better to write $\mathcal{L}_{comp}$ in formula.

5. What is the performance after each stage in your three-stage training pipeline. I would like to know which stage plays the most important role and how other stages improve the performance.

6. Gating is not a new technique for GNNs, e.g. [2]. It’s better to draw the difference with these methods and emphasize the novelty and contribution of your method. Note that I will not consider it novel if you only put some existing tricks into your model.

Typos:
Section 3.2, 3rd paragraph, graphes —> graphs

Bit-wise skipping in figure 1, $x_2$ —> $x_1$

In equation (4), should use parenthesis after $\odot$ to avoid confusion.

[1] Hamilton WL. Graph representation learning. Synthesis Lectures on Artifical Intelligence and Machine Learning. 2020 Sep 15;14(3):1-59.

[2] Li Y, Tarlow D, Brockschmidt M, Zemel R. Gated graph sequence neural networks. arXiv preprint arXiv:1511.05493. 2015 Nov 17.


summary_of_the_review: Some of the technical details need to be provided. The novelty need to be emphasized. I'll consider raise my score if my concerned can be properly addressed.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: In this work, the authors propose relatively low-cost GCNs in a data-dependent way.

Their framework has three main components: node-wise, edge-wise, and bit-wise skipping.

1. Node-wise skipping is determined by a binary decision for each node based on its features.

2. Edge-wise skipping is about the removal of connections between two nodes.

3. Bit-wise skipping is about the quantization precision of aggregated node features.


It boosts efficiency while achieving comparable performance over benchmark datasets.
main_review: I think D2-GCN is well-designed with simple components and the experimental results are impressive in that many recent graph data are at large-scale (especially, web data).

The authors focus on the efficiency of computation but I want to ask about the effectiveness of information processing.

1. Authors argue that it can be helpful to alleviate the over-smoothing issue. It makes sense because three gating (or blocking) functions might prevent mixing node information. However, is there any experimental evidence (e.g. measurement of smoothness)?

2. Can it be extended as (integrated with) a disentangled representation learning framework for GNNs? If so, could you briefly explain it? Because cutting connection and quantizing information seem like Disentangled GCN [1] in broad concept. I think it would be interesting to analyze which information is discarded or not.


The above questions are just for constructive discussion.


[1] Disentangled Graph Convolutional Networks (ICML 2019)
summary_of_the_review: I would recommend this paper to be accepted if other issues do not arise.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper  attempts to address the training efficiency and scalability of GCNs. To be specific, a so-called Data-Dependent dynamic GCN framework is proposed, in which node-wise skepping, edge-wise skipping, an bit-wise skipping are integrated via gate function to squeeze out (or reduce) the unimportant neighbor nodes in combinations, unimportant edge connections, and in the bit-precision, respectively. Extensive experiments are provided, showing new SOTA results on benchmark datasets. 
main_review: 
Strengths:
+ The paper is clearly written and easy to follow. 
+  Integrating node-wise skepping, edge-wise skipping, an bit-wise skipping  to reduce the nodes, prune the edges, and compress the bit-precision is interesting and somewhat novel.
+ Experiments are extensive and promissing.


Weaknesses: 

- In the discussion in Section 4.3 it is mentioned that "our proposed D$^2$GCN can resolve the over-smoothing issue." The reviewer is quite interesting how the over-smoothing issue is resolved in the proposed D$^2$GCN. More discussions, empirical or theoretical analysis on this point is needed.
- Furthermore, it is stated that  "such data-dependent skipping techniques serve as a regularization effect". It is appealing to mention of the regularization. However, the interpretation---" since different components of a graph may work together or independently in an data-dependent manner to avoid the over-smoothing issue" is disappointingly too vague. 
Could it possible to develop any theoretical analysis on the skipping techniques to verify whether it is equivalent to a special implicit regularization and, what kind of impilicit regularization?   Since that the idea of the paper is very simple, it would make the paper stronger if the theory behind could be developed. 


Minor issues:
There is a mistake in the subfigure of the right panel: $X_2 --> X_1$; 

summary_of_the_review: While the paper is clearly written and the empirical evaluation is convincing, it is not clear why and how the over-smoothing issue could be resolved or what kind of equivalent regularization behind.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: This paper proposes a Data-Dependent GCN framework (D$^2$-GCN) that integrates data-dependent dynamic skipping at multiple granularities. D$^2$-GCN is achieved by identifying the importance of node features via a low-cost indicator and thus is simple and generally applicable to various graph-based learning tasks. Experiments certify the effectiveness and efficiency of D$^2$-GCN. 
main_review: Strengths:

(1) This paper is well motivated. 

(2) The Introduction and Related works parts are well written.

(3) Experiments on top of various SOTA GCNs and datasets validate the effectiveness and advantages of D$^2$-GCN.


Weaknesses:

(1) The paper has limited technique contributions. The problem of boosting the efficiency and scalability of GCNs has been fully studied by existing works. The ideas of *Node-wise skipping* and *Edge-wise skipping* are not novel enough. Please refer to the following references:

- [KDD 2019] [Cluster-GCN] An Efficient Algorithm for Training Deep and Large Graph Convolutional Networks
- [ICCV 2019] [DeepGCNs] Can GCNs Go as Deep as CNNs
- [ICLR 2016] [GGS-NNs] Gated Graph Sequence Neural Networks
- [ICLR 2020] [DropEdge] Towards Deep Graph Convolutional Networks on Node Classification
- [NIPS 2020] [DropNode] Graph Random Neural Networks for Semi-Supervised Learning on Graphs

(2) The main contribution of this work is reducing the GCN inference cost. The data-dependent dynamic skipping strategies achieve this. I suggest the authors add theoretical complexity analysis comparison to better support this. 

(3) The skipping strategy are confusing: 
- the node-wise skipping parameter $G_l^n$ is 0 or 1. However, the authors claim $G_l^n \in R^{N*1}$. The problems also exist for $G_l^b$ and $G_l^e$.
- The gate function design part is important. However, the description is too brief. 

(4) In Eq.(6), $L_{comp}$ denotes the computational cost determined by the decisions made by the gating functions. The paper lacks a detailed calculation method.

(5) This paper lacks hyperparameters experiments about trade-off parameter $\alpha$, which can intuitively provide the trade-off between accuracy and efficiency.

(6) The writing needs improvements. There are many typos and problems with the mathematical symbols, sentence structure, such as

- The real number (M,N), vector ($G_l^n$) and matrix ($X$) are denoted as uppercase characters.
- The sentence in the first paragraph of page 6 is too long and hard to read.
- "Specially, where $X_1$ ∼ $X_4$ are the 4 nodes in the graph."?
- "input images or graphes"?

summary_of_the_review: Overall, this work is well motivated. However, the theoretical complexity analysis is missing and the methodology part remains confused. The technique contributions are limited.
review_title: null

Meta-review: The paper proposes Data-Dependent GCN (D2-GCN), which improves the efficiency of vanilla GCN by node-wise skipping, edgewise skipping, and bit-wise skipping. Gate functions are learned to prune the unimportant neighbor nodes in combinations, unimportant edge connections, and in the bit-precision. The proposed method boosts efficiency while achieving comparable performance over benchmark datasets. Most reviewers agree that the paper is well motivated, and the writing is clear. However, two of the reviewers found the novelty of the paper compared to previous work (for example, [1]) is limited. Three reviewers raised concerns about the lack of theoretical or empirical analysis on how D2-GCN can alleviate the over-smoothing problem, and how the proposed method can serve as an implicit regularization.

For the novelty concerns, the authors provided a detailed comparison with previous work during the rebuttal. For the lack of analysis on over-smoothing, the authors provided an additional empirical analysis using the distance of different intermediate layers’ output as the metric for measuring over-smoothing. But at least one reviewer is still not satisfied with those.

Given the current review scores (3, 5, 5, 6), the paper is below the acceptance threshold for the conference. The AC believes that the proposed method seems to be an effective and simple way towards more efficient graph neural networks and hence encourages the authors to submit the revised paper to another venue after addressing the reviewers’ concerns, especially on theoretical or empirical analysis on over-smoothing and implicit regularization.


[1]: Gated graph sequence neural networks
Decision: Reject