Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper analyses the phenomenon of oversmoothing in graph convolutional neural networks based on the graph laplacian. It propose a simple, but theoretically well-motivated, and experimentally well-validated solution to this problem. The solution builds upon previous theoretical analyses on how the variances in feature maps decreases with each graph convolution layer. Bounds are provided on how quickly this happens based on the singular values of the weight matrices. The theoretical analysis shows that one should simply scale the slope in ReLU activation functions with a factor 2. This solution greatly reduces the speed at which feature values converge to the oversmoothed sub-space in which feature eventually land. Thus, the simple trick allows to build deeper graph NNs.
strength_and_weaknesses: **Strengths**

1. The main idea of the paper is intuitive and simple, and can easily be incorporated in graph NNs.
2. The idea is theoreticall well motivated, and experimentally validated.
3. Although I did not check all proofs, most are based on existing results and the proof outlines sketched in the main text make sense, as such I believe them to be correct.

**Weaknesses**

1. The paper is, understandably, limited to graph NNs of the convolutional kind (based on spectral convolutions). Given the popularity of other graph NN approaches (s.a. message passing, transformers, ...) it would be interesting to learn how these results generalize to other type of graph NNs.
2. Section 4.1, models. The paper uses relatively basic architectures, which is important to ablate the effect of the modified activation function, however, it would still be nice to have models with residual connections just to have a baseline. It would give insight on how important it actually is to go deep with only basic building blocks.
3. The paper addresses the issue of oversmoothing, but only quantifies it indirectly be comparing test accuracies. Wouldn't it be possible to quantify the heterogeneity of features in graphs somehow, as a function of depth? E.g. via metrics present in this work:
*Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., & Sun, X. (2020, April). Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3438-3445).*
4. Small typo: page 4 "We use the notation of 'converging to a subspace'" -> "... notion of ..."
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written. Paper is of good quality. It is novel in its analysis as well as associated emperical study. Experiments are well explained and I consider the work reproducible.
summary_of_the_review: Considering the simplicity of the proposed solution, and the fact that graph convolutions are still very much prevalent in graph NN literature, the paper could have a high impact. Considering that the paper is also well written and sound, I recommend accept.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper consider the oversmoothing problem in deep GNNs and it showed activation function plays a crucial role in this phenomena. The paper proposes a simple modification to the slope of ReLU to reduce oversmoothing.  Their experimental results showed improvement on deep GNNs on classification tasks even though still a shallow GNNs works better than the deep network. They showed that deep GNNs, which do not suffer from oversmoothing, can be beneficial in the presence of the cold start problem. 

strength_and_weaknesses: Strength:
1- The paper developed a simple approach that reduces oversmoothing and enables deep architectures. It is known that in deep GNNs repetition of the Laplacian operator acts as a variance reducer and causes the oversmoothing. The paper transfers the idea of SeLU to GNNs, focusing on the part that increases the variance to alleviate oversmoothing. 

2- The paper provides a theoretical analysis that the convergence to a subspace where node representations are the same can be controlled by two factors 1) slope of the Relu function and 2) modifying the learning rate.

3- Since determining the right LR per layer is a non-trivial task, they focused on the slope of the Relu function. Increasing the slope of ReLU too much also causes exploding gradients and their analysis of the upper bound of the largest singular value of the weight matrix becomes too loose. 

Weakness:

1- Even though the paper provides an interesting analysis that correlates the slope of the Relu and the learning rate with the largest singular value of the weight matrix which affects the oversmoothing phenomena, it is still not clear how it is useful in the real scenario given the fact that even their experimental results show that still shallow GNNs performs better in classification task.

2- The paper claims that deep GNNs are beneficial in the presence of cold-start problems, but their experimental results sound artificial given that the features of the test data are removed. The cold start problem may appear on feature level (some features are missing) or edge level (some connections are not formed yet). So to show the impact of the deep GNNs I would suggest running some experiments either on a real dataset that has the cold start problem or evaluate different scenarios in existing datasets. Also it is not clear why when all features are removed why nodes which are further away are more important. 

3-  As mentioned in the paper the slope of the Relu affects the oversmoothing. That would be interesting how varying such slopes would affect the performance. It is mentioned that a large slope will cause exploding gradients. But I am wondering what the trend looks like as we increase the slope. Also can we say if the slope is smaller than one the oversmoothing happens earlier and all representations converge to a similar subspace?

clarity,_quality,_novelty_and_reproducibility: The paper is well written and easy to follow. There are some assumptions that are made through the paper that it is not clear why it holds for GNNs. For example the second assumption which holds for FFs is used for GNNs but I think GNNs are more restricted than FFNs because of the local aggregation and the probability that Relu(h)!=0 should be greater that Relu(h)!=0 in FFs. 

summary_of_the_review: In general the theoretical analysis the paper provides sounds interesting and it sheds more light on controllable parameters in GNNs to delay oversmoothness but still it is not clear how useful it is in practice since shallow GNNs are still better real scenarios.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The authors build upon the recent analysis in [1], which provides a theoretical foundation for the over-smoothing phenomenon in graph neural networks, to propose an adaptation of the activation function to counteract this effect which harms performance. Specifically, they propose multiplying the relu by a factor of 2, to counteract an exponential decay factor of 0.5, which contributes to over-smoothing. The authors provide both theoretical and empirical evidence that their proposed method indeed mitigates over-smoothing, thus enabling deeper graph neural networks to be used without significant performance drops.
strength_and_weaknesses: Strengths
- The paper is well presented. The theorems are clear, and the theoretical foundation for the proposed method is well established.
- Current experiments provide some support for the hypothesis that the method can mitigate over-smoothing in graph neural networks

Weaknesses
- Just a comment: The phenomenon of over-smoothing is a case where the graph feature maps become too smooth so that they lose the information necessary to make a prediction. It is not necessarily the phenomenon where the accuracy degrades as we add more layers. The latter can be caused simply by a failure to optimize, and not caused by a flawed architecture.
  
- The claims in this paper are too bold. Table 1 clearly demonstrates that over-smoothing still exists in the proposed method. 

- There are multiple new networks that do not suffer from over-smoothing. E.g., GCNII (2020), GRAND (Chamberlain et al., 2021), PDE-GCN (Eliasof et al. 2021), GraphCon (Rusch et al, 2022), EGNN (Zhou 2021). I am not sure that tackling the over-smoothing problem is still a pressing research question in this field. GCN and GAT are indeed over-smooth, but they are relatively old by now.

- The authors do not compare their method to other GNNs.    
- I believe that the authors do not provide sufficient investigations into the sensitivity and generalization of the proposed solution. Specifically, I believe several experiments are missing:

1. Generalization to modern architectures and activation functions: current
architectures and activation functions explored in this paper are from no later than 2019. Though the authors explicitly state that they intentionally do not compare to
modern architectures which use skip connections, I do not agree that these methods
are out of the scope of this paper. This is because the proposed method has the
potential to provide benefits to a vast variety of graph neural networks in overcoming
over-smoothing and I believe it is important to show that the method generalizes to
other more modern architectures. 
2. Sensitivity: I believe additional experimentation is required to support the choice of
the relu factor of 2 in this paper, such as trying different values of this factor. For
example, if I understand correctly, eq. (6) suggests that a factor of sqrt(2) may be more appropriate in
counteracting the exponential decay of the upper bound.

clarity,_quality,_novelty_and_reproducibility: - Clarity: The paper is well-written, and the method is clear, surprisingly simple, and well-established.

- Quality: The paper is of decent quality. However, the simplicity of the method is a bit underwhelming. That said, additional experimentation discussed above would definitely increase its quality.

- Novelty: Though simple, the adaptation of the activation function seems to be quite novel, especially in its application to mitigate over-smoothing.

- Reproducibility: The authors do not provide code. That said, the method is quite simple to replicate even with no code and training configurations provided in the main paper. However, it would help to have code to reproduce the strongest results of the “cold start” experiments.

summary_of_the_review: The paper is well written, the problem statement well established, and the proposed solution well founded. However, it is clear that the method is insufficient to mitigate over-smoothing, and at this point, many new architectures do not suffer from it. I also think that the authors do not perform sufficient investigations into the generalizability of their proposed method to modern architectures and activation functions (later than 2018), nor do they provide in-depth investigations into the sensitivity of the proposed solution to different choices of relu factors.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper studies the relevance of the slope of the ReLU activation function in deep GNNs with respect to the over smoothing issue.
The proposed strategy is simply to use a higher slope for the ReLU, setting its value to 2.
The paper presents an experimental analysis in which this idea is applied to GCN and GAT on a number of benchmark problems, and a modified version called cold start (in which the input features are removed from some nodes).

Update after rebuttal: I thank the authors for considering my concerns on their work. The rebuttal partially clarified doubts regarding the validity (especially) of the experimental analysis (and lack of fine-tuning). I am happy to slightly raise my score accordingly.
strength_and_weaknesses: Strenghts:
- The paper highlights the relevant role of the slope of the ReLU activation function in the design of deep GNN, showing the potentialities of increasing the value of the slope as a simple way to counter-balance the oversmoothing effects.

Weaknesses:
- The mathematical analysis lacks some rigor in the determination of the exact value of the slope.
In Section 3, which represents the core of this contribution, the paper elaborates on the importance of the value of the ReLU slope (i.e., alpha) on the upper bound for the largest singular value of the weight matrix. In a rather vague way, and without a formal statement it is concluded that the right value is alpha = 2. While this is a consequence of intuitive reasoning (and supported by some vague sentences, like "a slope of 2 makes the second exponential factor prevail over the first one", without a reference to the equation), I could not see a mathematical argument for which alpha needs to be exactly 2. Again, in a non-rigorous way it is stated that "we cannot make the slope too large", and "if we increase the slope of ReLU too much, the upper bound in eq. 6 becomes too loose". How much is too much? Is there any mathematically grounded approach to show that any larger (or smaller) value than alpha = 2 leads to gradient problems? The main message I got from the paper is that the actual value of alpha would be better tuned instead of being fixed to 1. This could be probably be treated as a hyperparameter and chosen accordingly.
- Hyperparameters tuning is essentially not performed in the experiments. Indeed, in Section 4.1 it is reported that the crucial values of the hyperparameters are all fixed (also, I could not find information on the used learning rate). Moreover, it could be interesting to tune the value of alpha as a hyperparameter.
- Results in the experimental section are reported without std across repetitions, and without any analysis of statistical significance of the reported average accuracy.
clarity,_quality,_novelty_and_reproducibility: Though the paper is sufficiently well written, the theoretical claims (in Lemmas and Theorems) are given without the necessary rigor.
summary_of_the_review: The paper has the merit of highlighting the role of the ReLU slope on the over smoothing problem. However, in its current form there are several issues with the mathematical analysis (and presentation), as well as with the experimental section.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper considers the role of activation function in oversmoothing problem in GNNs and provide a remedy for it(change the slope of ReLu activations). They provide theoretical justification for this phenomenon and show the benefits of the proposed method using experiments.

Strength:
- The main idea of the paper is intuitive and simple, and can easily be incorporated in graph NNs.
- The paper developed a simple approach that reduces oversmoothing and enables deep architectures
- The paper provides a theoretical analysis that the convergence to a subspace where node representations are the same can be controlled by two factors 1) slope of the Relu function and 2) modifying the learning rate.


Weakness:
- Even though the paper provides an interesting analysis that correlates the slope of the Relu and the learning rate with the largest singular value of the weight matrix which affects the oversmoothing phenomena, it is still not clear how it is useful in the real scenario given the fact that even their experimental results show that still shallow GNNs performs better in classification task.
- The paper claims that deep GNNs are beneficial in the presence of cold-start problems, but their experimental results sound artificial given that the features of the test data are removed.
- There are multiple new networks that do not suffer from over-smoothing. 
- the authors do not provide sufficient investigations into the sensitivity and generalization of the proposed solution. Specifically, several experiments are missing. see reviews for details.


justification_for_why_not_higher_score: See meta review for details.
justification_for_why_not_lower_score: Reject
Decision: Reject