REDUCING OVERSMOOTHING IN GRAPH NEURAL NETWORKS BY CHANGING THE ACTIVATION FUNCTION
Authors: Dimitrios Kelesis
Dimitrios Vogiatzis
Georgios Katsimpras
Dimitris Fotakis
Georgios Paliouras
Venue: Submitted to ICLR 2023 (2023)
Keywords: Graph Neural Networks
Oversmoothing

Abstract: The performance of Graph Neural Networks (GNNs) deteriorates as the depth of the network increases. That performance drop is mainly attributed to oversmoothing, which leads to similar node representations through repeated graph convolutions. We show that in deep GNNs the activation function plays a crucial role in oversmoothing. We explain theoretically why this is the case and propose a simple modification to the slope of ReLU to reduce oversmoothing. The proposed approach enables deep architectures without the need to change the network architecture or to add residual connections. We verify the theoretical results experimentally and further show that deep networks, which do not suffer from oversmoothing are beneficial in the presence of the “cold start” problem, i.e. when there is no feature information about unlabeled nodes.

Under review as a conference paper at ICLR 2023
REDUCING OVERSMOOTHING IN GRAPH NEURAL NET- WORKS BY CHANGING THE ACTIVATION FUNCTION
Anonymous authors Paper under double-blind review
ABSTRACT
The performance of Graph Neural Networks (GNNs) deteriorates as the depth of the network increases. That performance drop mainly attributed to over- smoothing, which leads to similar node representations through repeated graph convolutions. We show that in deep GNNs the activation function plays crucial role in oversmoothing. We explain theoretically why this is the case and propose a simple modification to the slope of ReLU to reduce oversmoothing. The pro- posed approach enables deep architectures without the need to change the network architecture or to add residual connections. We verify the theoretical results ex- perimentally and further show that deep networks, which do not suffer from over- smoothing, are beneficial in the presence of the "cold start" problem, i.c. when there is no feature information about unlabeled nodes.
INTRODUCTION
Graph Neural Networks (GNNs) utilize message passing or neighborhood aggregation schemes to extract representations for nodes and their neighborhoods. GNNs have achieved good results on a variety of graph analytics tasks, such as node classification (Zhang et al. 2018 Hamilton et al.) 2017). link prediction (Liben-Nowell & Kleinberg 2003) Zhang & Chen 2018) and graph classifi- cation (Kieperra et al. 2020). As a result, they play a key role in graph representation learning. One of the most prominent GNN models is the Graph Convolutional Network (GCN) (Kipf & Welling, 2017]. which creates node representations, by averaging the representations (embeddings) of its im- mediate neighbors. Several studies have shown that, the performance of GCNs deteriorates, when their architecture becomes deeper (Li et al., 2018).
The success of deep CNNs on many tasks, like image classification, naturally led to several attempts towards building deep GNNs for node classification (Kipf & Welling 2017 et al. 2018 Xu et al., 2018). Increasing the model's depth (and the number of parameters it has), would allow more ac- curate representational learning to occur. Most of the existing approaches have failed to develop sufficiently deep architecture that achieves good performance. Therefore, there is a need to design new models which can efficiently scale to a large number of layers. The aim of this paper is to investigate the contributing factors, that compromise the performance of deep GNNs and develop a method to address them.
The performance drop of deep GNNs, is associated with several factors, including vanishing gra- well as the phenomenon called oversmoothing (Li al.) 2018 Xu et al.)
dients, overfitting, 2018 Klicpera et al. 2018 Wu et al. 2019). Oversmoothing has been shown to be associated with graph convolution, type of Laplacian operator. Li al. (2018) proved that applying that opera- tor repeatedly, makes node representations converge to a stationary point. At that point, all of the initial information (i.e. node features' information) is lost through the Laplacian smoothing. Con- sequently, oversmoothing hurts the performance by making node features indistinguishable across different classes.
In this work, we address the oversmoothing problem in deep GNNs. We provide what is to the best of our knowledge the first study regarding the role of the activation function and the learning rate per layer of the model to oversmoothing and propose new method to address the problem. We confirm our hypotheses both experimentally and theoretically. The new method is shown to prevent node embeddings from converging to the same point, thus leading to better node representations. We summarize our main contributions as follows.
Role of Activation Function in Oversmoothing: We prove theoretically the connection between

---

Under review as a conference paper at ICLR 2023
the activation function and oversmoothing. In fact, we show the relation between the slope of ReLU and the singular values of weight matrices, which are known to be associated with oversmoothing
(Oono& & Suzuki 2020 Cai & Wang 2020). We have also verified our theoretical results experi-
mentally.
Our analysis the effect of the slope of ReLU to oversmoothing has direct extension to the learning rates used per layer of the network. We con- ducted further experiments study the effect of tuning the learning rates, showing that this approach could also reduce oversmoothing, but it is less practical.
Role of Learning Rate in Oversmoothing:
The power of Deep GNNs: We have performed extensive experiments using up to 64-layer net- works, tackling oversmoothing with the proposed method. We further show the benefits that such deep GNNs can provide in the presence of reduced information, such as in a "cold start" situation, where node features are available only for the labeled nodes in node classification setting.
2 NOTATIONS AND PRELIMINARIES
2.1 NOTATIONS
In order to illustrate the problem of oversmoothing, we consider the task of semi-supervised node classification graph. The graph to be analysed is G(V, E,X), with [V] N nodes u, E V, edges (u; uz) E EE and
[x1,...,< "Nn'' NxC denotes the initial node features. The edges form an adjacency matrix A N where edge (u, is associated with element A.,. At,5 can take arbitrary real values indicating the weight (strength) of edge (u, uy). Node degrees represented through a diagonal matrix D R NNN. where each element d, represents the sum of edge weights connected to node i. During training, only the labels subset V/2 E are available. The task is to learn node classifier, that predicts the label of each node using the graph topology and the given feature vectors.
GCN, originally proposed by Kipf & Welling (017), utilizes a feed forward propagation as:
H0+1) (AAI(
where H = [ N are node representations (or hidden vectors or embeddings) at the I-th layer, with I! standing for the hidden representation of node i; Å - denotes the augmented adjacency matrix after self-loop addition, where D corresponds the degree matrix; (-) nonlinear element-wise function, i.e. the activation function, ReLU; and w(l) is the trainable weight matrix of the 1-th layer. A generic ReLU function with slope a is defined as ReLU(x) mar(a- x, O).
(4)
2.2
UNDERSTANDING OVERSMOOTHING
GNNs achieve state-of-the-an performance in a variety of graph-based tasks, Despite their success, models like GCN (Kipf & Welling 2017) and GAT (Velickovic et al. 2018) experience a perfor- mance drop, when stacking multiple layers. To large extent, this is attributed to oversmoothing due to repeated graph convolutions. In their analysis Li et al. (2018). showed that graph convolution is a special form of Laplacian smoothing. In fact, they proved that the new representation of each node is formed by a weighted average of its own representation and that of its neighbors. This mechanism allows the node representations within each (graph) cluster, i.e. highly connected group of nodes, to become more similar and improves the performance on semi-supervised tasks on graphs. When stacking multiple layers, the smoothing operation is repeated multiple times leading to oversmooth- ing of node representations, i.e., the hidden representations of all nodes become similar, resulting in information loss.
2.3 DEEP GNN LIMITATIONS
Therefore, oversmoothing leads node representations to converge to fixed point the network's depth increases (Li et al. 2018). At that point, node representations contain information relevant to the graph topology and disregard the input features. Oono & Suzuki (2020) have generalized the idea in Li et al. (2018) by taking into consideration, that the ReLU activation function maps to positive cone. They explain oversmoothing as the convergence to subspace, instead of the
2

---

Under review as a conference paper at ICLR 2023
convergence to fixed point. A similar approach is presented in Cai & Wang 2020), offering different perspective to the oversmoothing problem using Dirichlet energy.
basis for our analysis. The main result in (Oono & Suzuki] 2020) is the definition of the convergence speed towards subspace M, where the distance between node representations tends zero. We denote as (x) the distance between the feature vector X and the subspace, where the oversmoothing is prevalent. When that distance approaches zero indicates that node representations have been over-smoothed. For this distance, Oono & Suzuki (2000) prove an interesting property.
We now look closer to the proposal of Oono & Suzuki (220)), which we will
Theorem (Oono & Suzuki (2020) Let the largest singular value of the weight matrix Wih be Hi
and II Sth. where Win is the weight matrix of layer h and Hı be the network's depth, h=1
following the notation the the paper. Then it holds that dm S sıdar (X) any X BNN where f(-) ) is the forward pass of a GNN layer (i.e. ((XX)))).
The above theorem indicates, that the deeper the network the smaller the distance of node represen- tations from the subspace M. If the maximum singular values are small then node representations asymptotically approach M, for any initial values of node features. Extending the above theorem, the authors conclude to the following estimate about the speed of convergence the oversmoothing subspace.
Corollary (Oono & Suzuki (2020) Let suPteN, 8t. then (x)(X()) = ((s)))) where I is the layer number and if sλ the distance from oversmoothing subspace exponentially approaches zero. Where X the smallest non-zero eigenvalue of A.
According to the authors, sufficiently deep GCN will certainly suffer from oversmoothing under some conditions (details can be found in Oono & Suzuki (2020). We build upon this result, aiming to develop a consistent approach that reduces oversmoothing and enables deep architectures.
2.4 NORMALIZATION DEEP NEURAL NETWORKS
Of relevance to oversmoothing are also methods that perform normalization for deep neural net- works (Ioffe & Szegedy 2015 Ba et al. 2016). In such methods, the output each neuron is normalized, in order to keep portion of the initial feature variance. Other recent work proposed Self Normalized Networks (SNN), that use a different activation function (SeLU) (Klambauer et al.) 2017). These models perform self normalization inside each neuron, in order keep the variance between consecutive layers stable. They have managed to enable deep Fully Connected models and achieve good performance.
A recent work regarding normalization in GNNs is the Pairnorm method (Zhao & Akoglu 2020), which aims to keep constant the total pairwise distance between node representations. Compared to SNNs, Pairnorm performs normalization needing constant value as hyper--prameete determined per dataset, while SNNs manage normalize the output utilizing their activation function, which can be used in family of neural networks. The activation function proposed in (Klambauer et al.) 2017) (i.e. SeLU) has saturating region reduce the variance of data and a slope slightly greater than in order to increase the data variance, when needed.
UNDERSTANDING AND DEALING WITH OVERSMOOTHING
Using the mathematical definition of oversmoothing in Oono & Suzuki 2020). this section we establish connection with the training process of GNNs. In particular, we analyse the role of the activation function and the learning rate and we propose modified versions of GNNs to address the issue.
3.1 THEORETICAL ANALYSIS
We start by establishing the connection between oversmoothing and variance reduction of node rep- resentations. Consider oversmoothing the convergence to subspace, where node representations
3

---

Under review as a conference paper at ICLR 2023
almost the same. In that particular subspace, the initial variance of feature vectors has been mas- sively reduced. This unwanted oversmoothing effect is 'harsh', due to the fact that ReLU performs zero mapping, when the imput is negative. We the term 'harsh' to indicate the smoothing case, where representations are mapped to a single point (i.e. zero), instead of converging to subspace. subspace' following the oversmoothing analysis presented in
We use the notion of "converging Oono & Suzukij 2020). where the subspace M (where oversmoothing is prevalent) is defined.
We will now show how the slope of the ReLU activation function affects oversmoothing, starting with some important assumptions: (a) the non-exploding gradients assumption, and (b) ReLU's probability not output zero. We use these assumptions extract bounds on the weights of the net- work and then use these bounds to determine the relationship between the largest singular value and the elements of each weight matrix. Given that relationship. we connect our analysis with existing literature regarding oversmoothing through Theorem T
Assumption For each layer I there exists number G1 which is the upper bound to the gradients of the output of the subsequent layer (I+1) with respect to the weight elements of W(1,, i.e.
d0
S GI,
Vw (1) olde.s
!1) dw olds.s
where O(++1) is the output of (I+1)a 1-th layer.
This assumption needs to hold in all cases, in order to avoid the exploding gradient case, that would not allow the learning process to converge.
Assumption The probability, that the output of the ReLU function not equals zero at layer I is independent of the outputs of the previous layers, in feed-forward ReLU neural network, i.e.
(( + .. <
pE(0,1)
(3)
where h is the node representation layer I, ReLU is applied piece-wise on that representation and Ö is the all zero vector.
Following the results of Lu al. 2019) we get that (()eLU()") + olt .j < -)) for feed-forward networks (FFs), on the condition that the output of the previous layers is positive. That condition does not necessary hold in GNNs, due the role of the adjacency matrix in the aggregation scheme. In FFs, when the output of a layer is zero, so are the outputs of all subsequent layers. In GNNs there might be cases, where the output of layer (a node representation) is zero but the aggregation using the adjacency matrix produces a non-zero representation in layer (1+1). Apart from this difference, GNNs can be considered a special case of FFs, with the proper wiring due to the adjacency matrix aggregation. Therefore, the above assumption holds also for GNNs.
Lemma For network of depth dep the total gradient reaching to the 1-th layer (i.... dua
dJ
in order to update (!)) is bounded by:
dJ
< (dep-1) Gi
(4)
(1) du old...
Where J is the model's s loss function (i.e. Cross Entropy), Or stands for the ReLUU slope and G: is the upper bound of gradients of the output of the subsequent layer (l+1) with respect to the weight elements of (t))
The proof of Lemma I is shown in Appendix A and is based on the chain rule for backpropagation.
Given the derivative of the ReLU function and the upper bound G; of Assumption [ we can derive
the bound of Equation To compute the gradient with respect to weight element (!) of layer we repeatedly differentiate nested ReLU functions, leading product of their slopes (or zero). In
the final differentiation step, we get the gradient of the output of (1+ 1)-th layer with respect to w!) which is bounded by Gt.
4

---

Under review as a conference paper at ICLR 2023
Lemma 2
While model's loss, through gradients. flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero) (Lu et al. 2019). The total number of weight elements getting updated at layer I is bounded by:
11) # S p d
(5)
where d is the largest of the two dimensions of W i.e. there are at most d2 elements in W if a square matrix.
The proof of Lemma 2 appears in Appendix B. Given Lemma 1 the gradient flowing backwards, with respect to a weight element ! contains product of ReLU derivatives. In order for w to get an update, all ReLUU derivatives need to be non-zero, otherwise the gradient will be zeroed and no update will be applied to w Additionally, due Assumption 2 weight elements located in the lower weight matrices tend to receive fewer updates. This is due to the fact, that the further the gradient flows backwards the more ReLU derivative factors appear in Based on Assumption ?
the probability of all of them to be non-zero decreases.
Regarding the singular values (denoted by s-)) of a weight matrix (((1) at the 1-th layer, the largest || VΣ
of them is given by: mor(s:(W())
|| I
|un[)1 where ||e
is the Frobenius norm and || ||2 is the spectral norm. Using the general weight updating rule where J is the model's loss and 17 is the learning rate, we arrive at the main theorem of this work.
i.e., wnew.. w atd;.; + 17
Theorem 2 The upper bound of the largest singular value of the weight matrix W at layer I for GNN model, utilizing a ReLU activation function, depends on the slope of the function. That bound is given per layer and shows the effect of each iteration of updates on the weight matrix. We denote with W Wdd and new the weight matrices before and after the update during an iteration of the training process respectively.
ma (W S &IWW oldll v33 V3.p/
p/ () (c B1
(6)
where Bi = G1, dep is the network's 's depth, d is the largest dimension of the W matrix, p is the upper bound of the probability of ReLU not outpul zero and is ReLU's slope.
The proof of Theorem 2 appears in Appendix C and uses the upper bound of the largest singular value by the Frobenius norm, expanded according to the weight update rule. Separating weight elements into two sets (updated and not updated) we identify the gradient values responsible for the weight updates. These values are bounded (Lemma []). as is the number of updated elements (Lemma 2). leading to Equation 6 The resulting upper bound shows the connection of the slope of ReLU and the upper bound of the probability of ReLU not to output zero with the largest singular value of the weight matrix. That largest singular value is connected with the oversmoothing problem and as we mentioned above could act as resource to reduce it.
3.2 ALLEVIATING OVERSMOOTIING
We transfer the idea of SeLU to GNNs, focusing on the part that increases the variance, because the repetition of the Laplacian operator acts as a variance reducer. In order to avoid oversmoothing in deep GNNs using this approach, we need to identify a 'sweet' spot, where variance reduction from graph convolution counteracted as needed by the slope of the activation function. Typically in GNNs, ReLU is used with a slope value = 1. As result, the second exponential factor in 6 can be ignored (it always equal to 1). This pushes the bound for the largest singular value of weight matrices towards a fixed low value, when the architecture of the network gets deeper (dep increases), and especially at the lower layers (small () of the network. This is due to the first exponential factor of (6) converging to zero for large values of dep.
Equation
The restriction of the largest singular values to low values, increases the speed of convergence to the oversmoothng subspace, as stated in Corollary from Oono & Suzuki (2020). According the corollary, the speed depends on À and s. The former parameter (A) is a property of the data (largest eigenvalue of the adjacency matrix), while the latter (s) is the product of the largest singular values
5

---

Under review as a conference paper at ICLR 2023
of the weight matrices.
Realizing the importance of the slope of ReLU in the training process, we move on to propose a modification of the activation function that reduces oversmoothing. Following (Lu et al. 2019), proceed our analysis with p = 1/2 (see Assumption 2). which simplifies calculations. In particular, observe that slope of 2 makes the second exponential factor prevail over the first one, leading
to new combined factor of 2 (===²) This in turn increases the upper bound for the largest singular value, restricting the influence of the layer index (l) and depth (dep) Equation 6 It worth noting, that the upper bound should not be constant across all layers (which could be achieved by setting a corresponding value for the slope), because different layers of the network have different roles and the goal of a GNN is to bring intra-class representations close, while keeping inter-class representations apart.
Using the results of (Oono & Suzuki 2020) we can further connect the choice of slope for the activation function to the speed of convergence to the oversmoothing subspace. Hence, the proposed method increases the bound for each largest singular value, which in turn decreases the speed of convergence to the oversmoothing subspace, according to Corollary
3.3 MODIFYING THE SLOPE OF RELU: LIMITATIONS
Oono & Suzuki 2020) have proved that any GNN deep enough will eventually reach the over- smoothing subspace. In theory, this also holds for our method, when the GNN gets very deep. Could this be avoided by increasing the slope of ReLU further? Unfortunately not, as we cannot make the slope too large. known, that by increasing the slope of ReLU too much, one may face the problem of exploding gradients (Bengio et al. 1994), that impedes the learning procedure. In fact, if we increase the slope of ReLU too much, the upper bound in Equation 6 becomes too loose and the performance degrades. Our proof suggests, that slope equal to 2 avoids oversmoothing and our experiments have shown that also avoids the exploding gradients problem.
3.4 MODIFYING THE LEARNING RATE, INSTEAD OF THE SLOPE OF RELU
Instead of changing the slope of ReLU, we could opt to modify the Learning Rate (LR) per layer in order to increase the upper bound in Equation 6 In particular, a different learning rate (1)) per layer leads to a different B3 (= G₁) per layer, counteracting the problematic first exponential factor of Equation 6
However, determining the right LR per layer is a non-trivial task. Intuitively. LR should be larger in the lower layers and smaller the upper ones. Keeping the slope equal to 1, the first problematic exponential term is smaller (note p < 1) in the lower layers reducing the value of the upper bound. To avoid this, one needs to tune LR carefully. Additionally, LR affects heavily the learning process and large LR values might lead to oscillations and poor learning performance. Modifying the slope and the learning rate combined led to negligible improvements. Preliminary results with modified LR values shown in Appendix D.
3.5 WHY DO WE NEED DEEP GNNs?
Oversmoothing is only a problem for deep GNNs, leading the question of whether and when deep GNNs are really needed. Most of the existing benchmark datasets for various graph analytics tasks do not seem to justify the need for deep networks. Due to their homophilic nature, useful informa- tion for each node resides in its close neighbors (usually 2 or 3 hops away). A task where deeper architectures could be needed the "cold start" problem; namely the situation where many node features are missing. The problem of missing features is called "cold start", because it resembles the situation of a new product/user arriving to a recommender system. The system has no prior knowledge about the new arrival, yet has make some recommendations. In this scenario, the hope is that deeper GNNs could recover features from more distant nodes, in order to create informative representations.

---

Under review as a conference paper at ICLR 2023
4 EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Datasets: Aligned most of the literature, use four well-known benchmark datasets: Cora, CiteSeer, Pubmed and the less homophilic dataset Texas. The statistics of the datasets are reported in the Appendix. For Cora, CiteSeer and Pubmed we use the same data splits in Kipt & Welling (2177). where all nodes except the ones used for training and validation are used for testing. For Texas we use the same splits as in (Pei et al., 2020).
Models: We utilize two different GNN models as our base models for the proposed methodology:
namely GCN (Kipf Welling, 2017) and GAT (Velickovic et al., 2018). We compare the results of these models with and without the modification of the slope of ReLU for varying number of layers. We do not compare against methods utilizing residual connections, because the aim of this work is to show that oversmoothing could be avoided without "short-cireuiting" initial information to latter layers. We use residual GNNs as a baseline in the Extended Experiments section, Appendix F. Hyperparameters: We set the number of hidden units (of each layer) of GCN and GAT to 128 for both models across all datasets. The Le regularization is set with penalty 5 - 10-4 for both models and learning rate set to 10 We vary the depth between 2 and 64 layers. The number of attention heads for GAT is set to 1.
Configuration: Each experiment run 10 times and we report the average performance over these runs. We train all models for 200 epochs using Cross Entropy as a loss function. For our experiments we used an RTX 3080ti GPU.
4.2 EXPERIMENTAL. RESULTS
Reducing oversmoothing:
Table I presents the classification performance of the two base models (GCN and GAT) on all four
datasets, with and without the modified slope of ReLU (called Slope2GNN) for varying number of layers. Additionally, it presents results using SeLU, instead of ReLU, since it also modifies the slope of the function, while additionally normalizing node representations, as explained in section 2.4 To the best of our knowledge, this is the first time SeLU is used with GNNs.
[sows that methods with a modified activation function reduce oversmoothing significantly. As expected, baselines maintain high performance for shallow architectures in homophilic datasets, where oversmoothing is not a problem. The modified activation functions (Slope2GNN and SeLU) consistently improve the testing accuracy as the number of layers (# L.) increases. Note that even the proposed method suffers performance degradation for very deep GNNs, due to the unavoidable nature of the oversmoothing (Oono & Suzuki, 2020). GCN and GAT average over information in the close neighborhood of the node, assuming homophily, which is not the case in the Texas dataset. Interestingly, low homophily in this dataset makes oversmoothing appear at larger depth compared to other datasets. This is because connected nodes do not have similar representations and the model needs more Laplacian operations to oversmooth them.
The effect of deeper networks on different activation functions:
Focusing on the use of SeLU, although it seems to make the models resistant to oversmoothing, the effect seems to be lost for GCNs with 64 layers. This may be due to the sensitivity of the function on the choice of slope value, which is lower than 2. Moreover, the saturating region (part of SeLU used to reduce variance of data, if needed) of SeLU might further reduce the variance of node representations, acting in favor of oversmoothing. Finally, SNNs aim to keep the variance of the model stable, ignoring the effect of the Laplacian smoothing in GNNs.
To further examine the behavior of each method, as the number of layers increases, in Figure [ we present more detailed progression of the accuracy of GCNs, as the networks get deeper on the Cora dataset. The effect of oversmoothing and its reduction by the two methods that modify ReLU is very apparent here. Furthermore, the simpler Slope2GNN approach seems to be more resistant to the increase in the depth of the network.
The need for deep networks:
Having shown the benefit of using the modified activation functions in deep GNNs, we move to a set of experiments that aims to highlight also the value of using such deep architectures. In par- ticular, we report our experimental results in the "cold start" scenario, as described in section 3.5 The cold-start datasets that we use in these experiments are generated by removing feature vectors

---

Under review as a conference paper at ICLR 2023
Accuracy (%)
Cora
CiteSeer
Pubmed
Texas
# Layers
Method
GCN
GAT
GCN
GAT
GCN
GAT
GCN
GAT
Original
81.38
77.79
70.52
69.04
77.65
77.33
59.01
59.54
2
Slope2GNN
81.84
77.81
70.55
68.24
78.34
77.02
57.93
56.67
SeLU
81.74
76.59
69.91
66.81
77.80
76.83
59.01
57.21
Original
78.09
78.06
65.21
63.98
76.75
76.34
59.55
58.28
Slope2GNN
80.39
79.54
67.57
67.27
76.46
75.59
57.66
57.39
SeLU
79.85
79.33
67.53
67.24
74.09
72.62
58.02
58.65
Original
23.56
33.11
32.54
31.33
49.08
55,86
57.48
57.65
Slope2GNN
76.54
78.33
65.80
65.55
75.00
74.20
57.18
56.85
SeLU
79.41
78.19
67.27
67.58
73.43
71.93
56.85
57.47
Original
14.92
15.53
17.73
17.80
28.95
29.88
39.91
41.71
16
Slope2GNN
76.81
75.00
57.10
56.13
76.35
73.81
57.21
57.53
SeLU
76.38
74.92
61.99
62.23
76.11
74.27
52.88
57.44
Original
14.02
14.67
16.69
18.67
38.49
29.34
25.50
18.64
32
Slope2GNN
73.21
69.62
47.20
49.16
75.78
74.63
55.79
54.14
SeLU
68.77
69.36
47.90
52.97
71.80
73.60
55.77
57.75
Original
12.48
15.18
17.65
13.35
31.73
28.62
27.39
06.76
64
Slope2GNN
61.59
24.17
46.46
26.24
71.68
38.01
46.31
41.44
SeLU
26.85
30.55
27.10
20.42
40.90
41.00
58.38
56.30
Table 1: Performance comparison of vanilla GCN and GAT against SeLU and Slope2GNN enhanced versions the same models, in Cora, CiteSeer, Pubmed, Texas. Average test node classification accuracy (%) for networks of different depth. With bold is the best performing model for each depth and each dataset.
Comparison of different activation function Slope2GNN Results
Results
SetU Results
0.
0.6
033
0.1
20
50
Model Depth
Figure 1: Comparison between three different activation functions for a GCN on Cora dataset. Y- axis shows model accuracy test nodes, while varying the model's depth (shown in x-axis).
from unlabeled nodes and replacing them with all-zero vectors. For each combination of GNN and activation function, we present in Table the best performance achieved and the depth at which the model attains that performance. The main observation in these results is the improvement in terms of accuracy with the use of deeper GNNs. This improvement is only attainable with the modified activation functions that reduce the effect of oversmoothing. The vanilla versions of GCN and GAT cannot go further than the performance of their shallow versions. Worth-mentioning is also the fact that the simple Slope2GNN approach manages benefit from the deeper architectures even in the hard Texas dataset. This is not the case for SeLU, which may require tuning of the slope of the activation function, discussed above.
RELATED WORK
Li et al. 20188) were the first to introduce the problem of oversmoothing in GNNs. An initial ap- proach of activation function alternatives to avoid reduction of the rank of the feature space is the
8

---

Under review as a conference paper at ICLR 2023
Model
GCN
GAT
Dataset
Method
Accuracy (%)
#L
Accuracy (%)
#L
Original
64.85
59.74
3
Cora
Slope2GNN
73.47
19
72.78
20
SeLU
73.00
22
72.65
23
Original
41.94
38.50
CiteSeer
Slope2GNN
49.88
21
49.63
26
SeLU
49.30
19
49.38
20
Original
60.16
50.45
Pubmed
Slope2GNN
72.61
32
71.14
32
SeLU
72.50
23
71.41
26
Original
32.40
4
30.10
2
Texas
Slope2GNN 33.33
31.00
SeLU
31.62
30.81
2
Table 2: Comparison of different models and activation functions on the "cold start" problem. We show accuracy percentage (%) for the test set using GCN and GAT as backbone GNN models. Only the features of the nodes in the training set available to the model. We also show at what depth (i.e. # Layers) each model achieves its best accuracy.
one provided by Luan et al. (2019). where tanh is used instead of ReLU. Subsequent work, (Oono & Suzuki 2020: Cai & Wang (020) further analyzed and theoretically proved the existence of the problem and showed that it unavoidable as the depth increases, Xu et al. (2018) proposed Jumping Knowledge Networks (K-Networks) as the first attempt to address the problem, by injecting skip connections to the GNNs. The model kept information from lower layers and combined them di- rectly with the final layer, before labeling the node. Following a similar approach, Chen et al. (2020) proposed the of residual connections between layers to enable deep architectures and alleviate the problem. Both these approaches explicitly inject part of the initial information into higher layers of the network, implicitly downgrading the importance of intermediate layers. On the other hand, our method enables deep architectures without the need to inject initial information and allows the model learn how to compress the initial information and what amount of maintain. We do not compare against these methods, because the aim of this work is to show that oversmoothing could be avoided without sshort-circuiting initial information to latter layers.
DropEdge (Rong et al. 2020) and DropNode (Do al. 2021) are two alternative methods that address oversmoothing by altering the graph topology. particular, they remove either edges or nodes at random, in order slow down the message passing speed. In contrast to such approaches, our method does not change the graph topology. which may have unpredictable effects, but rather relies small adjustment of the slope of the activation function. An additional advantage of the approach proposed in this paper is that it does not have any syper-parametent to be tuned, in contrast all of the alternatives mentioned above. Hyperparameter tuning is also required by the Pairnorm method, which was mentioned in section 2.4
6 CONCLUSION
In this paper we have shown the important role that the slope of the ReLU activation function plays in the oversmoothing problem in GNNs. We have further proposed a simple modification that reduces drastically the problem. We have illustrated the benefits of the approach in set of experiments with different datasets and GNNs of different depth. Additionally, we showed the improvement in accuracy one can achieve through deep architectures that do not suffer from oversmoothing. This was evident in set of experiments that simulated the "cold start" problem of missing node features GNNs.
The simple link between the activation function and oversmoothing unveiled in this paper, opens up range of interesting hypotheses to be investigated in the future. In particular, we would like to study the effect of changing the activation function in various GNN architectures and learning methods. Additionally, we would like to study altermative activation functions that could provide further benefits. Finally, aim to study in more detail the approach of varying LR per layer.

---

Under review as a conference paper at ICLR 2023
REFERENCES
Lei Jimmy Ba, Jamie Ryan Kiros, and Geoffrey E. Hinton. Layer normalization. abs/1607.06455 2016. htKllttt:////xxi..oo
CoRR,
abs/ 1607. 06450
1607.06450
Yoshua Bengio, Patrice Y. Simard, and Paolo Frasconi. Learning long-term dependencies with gradient descent is difficult. IEEE Trans. Neural Networks, 5(2):177-166, 1994. doi: 10.1 1091 72.279181.
Chen Cai and Yusu Wang. A note over-smoothing for graph neural networks. CoRR, abs/2006. 13318, 2020. URL
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Proceedings of the 37th International Conference on Machine Learn- ing, ICML 2020. 13-18 July 2020, Virtual Event, volume 119 of Proceedings of Machine Learning PMLR, 2020. URL WRLhhtp::// http:://ppooeeeinnnn.....rrr... mlr. ptt.s//11s...
Research, pp. chen20v. html
Tien Huu Do, Due Minh Nguyen, Giannis Bekoulis, Adrian Munteanu, and Nikos Deligiannis. Graph convolutional neural networks with node transition probability- -based message passing and dropnode regularization. Expert Syst. Appl., 174:114711, 2021. doi: 10. 00.1016/j.esaa2221.. 114711. /ttt/t:/p///o/ wtw/:.i...g....ooo 10e11111..... eswa 2021. 114711
William L. Hamilton, Zhitao Ying, and Jure Leskovee. Inductive representation learning on large graphs. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 30: Annual Conference on Neural Infor- mation Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pp.
www..ngg......nu.ne... ips. CC/ wccc//ppee/211///aaaae 1024-1034, 2017. URL wew.ea/77eeeeeaeeeeaa Abstract .htaalctcttmlll.
Sergey Ioffe and Christian Szegedy. Batch normalization:
Accelerating deep network training by reducing internal covariate shift. In Francis R. Bach and David M. Blei (eds.), Proceedings of the 32nd International Conference on Machine Learning, ICMI. 2015, Lille, France, 6-11 July 2015, volume 37 of JMLR Workshop and Conference Proceedings, pp. 448 456. JMLR.org, 2015. URL. http:://proceedings mlrr wwtps//////ooeeeeeeee
Thomas N. Kipf and Max Welling. Semi-supervised classification with graph convolutional net- works. In 5th International Conference on Learning Represemaations, ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings. OpenReview.net, 2017. URL https: //openrevies ew www.//frur// dot.
Günter Klambauer, Thomas Unterthiner, Andreas Mayr, and Sepp Hochreiter. Self-normalizing neural networks. CoRR, abs/1706.025155 2017. URL http://arxiv.orr/aab///1
02515
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Personalized embed- ding propagation: Combining neural networks on graphs with personalized pagerank. CORR, abs/1810.05997, 2018. URL/http://arxt org/ abs/1810. ottps/////aaaa/a////////////
05997
Johannes Klicpera, Janck Groß, and Stephan Günnemann. Directional message passing for molec- ular graphs. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL https::
Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Sheila A. McIlraith and Kilian Q. Weinberger (eds.), Proceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), and the Sth AAAI Symposium on Educational Advances in Artificial Intelligence (EAAI-18), New Orleans. Louisiana, USA, February 2-7, 2018, AAAI Press, 2018 URL https: www. aaai. orgg /cs /w/iiinxxx../ php/ 6098
pp.
AAAT AAA.AAAATTTTA BAAT/AAAT18/papr////1
10

---

Under review as a conference paper at ICLR 2023
David Liben-Nowell and Jon M. Kleinberg. The link prediction problem for social networks. In Proceedings of the 2003 ACM CIKM International Conference on Information and Knowledge Management, New Orleans, Louisiana, USA. November 2-8, 2003, pp. 556 559. ACM, 2003. doi: 10.1145/956863. 956972. URL htt.ttts:// //c::///do. orgeloorgrlio 10. 10. 11 45/ 41/956663, 0
Lu Lu, Yeonjong Shin, Yanhui Su, and George E. Karniadakis. Dying relu and initializaaton: Theory and numerical examples. CoRR, abs/ 1es/1993.06733, 2019. URL
1903. 06733
Sitao Luan, Mingde Zhao, Xiao-Wer Chang, and Doina Precup. Break the ceiling: Stronger multi-scale deep graph convolutional networks. In Hanna M. Wallach, Hugo Larochelle, Alina Beygelzimer, Florence d' Alché-Buc, Emily B. Fox, and Roman Garnett (eds.), Ad- vances in Neural Information Processing Systems 32: Annual Conference on Neural Informa- tion Processing Systems 2019, NeurIPS 2019, December 8-14, 2019, Vancouver, BC, Canada. pp. 10943 10953, 2019. URL https: https::/proceeednngggggoooo neurips. httppppscccpppppppppppaeeee 1ttml
Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL mert/foummiin frformmiidd) d02EFFP
Hongbin Pei, Bingzhe Wei, Kevin Chen-Chuan Chang, Yu Lei, and Bo Yang. Geom-gen: Geometric graph convolutional networks. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.ner, 2020. URL. https:
//openzeview.
Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In 8th International Conference Learning Rep- resentations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
htttps://cpenreeviieet net Forum? fowrmmmiddtkxxlkkee........
Laurens van der Maaten and Geoffrey Hinton.
Visualizing data using t-SNE. Journal of Ma- chine Learning Research, 9:2579 2605, 2008. URLhttp://ww// jml 1.. wwtppppers.//popooeooo /epeeersvvv
wandermaaten html
Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph attention networks. In 6th International Conference on Learning Representations, ICLR 2018, Vancouver, BC, Canada, April 30 May 3, 2018, Conference Track Proceedings. OpenReview.net, 2018. URL wwwooopeeeiiieeee / foet///ormmddd-XXXpp
Felix Wu, Amauri H. Souza Jr., Tianyi Zhang, Christopher Fifty, Tao Yu, and Kilian Q. Weinberger. Simplifying graph convolutional networks. In Kamalika Chaudhuri and Ruslan Salakhutdinov (eds.), Proceedings of the 36th International Conference on Machine Learning, ICML 2019, 9- 15 June 2019, Long Beach, California, USA, volume 97 of Proceedings of Machine Learning Research. pp. 6861-6871. PMLR. 2019. URL http: :ttE:://proceedingg.o mir mww.ppr...p.// wttp://wwweiiinnmm...eeeeeeeeeee
Wul 9e ht.l
Keyulu Xu, Chengtao Li. Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Ste- fanie Jegelka. Representation learning on graphs with jumping knowledge networks. In Jen- nifer G. Dy and Andreas Krause (eds.). Proceedings of the 35th International Conference on Machine Learning, ICML 2018, Stockholmsmissan. Stockholm, Sweden, July 10-15, 2018, vol- ume 80 of Proceedings of Machine Learning Research, pp. 5449-5458. PMLR, 2018. URL http: /ttp://e/cccce..s.c/// .ww...r....mmm ppress/v80) htl//uu888 xul8c. html
Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. In Samy Bengio, Hanna M. Wallach, Hugo Larochelle, Kristen Grauman, Nicolò Cesa-Bianchi, and Roman Garnett (eds.), Advances in Neural Information Processing Systems 31: Annual Conference on Neural In- formation Processing Systems 2018, NewrlPS 2018. December 3-8, 2018, Montréal. Canada, pp. 5171-5181, 2018 URL ht h oat. neurisnssssnnnnnnssop wwtpeeewiss..p.papeeeeeeeeeeeeee 33Fo77c331a09b333222444999922223 2tw.a//000000022222422222-...... http://ps...pppppp//////////////

---

Under review as a conference paper at ICLR 2023
Muhan Zhang, Zhicheng Cui, Marion Neumann, and Yixin Chen. An end-to-end deep learning architecture for graph classification. In Sheila A. Mcllraith and Kilian Q. Weinberger (eds.), Pro- ceedings of the Thirty-Second AAAI Conference Artificial Intelligence, (AAAI-18), the 30th innovative Applications of Artificial Intelligence (IAAI-18), 18), and the 8th AAAI Symposium on Ed- ucational Advances in Artificial Intelligence (EAAI-18), New Orleans, Louisiana, USA, Febru- ary 2-7, 2018, pp. 4438-4445, AAAI Press, 2018. URL https: /tt:://wwwww...
www. aww. hwtp://wwwaaa..ggoccoo//////////
index. Pww.xx.pppAAAAA.A AAAII8 www.s/ppppppaaaaa
17146
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net, 2020. URL
APPENDIX
A: LEMMA PROOF
Lemma For network of depth dep the total gradient reaching to the 1-th layer iie., order to update W!)) bounded by:
as
in
dJ
S (dep-l) Gt
(1) dw otds.
Where is the model's loss function (i.e. Cross Entropy), stands for the ReLU slope and G1 is the upper bound of gradients of the output of the subsequent layer (I+ with respect to the weight elements of W())
Proof: Firstly, let define the gradient of the output of a ReLU function with respect the input as follows:
((ReLU(x))
{ a, x 0,
dx
0
Let us consider now a function of nested ReLUs, like the neural networks under investigation, i.. fz) ((o(g((((g(((((()))))))))) and g() is function that multiplies its input with a value. Then the gradient of f(-) with respect to (x is vector) given as:
where a--) is the ReLU function applied repeatedly times,
d(f(z)) dr
a=
d(g(a)) dz
g(x) > (,)((())))
0,
otherwise
This formula indicates the relation between slope, layer index, depth and largest singular value (we only care about the largest). Let the notion of J for the loss function of the network (i.e. Cross Entropy). The relation comes from the following formulas (for each weight element in the weight matrix):
w news, w oodd,, + dJ
dw (i) olds
dJ
(depth-1) (O(++) or 0
dw
(1) old..;
w Wod,.,
Where the 0 value comes from Equation GNNs' forward pass is similar to f(-), hence performing the backpropagation leads to gradient calculation of the form of Equation 7 So the update rule adds (or subtracts) to each weight big number (contains exponential factor). We prove, that dJ tends to be closer to zero as the layer index gets smaller, because when the gradient travels dwaldd backwards the more distant it travels the more probable it is to die. There exist many components (gradients of ReL.U functions) in the total gradient it is probable one of them be zero and the
12

---

Under review as a conference paper at ICLR 2023
total gradient be zero. Given Assumption 1, we get an upper bound regarding the gradient of the output of the subsequent
layer with respect each weight element of w i..., So will have that:
dou+2)
20 S Gi
dJ
11 dwaddss
1 (depthh1)) Gi S (deptth-1) Bt
(8)
where B, is an upper bound on the product of the learning rate with G1, i.. Bi = 17 Gt.
B: LEMMA 2 PROOF
Lemma 2 While model's 's loss, through gradients, flows backwards some weight elements do not receive updates, because we have dying ReLUs (i.e. ReLUs outputting zero) (Lu et al. 2019). The total number of weight elements getting updated at layer is bounded by:
#(w (1)
(dep- (dop-1))d
where d is the largest of the two dimensions of W i.e. there are at most df elements in W( if a square matrix.
Let us assume that the number of weights that get updated reduced as the gradient flows backward. This is due to the fact that, the further the gradient 'travels' the more gradients of ReLUs will exist within Hence greater is the probability at least one of them to zero. In fact, we define the probability of weight element to get updated and the number of weight elements to be updated layer
Proof:
(1) = updoled) = p(depth-1) P(
#(w p(lepthh)) . (i. S. (ddepth-l) de
(9)
Where j the dimensions of the weight matrix and d is the largest of them. The probability for weight element to get updated has the aforementioned form, because we have used the Assumption 2 regarding the probability of ReLU unit
output zero, which is at most Thus, the probability of getting updated is the probability of the gradient flowing backwards to reach the weight element, which in turn means not have any ReLU component equal zero. The total number of elements that get updated is upper bounded (using union bound) on the sum of the probabilities all elements of the weight matrix. In fact, every weight matrix has i elements where i, are the matrix dimensions and max(i,
C: THEOREM 2 PROOF
Theorem The upper bound of the largest singular value of the weight matrix W at layer I for GNN model, utilizing ReLU activation function, depends the slope of the function. That bound is given per layer and shows the effect of each iteration of updates on the weight matrix. We denote with Wotd old and Wnew the weight matrices before and after the update during an iteration of the training process respectively.
max(s:(W)) )) $1I 1. + 13. 4 (T) d (dep-l) B
new
where B1 G1. dep is network's 's depth, d is the largest dimension of (()) matrix, is the upper bound of the probability of ReLU not to output zero and a is ReLU's slope.
Proof: Regarding the singular values matrix (denoted by si), is known that:
moz(s((W(1)) || ||
|w
13

---

Under review as a conference paper at ICLR 2023
the matrix here being W the weight matrix at layer 1. Based on Lemma 1, new weight, during
the weight update process will be given by: Wnew (Idepth-1). Br. Since only # #(w weight
elements are getting updated per layer (Lemmaa 2), in each iteration the Frobenius norm increases by
a value, that depends on layer index, depth and slope. Specifically, if we define a (depth-)) Bi Ki. in order simplify the formulas, we have:
maz(s: (Wi
|| new ||
(1) Wineuss 12
ma new! z
Wunews, 12
Σ not updated
(1) Wnnwur
updated
max(s:( (W S
Σ
(wi
+ Kil2
Σ notaupated
w ( old..
12
oldus
updated
max( (st (W (ow))
Σ
(w (
12 Kil2 + 2w olds,
Ki)
+
Σ not.upaatdd
(1) old.,
uppltted
Using the fact that w old..;
(2)
Ki. because K₁ contains an exponential term and weights are
initialized values close to zero we get:
(W)l) remm
=
Σ updated
(Iw
+ 3|K/
+
Σ
(w
old,, 12
VA=BEVAtVB
not.updated
mar(s((W(0) n)w
Σ
Iw
(1) atd
12+
Σ
( old
Σ
3|Ki|
Lemma2
updated
.updated
updated
In the second square root the updated weight elements and Ki is independent of them. Hence, we get:
maz(s:(W:il)) (Wew)) $ || (0) 3//K Hw!!))
Marr(s(WWih new)) < || otalle + v3. |Ki| .pl donth d
max(s (W)l)) )) < || dille + v3
depth )). d (depth-t). B
So the slope of the function determines the upper bound of the Frobenius norm while train- ing. In turn this norm is directly connected as an upper bound to the largest singular value of the matrix. the aforementioned proof have used previously defined Lemmas and Assumptions.
D: MODIFYING LEARNING RATE: LR2GNN
As discussed section 3.4, an alternative to changing the slope ReLU to modify the learning rate. In order to test this hypothesis and compare the results to the modified activation function, we performed some preliminary experiments with GCN. In these experiments, we determined different learning rate per layer, using the validation part of each dataset and the chosen values
14

---

Under review as a conference paper at ICLR 2023
shown in Table 3 We did this only for the first 8 layers, the process of tuning the rate proved very cumbersome and time-consuming. The main observation seems to be the improved performance the vanilla GCN for the three simpler datasets, coupled with low accuracy for the Texas dataset. The comparison against the modified activation function seems inconclusive, but the latter approach wins due its simplicity. If one opted for the modified learning rate, method to automatically adapt the rate in different layers would be needed. Deeper architectures extend the complexity and the pool, from which we would have find the proper values of learning rates. Worth to mention, combining different values of LR per layer with Slope2GNN results in negligible improvements.
Layer index
3
5
Learning Rate
3 10
10
10-5
10
5
10-5
10
10
10
10
Cora Accuracy: 76.43
CiteSeer Accuracy: 64.96
Pubmed Accuracy: 78.82
Texas Accuracy: 16.32
Table 3: Preliminary results of modifying the learning rate of GCN Cora. CiteSeen Pubmed and Texas datasets. The graph displays test accuracy of an 8-layer GCN, using the aforementioned learning rates.
E: DATASETS STATISTICS
Datasets
Nodes
# Edges
# Classes
# Features
Cora 2708
5429
7
1433
CiteSeer
3327
4732
6
3703
Pubmed
19717
44338
3
500
Texas
183
309
5
1703
Table 4: The statisties of all datasets used in this work.
15

---

Under review as a conference paper at ICLR 2023
F: EXTENDED EXPERIMENTS
Extended experimentation, regular datasets:
Table is expanded version of Table [1 showing average test node classification accuracy along with the respective standard deviations. We have also included GCNII (Chen et al,, 2020), which one of the best performing GNNs utilizing residual connections. GCNII reduces oversmoothing through "short circuiting' initial information to subsequent layers of the model. The aim of the comparison against GCNII show that by changing the slope of ReLU simple models are capable of producing comparable (up to limit) results to more sophisticated architectures, that use residual connections.
Accuracy (%) and standard deviation
Layers
Dataset
Method
8
32
64
GCN
81.3
=8.0
23.56=9.2
14.92+6.4
14.02+3.6
12.48+2.9
GCN(SIoPe2GNN)
80.39 00.8
26.54
76.
23.21
61.59+4.7
GCN(SELU)
81.74.0.4
79.85 .0.7
79.41
76.38
68.7715.7
26.8514.0
Cora
GAT
77.7941.5
78.06=1.6
19.2
15.5319.0
13.1815.5 14.67
GAT(Slope2GNN)
77.
79.54
1.2
78.33
75.00
69.62
24.1749.5
GAT(SELU)
2..59
79.33
78.19
74.92+1.0
69.36+4.3
30.55++..9
GCNII
82.3210.6
82.89
83.99 20.5
84.77.0.7
84.9620.
85.3940.6
GCN
70.52+0.8
65.21
+2.3
17.73+4.9
16.69
17.65+4.2
GCN(Slope2GNN)
70.55+0.4
67.57
-0.7
65.80
57.10+4.3
47.20
46.4614.2
GCN(SELU)
69.91
67.53
67.27
61.99+1.9
47.90
27.1047.6
CiteSeer
GAT
69.04+1.4
63.98=2.
31.33=7.0
17.80+1.
18.67
13.35+4..3
GAT(Slope2GNN)
68.24
67.27 110
65.55
56.13
26.24 49.1
GAT(SELU)
66.81
67.24
67.58
62.23
52.97 26.4
20.4212.8
GCNII
68.05 40.7
67.75
1.8
70.90
72.69
72.68
+0.4
72.77+0.9
GCN
77.65
76.75
49.08
28.9549.9
38.49
31.7329.9
GCN(Slope2GNN
78.34
76.46
1.0
75.00
76.35
75.78 22
71.68
GCN(SELU)
0..
74.09
73.43
76.11
71.80
40.9040.9
Pubmed
GAT
77.33+0.7
1.3 76.34
55.8627.5
29.88+9.0
29.34
28.6249.9
GAT(SSope2GNN)
77.02+0.7
75.
74.20
73.81+2.
38.01+5.7 74.63
GAT(SELU)
76.83
72.62
71.93
74.27
41.0010.33 73.60
GCNII
78.57
79.09 20.4
79.85
39.7010.3
79.77
79.6540.3
GCN
59.0112.
59.55
2.7
57.4826.3
39.91
27.3910.9 25.5020.6
GCN(Slope2GNN)
57.93142
57.66=5.0
57.1816.2
57.21
55.79
46.31
GCN(SELU)
59.01
58.0225.0
56.8526.2
52.8826.6
55.77
58.3816.8
Texas
GAT
58.28 19.54
26.5
57.65 16.7
41.7114.0
06.76.119 18.64
56.67
57.39 25.
56.85
57.53
41.4412.3 1..15
GAT(SELU)
58.65 57.21+5.0
5.8
57.47
57.44
57.75 26.4
56.30
GCNII
69.37+2.9
70.72
72.07
70.00+2.2
71.26
71.35+1..
Table 5: Performance comparison of vanilla GCN and GAT against SeLU and Slope2GNN enhanced versions the same models, in Cora, CiteSeer. Pubmed, Texas. We include GCNII (Chen et al.]
2020) in the comparison. Average node classification accuracy (%) and standard deviation for networks of different depth. With hold the best performing method for each model (i.e., GCN, GAT and GCNII), depth and dataset.
16

---

Under review as a conference paper at ICLR 2023
Extended "Cold start":
Table 6 is an expanded version of Table including GCNII's performance and the standard devi- ations for all models. The "cold start" problem requires deeper models, because nodes containing informative feature vectors are located further from each test node. Thus, architectures that utilize deep GNNs could effectively address the problem, if they avoid oversmoothing. Once again method effectively enables deep architectures using simple models (i.e., GCN, GAT).
Dataset
Method
Accuracy (%) & std
#L
GCN
64.85
GCN(SIope2GNN)
73.47
19
GCN(SeLUU)
73.00+1.5
22
Cora
GAT
59.74
GAT(Slope2GNN)
72.78
20
GAT(SeLU)
72.65
23
GCNII
73.40+1.0
GCN
41.9410.3
GCN(SloPE2GNN)
49.88
GCN(SeLU)
49.30+1.2
19
CiteSeer
GAT
38,50+4.7
GAT(Slope2GNN)
49.63
26
GAT(SeLU)
49.38
20
GCNII
59.8413.0
31
GCN
60.16.5.4
GCN(SIOPE2GNN)
72.61
GCN(SeLU)
72.50+0.8
23
Pubmed
GAT
50.45+s.9
GAT(Slope2GNN)
71.14+1.5
32
GAT(SeL.U)
71.41
26
GCNII
78.25+0.4
GCN
32.40+6.8
GCN(SIOPe2GNN)
33.33
GCN(SeLU)
31.62.1.9
Texas
GAT
10.60...5
GAT(Slope2GNN)
31.00
GAT(SeLU)
30.8145.8
GCNII
57.66:2..0
Table 6: Comparison of different models and activation functions the "cold start" problem. We show accuracy percentage (%) and standard deviation on the test set using GCN and GAT as back- bone GNN models and GCNII representative of residual GNNs. Only the features of the nodes in the training set are available to the model. We also show at what depth (i.e. Layers) each model achieves its best accuracy.
Slope sensitivity results:
The choice of slope (i.e., experimentally that this slope value helps to reduce the effect of oversmoothing. However, this is experimented with other slope values and
2) that proposed is based Equation 6 We have verified
not the only slope value that could be used. Therefore, present the results in Figure 2 Based on the Figure, it evident that any value between 1.8 and 2.2 reduces oversmoothing, making the exact choice of value less important.
17

---

Under review as a conference paper at ICLR 2023
Varying ReLU Slope results Cora
GCN Results
GAT Results
0.8
0.7
0.6
0.5
Slape: 0.8
Sloge: 0.8
0.4
Slape:
Slope:
Slape:
Slope:
0.3
Slope 1.4
Slope 1.4
Slope:
Slope: 1.8
Sloge: 1.8
Slope: 2.0
Slope: 2.0
Slope:
10
15
20
25
30
0
10
15
20
25
30
Model Depth
Model Depth
Varying ReLU Slope results Citeseer
GCN Results
GAT Results
0.7
0.6
0.4
Slope: 0.8
Slope: 0.8
Slope:
Slope:
Slope: 1.2
Slope
0.3
Slape:
Sloge:
Slope:
Slope: 1.8
Slope: 1.8
0.2
Slope:
Sloge:
Slope:
Slope:
5
10
15
20
25
30
10
15
20
25
Model Depth
Model Depth
Varying ReLU Slope results on Pubmed
GCN Results
GAT Results
0.8
Slope: 0.8
Slope: 0.8
Slope:
1.0
Slope: 1.2
Slope: 1.2
0.4
Slope 1.4
Slope: 1.4
Slope: 1.6
1.6
Slope: 1.8
Slope: 1.0
0.3
Slope:
Slope:
Slope: 2.2
Slope: 2.2
15
20
25
30
Model Death
Model Depth
Figure 2: Comparison of test performance accuracy (%) of GCN and GAT with varying slope values in Cora, CiteSeer and Pubmed. We show the accuracy of the models in y-axis, while increasing their depth.
18

---

Under review as a conference paper at ICLR 2023
Visualising the effect of oversmoothing:
In order to understand the effect of oversmoothing in GNNs and show how our method alleviates it, visualise node representations using t-SNE (van der Maaten & Hinton 2008 for varying depth models. Figures and 3 visualise the representations for different depth levels and different ReL. slope values. Ideally, would like nodes of the same color (same class) be close together and well-separated from other classes, In all three datasets, we observe how oversmoothing (at slope 1) leads to mixing of the classes. We also see how the situation changes for slope 2. In those graphs, node representations do not rapidly mix as the depth of the network increases.
19

---

Under review as a conference paper at ICLR 2023
Slope: on Cora
2-layer
4-layer
-40
8-layer
16-layer
20
-40
-60
-40
-20
20
-60
-40
-20
20
Slope: Cora
2-layer GCN
4-layer GCN
60
40
20
-20
8-layer GCN
16-layer GCN
60
20
-20
-40
-20
20
40
-40
-20
60
Figure 3: t-SNE of node representations of GCN and GAT on Cora, while increasing model's 's depth. Upper (lower) 4 figures show the results without (with) our method.
20

---

Under review as a conference paper at ICLR 2023
Slope: 1 on CiteSeer
2-layer
4-layer
40
20
-20
400
8-layer
16-layer
20
-20
-60
-40
-20
20
-60
-40
-20
Slope: CiteSeer
2-layer GCN
4-layer GCN
20
0-
8-layer GCN
16-layer GCN
20
0-
-20
-40
-60
-40
-20
40
-60
-40
-20
Figure 4: t-SNE of node representations of GCN and GAT on CiteSeer, while increasing model's depth. Upper (lower) 4 figures show the results without (with) our method.
21

---

Under review as a conference paper at ICLR 2023
Slope: on Pubmed
2-layer
4-layer
-10
-20
-30
-00
8-layer
16-layer
-10
-20
-30
-40
-40
-30
-2
-10
10
20
30
-40
-30
20
110
10
20
Slope: Pubmed
2-layer GCN
4-layer GCN
40
30
-20
-30
-40
8-layer GCN
16-layer GCN
40
30
20
-30
-40
440
-30
-20
10
40
-30
-20
-10
20
Figure 5: t-SNE of node representations of GCN and GAT on Pubmed, while increasing model's depth. Upper (lower) 4 figures show the results without (with) our method.
22


Reviews:
[{"review_id": "tRvFjhKK-q", "reviewer": "Reviewer_rfT6", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper consider the oversmoothing problem in deep GNNs and it showed activation function plays a crucial role in this phenomena. The paper proposes a simple modification to the slope of ReLU to reduce oversmoothing.  Their experimental results showed improvement on deep GNNs on classification tasks even though still a shallow GNNs works better than the deep network. They showed that deep GNNs, which do not suffer from oversmoothing, can be beneficial in the presence of the cold start problem. \n", "summary_of_the_review": "In general the theoretical analysis the paper provides sounds interesting and it sheds more light on controllable parameters in GNNs to delay oversmoothness but still it is not clear how useful it is in practice since shallow GNNs are still better real scenarios.", "strength_and_weaknesses": "Strength:\n1- The paper developed a simple approach that reduces oversmoothing and enables deep architectures. It is known that in deep GNNs repetition of the Laplacian operator acts as a variance reducer and causes the oversmoothing. The paper transfers the idea of SeLU to GNNs, focusing on the part that increases the variance to alleviate oversmoothing. \n\n2- The paper provides a theoretical analysis that the convergence to a subspace where node representations are the same can be controlled by two factors 1) slope of the Relu function and 2) modifying the learning rate.\n\n3- Since determining the right LR per layer is a non-trivial task, they focused on the slope of the Relu function. Increasing the slope of ReLU too much also causes exploding gradients and their analysis of the upper bound of the largest singular value of the weight matrix becomes too loose. \n\nWeakness:\n\n1- Even though the paper provides an interesting analysis that correlates the slope of the Relu and the learning rate with the largest singular value of the weight matrix which affects the oversmoothing phenomena, it is still not clear how it is useful in the real scenario given the fact that even their experimental results show that still shallow GNNs performs better in classification task.\n\n2- The paper claims that deep GNNs are beneficial in the presence of cold-start problems, but their experimental results sound artificial given that the features of the test data are removed. The cold start problem may appear on feature level (some features are missing) or edge level (some connections are not formed yet). So to show the impact of the deep GNNs I would suggest running some experiments either on a real dataset that has the cold start problem or evaluate different scenarios in existing datasets. Also it is not clear why when all features are removed why nodes which are further away are more important. \n\n3-  As mentioned in the paper the slope of the Relu affects the oversmoothing. That would be interesting how varying such slopes would affect the performance. It is mentioned that a large slope will cause exploding gradients. But I am wondering what the trend looks like as we increase the slope. Also can we say if the slope is smaller than one the oversmoothing happens earlier and all representations converge to a similar subspace?\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "qo8NG3Mumd", "reviewer": "Reviewer_WALS", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper analyses the phenomenon of oversmoothing in graph convolutional neural networks based on the graph laplacian. It propose a simple, but theoretically well-motivated, and experimentally well-validated solution to this problem. The solution builds upon previous theoretical analyses on how the variances in feature maps decreases with each graph convolution layer. Bounds are provided on how quickly this happens based on the singular values of the weight matrices. The theoretical analysis shows that one should simply scale the slope in ReLU activation functions with a factor 2. This solution greatly reduces the speed at which feature values converge to the oversmoothed sub-space in which feature eventually land. Thus, the simple trick allows to build deeper graph NNs.", "summary_of_the_review": "Considering the simplicity of the proposed solution, and the fact that graph convolutions are still very much prevalent in graph NN literature, the paper could have a high impact. Considering that the paper is also well written and sound, I recommend accept.", "strength_and_weaknesses": "**Strengths**\n\n1. The main idea of the paper is intuitive and simple, and can easily be incorporated in graph NNs.\n2. The idea is theoreticall well motivated, and experimentally validated.\n3. Although I did not check all proofs, most are based on existing results and the proof outlines sketched in the main text make sense, as such I believe them to be correct.\n\n**Weaknesses**\n\n1. The paper is, understandably, limited to graph NNs of the convolutional kind (based on spectral convolutions). Given the popularity of other graph NN approaches (s.a. message passing, transformers, ...) it would be interesting to learn how these results generalize to other type of graph NNs.\n2. Section 4.1, models. The paper uses relatively basic architectures, which is important to ablate the effect of the modified activation function, however, it would still be nice to have models with residual connections just to have a baseline. It would give insight on how important it actually is to go deep with only basic building blocks.\n3. The paper addresses the issue of oversmoothing, but only quantifies it indirectly be comparing test accuracies. Wouldn't it be possible to quantify the heterogeneity of features in graphs somehow, as a function of depth? E.g. via metrics present in this work:\n*Chen, D., Lin, Y., Li, W., Li, P., Zhou, J., & Sun, X. (2020, April). Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference on Artificial Intelligence (Vol. 34, No. 04, pp. 3438-3445).*\n4. Small typo: page 4 \"We use the notation of 'converging to a subspace'\" -> \"... notion of ...\"", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "TePQuU3875", "reviewer": "Reviewer_XeNT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper studies the relevance of the slope of the ReLU activation function in deep GNNs with respect to the over smoothing issue.\nThe proposed strategy is simply to use a higher slope for the ReLU, setting its value to 2.\nThe paper presents an experimental analysis in which this idea is applied to GCN and GAT on a number of benchmark problems, and a modified version called cold start (in which the input features are removed from some nodes).\n\nUpdate after rebuttal: I thank the authors for considering my concerns on their work. The rebuttal partially clarified doubts regarding the validity (especially) of the experimental analysis (and lack of fine-tuning). I am happy to slightly raise my score accordingly.", "summary_of_the_review": "The paper has the merit of highlighting the role of the ReLU slope on the over smoothing problem. However, in its current form there are several issues with the mathematical analysis (and presentation), as well as with the experimental section.", "strength_and_weaknesses": "Strenghts:\n- The paper highlights the relevant role of the slope of the ReLU activation function in the design of deep GNN, showing the potentialities of increasing the value of the slope as a simple way to counter-balance the oversmoothing effects.\n\nWeaknesses:\n- The mathematical analysis lacks some rigor in the determination of the exact value of the slope.\nIn Section 3, which represents the core of this contribution, the paper elaborates on the importance of the value of the ReLU slope (i.e., alpha) on the upper bound for the largest singular value of the weight matrix. In a rather vague way, and without a formal statement it is concluded that the right value is alpha = 2. While this is a consequence of intuitive reasoning (and supported by some vague sentences, like \"a slope of 2 makes the second exponential factor prevail over the first one\", without a reference to the equation), I could not see a mathematical argument for which alpha needs to be exactly 2. Again, in a non-rigorous way it is stated that \"we cannot make the slope too large\", and \"if we increase the slope of ReLU too much, the upper bound in eq. 6 becomes too loose\". How much is too much? Is there any mathematically grounded approach to show that any larger (or smaller) value than alpha = 2 leads to gradient problems? The main message I got from the paper is that the actual value of alpha would be better tuned instead of being fixed to 1. This could be probably be treated as a hyperparameter and chosen accordingly.\n- Hyperparameters tuning is essentially not performed in the experiments. Indeed, in Section 4.1 it is reported that the crucial values of the hyperparameters are all fixed (also, I could not find information on the used learning rate). Moreover, it could be interesting to tune the value of alpha as a hyperparameter.\n- Results in the experimental section are reported without std across repetitions, and without any analysis of statistical significance of the reported average accuracy.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JwME-gwce1A", "reviewer": "Reviewer_kwRc", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The authors build upon the recent analysis in [1], which provides a theoretical foundation for the over-smoothing phenomenon in graph neural networks, to propose an adaptation of the activation function to counteract this effect which harms performance. Specifically, they propose multiplying the relu by a factor of 2, to counteract an exponential decay factor of 0.5, which contributes to over-smoothing. The authors provide both theoretical and empirical evidence that their proposed method indeed mitigates over-smoothing, thus enabling deeper graph neural networks to be used without significant performance drops.", "summary_of_the_review": "The paper is well written, the problem statement well established, and the proposed solution well founded. However, it is clear that the method is insufficient to mitigate over-smoothing, and at this point, many new architectures do not suffer from it. I also think that the authors do not perform sufficient investigations into the generalizability of their proposed method to modern architectures and activation functions (later than 2018), nor do they provide in-depth investigations into the sensitivity of the proposed solution to different choices of relu factors.\n", "strength_and_weaknesses": "Strengths\n- The paper is well presented. The theorems are clear, and the theoretical foundation for the proposed method is well established.\n- Current experiments provide some support for the hypothesis that the method can mitigate over-smoothing in graph neural networks\n\nWeaknesses\n- Just a comment: The phenomenon of over-smoothing is a case where the graph feature maps become too smooth so that they lose the information necessary to make a prediction. It is not necessarily the phenomenon where the accuracy degrades as we add more layers. The latter can be caused simply by a failure to optimize, and not caused by a flawed architecture.\n  \n- The claims in this paper are too bold. Table 1 clearly demonstrates that over-smoothing still exists in the proposed method. \n\n- There are multiple new networks that do not suffer from over-smoothing. E.g., GCNII (2020), GRAND (Chamberlain et al., 2021), PDE-GCN (Eliasof et al. 2021), GraphCon (Rusch et al, 2022), EGNN (Zhou 2021). I am not sure that tackling the over-smoothing problem is still a pressing research question in this field. GCN and GAT are indeed over-smooth, but they are relatively old by now.\n\n- The authors do not compare their method to other GNNs.    \n- I believe that the authors do not provide sufficient investigations into the sensitivity and generalization of the proposed solution. Specifically, I believe several experiments are missing:\n\n1. Generalization to modern architectures and activation functions: current\narchitectures and activation functions explored in this paper are from no later than 2019. Though the authors explicitly state that they intentionally do not compare to\nmodern architectures which use skip connections, I do not agree that these methods\nare out of the scope of this paper. This is because the proposed method has the\npotential to provide benefits to a vast variety of graph neural networks in overcoming\nover-smoothing and I believe it is important to show that the method generalizes to\nother more modern architectures. \n2. Sensitivity: I believe additional experimentation is required to support the choice of\nthe relu factor of 2 in this paper, such as trying different values of this factor. For\nexample, if I understand correctly, eq. (6) suggests that a factor of sqrt(2) may be more appropriate in\ncounteracting the exponential decay of the upper bound.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject