Revisiting Over-smoothing in Graph Neural Networks
Authors: Pantelis Elinas
Edwin V. Bonilla
Venue: Submitted to ICLR 2023 (2023)

Abstract: Shallow graph neural networks (GNNs) are state-of-the-art models for relational data. However, it is known that deep GNNs suffer from over-smoothing where, as the number of layers increases, node representations become nearly indistinguishable and model performance on the downstream task degrades significantly. Despite multiple approaches being proposed to address this problem, it is unclear when any of these methods (or their combination) works best and how they perform when evaluated under exactly the same experimental setting. In this paper, we systematically and carefully evaluate different methods for alleviating over-smoothing in GNNs. Furthermore, inspired by standard deeply supervised nets, we propose a general architecture that helps alleviate over-smoothing based on the idea of layer-wise supervision. We term this architecture deeply supervised GNNs (or DSGNNs for short). Our experiments show that deeper GNNs can indeed provide better performance when trained on a combination of different approaches and that DSGNNs are robust under various conditions and can provide the best performance in missing-feature scenarios. 

Under review as a conference paper at ICLR 2023
REVISITING OVER-SMOOTHING IN GRAPH NEURAL NETWORKS
Anonymous authors Paper under double-blind review
ABSTRACT
Shallow graph neural networks (GNNs) are state-of-the-art models for relational data. However, it is known that deep GNNs suffer from over-smoothing where, as the number of layers increases, node representations become nearly indistin- guishable and model performance on the downstream task degrades significantly. Despite multiple approaches being proposed to address this problem, it is unclear when any of these methods (or their combination) works best and how they per- form when evaluated under exactly the same experimental setting. In this paper we systematically and carefully evaluate different methods for addressing over- smoothing in GNNs. Furthermore, inspired by standard deeply supervised nets, we propose a general architecture that deals with over-smoothing based on the idea of layer-wise supervision. We term this architecture deeply supervised GNNs (or DSGNNs for short). Our experiments show that deeper GNNs can indeed provide better performance when trained combination of different approaches and that DSGNNs are robust under various conditions and can provide best performance in missing features scenarios.
INTRODUCTION
Graph Neural Networks, first introduced by Scarselli et al. (2009), have emerged as the de facto standard for representation learning on relational or graph-structured data. GNÑs find many useful applications in building predictive models in traffic speed prediction, product recommendation, and drug discovery (Zhou al. 2020a; Gaudelet et al. 202T). One of the most important applications of GNNs is that of node property prediction, as in semi-supervised classification of papers (nodes) in citation network (see, e.g., Kipf & Welling 2017). In this case, we are given labels for a subset of the nodes and aim to learn an algorithm that can accurately predict the labels for the remaining nodes using the network structure and (if available) node features.
Even though GNNs, and especially those based on the Graph Convolutional Network (GCN) formu- lation (Kipf & Welling (017)) and its extensions (Veličković et al. 2018 Hamilton et al. 2017 Wu al. 2019|]Klcppers et al., (018), have been shown be powerful tool for graph representation learning, they are limited in depth, that the number of graph convolutional layers. Indeed, deep GNNs suffer from the problem of over-smoothing where, as the number of layers increases, the node representations become nearly indistinguishable and model performance on the downstream task deteriorates significantly. Increasing model depth is necessary in order allow information to travel between distant nodes in the graph where each graph convolutional layer corresponds to propagating information from node's one-hop neighbourhood.
Previous work has analyzed and quantified the smortsmgg problem (Liu et al. 2020 Zhao
Akoglu 2020 Chen al.) (020a) as well as proposed methodologies to address it explicitly (Li et al.] 2018 Zhao & Akoglu 2020) Xu et al. 2018). Some of the most recent approaches have focused on forcing diversity on latent node representations via residual connections (see, e.g,
Xu et al., 2018; Chen et al.] (020b), normalization (see, e.g, Zhou et al.) 2020b Zhao & Akoglu 2020), and enforced sparsity (see, e.g, Rong et al. (020). However, the proposed solutions have mostly been shown to alleviate over-smoothing but do not completely eliminate it, with shallow networks usually performing best. In this context, alleviating means that performance does not catastrophically deteriorate function of network depth. One notable exception to this is the

---

Under review as a conference paper at ICLR 2023
GCNII architecture (Chen et al. 2020b). which was shown improve the performance of standard GNNs on classification tasks when using deeper architectures.
One interesting scenario for the analysis of deep GNNs is when only subset of the nodes have features, which we will refer to as the missing-featuus setting (Zhao & Akoglu 2020). In this case, most of the solutions mentioned above (with the exception of GCNII) have been shown superior to the basic GCN architecture. Intuitively, as pointed out by Zhao & Akoglu (2020). large number of propagation steps (i.e., deeper GNNs) may be required to obtain useful feature node representations.
While acknowledging the significant advances towards making GNNs more robust to over- smoothing, we have found important gaps in the literature that, we believe, need to be recognized and addressed by the community. Firstly, and naturally. most previous work has focused on de- veloping new algorithms and showing that they outperformed previous approaches. While there is nothing inherently wrong with such approaches, this usually has come at the expense of empirical results across different algorithms not using exactly the same settings (such as hyper- sypee--aramettr op- timization). Secondly, we have also found that, in fact, all proposed solutions are general enough that can be combined together. However, these combinations and their performance have not been studied in detail. Finally, another crucial gap in the GNN over-smoothing literature is that, previous approaches that have tackled (scemingly different but) related problems in standard neural networks have not been investigated. In particular, refer to the work deeply supervised nets Lee et al. (2015) for learning discriminative and robust features and for dealing with vanishing/exploding gra- dients.
In this paper, (i) we address the above gaps by systematically and carefully eval- uating several proposed GNN over-smoothing solutions and their combinations. We analyze their performance in the transductive, semi-supervised node classification setting in both the standard fully observed and missing-fcature settings. Furthermore, inspired by the work of et al. (2015), (ii) we propose new general architecture for tackling over-smoothing, Our architecture trains pre- dictors using node representations from all layers, each contributing to the loss function equally, therefore encouraging the GNN to learn discriminative features at all network depths. We name our approach seeply-supervised graph neural networks (DSGNNs). (iii) We show that DSGNNs are resilient to the over-smoothing problem in deep networks and can outperform competing meth- ods on challenging datasets. Finally, (iv) we provide recommendations for the selection of a GNN architecture for practical applications.
Contributions:
2 GRAPH NEURAL NETWORKS
Let graph be represented as the tuple G= =V, where V is the set of nodes and E the set of edges. The graph has [V| = N nodes. We assume that each node E V is also associated with an attribute vector X₂ E Rd and let X Nxd represent the attribute vectors for all nodes in the graph. Let A W represent the graph adjacency matrix; here we assume that A is symmetric and binary matrix such that Aij E {0, where Aij if there is an edge between nodes and j. i.c., (v,, U1) E, and Aig = 0 otherwise. Also, let D represent the diagonal degree matrix such that D& === N Aij.
Typical GNNs learn node representations via a neighborhood aggregation function. Assuming a GNN with K layers, we define such a neighborhood aggregation function centred on node at layer
centred
1 as follows,
h = (f(g (onnh--- (h.--),, "(--",,("" Vu E &N..))
where No the of node v's neighbors in the graph, g is an aggregation function, is linear transformation that could be the identity function, and (!) is non-linear function applied element- wise. Let ( E Nx the representations for all nodes at the I-th layer with output dimension d() we set ()) def X and ()) del d. A common aggregation function y that calculates the weighted average of the node features where the weights are deterministic function of the node degrees ÁH as proposed by Kipf & Welling (0117). Here Å represents the twice normalized adjacency matrix with self loops given by A = 172 (A + 1)D- 1/2 where is the degree matrix for A + I and I E N is the identity matrix. Substituting this aggregation function in and specifying
2

---

Under review as a conference paper at ICLR 2023
X Nxd
X E R
Graph Convolution
Graph Convolution
H E
R2xx
H E R
Graph Convolution
Graph Convolution
H E R
H
Classifier
Classifier
Classifier
4 Z
Z
Figure I: GNN architectures for node classification. Left: the standard architecture using two graph convolutional layers but also shown with optional jump connections (dashed lines). Right: the proposed architecture with deep supervision.
be linear projection with weights W gives rise to the graph convolutional layer of Kipf & Welling (2017).
H = 10 (AH(!
where, as before, (1) non-linear function, typically the element-wise rectified linear unit (ReLU) activation (Nair & Hinton 2010).
Many other aggregation functions have been proposed, most notably the sampled aggregator in GraphSAGE (Hamilton et al. (017)) and the attention-based weighted mean aggregator in graph attention networks (GAT, Veličković al. 2018).
2.1 NODE PROPERTY PREDICTION
Equation is realization of I and constitutes the so-called spatial graph convolutional layer. More than one such layers can be stacked together to define GNNs. When paired with a task-specific loss function, these GNNs can be used to learn node representations in semi-supervised setting using full-batch gradient descent. For example, in semi-supervised node classification, it customary to use the row-wise softmax function the output layer along with the cross-entropy loss over the training (labeled) nodes.
Figure (left) shows diagram of the standard GNN architecture with optional jump connections as proposed in Xu et al. (2018). Furthermore, the last hidden layer can be followed by multi-layer Perceptron (MLP) that functions as the classifier. The MLP is optional when using the standard GCN architecture (Kipf & Welling, 2017) but necessary when employing jump connections. Given a suitable loss function such the cross-entropy for classification, we can train predictive models in semi-supervised setting for node-level tasks.
3
IN GRAPH NEURAL NETWORKS
Intuitively, deeper networks should allow information travel between distant nodes in the graph and we would like to learn node representations that take into account a larger neighborhood of a node. In other words, nodes further away in the graph should influence the node's representation and take advantage the graph's 's structure beyond the local neighborhoods. However, in practice, this is not the case due over-smoothing. As mentioned in section T over-smoothing in GNNs is

---

Under review as a conference paper at ICLR 2023
the phenomenon where model performance on the downstream task deteriorates significantly the number of layers increases. This can be attributed to the repeated application of the neighborhood aggregation operation and, therefore, node representations becoming nearly indistinguishable for very deep architectures.
3.1 PREVIOUS APPROACHES
We briefly overview some of the techniques proposed in the literature to address smoothing in GNNs, which we will use in our experimental evaluation.
Graph convolutional networks (GCNs): Kipf & Welling 2017) identified over-smoothing in deep GNNs when using GCN layers, showing empirically that such architectures performed best with only 2 or 3 layers, while performance degraded substantially for or more layers. They associated this problem with larger node-context sizes and overfitting, due to the significant increase in model parameters. They proposed the addition of residual connections for each layer such that the input node features where combined with the layer's node output features. Residual connections alleviated the over-smoothing problem but the best performance was still achieved using only 2 or 3 layers.
Jumping knowledge networks (JKNets): Xu et al. (2018)'s 's JKNets were specifically proposed as a solution to smootsont and centered on the introduction of residual connections. JKNet intro- duces residual connections from all hidden layers the output layer. Node attributes are combined using one of several schemes, c.g., concatenation and clement-wise max-pon moo-lio.ing. to generate node representations adapted independently to different context sizes for each node. JKNet was shown empirically to alleviate over-smoothing but best performance was limited to shallow networks.
Pair normalization (PN): Zhao & Akoglu 2020) provided first attempt at quantifying over- smoothing. They proposed 2 metrics that measure the pairwise similarity of latent node repre- sentations, row-diff, and the variance of latent node features, col-dif, across all nodes in the graph. The authors propose that the node representations output at each hidden layer should preserve the total pairwise squared distance (TPSD) of the input node attributes. They propose pair normal- ization (PN) layer that aims at keeping the TPSD constant across each hidden layer in the GNN. They demonstrate empirically that equipping GNN with PN layers alleviates mmothing but does not improve performance over shallow networks without layer normalization. In fact, adding pair normalization layers a shallow GNN is shown to decrease performance. However, they intro- duce the missing- feature setting where layer normalization is shown to improve model accuracy as a function of increased depth. In this setting, only the attributes for the nodes in the training set are known whereas the attributes for the nodes in the test set are unknown. We note that, in this setting, model performance is lower than fully-observed setting.
Group normalization (GN): Generalizing the pair normalization idea, differentiable group normal- ization (Zhou et al. 2020b) introduces a constraint that attempts to force latent node representations into groups such that nodes with similar labels cluster together whereas groups of nodes with differ- ent labels are well separated. Such grouping of latent representations should benefit the performance of downstream classifier. GN alleviates over-smoothing as network depth increases but does not help improve performance over shallow models. However, on the feastne-fcature setting, GN was shown to outperform pair normalization and benefit from added network depth.
The GCNII architecture, as proposed by Chen et al. (2020b), extends GCN in two ways. First, it modifies the residual connections to carry information from the input features to each hidden layer rather than the latent representations of the previous hidden layer. Second, it introduces an identity mapping to the hidden layer weights as initially proposed by [e et al.] (2016). The latter guarantees that GCNII will perform as well as an equivalent (in terms of number and size of hidden layers) GCN model. GCNII the only architecture shown empirically to benefit from network depth. However, this architecture has not been evaluated in the missing-fcature setting where it is not clear the lack of input features will limit its performance.
GCNII:
Unavoidably, there are other techniques for dealing with over-smoothing in GNNs that we do not investigate here. Of notable mention is the work of Rong et al. (2020), who proposed DropEdge as general heuristic for alleviating over- smoohhnng by modifying the message passing mechanism in GNNs. The core idea is to prevent some of the information from propagating through the graph by randomly eliminating subset of the graph's edges. At each training epoch, a random subset of the
4

---

standard neu- Under review as a conference paper at ICLR 2023
graph's 's edges are removed preventing information flow through them; these edges restored for the next epoch before another subset of edges is removed. They show that DropEdge is effective at alleviating over-smoothing and that small performance boost is possible for shallow GCN models augmented with DropEdge. However, they do not evaluate DropEdge in the low-label regime that is the most common evaluation setting in the GNN literature.
3.2 POTENTIAL PITFALLS OF PREVIOUS APPROACHES
One common characteristic of all these previous approaches to over-smoothing in GNNs is that they have not been evaluated using a common framework. For example, some methods are evaluated in the fully-observed setting using random splits whereas others have been evaluated in the low label regime using the node splits published by Yang et al. (2016). In addition, few of these works have been evaluated both the fulll-observed and the missing-featun of the different methods have not been studied empirically and, for example, is unclear if adding layer normalization GCNII will further improve hinder downstream task performance.
settings. Lastly, combinations
is certainly not our intention to focus on the potential flaws in previous approaches to over- smoothing in GNNs. We recognize machine learning is heavily driven by practical performance and that the methods mentioned above have provided significant advances to the field. However, it imperative that we evaluate our methods fairly and provide advice practitioners and researchers on best practices and what works best under different scenarios. In fact, the difficulty of consistently evaluating GNNs has been considered previously in works such as Shchur et al. (2018) and Errical et al. (2019).
Despite the issues about fairness in evaluating GNNs raised the latter works, inconsistency in reporting GNN performance has persisted. For example, when Chen et al. (2020)) compares with prior works the performance values as reported in Fey & Lenssen (2019) and Rong et al. (2020) are re-used. As second example, Zhao & Akoglu (2020) uses 32 hidden units for each layer, [hou jet al. (2020b) uses 16 hidden units and Chen et al. (2020b) claims to tune the number of units as bypeer-arammeter but the range of values considered is not reported. For these reasons, and in the spirit of fairness, we perform a careful empirical evaluation using the same framework com- paring these methods, their combinations and our newly-proposed deeply-supervised graph neural networks (DSGNNs) for both the fully-observed and the miissing-featur settings. In the next sec- tion we describe the intuition underlying DSGNNs and elaborate the details on how to train these architectures.
4 DEEPLY-SUPERVISED GRAPH NEURAL NETWORKS
Deeply-supervised nets (DSNs, Lee al. 2015) were proposed a solution to several problems in training deep neural networks. By using companion objective functions attached to the output of each hidden layer. DSNs tackle the issue of vanishing gradients. Furthermore, ral networks with shallow architectures, deep supervision operates as a regularizer of the loss at the last hidden layer. Lastly, and more importantly, for deep networks, encourages the estima- tion of discriminative features at all network layers (Lee et al. 2015). Therefore, inspired by this work, we introduce deeply supervised graph neural networks (DSGNNs), i.c., graph neural network architectures trained with deep supervision. Thus, we hypothesize that DSGNNs are resilient to ovor-smog and test this hypothesis by evaluating and analyzing their performance in training shallow and deep networks in section 5
Figure I (right) shows a diagram of the proposed DSGNN architecture. We introduce jump con- nections from each hidden layer where node-level representations, as described in section 2 estimated and given to an MLP that predicts the class probabilities for each node. We note that we use one MLP for each hidden layer as in DSNs. Let ( = (1) (H()) denote the node representations output by lne-layer MLP attached to the I-th hidden layer where ((((()) is the soft- max function for classification task. Finally, given loss function, e.g., cross entropy, over true and predicted outputs, we formulate deyeeddependent losses and learn all model parameters by optimizing the average loss function across all layers.
5

---

Under review as a conference paper at ICLR 2023
4.1 NODE CLASSIFICATION WITH 2-LAYER NETWORK
As an illustrative example, here we consider a node classification problem with C classes using 2 GCN layers as shown in figure 1 (right). We given a graph represented the tuple G (V, E) where V is the set of nodes and the set of edges, as described in section 2 A subset of M nodes, Vi C V. has known labels. Each label represents one of C' classes using a one-hot vector representation such that Y E RMCC The node property prediction task is to learn a function f:V Y that maps node representations to class probabilities.
Consider the case of 2-layer GNN with GCN (Kipf & Welling 2017) layers. The node represen- tations output by each of the GCN layers are given by,
H 1) Rel.U(AXW")) H ReI.(ÁH(
(3)
(4)
where the ReLU activations are element-wise, Å are the edge weights given by equation 2 W()) are trainable layer weights.
and
Let each GCN layer be followed by linear layer with softmax activation calculating class proba- bilities for all nodes in the graph such that,
Z ( softmax (H (1) W()), 1= 1,2,
(5)
where ( are the class probabilities for all nodes as predicted by the Ith layer, and W are the layer's 's trainable weights.
Now we can compute auyer-dependent losses as:
C -- Σ Σ Y ooolog (Z (0)) 1= 1,2.
DEVi
For standard GNN, in order to estimate the weights (W ( W J. we optimize the cross- entropy loss calculated over the set of nodes with known labels only using C
Deep supervision adds a linear layer corresponding to each GCN layer in the model such that, in our example, the model makes two predictions for each node, ( and 2 We now estimate the weights (W(I), W(2), W(1), W(2),, }, and optimize the mean loss given by,
CN 2
(k) N
where, in our example, = 2. We estimate the model parameters using gradient-based optimiza- tion minimize the total loss in equation 7 Unlike Lee et al. (2015), do not decay the contribution of the surrogate losses as a function the training epoch. Consequently, at prediction time we average the outputs from all classifiers and then apply the softmax function to make single prediction for each node.
4.2 ADVANTAGES OF DEEP SUPERVISION
As mentioned before, over-smoothing leads node representations with low discriminative power at the last GNN layer. This hinders the deep GNN's ability to perform well on predictive tasks. DSGNNs circumvent this issue as the learned node representations from all hidden layers inform the final decision. The distributed loss encourages node representations learned at all hidden layers to discriminative such that network predictions do not rely only on the discriminative power of the last layer's 's representations.
Furthermore, deep supervision increases the number of model parameters linearly to the number of MLP layers. Consider a classification model with K graph convolutional layers, de dimensional node representations, and a single layer MLP, If the number of classes is C, then a DSGNN model requires K x dø x C' parameters more than standard GNN. This is in sharp contrast with other architectures such as that Liu et al. (2020). where the number of additional parameters grow quadratically with the number of GNN layers.

---

Under review as a conference paper at ICLR 2023
Table 1: Mean test accuracy = one standard deviation for the fully observed setting. The notation
+PN and +GN indicates the addition of pair and group normalization layers respectively. In paren- theses we indicate the architecture depth that achieved the highest test accuracy.
Method
Cora
Citeseer
Amazon Photo
Pubmed
GCN
82.3+0.5 (3)
70.4+0.7 (2)
90.7=0.5 (2)
0.. (2)
JKNet
81.6+0.6 (2)
69.2+11.4
90.3=0.4 (8)
79.6+0.6 (2)
DSGNN
82.610.6 (3)
69.9 (2)
90.6=0.6 (3)
78.8 (4)
GCNII
83.1 (32)
72.1 = (.4(32)
92.0 0.3
80.1 (16)
GCN+PN
78.4 (2)
62.3 (2)
90.0=0.4 (2)
77.2:10.6 (2)
DSGNN+PN
77.7 (16)
63.1 (2)
90.6 (3)
76.9 (32)
GCNII+PN
80.9+1.0
62.7:22. (16)
0.6
76.0++1.00)
GCN+GN DSGNN+GN
83.1+0.7 (2)
69.3+0..
9..7=0.5
79.6+ (3)
82.6++0.6 (3)
70.4+00.7 (2)
9..6 (4)
79.4+0.6 (4)
GCNII+GN
83.6 0.7(32)
71.9 (32)
92.0 = 0.4(8)
80.2 0.3
Table 2: Mean test accuracy + one standard deviation for the missing features setting. The no- tation +PN and +GN indicates the addition of pair and group normalization layers respectively. In parentheses we indicate the architecture depth that achieved the highest test accuracy.
Method
Cora
Citeseer
Amazon Photo
Pubmed
GCN
68.2+ (4)
38.8 (4)
84.6 (4)
49.7+2.0 (4)
JKNet
66.1 (8)
36.7+3.0 (4)
65.4 (8)
48.0 (8)
DSGNN
72.6+ (8)
38.2+2.8 (8)
85.8:+0..8 (8)
56.6+1922 (32)
GCNII
75.7 (32)
46.1 (32)
60.1 (4)
62.3 (16)
GCN+PN
69.8 (8)
39.4 (8)
85.3+1.2 (8)
64.9 (32)
DSGNN+PN
7.. (16)
50.0 (32)
8
73.8 + 1.2(32)
GCNII+PN
71.0=33.4 (16)
40.6 (32)
48.6+3.4 (3)
67.5#2.3 (32)
GCN+GN
70. (4)
42.5 (4)
84.4+0.8 (4)
66,213.6 (32)
DSGNN+GN
76.2+2.0 (64)
48.8+5.0 (64)
8.. (16)
69.3 (32)
GCNI+GN
78.3 + 1.6(32)
52.7 + 2.3664
85.7+1.0
71.9+ (32)
EXPERIMENTS
We implemented all architectures using PyTorch and the Deep Graph Library (DGL., Wang al. 2019). The version of the datasets we use is that available via DGI 2 All experiments were run on workstations with 32GB of RAM. Nvidia Telsa P100 GPU, and Intel Xeon processor.
We evaluate the performance of all architectures under two settings that we call (a) fully observed, and (b) missing features. In the fully observed setting, the node features for all nodes are available during training. In the missing-featue set are available. For all other nodes, we follow Zhao & Akoglu (2020) and set the node attributes to zero vectors. The mhissin-feattur setting reflects scenario common in real-world applications where obtaining node attribute data can challenging.
setting, only the node features for those nodes in the training
Datasets: We use 4 benchmark datasets common in the GNN literature. These are the cita- tion networks Cora, Citeseer, Pubmed and the co-purchase network Amazon Photo. The splits for the citation networks from Yang et al. (2016). For Amazon Photo, we split nodes into train/validation/test following Shehur et al. (2018); we use 20 examples per class for training, 30 examples per class for validation, and all remaining nodes as test. See appendix A.I for details.
Experimental Setup: We consider architecture depths in {2, 3, 8,8, 16, 32, 64) except for Pubmed and Amazon Photo where we used maximum 32 layers to limit total training time. All architectures
We will release the source code upon publication acceptance. Available htttpp::/ / / thub. com dd 1m

---

Under review as a conference paper at ICLR 2023
use the graph convolutional layers from Kipf & Welling (017). For JKNet, DSGNN, and GCNII the last layer is dense with softmax activation. GCNII also has a dense layer at the imput in order to map the imput node attributes to the same dimension the hidden layer representations which are combined via addition. For JKNet we used element-wise max pooling before the final dense layer to combine the node representations from all convolutional layers. For all architectures, set the number of hidden units to 64, 256, 256, 64 for Cora, Citeseer, Pubmed, and Amazon Photo respectively, Optimization and hyper-parameter tuning details can be found in appendix A.3
5.1 RESULTS AND ANALYSIS
Fully observed setting: We see in table in this setting, the best method is GCNII for Citeseer and Amazon Photo and only marginally improved from the addition of group normalization for Cora and Pubmed. For Citeseer and Amazon Photo, GCNII outperforms the other architectures by margin of approximately 0.2% and 1.3% using 32 and 4 layers respectively as compared to GCNII+GN and GCN with 8 and layers. GCNII is the only method to benefit from added depth in the fully observed setting where, with the exception of Amazon Photo, performance was maximized for or more layers. Augmenting any of the architectures with pair normalization caused a reduction in test accuracy. On the other hand, the addition of group normalization resulted in a marginal improvement for some combinations of dataset and architecture such as, DSGNN vs DSGNN+GN on Pubmed and GCN vs GCN+GN on Cora. These marginal improvements, if any, offered by group normalization come at the cost of longer training times due to an increase in the number of hyper-parameter sets that must be evaluated during grid-based cross-validation (see appendix AA.3 for details and discussion). Thus, the increased computation time coupled with marginal improvements suggests that there is justification for using group normalization in the fully observed setting.
Missing-featur setting: Table 2 shows that all architectures benefited from increased network depth in the missing-feature setting. We note that test accuracy is lower than in the fully observed setting which indicates that input node attributes contain useful information that GNNs exploit to improve prediction accuracy. some cases, such as Citeseer, the gap in test accuracy between the fully observed and missing-feature settings is as much as 20% (GCNII+GN). Generally, all archi- tectures except GCN achieve the highest test accuracy at a larger network depth. This is expected since information from nodes with features must propagate further in the graph to inform the repre- sentations for the nodes with missing features.
Additionally, we observe that DSGNN's s ability to learn discriminative representations across all hid- den layers yields benefits especially in combination with pair normalization. DSGNN enhanced with pair normalization (DSGNN+PN) was best for Amazon Photo and Pubmed by margins 2.2% and 1.9% from the second best architecture (GCNII+GN) respectively. DSGNN+PN was second best on Cora and Citeseer lagging by margins of 0.9% and 2.7% respectively from the best architecture GCNII+GN. However, as we can see in Figure (right, see also appendix (A.3) GCNII+GN requires approximately factor of approximately 3 times longer train as compared to DSGNN+PN; for practical applications with a constrained computational budget, the trade-off between a small loss in test accuracy and large gain in training speed suggests a preference for DSGNN+PN as the architecture of choice.
Ove--smog resilience: In order to examine the architectures' resilience to over-smoothing, we plot test accuracy vs network depth for a subset of the methods in fig. (1eft) on Amazon Photo under the fesstug-featureee setting. Here we have selected vanilla GCN method (DSGNN+PN) and the top performing methods in the fully observed setting (GCNII and GCNII+GN). A comprehensive set of results is given in the appendix (fig. 3 and fig. + We see in fig. 2 that all architectures achieve an increase in performance in step with larger number of layers up to a point. However, we note that GCNII (which performed best on this dataset in the fully observed case), does very poorly in comparison with the other methods when using eight or more layers. This behavior is significantly improved when using it in conjunction with group normalization (GCNII+GN). In contrast with GCNII and similarly GCNII+GN, DSGNN+PN monotonically increases performance up to sixteen layers and only slightly worsens after that.
baseline; the top performing
Additional results in appendix: Analyzing figs. and we can see that all architectures with the exception of vanilla GCN alleviate svor-smoothing. Generally, adding residual connections and/or some form of layer normalization is beneficial even though it does not always increase test accuracy,
8

---

Under review as a conference paper at ICLR 2023
90
X
80
GCN
Test Accuracy
Test Accuracy
DSGNN-PN
7g-
GCN DSGNN PN
GCNII
GONII+ON
60
JKNet
GONII+GN
DSGNN
60-
GCNII+PN
GON+GN
40
DSGNN+GN
16 * Layers
10 20
Total Training Time (rel. to GCN, log scale)
Figure 2: (Left) Plot of test accuracy vs GNN depth for subset of the models in table (Right) Plot of test accuracy vs total training time (including hyper-parametto tuning, see appendix A.3) relative to GCN. For each model, we plot the training time and accuracy for the depth where test accuracy was highest as listed in table Both plots are for Amazon Photo and the missing-featur feature setting.
as can be seen in the fully observed setting, Similarly to the Amazon Photo dataset, in the missing- feature setting, for all architectures except vanilla GCN, there is increase in performance in step with the increase in the number of network layers up to a point depending on the dataset. We further illustrate how all architectures except vanilla GCN are resilient over-smoothing in figure where we plot the instance information gain metric proposed by Zhou et al. (2020a) vs the number of network layers for models trained on Amazon Photo. Instance information gain measures the amount of information in the input node features that is carried through to the network's output. For architectures such as vanilla GCN that suffer from wrer-smoohhing, this metric tends to zero as network depth increases. As we can see in Figure fig. 5 all architectures with some form of residual connections and/or normalization layers propagate information in the input node features through to the output layer as information gain is constant as a function of depth.
Deep supervision: Our proposed DSGNN alleviates but does not eliminate over-smoothing simi- larly JKNet. This can be seen in the top of Figure where, for of the datasets DSGNN exhibits only a small reduction in test accuracy as a function of network depth. Furthermore, as shown in table I DSGNN performs competitively with the other architectures. In the missing- feature setting (see table 2). DSGNN in combination with pair normalization achieves the highest test accuracy for 2 of the 4 datasets while it is second highest for the other 2.
6 CONCLUSIONS & RECOMMENDATIONS
We have revisited the over-smoothing problem in deep GNNs, carrying out a systematic evaluation of several ofte--of-the-at solutions and their combination in the fully observed and missing- missing-feature settings on several benchmark datasets. Additionally, we have proposed DSGNN, a new architecture based on deep supervision possible solution.
Recommendations: Based on experiments and analysis results, we can make the following rec- ommendations for training deep GNNs resilient over-smoothing: (a) residual connections in any of the many forms as present in JKNet, DSGNN, and GCNII, are essential to alleviating over- smoothing in deep GNNs. Residual connections alone do not guarantee improved generalization accuracy in the fully observed setting but are essential in the missing-feature setting. (b) In the fully observed setting GCNII achieves highest test set performance and it is the only architecture that can exploit model depth. Since GCNII has the same number of trainable parameters as GCN and fewer than DSGNN and JKNet (see appendix A.3) should be the preferred architecture. (c) In the fully observed setting, pair normalization negatively impacts performance for all architectures and its use should be avoided. (d) In the missing-feate GCNII with group normalization should be preferred. Since GCNII and group normalization have a large number of hhper-paramen parameters that require tuning, DSGNN with pair normalization should be preferred when computation budget is limited. We hope our analysis helps guide practitioners and researchers in their application of deep GNNs to real-world problems and promotes a more systematic and careful evaluation of existing and new GNN methods.
setting, DSGNN with pair normalization and/or

---

Under review as a conference paper at ICLR 2023
REFERENCES
Deli Chen, Yankai Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over- smoothing problem for graph neural networks from the topological view. In Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pp. 3438-3445, 2020a.
Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. In Hal Daumé III and Aarti Singh (eds.). Proceedings of the 37th In- ternational Conference on Machine Learning, volume 119 of Proceedings of Machine Learning Research. pp. 1725 1735. PMLR, 13-18 Jul 2020b. URL press/ http:/v119///heeee html
.mlr.
Federico Errica, Marco Podda, Davide Bacciu, and Alessio Micheli. A fair comparison of graph neural networks for graph classification. arXiv preprint 0rXiv:1912.09893 2019.
Matthias Fey and Jan Eric Lenssen. Fast graph representation learning with pytorch geometric. arXiv preprint wrXiv:1933.02428 2019.
Thomas Gaudelet, Ben Day. Arian R Jamasb, Jyothish Soman, Cristian Regep, Gertrude Liu, Jeremy B R Hayter, Richard Vickers, Charles Roberts, Jian Tang, David Roblin, Tom L Blun- dell, Michael M Bronstein, and Jake P Taylor-King. Utilizing graph machine learning within drug discovery and development. Briefings in Bioinformatiics, 22(6), 05 2021. ISSN 1477-4054. doi: 10. 10.1093/bib/bbabl55/ URL https::: /ttt:p///oolloorr//00000//////// btbpbbbbbbbbbbbbbb bbab159.
Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Advances Neural Information Prucessing Systems, pp. 1024-1034, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Thomas N Kipf and Max Welling, Semi-Supervised Classification with Graph Convolutional Net- works. In International Conference on Learning Representations, 2017.
Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Com- bining neural networks with personalized pagerank for classification on graphs. In International Conference Learning Representations, 2018
Chen-Yo Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeply- Supervised Nets. In Guy Lebanon and S. V. N. Vishwanathan (eds.), Proceedings of the Eigh- teenth International Conference Artificial Intelligence and Statistics, volume 38 of Proceed- ings of Machine Learning Research, pp. 562-570, San Diego California, USA, 09-12 May 2015. PMLR. URL TRRhhtp::// BRL:httt://porceeeiiii pttpe/ediooeeeeecsoc/ee wwe..pres....mmm.......m. 11a.:/eeeeaaa.tmmm
Qimai Li, Zhichao Han, and Xiao-ming Wu. Deeper insights into graph convolutional networks for semi-supervieed learning. Proceedings of the AAAI Conference Artificial Intelligence, 32 (1), Apr. 2018. URL https::/ojss.oo aaai org/ begg/index...// 11604
Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Pro- ceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, pp. 338 348, New York, NY, USA, 2020. Association for Computing Machin- ery. ISBN 47814993799 doi: 00.1145/3394488.4400000 URL htt.ttt::////...rgg///
1145/3394488 3403076
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann machines. In ICML, 2010.
Yu Rong. Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: Towards deep graph convolutional networks on node classification. In International Conference on Learning Repre- sentations, 2020.
10

---

Under review as a conference paper at ICLR 2023
Franco Scarselli, Marco Gori, Ah Chung Tsoi, Markus Hagenbuchner, and Gabriele Monfardini. The graph neural network model. IEEE Transactions on Neural Networks, 20(1):61-80, 2009. doi:
2005605.
Oleksandr Shchur, Maximilian Mumme. Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprinl arXiv: 1.05868, 2018.
Petar Veličković, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Liò, and Yoshua Bengio. Graph Attention Networks. In International Conference on Learning Representations, 2018.
Minjie Wang. Da Zheng, Zihao Ye, Quan Gan, Mufei Li, Xiang Song, Jinjing Zhou, Chao Ma, Lingfan Yu, Yu Gai, Tianjun Xiao, Tong He, George Karypis, Jinyang Li, and Zheng Zhang. Deep graph library: A graph-centrie, highly-performant package for graph neural networks. arXiv preprint wrXiiv:/99.0f1
2019.
Felix Wu, Amauri Souza, Tianyi Zhang, Christopher Fifty. Tao Yu, and Kilian Weinberger. Sim- plifying graph convolutional networks. In International Conference on Machine Learning, pp- 6861-6871, 2019.
Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanic Jegelka. Representation learning on graphs with jumping knowledge networks. In Jennifer Dy and Andreas Krause (eds.), Proceedings of the 35th International Conference Machine Learning, volume 80 of Proceedings of Machine Learning Research, pp. 5453-5462. PMLR, 10-15 Jul 2018. httt:tt/pproeeeinnggg. www.innggmmmr...omm pte.s.ess....p..p.o..e...e.....e /vteesss//0// ww00vv000..1...0
Zhilin Yang, William Cohen, and Ruslan Salakhudinov. Revisiting semi-supervised learning with graph embeddings. In International conference on machine learning. pp. 40-48 40-48. PMLR, 2016.
Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in GNNs. In International Conference on Learning Representations, 2020. URUjatpps://openreviiw net forum?
lddrleel1
Jie Zhou, Ganqu Cui, Shengding Hu, Zhengyan Zhang, Cheng Yang, Zhiyuan Liu, Lifeng Wang. Changcheng Li, and Maosong Sun. Graph neural networks: A review of methods and applica- tions. Al Open, 1:57-81, 2020a.
Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. To- wards deeper graph neural networks with differentiable group normalization. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 4917-4928. Curran Asso- ciates, Inc., 2020b. URL htssttttss//ppooeeedii ttt.o//ooooooo///oo.o....oo.oo.. 333d6bbaalbbee222aaaloof222ooood Paper. pdf

---

Under review as a conference paper at ICLR 2023
A APPENDIX
A.I
DATASETS
Table 3: Dataset statistics.
Name
Nodes
Classes
Node features features
# Edges
Avg. Node degree
Median
train/val/test
Node
degree
Cora
2708
1433
10556
3.9
3
140/500/1000
Citeseer
3327
3703
9228
2.7
2
120/500/1000
Amazon Photo
7650
8
745
238,163
31.1
22
160/240/7250
Pubmed
19717
500
88651
4.5
2
60/500/1000
Cora, Citeseer, and Pubmed are citation networks where the goal is to predict the subject of a paper.
Edges represent citation relationships. The datasets have known train/val/test splits from Yang et al.) (0016) Training sets are small with the number of labeled nodes equal to 140 (20 for each of classes), 120 (20 for each of 6 classes), and 60 (20 for each of classes) for Cora, Citeseer, and Pubmed respectively.
Amazon Photo subset of the Amazon Co-purchase product dataset. Nodes represent items and edges connect items that are bought together. Node attributes are bag-of-words features derived from product reviews. The goal is to predict the product category.
We treat all graphs undirected as it is common in the GNN literature.
A.2 ADDITIONAL RESULTS
We demonstrate the architectures' resilience to over-smoothing in the fully observed and missing- feature settings by plotting test accuracy vs network depth in Figures aadd
In the fully observed setting, the performance of GCN drops sharply for all datasets for deep models. The combination of GCN with pair (GCN+PN) or group (GCN+GN) normalization alleviates the problem to some degree but performance still degrades rapidly with depth (middle and bottom of Figure 3 GCN+GN more resilient to over-smoothing than GCN+PN, GCNII and JKNet both alleviate over-smoothing. DSGNN better than GCN, GCN+PN, and GCN+GN but still only reduces the rate of over-smoothing
In the missing-feature setting, all architectures initially benefit from increased depth more graph convolutional layers are necessary to allow for information to propagate through the graph given that the majority of nodes do have associated input node attributes (which have been set zero vectors). However, GCN still suffers from over-smoothing as network depth increases while the other architectures are more resilient. Some combinations such as DSGNN+PN, DSGNN+GN, and GCNII+GN on Pubmed continue to see an increase on test accuracy for up to 32 layers.
A.3 TRAINING DETAILS
We used the Adam optimizer with learning rate set to 0.01. We trained all models for maximum 2500 epochs and used early stopping with patience 250 epochs monitoring validation set accuracy. We run each experiment 20 times and report the mean performance and standard deviation of test accuracy.
The training time for each architecture depends two main factors. One is the total number of trainable parameters and the other is the number of haper-parameters that require tuning. In table
4
we report for each architecture the total number of syper-parameters, hyper-parameter settings considered using grid search and the total number of trainable parameters.
For all architectures tune dropout (4 values from {0.2, 0.5, 0.6, 0.7)) and weight decay (5 values from (0.01, 0.001, 0.0001, 0.0005, 0.00001)). Weight decay for the dense layers in GCNII was fixed to 0.0005 as suggested by Chen al. (2020b). GCNII requires selection of one additional
12

---

Under review as a conference paper at ICLR 2023
Citeseer
Amazon Photo
Pubmed
Thst Accuracy
70
85
70-
60-
%
80-
60-
10
DSRMY
30-
32
16
32
60
77.
Accurccy
76-
70-
50
60
86
74.
.50
GC18
84
7
8
16
32
64
2 8
64
8
16
16
32
85
70
92
accurscy
80
90
75
60
76
70
50
86
74
84
SC
250MW-C
72
16
32
2
64
16
32
# Layers
Layers
# Layers
# Layers
Figure 3: Test accuracy vs architecture depth for the fully observed setting. Top row: vanilla architectures. Middle row: architectures with pair normalization (+PN). Bottom row: architectures with group normalization (+GN). Note that the y-axis for the different plots have different range.
Cora
Citeseer
Amazon Photo
Pubmed
45
60-
60
40
70
35-
50
SCN
30-
GCMB
50-
45
25
8
16
32
64
8
166
32
611
16
32
16
32
80-
70
70
40
60-
60-
30
SCH-PM
40
260
20
30
80
Accuracy
70
45
70-
60
40
35
3CH-BM
40
16
1'6
Layers
Figure 4: Test accuracy architecture depth for the missing-feature setting. Top row: vanilla architectures. Middle row: architectures with pair normalization (+PN), Bottom row: architectures with group normalization (+GN). Note that the y-axis for the different plots have different range.
prper-parameter the identity mapping weight, (3 values from (0.4, 0.6)). For GCNII, we fixed the value indicating the fraction of input features retained at each layer to 0.1 in Chen et al. (20206). Pair normalization layers do not add any hyper-parameters Group normalization layers require selection of an additional hyper-paramettr, the group normalization balancing factor, (9
13

---

Under review as a conference paper at ICLR 2023
Table 4: Number of hyper-parameter sets and number of trainable parameters for each architecture.
Architecture
# hhyper-paammeters
# settings
# of trainable parameters
GCN
20
xK-1)x&+dxx xx xxdx
JKNet
DSGNN
2
20
GCNII
60
(K+1)xd+dxC
GCN+PN
2
20
(K-1)x&4dxC
DSGNN+PN
(x+x(
GCNII+PN
60
(K+1)x
GCN+GN
180
(K - x +xx(xxx=+x
DSGNN+GN
3
180
K X xxdx((+C+G))
GCNII+GN
540
(K [K+])x&&dx(C++X
values from {0.0005, 0.001, 0.002, 0.003, 0.005, 0.01, 0.02, 0.03, 0.05)). For group normalization set the number of groups to 10, 10, 5 for Cora, Citeseer, and Pubmed respectively suggested by Zhou et al. (2020b). For Amazon Photo, we set the number of groups to 10 since Zhou et al. 20206) recommends that the number of groups should be close the number of classes predict.
The total number of hyper-parameters that require tuning varies based on the architecture. For example, the vanilla GCN architecture only requires tuning over set of x 5 20 hyper- parameerr settings whereas GCNII+GN requires tuning over a set of 4 x 5 x 3 x 9 = 540 settings.
We estimate the total number of trainable parameters for each architecture and report them in table Let K be the number of GNN layers, the number of classes, G the number of node clusters, and d the dimensionality of latent node representations. For clarity, we assume that nodes have input attributes also with dimensions.
In our experimental setup, K K-layer GCN model comprises of K graph convolutional layers where the last layer is considered the output layer predicting class probabilities for each node. For JKNet and GCNII, K K-layer model comprises of K graph convolutional layers followed by dense layer for classification. Figure [ (left) shows an example of JKNet with K = 2 layers where are graph convolutional. For DSGNN, K K-layer model comprises of K graph convolutional layers and K dense layers attached to each hidden layer. Figure | (right) shows an example of DSGNN with K layers where 2 graph convolutional and 2 dense with softmax activation.
The number of trainable parameters for GCN are (K 1) x d² + d x C. Here we consider the last graph convolutional layer the output layer. JKNet has K graph convolutional layers followed by dense layer. The total number of trainable parameters is K x de + d x C. DSGNN adds dense layer for each graph convolutional layer so the number of trainable parameters is K d² + K x d x C. GCNII adds dense layer at the input followed by K graph convolutional layers and dense layer for classification. Hence, the total number of trainable parameters (K 1) df + +d x C. Group normalization adds a dense layer with softmax activation for each graph convolutional layer; the dense layer calculates soft assignment of nodes to clusters. For a model with K graph convolutional layers, group normalization adds K x d x G trainable parameters.
A.4 QUANTIFYING OVER-SMOOTHING
We can quantify the propensity of an architecture to over-smooth by considering the amount of information in the input node features that is not preserved at the output as proposed by Zhou et al. 2020b). Instance information gain measures the amount of information in the input node features that is preserved in the output of a GNN layer. Zhou et al. (2020b) propose instance information gain goes
function of GNN depth indicating over-smoothing. In Figure we plot instance information gain as a function of architecture depth for Amazon Photo in the fully observed setting. We that information gain for GCN decreases network depth increases. This is not the case for
14

---

Under review as a conference paper at ICLR 2023
0.115
0.110
Instance Information Gain
0.105
0.100
GCN DSGNN
0.095
GCNII
0.090
JKNet
GCN+
DSGNN+
0.085
GCNII+PN
GCN+GN
0.080
DSGNN+GN
GCNII+GN
5
10
15
20
25
30
Layers
Figure 5: Instance Information Gain as function GNN depth. All results are for Amazon Photo dataset in the fully observed setting.
all other architectures. Notably, just the addition of pair normalization to GCN, GCN+PN, alleviates information loss. This results correlates with the test accuracy GNN depth curves in Figure 3
15


Reviews:
[{"review_id": "k8G7KrIyzOl", "reviewer": "Reviewer_NBPk", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies the over-smoothing problem in deep graph neural networks. The authors systematically evaluate different methods including graph convolutional networks, jumping knowledge networks, pair normalization, group normaliztion, and GCNII. Besides, the authors propose an architecture called architecture deeply supervised GNNs (DSGNNs) based on the idea of layer-wide supervision. The experimental results show that the proposed method can achieve better performance on deeper GNNs, especially providing best performance in missing feature scenarios. \n", "summary_of_the_review": "Although this paper is complete in content, the fatal problem is that there is no effective analysis to demonstrate the superiority of the method. I suggest that the author think clearly about the advantages of the method in this work, and reorganize the presentation of the article based on this. Based on the current version, I don't think it meets the ICLR requirements for reception. Authors are encouraged to further improve this article from originality and technicality.", "strength_and_weaknesses": "The main strength of this work is the clear writing. It is not hard to understand the main contribution of this work. Overall I like the presentation of this work. I believe that the authors make an effort in polishing the writing.\n\nOn the other hand, I think that the experimental results presented in this work are systematic. The authors select the most famous deep GNN method and make a detailed comparison between them.\n\nHowever this work lacks in-depth analysis on the difference between previous methods and the proposed method. \n\nFirst, the authors claim that previous approaches may have potential pitfalls because of the evaluation and fairness problem. I understand the author may try to emphasise the advantage of the proposed method in missing features scenarios. However, the discussion in Section 3.2 is not convincing. The author only explains the possible risks of the previous method, and does not give us a specific and in-depth analysis to tell you why the previous method has shortcomings and the reasons for the shortcomings.\n\nSecond, when expounding the advantages of deep supervision, the author does not provide effective and detailed arguments to be convincing. What I mean by effective arguments include theoretical proofs or elaborate experimental analyses. However I don't see any analysis. Therefore, the lack of this paper is able to show the proposed method well.\n\nFinally, from the reported results in Table 1 and 2, the best performance usually is achieved by GCNII+GN instead of the proposed method (DSGNN). Even in the missing features setting, the proposed method of this work only achieves best performance on 2 out of 4 dataset. This is not enough to demonstrate the effectiveness of the method proposed in the article.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "ag6X-LOYlM", "reviewer": "Reviewer_K9Pi", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the over-smoothing problem in graph neural networks and proposes a new architecture called deeply-supervised GNN (DSGNN) as a solution. Inspired by deeply-supervised nets (Lee et al., 2015), DSGNN has a classifier attached to each hidden layer and gets supervision during training. During inference, DSGNN makes predictions based on averaged prediction across all layers. The authors conduct experiments on Cora, Citeseer, Amazon Photo, and Pubmed datasets with both fully observed and missing feature settings to compare GCN, JKNet, GCNII, and their proposed DSGNN. The application of pair normalization and group normalization are also considered in the experiments.", "summary_of_the_review": "To sum up, the main claim of the paper is not well-supported and a lot of baselines are missing. Therefore, I recommend rejecting this paper.", "strength_and_weaknesses": "### Strengths\n- The paper is written clearly and easy to follow. I like section 3 which summarizes the prior efforts on tackling over-smoothing in a clear manner.\n- The proposed method is simple.\n- This paper studies both fully observed and missing feature settings.\n\n### Weaknesses\n- No evidence is provide to support this paper's claim that DSGNN can address over-smoothing. It is well known that deep supervision helps the training of deep nets. It is not clear whether deep supervision actually addresses the over-smoothing problem. There are two other potential explanations of the good: 1) deep supervision alleviates the gradient vanishing problem of the bottom layers and 2) the model makes predictions by taking the average which is an ensemble method that usually improves the performance of models. \n- The author should report the performance of the last classifier of DSGNN. It is possible that the top layers are as bad as deep GCNs and DSGNN does well relying on the predictions of its earlier layers.\n- The author should consider a simple baseline that averages a couple of different shallow GCNs which also has the advantage of ensemble. For example, comparing a 32 layer DSGNN with these baselines\n   - 16 2-layer GCNs\n   - 4 2-layer GCNs + 4 3-layer GCNs + 4 4-layer GCNs\n   - a combination of 1, 2, 3, ..., 32 layer GCNs\n- The authors only consider architecture depths in {2, 3, 4, 8, 16, 32, 64}. However, as we can see in Table 2, several models have the best depth being 64. It is weird that the author didn't consider increasing the depth further to find the optimal depth.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "UXLfH7HD8la", "reviewer": "Reviewer_QdpM", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies the over-smoothing problem in GNN, which means the node representations tend to become indistinguishable as the number of layers increases. The key contributions of these papers are two-folded. First, they do a benchmarking to evaluate and analyze several methods. Second, motivated by deeply supervised nets, the paper proposed deeply supervised GNNs. Experimental results on standard and missing feature scenarios show the method's advantages.", "summary_of_the_review": "The paper tackles an important problem. However, there is a very similar related work that also focuses on benchmarking deep GNNs. The previous work considers more datasets and more scenarios than this work. So it is unclear to me whether the contribution of this paper is significant. The paper should also consider more datasets and baselines. Thus, overall I can not recommend acceptance.", "strength_and_weaknesses": "Strengths:\n1. Over-smoothing is an important problem for GNNs.\n2. The idea of deeply supervised GNNs (DSGNN) is well-motivated. The performance of DSGNN improves in most cases as the number of layers increases.\n\nWeaknesses:\n1. The benchmarking experiments are not enough. Particularly, there is a previous work that conducts much more comprehensive benchmarking than this paper [1].\n2. Only 4 datasets are considered in the evaluation and benchmarking. More large-scale datasets should be considered.\n3. DSGNN seems to only be comparable with GCNII. It is unclear what advantages DSGNN has.\n4. A missing related work should be compared since it can also improve performance when using deeper architectures [2].\n\n\n[1] Chen, Tianlong, et al. \"Bag of tricks for training deeper graph neural networks: A comprehensive benchmark study.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2022).\n[2] Zhou, Kaixiong, et al. \"Dirichlet energy constrained learning for deep graph neural networks.\" Advances in Neural Information Processing Systems 34 (2021): 21834-21846.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "P6dUYzdU713", "reviewer": "Reviewer_FUiX", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper tries to study the over-smoothing issue of GNNs. A detailed discussion on the potential 'pitfalls' of the existing works (focusing on preventing over-smoothing) is given. Then, a deeply-supervised GNN framework, based on the idea of layer-wise supervision, is proposed for problem-solving. Extensive experiments are conducted to verify the effectiveness and advantage of the proposed deeply-supervised trick. Based on the experimental results, some (initial) recommendations for the selection of a GNN architecture for practical applications are presented.", "summary_of_the_review": "Overall, based on the weaknesses mentioned above, I vote for rejection.", "strength_and_weaknesses": "Strengths:\n\n+ The discussion on the characteristics of  several existing GNN over-smoothing solutions and their combinations is informative\n+ The presentation and readability are easy to follow\n\nWeaknesses:\n+ The technical contribution is limited since the deeply-supervised idea is already a widely-used trick for designing very deep nets, like DenseNet (CVPR, 2017) and its follow-up works\n+ More in-depth (theoretical) analysis on why the proposed DSGNN is powerful is missing. The arguments presented in Section 4.2 are not convincing. \n+ Experimental studies are not strong. More large-scale graph datasets are needed to further verify the effectiveness and efficiency of DSGNN. Also, it seems that GCNII+GN outperforms DSGNN in most cases, which contradicts the merits of DSGNN.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Reject