Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
Authors: Kaixiong Zhou
Xiao Huang
Daochen Zha
Rui Chen
Li Li
Soo-Hyun Choi
Xia Hu
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep graph neural networks
Over-smoothing
Node classification
Dirichlet energy
Orthogonal weight

Abstract: Graph neural networks (GNNs) integrate deep architectures and topological structure modeling in an effective way. However, the performance of existing GNNs would decrease significantly when they stack many layers, because of the over-smoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations of neighbors. To enable deep GNNs, several methods have been explored recently. But they are developed from either techniques in convolutional neural networks or heuristic strategies. There is no generalizable and theoretical principle to guide the design of deep GNNs. To this end, we analyze the bottleneck of deep GNNs by leveraging the Dirichlet energy of node embeddings, and propose a generalizable principle to guide the training of deep GNNs. Based on it, a novel deep GNN framework -- Energetic Graph Neural Networks (EGNN) is designed. It could provide lower and upper constraints in terms of Dirichlet energy at each layer to avoid over-smoothing. Experimental results demonstrate that EGNN achieves state-of-the-art performance by using deep layers.

Dirichlet Energy Constrained Learning for Deep Graph Neural Networks
Kaixiong Zhou
Xiao Huang The Hong Kong Polytechnic University xiaohuang@comp polyu. edu. hk
Rice University
Kaixiong. Zhou@rice.edu
Daochen Zha Rice University Daochen. Zha@rice.cc.o edu
Rui Chen
LiLi Samsung Research America 1i li www.l18saa Liissamsung..... com
Samsung Research America
rui. chen chenn@ssame @neemnsammung.... com
Soo-Hyun Choi" Samsung Electronics soohyunc@gmaill com
Xia Hu Rice University
xia. huâri ce edu
Abstract
Graph neural networks (GNNs) integrate deep architectures and topological struc- ture modeling in an effective way. However, the performance of existing GNNs would decrease significantly when they stack many layers, because of the over- smoothing issue. Node embeddings tend to converge to similar vectors when GNNs keep recursively aggregating the representations of neighbors. To enable deep GNNs, several methods have been explored recently. But they are developed from either techniques in convolutional neural networks or heuristic strategies. There is generalizable and theoretical principle to guide the design of deep GNNs. To this end, we analyze the bottleneck deep GNNs by leveraging the Dirichlet energy of node embeddings, and propose generalizable principle guide the training of deep GNNs. Based on it, novel deep GNN framework Energetic Graph Neural Networks (EGNN) is designed. It could provide lower and upper constraints terms Dirichlet energy at each layer to avoid over-smoothing. Experimental results demonstrate that EGNN achieves oftte-of-hh-art performance by using deep layers.
Introduction
Graph neural networks (GNNs) are promising deep learning tools analyze networked data, such as social networks 2 3
academic networks (5 7). and molecular graphs (8) [ I..
Based on spatial graph convolutions. GNNs apply recursive aggregation mechanism to update the representation of each node by incorporating representations of itself and its neighbors of GNN variations have been explored for different real-world networks and applications 13
[12). A variety
key limitation of GNNs is that when we stack many layers, the performance would decrease significantly. Experiments show that GNNs often achieve the best performance with less than layers 15 13 As the layer number increases, the node representations will converge to indistin- guishable vectors due the recursive neighborhood aggregation and non-linear activation [6 17
"Conesponding author.
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

Such phenomenon is recognized as over-smoothing issue [8 20 21 22 It prevents the stacking
of many layers and modeling the dependencies to high-order neighbors.
number of algorithms have been proposed to alleviate the over-smoothing issue and construct deep GNNs, including embedding normalization
23 24 25], residual connection [26] 27 28 and
random data augmentation [29] 30, 31]. However, some of them are motivated directly by techniques in convolutional neural networks (CNNs) [32), such as the embedding normalization and residual connection. Others are based on heuristic strategies, such as random embedding propagation [30 and
dropping edge [29]. Most of them only achieve comparable or even worse performance compared to their shallow models. Recently, a metric of Dirichlet energy has been applied to quantify the over-smoothing (33, which is based on measuring node pair distances. With the increasing of layers, the Dirichlet energy converges to zero since node embeddings become close to each other. But there a lack of empirical methods to leverage this metric to overcome the over-smoothing issue.
Therefore, it remains a non-trivial task to train a deep GNN architecture due to three challenges. First, the existing efforts are developed from diverse perspectives, without generalizable principle and analysis. The abundance of these components also makes the design of deep GNNs challenging, i.e. how should we choose suitable one combinations for real-world scenarios? Second, even if an effective indicator of over-smoothing is given, it is hard to theoretically analyze the bottleneck and propose generalizable principle guide the training of deep GNNs. Third, even if theoretical guidance is given, it may be difficult to be utilized and implemented to train GNNs empirically.
To this end, in this paper, target develop a generalizable framework with theoretical basis, to handle the over-smoothing issue and enable effective deep GNN architectures. In particular, we will investigate two research questions. 1) Is there theoretical and generalizable principle guide the architecture design and training of deep GNNs? 2) How can we develop an effective architecture to achieve state-of-the-an performance by stacking a large number of layers? Following these questions, we make three major contributions as follows.
We propose a generalizable principle - Dirichlet energy constrained learning, to guide the training of deep GNNs by regularizing Dirichlet energy. Without proper training, the Dirichlet energy would be either too small due to the over-smoothing issue, or too large when the node embeddings are over-separating. Our principle carefully defines an appropriate range of Dirichlet energy at each layer. Being regularized within this range, a deep GNN model could be trained by jointly optimizing the task loss and energy value.
We design novel deep architecture- - Energetic Graph Neural Networks (EGNN). It follows the proposed principle and could efficiently learn an optimal Dirichlet energy. It consists of three components, i.e., orthogonal weight controlling, bounded residual connection, and shifted ReLU (SReLU) activation. The trainable weights at graph convolutional layers are orthogonally initialized as diagonal matrices, whose diagonal values are regularized to meet the upper energy limit and eliminate the over-separating. The residual connection strength is determined by the lower energy limit to avoid the over-smoothing. While the widely-used ReL.U activation causes the extra loss of Dirichlet energy, the linear mapping worsens the learning ability of GNNs. We apply SReLU with a trainable shift to provide trade-off between the non-linear and linear mappings.
We show that the proposed principle and EGNN can well explain most of the existing techniques for deep GNNs. Empirical results demonstrate that EGNN could be casily trained to reach 64 layers and achieves surprisingly competitive performance on benchmarks.
2 Problem Statement
Notations. Given an undirected graph consisting of n nodes, it is represented as G = X), where A E Raxe denotes the adjacency matrix and X E R denotes the feature matrix. Let Á == A+ I, and D == D D+I I, be the adjacency and degree matrix of the graph augmented with self- loops. The augmented normalized Laplacian then given by Á == I,- P, where P := Aカ- is an augmented normalized adjacency matrix used for the neighborhood aggregation in GNN models.
2

---

The detailed proof is provided in the Appendix. A and An are the non-zero eigenvalues of matrix Á that are most close to values 1 and 0, respectively. (k) and (&)) are the squares of minimum and maximum singular values of weight Wi(), respectively. Note that the eigenvalues of Á vary with the Node classification task. GNNs have been adopted in many applications 66 11 [4]. Without loss of generality, we take node classification as an example. Given a graph G (A, and a set of its nodes with labels for training, the goal is to predict the labels of nodes in test set.
We now the graph convolutional network (GCN) 15
typical example, to illustrate how traditional GNNs perform the network analysis task. Formally, the layer-wise ppooaddpropagation operation in GCN the k-th layer is defined as:
x -(X
(k) and ((==1) are node embedding matrices at layers k and k - 1, respectively; W E Rdxd denotes trainable weights used for feature transformation; a denotes an activation function such as ReLUU ( X the initial layer of GCN. The embeddings at the final layer optimized with node classification loss function, e.g., cross-entropy loss. The recursive neighborhood aggregation in Eq. ① will make node emheddings similar each other as the number of layer k increases. This property, i.e., over-smoothing, prevents traditional GNNs from exploring neighbors many hops away. In practice, the dependencies high-order neighbors are important the node classification. The traditional shallow GNNs may have sub-optimal performances in the downstream tasks [6 28
3 Dirichlet Energy Constrained Learning
In this paper, we aim develop an effective principle alleviate the over-smoothing issue and enable deep GNNs to leverage the high-order neighbors. We first theoretically analyze the over-smoothing issue, and then provide principle to explain the key constraint in training deep GNNs.
Node pair distance has been widely adopted to quantify the over-smoothing based on embedding similarities 119 23 Among the series of distance metries, Dirichlet energy simple and expressive
for the over-smoothing analysis [33] Thus, we adopt Dirichlet energy and formally define it as below.
Definition 1. Given node embedding matrix (*) = [z] at the k-th layer, the Dirichlet energy ((x())) is defined as follows:
In WT E Randd learned from GCN
(R)
= Ex(()) 2 Σ aij V1 d
2
(2)
V1
where tr(-) denotes trace of matrix; aij is edge weight given by the (t.j) -tth element in matrix A; d node degree given by the i-th diagonal element in matrix D. Dirichlet energy reveals the embedding smoothness with the weighted node pair distance. While smaller value of ((x( is highly related to the over-smoothing, a larger one indicates that the node embeddings are over-separating even for those nodes with the same label. Considering the node classification task, one would prefer have an appropriate Dirichlet energy at each layer to separate the nodes of different classes while keeping those of the same class close. However, under some conditions, the upper bound of Dirichlet energy theoretically proved to converge to 0 in the limit of infinite layers 33 In other words, all nodes converge to a trivial fixed point in the embedding space.
Based on the previous analysis, we derive the corresponding lower bound and revisit the over- smoothing/separating problem from the model design and training perspectives. To simplify the derivation process, we remove the non-linear activation a, and re-express GCN as: x²k) p... (XW(1) Wik), The impact of non-linear function will be considered in the model design.
Lemma 1. The Dirichlet energy at the k-th layer is bounded as follows:
(4) (1 (1- & E(X(k-1) < (k) < 22 Smax ((x(k-1)).
(k)
real-world graphs, and locate within range [0, 2). We relax the above bounds as below.
3

---

Lemma 2. The lower and upper bounds of Dirichlet energy at the k-th layer could be relaxed as: ((k) mnx E(x((-1)).
(4) Besides the uncontrollable eigenvalues determined by the underlying graph, it is shown that the Dirichlet energy can be either too small or too large without proper design and training on weight W(K), On one hand, based on the common Glorot initialization [35] and L2 regularization, we empir- ically find that some of the weight matrices approximate to zero in a deep GCN. The corresponding square singular values are hence close to zero in these intermediate layers. That means the Dirichlet energy will become zero at the higher layers of GCN and causes the over-smoothing issue. On the
0 S ((x())) I
other hand, without the proper weight initialization and regularization, large (k) may lead the
energy explosion and the over-separating.
The Dirichlet energy plays key role in training deep GNN model. However, the optimal value of Dirichlet energy varies in the different layers and applications. It is hard to be specified ahead and then enforces the node representation learning. Therefore, constrained learning, defined in Proposition 1. It provides appropriate lower and upper limits of Dirichlet energy. Regularized by such given range, deep GNN model could be trained by jointly optimizing the node classification loss and Dirichlet energy at each layer.
propose principle Dirichlet energy
Proposition 1. Dirichlet energy constrained learning defines the lower & upper limits at layer & as:
(5) We apply the transformed initial feature through trainable function f: x f(X) E gaxd Both Cmin and Cmax are positive hyperparameters. From value interval (0, 1), hyperparameter Cmin is selected by satisfying constraint of (xx()))= 2 et E(X())) > O. In such a way, the over-smoothing overcome since the Dirichlet energies of all the layers are larger than appropriate limits related to -uin- Compared with the initial transformed feature ( the intermediate node embeddings of the same class are expected to be merged closely to have a smaller Dirichlet energy and facilitate the downstream applications. Therefore, we exploit the upper limit E to avoid over- separating, where Cmax is usually selected from (0, 1]. In the experiment part, show that the optimal energy accompanied with the minimized classification loss locates within the above range at each layer. Furthermore, hyperparameters Cuin and Cmax could be easily selected from the large and appropriate value scopes, which do not affect the model performance.
Cmin E(X S E(X(1) S Cmax (()
Given both the low and upper limits, an intuitive solution to search the optimal energy is to train GNNs by optimizing the following constrained problem:
min Lraak +7 Eg [|
(6)
s.t. Cmin EminE(X E(X ((x(k-)) S {(((()) S. E(x(x for k k= 1,... K.
Crask denotes the cross-entropy loss node classification task; K is layer number of GNN; ||-1|F denotes Frobenius norm of matrix; and y is loss hyperparameter. Note that Dirichlet energy has also been adopted to regularize the node representation learning in shallow neural networks 33 37 38
We instead focus optimizing deep GNNs as shown in Eq. (6) where K is often large.
4 Energetic Graph Neural Networks - EGNN
is non-trivial to optimize Problem (6) due to the expensive computation of ((X())). Furthermore, the numerous constraints make the problem very complex optimization hyper-planes, at which the raw task objective tends to fall into local optimums. Instead of directly optimizing Problem (6). we propose an efficient model EGNN to satisfy the constrained learning from three perspectives: weight controlling, residual connection and activation function. We introduce them one by one as follows.
4.1 Orthogonal Weight Controlling
According to Lemma 2, without regularizing the maximum square singular value (k) of matrix ((4), the upper bound of Dirichlet energy can be larger than the upper limit, i.e., 1ie E(x Gmax (0)). That means the Dirichlet energy of a layer may break the upper limit of constrained learning, and makes Problem (6) infeasible. this section, we show how to satisfy such limit by controlling the singular values during weight initialization and model regularization.

---

Orthogonal initialization. Since the widely-used initialization methods (e.g.. Glorot initialization) fail restrict the scopes of singular values, we adopt the orthogonal approach that initializes trainable weight W((k) as a diagonal matrix with explicit singular values [39]. To restrict Smax and meet the
constrained learning, we apply equality constraint of 8max (((x(-11)) Cmax F (() each layer. Based this condition, we derive Proposition 2 initialize those weights W and their square singular values for all the layers of EGNN, and give Lemma 3 to show how we can satisfy the upper limit of constrained learning. The detailed derivation and proof are listed in Appendix.
(R)
At the first layer, weight W(1) is initialized as diagonal matrix /Cmax La, where la is identity matrix with dimension and the square singular values are Cmax. At the higher layer k > 1, weight (*)) is initialized with an identity matrix Ia. where the square singular values are 1.
Proposition 2.
Lemma 3. Based on the above orthogonal initialization, at the starting point of training, the Dirichlet energy of EGNN satisfies the upper limit at each layer k: ((x(k)) S Cmax E(
Orthogonal regularization. However, without proper regularization, the initialized weights cannot guarantee they will still satisfy the constrained learning during model training. Therefore, we propose training loss that penalizes the distances between the trainable weights and initialized weights Im or Ls. To be specific, we modify the optimization problem 6 as follows:
K
min Crask + - -Cmaaxlllle +
-
|| Lalle.
we instead use the weight penalization to meet the upper limit of constrained learning, and make the model training efficient. While larger y highly regularizes the trainable weights around the initialized ones to satisfy the constrained learning, smaller y assigns the model more freedom to adapt to task data and optimize the node classification loss. Considering the above orthogonal initialization where weight W**) is diagonal and sparse, we use the simplest distance constraint in Eq. (7) to update weight at the vicinity of its initialization. The singular values of updated sparse weight will be mainly determined by the dominant diagonal values, which are potentially close to the initialized ones. Therefore, we are able to control the singular values and regularize the upper limit of Dirichlet energy even at the model training phase. In the future work, more the advanced orthogonal initialization and regularization approaches could be explored to further boost performance deep GNNs 40
Comparing with the original problem 66.
4.2 Lower-bounded Residual Connection
Although the square singular values are initialized and regularized properly. we may still fail to guarantee the lower limit of constrained learning in some specific graphs. According Lemma the lower bound of Dirichlet energy is (1 A (()) E ((-())))) In the real-world applications, eigenvalue A may exactly equal to and relaxes the lower bound as zero as shown in Lemma 2. For example, in Erdös-Rén graph with dense connections [43) the eigenvalues of matrix À converge
to with high probability [17]. Even though (4) > O, the Dirichlet energy can be smaller than the lower limit and leads to the over-smoo To tackle this problem, we adopt residual connections
to the initial layer X and the previous layer X (k-1), To be specific, we define the residual graph convolutions as:
X (k) a <[(1 Cmin 1x(1-11) + (x-11)) + +3x(0))w(())).
(8)
and [ are residual connection strengths determined by the lower limit of constrained learning, i.c., a + Cuin-- We are aware that the residual technique has been used before set up deep However, they either apply the whole residual components, or combine an arbitrary fraction without theoretical insight. Instead, we use an appropriate residual connection according to the lower limit of Dirichlet energy. In the experiment part, we show that while a strong residual connection overwhelms information the higher layers and reduces the classification performance, a weak one will lead to the over-smoothing. In the following, we justify that both the lower and upper limits in the constrained learning can be satisfied with the proposed bounded residual connection. The detailed proofs are provided in Appendix.
GNNs 26 44 28
5

---

Lemma 4. Suppose that Cuax 2 Cmin/ (2in/(2Cmin - 1)2. Based upon the orthogonal controlling and residual connection, the Dirichlet energy of iiitialized EGNN larger than the lower limit each layer k i... ((x())) 2 Cmin E (xx(k=1))
Lemma 5. Suppose that /Cmax (m-omin (I-omin]A0+2 + Being augmented with the orthogonal controlling and residual connection, the Dirichlet energy of initialized EGNN smaller than the upper limit at each layer k. i... ((x(k)) I E
4.3 SReLU Activation
Note that the previous theoretical analysis and model design are conducted by ignoring the activation function, which is usually given by ReLU in GNN. In this section, we first theoretically discuss the impact of ReLU on the Dirichlet energy, and then demonstrate the appropriate choice of activation.
Lemma 6. We have ((o( S (k) activation function a is ReLU or Leaky-ReLU
It is shown that the application of ReLU further reduces the Dirichlet energy, since the negative embeddings are non-linearly mapped to zero. Although the trainable weights and residual connections are properly designed, the declining Dirichlet energy may violate the lower limit. On the other hand, a simplified GNN with linear identity activation will have limited model learning ability although it does not change the energy value. For example, simple graph convolution (SGC) model achieves comparable performance with the traditional GCN only with careful hyperparameter tuning [45]. We propose to apply SRcLU to achieve good trade-off between the non-linear and linear activations [46]]] 47 SReLU is defined ddement-wiselyy as:
(xx())) max(b, X (()),,
where is a trainable shift shared for each feature dimension of (*) SReLU interpolates between the non- non-linearity and linearity depending on shift While the linear identity activation is approximated if b is close to cc, the non-linear mapping is activated if node embedding is smaller than the specific In our experiments, we initialize with a negative value to provide an initial trade-off, and adapt to the given task by back propaprppggaiing the training loss.
4.4 Connections to Previous Work
Recently, various techniques have been explored to enable deep GNNs 16 are designed heuristically from diverse perspectives, and others are analogous to CNN components without theoretical insight tailored to graph analytics. In the following, we show how our principle and EGNN explain the existing algorithms, and expect to provide reliable theoretical guidance to the future design of deep GNNs.
30
Some of them
The general normalization layers, such as pair [23 batch 25 and group 24 normalizations, have been used to set up deep GNNs. The pair normalization (PairNorm) aims to keep the node pair distances as constant in the different layers, and hence relieves the over- smoothing. Motivated from CNNs. the batch and group normalizations re-scale the node embeddings of a batch and a group, respectively. Similar to the operation in PairNorm, they learn to maintain the node pair distance in the node batch or group. The adopted Dirichlet energy is also variant of the node pair distance. The existing normalization methods can be regarded as training GNN model with a constant energy constraint. However, this will prevent GNN from optimizing the energy as analyzed in Section 3 We instead regularize it within the lower and upper energy limits, and let model discover the optimum.
Embedding normalization.
Dropping edge. As data augmentation method, dropping edge (DropEdge) randomly masks a fraction of edges at each epoch (29). makes graph connections sparse and relieves the over- smoothing by reducing information propagation. Specially, the contribution of DropEdge could be explained from the perspective of Dirichlet energy. In Erdós-Rémp graph, eigenvalue Ag converges to 1 if the graph connections are more and more dense [17]. DropEdge reduces the value of Ao
and helps improve the upper bound of Dirichlet energy (1 - >a) (k) (*-(()))) to slow down the energy decreasing speed. In the extreme case where all the edges dropped in any graph.

---

Laplacian becomes zero matrix. As a result, we have eigenvalue Ào of zero and maximize the upper bound. In practice, the dropping rate has be determined carefully depending on various tasks. Instead, our principle assigns model freedom to optimize the Dirichlet energy within large and appropriate range.
Residual connection. Motivated from CNNs, residual connection has been applied to preserve the previous node embeddings and relieve the over-smoothing. Especially, the embedding from the last layer is reused and combined completely in related work [26] 48 49) A fraction of the initial embedding is preserved in model GCNII (28) and APPNP [50]. Networks JKNet (27] and DAGNN [5T] aggregate all the previous embeddings at the final layers. The existing work uses the residua connection empirically. In this work, we derive and explain the residual connection to guarantee the lower limit of Dirichlet energy. By modifying hyperparameter Cuins our EGNN can easily evolve to the existing deep residual GNNs, such as GCNII and APPNP.
Model simplification. Model SGC 45 removes all the activation and trainable weights to avoid over-fitting issue, and simplifies the training of deep GNNs. It is equivalent to EGNN with Cmaa 1 and b oo, where weights W and shifts b are remained as constants. Such simplification will reduce the model learning ability. As shown in Eq. 7 we adopt loss hyperparameter y to learn the trade-off between maintaining the orthogonal weights or updating them to model data characteristics.
Experiments
In this section, we empirically evaluate the effecttivneess of EGNN real-world datasets. We aim to answer the following questions. Q1: How does our EGNN compare with the state-of-the-ar deep GNN models? Q2: Whether not the Dirichlet energy at each layer of EGNN satisfies the constrained learning? Q3: How does each component of EGNN affect the model performance? Q4: How do the model hyperparameters impact the performance of EGNN?
5.1 Experiment Setup
Datasets.
Following the practice of previous work, we evaluate EGNN by performing node classifi- cation on four benchmark datasets: Cora, Pubmed [52), Coauthor-Physies The detailed statistics are listed in Appendix.
[53] and Ogbn-arxiv [54).
Baselines. We consider seven state-of-the-ar baselines: GCN [15], PairNorm (3). DropEdge (2) SGC [45), JKNet (27), APPNP (50), and GCNII (28). They are implemented based on their open repositories. The detailed descriptions of these baselines are provided in Appendix.
Implementation. We implement all the baselines using Pytorch Geometric (55) based on their official implementations. The model hyperparameters are reused according the public papers or are fine-tuned by ourselves if the classification accuracy could be further improved. Specially, we apply max- max-pooling to obtain the final node representation at the last layer of JKNet. In Ogbn-arxiv. we additionally include batch normalization between the successive layers in all the considered GNN models except PairNorm. Although more tricks (e.g., label reusing and linear transformation as listed in leader board) could be applied to improve node classification in Ogbn-arxiv, we focus on comparing the original GNN models enabling deep layer stacking. The training hyperparameters are carefully set by following the previous common setting and are listed in Appendix.
We implement our EGNN upon GCN, except for the components of weight initialization and regularization, lower-bounded residual connection and SReLU. We choose hyperparameters Cmax. Cuin. 7 and based on the validation set. For the weight initialization, datasets: that is, the trainable weights are initialized as identity matrices at all the graph convolutional layers. The loss hyperparameter y is 20 in Cora, Pubmed and Seauthor-Physice to strictly regularize towards the orthogonal matrix; and it is 10-4 in Ogbn-arxiv to improve the model's 's learning ability. For the lower-bounded residual connection, we choose residual strength Gmin from range 10.1, 0.75] and list the details in Appendix. The trainable shift b initialized with -10 in Cora and Pubmed; it initialized to -5 and -1 in Coauthor-Physic and Ogbn-arxiv, respectively. We also study these hyperparameters in the following experiments, All the experiment results are the averages of 10 runs.
set Cuax be for all the
7

---

Table Node classification accuracies in percentage with various depths: 2, 16, 32/64. The highest accuracy at each column is in bold.
Datasets
Cora
Pubmed
Coauthors- Phynsics
Ogbn-arxiv
Layer Num
2
16
64
16
64
2
16
32
2
16
32
GCN
82.5
22.0
21.9
79.7 7
37.9
38.4
92.4
13.5
13.1
70.4
70.6
68.5
PairNorm
74.5
44.2
14.2
73.8
68.6
60.0
86.3
84.0
83.6
67.6 6
70.4
69.6
DropEdge
82.7
23.6
25.2
79.6
45.9
40.0
92.5
85.1
35.2
70.5
70.4
67.1
SGC
75.7
72.1
24.1
76.1
70.2
38.2
92.2
91.7
84.8
69.2
64.0
59.5
JKNet
80.8
74.5
70.0
77.2
70.0
66.1
92.7
92.2
91.6
70.6
71.8
71.4
APPNP
82.9
79.4
79.5
79.3
77.1
76.8
92.3
92.7
92.6
68.3
65.5
60.7
GCNII
82.4
84.6
85.4
77.5
79.8
79.9
92.5
92.9
92.9
70.1
71.5
70.5
EGNN
83.2
85.4
85.7
79.2
80.0
80.1
92.6
93.1
93.3
68.4
72.7
72.7
5.2
Experiment Results
[ [ the test classi- fication accuracies. Each accuracy is averaged over 10 random trials. We report the results with 2/ /16/64 layers for Cora and Pubmed, and 2/16/32 16/32 layers for to space limit, we report the detailed results of mean accuracy and standard deviation in Appendix.
Node classification results. answer research question Q1,
and Oghn-arxiv. Due
We observe that our EGNN generally outperforms all the baselines across the four datasets, especially in the deep cases (K > 16). Notably, the node classification accuracy is consistently improved with the layer stacking in EGNN until K 32 or 64, which demonstrates the benefits of deep graph neural architecture to leverage neighbors multiple hops away. While the state-of-the-ae models PairNorm, DropEdge. SGC, JKNet, and APPNP alleviate the over-smoothing issue some extend, their performances still drop with the increasing of layers. Most of their 32/64-layer models are even worse than their corresponding shallow versions. As the most competitive deep architecture in literature, GCNII augments the transformation matrix as (1 (1--00 Φ) (W(k), where 0 & < 1 is a hyperparameter to preserve the identity mapping and enhance the minimum singular value of the augmented weight. Instead of explicitly defining the strength of identity mapping, we propose the orthogonal weight initialization based on the upper limit of Dirichlet energy and apply the orthogonal weight regularization. Based on Eq. 7 EGNN automatically learns the optimal trade-off between identity mapping and task adaption. Furthermore, we use SReLU activation and the residual connection to theoretically control the lower limit of Dirichlet energy. The experimental results show that EGNN not only outperforms GCNII in the small graphs Cora, Pubmed and Saauthor-Physieee but also delivers significantly superior performance in the large graph Obgn-arxiv, achieving 3.1% improvement over GCNII with 32 layers.
Dirichelet energy visualization. To answer research question Q2, we show the Dirichlet energy at each layer of 64-layer EGNN in Cora and Pubmed datasets in Figure []] To have better visualization purposes, by keeping other default hyperparam- eters unchanged, EGNN trained with /max//Cu
Bini
GCN)
G24
Upper Lower
Upper
Layem Figure 1: Dirichelet energy variation with layers in Cora (Left) and Pubmed (Right). The upper and lower denotes the energy limits.
0.4/0.15 and Cmax/ = 0.4/0.11 in Cora and Pubmed, respectively. We only plot and compare with the baseline approaches of GCN and GCNII due to space limit. For other meth- ods, the Dirichlet energy is either close to zero overly large due to the over-smoothing issue or over-separating issue of node embeddings, respectively.
It is shown that the Dirichlet energies of EGNN are strictly constrained within the range determined by the lower and upper limits the constrained learning. Due to the over-smoothing issue in GCN, all the node embeddings converge zero vectors. GCNII has comparable or smaller Dirichlet energy by carefully and explicitly designing both the initial connection and identity mapping strengths. In contrast, our EGNN only gives the appropriate limits of Dirichlet energy, and let the model learn the

---

Table 2: Ablation studies weight initialization, lower limit Cuin and activation function of EGNN.
Cora
Pubmed
Seauthors-Physiee
Ogbn-arxiv
Component
Type
16
2
16
64
16
32
16
32
Weight
Glorot
77.8 40.2
23.6
68.4
62.6 60.2
92.6
81.7
73.4
68.4
72.8
72.7
initialization
Orthogonal
83.2 85.4 85.7
79.2
80.0 80.1 92.6
93.1
93.3
68.4
72.7
72.7
Lower
0.
83.6 68.6 12.9
78.9
77.1 44.1
92.88
91.4
79.7
70.9
69.4 62.4
limit setting 0.1
0.75
83.2
85.4 85.7
79.2
80.0 80.1 92.6
93.1
93.3
68.4
72.7
72.7
Cuin
0.95
65.4 72.0 71.5
74.0
75.3 75.7
89.4
90.4
90.5
56.5
66.8
69.5
Linear
83.1
85.6 85.5
79.2
79.9 79.9
92.6
93.1
93.1
64.8
72.5
71.0
Activation
SReLU
83.2
85.4 85.7
79.2 80.0 80.1
92.6
93.1
93.3
68.4
72.7
72.7
ReLU
83.1 85.2 85.0
79.1 79.7 79.9
92.6 93.1
93.1
68.6
72.4
72.4
optimal energy each layer for specific task. The following hyperparameter studies will show that the values of Cmin and Cmax could be easily selected from a large appropriate range.
Gyrnn
Cmax
80
40
20
0
1
0.0
0.5
1.0
0.25
0.50
0.75
1.00
Figure 2: The impacts of hyperparameters b, Y, Gmin and Cmax on 64-layer EGNN trained Cora.
Y-axis is test accuracy in percent.
Ablation studies of EGNN components. To demonstrate how each component affects the training of graph neural architecture and answer research question Q3, we perform the ablation experiments with EGNN all the datasets. For the component of orthogonal weight initialization and regular- ization, we compare and replace them with the traditional Glorot initialization and Frobenius norm regularization as shown in Eq. (6) Considering the component of lowerbbounded residual connection, we vary the lower limit hyperparameter Cmin from 0, 0.1 ~ 0.75 and 0.95. Within the range of 0.1 ~ 0.75, the adoption of specific values specified for each dataset in Appendix. The component of the activation function is studied from candidates of linear identity activation, SReLU, and ReLU. Table reports the results of the above ablation studies.
The orthogonal weight initialization and regularization are crucial to train the deep graph neural architecture. In Cora, Pubmed, and Seauthor-Physicee Glorot initialization and Frobenius norm regularization fail to control the singular values of trainable weights, which may lead to overly large or small Dirichlet energy and affect the node classification performance. In Ogbn-arxiv, the input node features are described by dense word embeddings of paper (56), where the trainable weights GNN are required to capture data statistics and optimize the classification task. EGNN applies a small loss hyperparameter 7 of 10-4 to let the model adapt to the given task, which is equivalent to the traditional regularization. Therefore, the two approaches have comparable performances.
An appropriate lower limit could enable the deep EGNN. While the Dirichlet energy may approach zero without the residual connection, the overwhelming residual information with Cmin 0.95 prevents the higher layer from learning the new neighborhood information. Within the large and appropriate range of [0.1, 0.75), Cmin could be easily selected achieve superior performance.
Activation SReLUU performs slightly better than the linear identity activation and ReLU. This is because SReLU could automatically learn the trade-off between linear and non-linear activations, which prevents the significant dropping of Dirichlet energy and ensures the model learning ability.
Hyperparameter analysis. To understand the hyperparameter impacts on 64-layer EGNN and answer research question Q4, we conduct experiments with different values of initial shift 6, loss

---

factor 7, lower limit factor Cmin and upper one Cmax. We present the hyperparameter study in Figure for Cora, and show the others with similar tendencies in Appendix.
We observe that our method is not sensitive to the choices of b, 7, Cmin and Cmax in a wide range: (i) The initial shift value should be b I 0, in order to avoid the overly nonlinear mapping and Dirichlet energy damage. (ii) It is shown that EGNN approximates the optimal performance once the loss factor 7 larger than a specific threshold. The thresholds are 0.3 in Cora, 0.1 in Pubmed and Coauthor-Physics, and les-4 in Ogbn-arxiv, respectively. The threshold depends on the specific dataset: while a larger potentially works in the small dataset to strictly regularize Dirichlet energy, smaller one would be preferred for the large dataset to capture the complex data manifold. (iii) Cmin within the appropriate range [0.1, 0.75] allows the model expand neighborhood size and preserve residual information to avoid the over-smoothing, (iv) As shown in Figure | since energy (xx()))
the hidden layer is much smaller than E(X from the input layer, we could easily satisfy the upper limit with Cuax in large range [0.2, 1]. Given these large hyperparameter ranges, EGNN could be easily trained with deep layers.
6 Conclusions
In this paper, we propose a Dirichlet energy constrained learning principle to show the importance of regularizing the Dirichlet energy each layer within reasonable lower and upper limits. Such energy constraint is theoretically proved to help avoid the over-smoothing and over-separating issues. We then design EGNN based on theoretical results and empirically demonstrate that the constrained learning plays key role in guiding the design and training of deep graph neural architecture. The detailed analysis presented illustrate how our principle connects and combines the previous deep methods. The experiments on benchmarks show that EGNN could be easily trained to achieve superior node classification performances with deep layer stacking. We believe that the constrained learning principle will help discover deeper and more powerful GNNs in the future.
References
[1] Yujia Li, Daniel Tarlow, Mare Brockschmidr, and Richard Zemel. Gated graph sequence neural networks. arXiv preprint 1rXiv:5511.05493 2015.
[2] Rex Ying, Ruining He, Kaifeng Chen, Pong Eksombatchai, William L Hamilton, and Jure Leskovec. Graph convolutional neural networks for web-scale recommender systems. In Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 974-983, 2018
[3] Xiao Huang, Qingquan Song, Yuening Li, and Xia Hu. Graph recurrent networks with at- tributed random walks. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pages 732-740, 2019.
[4] Wenqi Fan, Yao Ma, Qing Li, Yuan He, Erie Zhao, Jiliang Tang. and Dawei Yin. Graph neural networks for social recommendation. In The World Wide Web Conference, pages 417-426, 2019.
[5] Hongyang Gao, Zhengyang Wang, and Shuiwang Ji. Large-scale learnable graph convolutional networks. In Proceedings of the 24th ACM SIGKDD International Conference Knowledge Discovery & Data Mining, pages 1416-1424, 2018
[6] Hongyang Gao and Shuiwang Ji. Graph u-nets, international conference on machine learning. pages 2083-2092. PMLR, 2019.
[7] Kaixiong Zhou, Qingquan Song, Xiao Huang, and Xia Hu. Auto-gnn: Neural architecture search of graph neural networks. arXiv preprini arXiv: 1909.9998938 2019.
[8] Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In Proceedings of the 34th International Conference on Machine Learning- Volume 70, pages 1263-1272. JMLR. org, 2017.
10

---

[9] Marinka Zitiik and Jure Leskovec. Predicting multicellular function through multi-layer tissue networks. Bioinformatics, 33(14):190-i198, 2017.
[10] Federico Monti, Michael M Bronstein, and Xavier Bresson. Geometric matrix completion with recurrent multi-graph neural networks. arXiv preprini 170.00370,, 2017.
[11] Kaixiong Zhou, Qingquan Song, Xiao Huang, Daochen Zha, Na Zou, and Xia Hu. Multi-channel graph neural networks. arXiv preprint arXiv:1912.08306, 2019.
[12] Will Hamilton, Zhitao Ying, and Jure Leskovec. Inductive representation learning on large graphs. In Neu/PS, pages 1024-1034, 2017.
[13] Petar Velickovic, Guillem Cucurull, Arantxa Casanova, Adriana Romero, Pietro Lio, and Yoshua Bengio. Graph attention networks. arXiv, 1(2). 2017.
[14] Keyulu Xu, Weihua Hu, Jure Leskovec, and Stefanie Jegelka. How powerful are graph neural networks? arXiv preprint arXiv:
2018.
[15] Thomas N Kipf and Max Welling. Semi-supervised classification with graph convolutional networks. ICLR, 2017.
[16] Qimai Li, Zhichao Han, and Xiao-Ming Wu. Deeper insights into graph convolutional networks for semi-supervised learning. In Thirty-Second AAAI Conference on Artificial Intelligence, 2018.
[17] Kenta Oono and Taiji Suzuki. Graph neural networks exponentially lose expressive power for node classification. International Conference on Learning Representations, 2020.
[18] Hoang NT and Takanori Maehara. Revisiting graph neural networks: All we have is low-pass filters. arXiv preprint 1exXiv:905509990 2019.
[19] Deli Chen, Yankai I. Lin, Wei Li, Peng Li, Jie Zhou, and Xu Sun. Measuring and relieving the over-smoothing problem for graph neural networks from the topological view. arXiv preprint arXiv: 190. 2019.
[20] Uri Alon and Eran Yahav. On the bottleneck of graph neural networks and its practical implications. arXiv preprint arXiv:2006.0522 2020.
[21] Eli Chien, Jianhao Peng, Pan Li, and Olgica Milenkovic. Adaptive universal generalized pagerank graph neural network. In International Conference on Learning Representations. https://openreview. netiforum, 2021.
[22] Wenbing Huang, Yu Rong, Tingyang Xu, Fuchun Sun, and Junzhou Huang. Tackling over- smoothing for general graph convolutional networks. arXiv e-prints, pages arXiv-2008 2020.
[23] Lingxiao Zhao and Leman Akoglu. Pairnorm: Tackling oversmoothing in gnns. arXiv preprint /909 /2223, 2019.
[24] Kaixiong Zhou, Xiao Huang, Yuening Li, Daochen Zha, Rui Chen, and Xia Hu. Towards deeper graph neural networks with differentiable group normalization. Advances in Neural Information Processing Systems, 33, 2020.
[25] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. ICML, 2015.
[26] Guohao Li, Matthias Muller, Ali Thabet, and Bernard Ghanem. Deepgens: Can gens go as deep as cnns? In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 9267-9276. 2019.
[27] Keyulu Xu, Chengtao Li, Yonglong Tian, Tomohiro Sonobe, Ken-ichi Kawarabayashi, and Stefanie Jegelka. Representation learning on graphs with jumping knowledge networks. In International Conference on Machine Learning, pages 5453-5462 PMLR, 2018
[28] Ming Chen, Zhewei Wei, Zengfeng Huang, Bolin Ding, and Yaliang Li. Simple and deep graph convolutional networks. arXiv preprint arXiv:2007.02113,, 2020.

---

[29] Yu Rong, Wenbing Huang, Tingyang Xu, and Junzhou Huang. Dropedge: graph convolutional networks on node classification. In International Conference on Learning Representations. https://openreview neifforuum, 2020.
Towards deep
[30] Wenzheng Feng, Jie Zhang, Yuxiao Dong. Yu Han, Huanbo Luan, Qian Xu, Qiang Yang, Evgeny Kharlamov, and Jie Tang. Graph random neural networks for semi-supervised learning on graphs. Advances in Neural Information Processing Systems, 33, 2020.
[31] Arman Hasanzadch, Ehsan Hajiramezanali, Shahin Boluki, Mingyuan Zhou, Nick Duffield, Kr- ishna Narayanan, and Xiaoning Qian. Bayesian graph neural networks with adaptive connection sampling. In International Conference Machine Learning, pages 4094-4104. PMLR, 2020.
[32] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In CVPR, pages 770-778, 2016.
[33] Chen Cai and Yusu Wang. A note on over-smoothing for graph neural networks. arXiv preprint arXiv:2006. 13318, 2020.
[34] Muhan Zhang and Yixin Chen. Link prediction based on graph neural networks. arXiv preprint 1oXiv:1802.09691, 2018.
[35] Xavier Glorot and Yoshua Bengio. Understanding the difficulty of training deep feedfor- ward neural networks. In Proceedings of the thirteenth international conference on artificial intelligence and statistics, pages 249-256, 2010.
[36] Chenhui Deng, Zhiqiang Zhao, Yongyu Wang, Zhiru Zhang, and Zhuo Feng. Graphzoom:
A multi-level spectral approach for accurate and scalable graph embedding. arXiv preprint 19X1v:19100023300 2019.
[37] Yu Chen, Lingfei Wu, and Mohammed Zaki. Iterative deep graph learning for graph neural networks: Better and robust node embeddings. Advances in Neural Information Processing Systems, 33, 2020.
[38] Mikhail Belkin and Partha Niyogi. Laplacian eigenmaps and spectral techniques for embedding and clustering. In Nips, volume 14, pages 585-591, 2001.
[39] Quoc V Le, Navdeep Jaitly, and Geoffrey E Hinton. A simple way to initialize recurrent networks of rectified linear units. arXiv preprint 104404150, 2015. 2015.
[40] Asher Trockman and Zico Kolter. Orthogonalizing convolutional layers with the cayley transform. arXiv preprint arXiv:2104.07167 2021.
[41] Lei Huang, Li Liu, Fan Zhu, Diwen Wan, Zehuan Yuan, Bo Li, and Ling Shao. Controllable orthogonalization in training dnns. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6429-6438, 2020.
[42] Jeffrey Pennington, Samuel S Schoenholz, and Surya Ganguli. Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practice. In Proceedings of the 31 International Conference on Neural Information Processing Systems, pages 4788-4795 2017.
[43] Paul Erdös and Alfréd Rényi. On the evolution of random graphs. Publ. Math. Inst. Hung Acad. Sci, 5(1):17-60, 1960.
[44] Guohao Li, Chenxin Xiong, Ali Thabet, and Bernard Ghanem. Deepergen: All you need to train deeper gens. arXiv preprint N00v::2000.07739, 2020.
[45] Felix Wu, Tianyi Zhang, Amauri Holanda de Souza Jr, Christopher Fifty, Tao Yu, and Kilian Q Weinberger. Simplifying graph convolutional networks. arXiv preprint arXiv: 1902.07 153, 2019.
[46] Sitao Xiang and Hao Li. On the effects of batch and weight normalization in generative adversarial networks. arXiv preprint 1rXvv:1704.03971, 2017.
12

---

[47] Haozhi Qi, Chong You, Xiaolong Wang, Yi Ma, and Jitendra Malik. Deep isometric learning for visual recognition. In International Conference on Machine Learning, pages 7824-7835. PMLR, 2020.
[48] Lei Chen, Le Wu, Richang Hong, Kun Zhang, and Meng Wang. Revisiting graph based collaborative filtering: A linear residual graph convolutional network approach. Proceedings of the AAAI Conference Artificial Intelligence, volume 34, pages 27-34, 2020.
[49] Kuangqi Zhou, Yanfei Dong, Kaixin Wang, Wee Sun Lee, Bryan Hooi, Huan Xu, and Jiashi Feng. Understanding and resolving performance degradation in graph convolutional networks, 2020.
[50] Johannes Klicpera, Aleksandar Bojchevski, and Stephan Günnemann. Predict then propagate: Graph neural networks meet personalized pagerank. arXiv preprint arXiv:/810.05999, 2018.
[51] Meng Liu, Hongyang Gao, and Shuiwang Ji. Towards deeper graph neural networks. In Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery Data Mining, pages 338-348, 2020.
[52] Zhilin Yang. William W Cohen, and Ruslan Salakhutdinov. Revisiting semi comi-supervised learning with graph embeddings. arXiv preprint arXiv:1603.0880 08861, 2016.
[53] Oleksandr Shehur, Maximilian Mumme, Aleksandar Bojchevski, and Stephan Günnemann. Pitfalls of graph neural network evaluation. arXiv preprint 1rXiv:/811.055 1.05668 2018.
[54] Weihua Hu, Matthias Fey, Marinka Zitmik, Yuxiao Dong, Hongyu Ren, Bowen Liu, Michele Catasta, and Jure Leskovec. Open graph benchmark: Datasets for machine learning on graphs. arXiv preprint arXiv:2005.00 00687, 2020.
[55] Matthias Fey and Jan E. Lenssen. Fast graph representation learning with PyTorch Geometric. In ICLR Workshop, 2019.
[56] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg Corrado, and Jeffrey Dean. Distributed representations words and phrases and their compositionality. arXiv preprint arXiv: 13 10.4566, 2013.
13


Reviews:
[{"review_id": "xB-TFLoKxBh", "reviewer": "Reviewer_DaEi", "summary": "This paper proposes a Dirichlet energy-based method to alleviate the over-smoothing and over-separating problems which are commonly observed in deep GNNs. Different from existing methods that rely on techniques from CNNs or heuristic strategies, this work provides theoretical proof of the lower and upper bound of the Dirichlet energy in GNNs and proposes EGNN which enables deeper GNNs. The effectiveness of EGNN is evaluated on commonly used benchmarks.", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Pros:\n1.\tThe paper points out that the Dirichlet energy of each layer of GNN should not too large or too small.\n2.\tAccording to the theoretical findings, the paper proposes constraints on Dirichlet energy. Especially, it smartly binds the lower bound of the Dirichlet energy to avoid over-smoothing.\n3.\tThe experimental results show the great impact of Dirichlet energy constraints for deep GNNs.\n\nCons:\n1.\tIn deep graph learning, solving the Dirichlet Energy of embedding matrix is equivalent to a set of linear equations in terms of Laplacian matrix and embedding matrix. Therefore, the theoretical findings of Lemma 2 and Lemma 3 are equivalent to the theoretical findings of Theorem 3 in [1]. In [1], it also proves the upper bound of $s_max$ in terms of node size. Therefore, the theoretical contribution is not significant. \n2.\t[2] also constrains the Dirichlet energy by minimizing it although it does not constrain the lower bound of the Dirichlet energy (see Eq. 5 in [2]). However, the paper does not discuss with [2].\n3.\tThe performance improvement is insignificant compared with GCNII [27]. Furthermore, the classification accuracy of GCNII shown in Table 1 is not consistent with the results reported in the original paper [27]. For example, GCNII achieves 80.2 on Pubmed with num_layer=16 [27], while Table 1 reports 79.8. \n\nQuestions:\n1.\tWhat is your main Theorem? There are only Lemmas in the paper.\n2.\tIn Figure 1, why the Dirichlet energy of GCN is zero at the very beginning? It is supposed to be decreasing to zero after several layers.\n\n[1] Oono, Kenta, and Taiji Suzuki. \"Graph Neural Networks Exponentially Lose Expressive Power for Node Classification.\" International Conference on Learning Representations. 2019.\n[2] Deng, C., et al. \"GraphZoom: A Multi-level Spectral Approach for Accurate and Scalable Graph Embedding.\" The International Conference on Learning Representations (ICLR). 2020.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "see above", "review_text": "", "extra_scores": {"ethical_concerns": "n/a", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "6 HOURS"}, "extra_text": {}}, {"review_id": "ENF2l_0LGTU", "reviewer": "Reviewer_vMYT", "summary": "This paper proposes Dirichlet energy constrained learning as a generalizable principle to guide the training of deep GNNs through regularizing Dirichlet energy at each layer. Following this principle, a novel deep architecture called Energetic Graph Neural Networks (EGNN) is proposed, which could learn an optimal Dirichlet energy to enable effective training of deep networks. Empirical results demonstrate that EGNN can be trained to achieve better results with more layers and outperform other GNN deepening approaches. ", "questions": "", "limitations": "", "rating": 7, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "This paper leverages a metric of Dirichlet energy of node embeddings to provide a better understanding of the over-smoothing and over-separation issues of deep GNNs. The new Dirichlet energy constrained learning framework is well presented to explain the key constraints in training deep GNNs. Three learning strategies (orthogonal weight controlling, lower-bounded residual connection, and shifted ReLU activation) are proposed to ensure that the training of the proposed EGNN model satisfies the constrained learning. Theoretical proofs and empirical results on node classification both demonstrate the effectiveness of the proposed solution. This paper is well-written and has made solid technical contributions towards effectively training deep GNNs. Its main claims are theoretically and experimentally supported. \n\nI have carefully read the authors' responses -- Although there are a few potential issues raised by other reviewers, I think overall this is a pretty good paper in my batch, so I retain my recommendation.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "3 hours"}, "extra_text": {}}, {"review_id": "DIUARR-nKlf", "reviewer": "Reviewer_TzoX", "summary": "The authors suggest a way to prevent over smoothing in many graph neural networks that use the graph Laplacian as the spatial convolution kernel. The idea is to keep the Dirichlet Energy under control throughout the network, and hence not allow the network to oversmooth. ", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper is interesting with several theoretical results that look correct as far as I can tell. The results are nice but a bit limited to four data sets only of the same type, and the method still relies on the x^0 input term of GCNII at every layer, which looks specific to this type of data sets. More specific details are given below:\n\n1) \"Orthogonal initialization\": the authors apply a diagonal initialization, probably with diagonal entries of +-1. Is that right? The writing is a bit confusing. What happens when the matrices W are not square? \n\n2) Eq. (7) the authors regularize W to be a scaled identity. I would not necessarily call it \"orthogonal\". In orthogonal regularization, I would expect something like ||W^T*W - c*I||. Why call it or even regularize it this way? Am I missing something? \n\n3) In continuation to the previous remark, the authors say that for three of the four data sets (e.g. cora, pubmed and Coauthors-Physics), they use a high regularization parameter gamma. It means that many W's are close to the identity. That also means that the network computes a lot of FLOPs but the weights W are not so meaningful. Is that a good network design? What happens if there are no W's at all (just W=c*I)?\n\n4) In continuation to the previous points, you penalize the energy, not necessarily constrain it. For a low gamma, the energy can be high. Am I missing something? In your fourth experiment, you use a very small gamma. Is the energy bounded there?\n\n5) Page 9: overwhelming residual information. This is quite obvious. By setting cmin=0.95 you more or less ignore the Laplacian and weight, and just propagate the input through the network without doing anything. I'm not so sure this is even worthy of discussing.\n\n6) In figure 1, it looks like the legend is incorrect. \"GCN\" and \"Lower\" are flipped. Am I wrong?\n\n7) You use gamma=20 for the three data sets and the gamma=10^-4 for the last one. That is a huge difference. Can you comment on why is that? \n\n8) \"We implement our EGNN upon GCN\". The additions in (8) look very similar to those applied in GCNII, so it looks like most of the performance boost is coming from there. Now, the beta term in GCNII (having the input x^0 in all layers) is a bit specific to these data sets (i.e. not used in other scenarios like in standard CNNs or other GNN tasks). How does the method perform without beta? (which is learnable, right?)  Also - in Fig 1 it looks like GCNII does not lose energy because of such additions - so why do we still need to penalize the energy?\n\n9) Are the results for the semi-supervised or fully supervised node classification. I guess the former, but this is not written in the paper.\n\n\nPost rebuttal:\n==========\nI upgrade my score to 6. \n\n ", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "No mentioning.", "review_text": "", "extra_scores": {"ethical_concerns": "No", "needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}, {"review_id": "-C5EiCUOW8", "reviewer": "Reviewer_tjhZ", "summary": "In order to solve the over-smoothing issue for graph neural network training, this paper propose a way with lower and upper constraints in terms of Dirichlet energy at each layer.  According to their experiment results, they achieve state-of-the-art performance by using deep layers in this Energetic Graph Neural Networks (EGNN) framework.", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Originality:\n\nIn this work,  lower and upper constraints in terms of Dirichlet energy at each layer are used for solving over-smoothing issue. This is a new and interesting approach. \n\nQuality:\n\n Claims are reasonably supported. The initializations satisfy the lower and upper constraints. However, learning with strong regularizations respect to initializations does not mean that the lower and upper constraints hold after the parameters are updated.\n\nClarity:\n\nThe writing could be improved. For example, the abbreviation \"EGNN\" appear in the abstract, however, it does not state that it means \"Energetic Graph Neural Networks\".\n\nSignificance:\n\nThe results seems promising. However, learning with strong constraints respect to the distance with initializations  could limit the ability of graph neural network. Some discussions could be done.\n\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "Limitations from previous reviewers are almost solved.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "5"}, "extra_text": {}}]

Author rebuttal:

Meta-review: Congratulations! Your paper is accepted to NeurIPS 2021. 
Please incorporate the edits and corrections as discussed in the rebuttal. 
Decision: Accept (Poster)